Predictive Patterns and Market Efficiency: A Deep Learning Approach to Financial Time Series Forecasting

Abstract

This study explores market efficiency and behavior by integrating key theories such as the Efficient Market Hypothesis (EMH), Adaptive Market Hypothesis (AMH), Informational Efficiency and Random Walk theory. Using LSTM enhanced by optimizers like Stochastic Gradient Descent (SGD), Adam, AdaGrad, and RMSprop, we analyze market inefficiencies in the Standard and Poor’s (SPX) index over a 22-year period. Our results reveal “pockets in time” that challenge EMH predictions, particularly with the AdaGrad optimizer at a size of the hidden layer (HS) of 64. Beyond forecasting, we apply the Dominguez–Lobato (DL) and General Spectral (GS) tests as part of the Martingale Difference Hypothesis to assess statistical inefficiencies and deviations from the Random Walk model. By emphasizing “informational efficiency”, we examine how quickly new information is reflected in stock prices. We argue that market inefficiencies are transient phenomena influenced by structural shifts and information flow, challenging the notion that forecasting alone can refute EMH. Additionally, we compare LSTM with ARIMA with Exponential Smoothing, and LightGBM to highlight the strengths and limitations of these models in financial forecasting. The LSTM model excels at capturing temporal dependencies, while LightGBM demonstrates its effectiveness in detecting non-linear relationships. Our comprehensive approach offers a nuanced understanding of market dynamics and inefficiencies.

Keywords

• forecasting
• machine learning
• efficient market hypothesis
• LSTM optimization
• dynamics in market efficiency