Lesson 2.6: Long Short-Term Memory (LSTM) Networks and Gated Architectures

Introduction: Overcoming Short-Term Memory in Financial Forecasting

In Lesson 1, we established that standard Recurrent Neural Networks fail to model long-term financial sequences due to the vanishing gradient problem. In real-world financial forecasting—such as predicting multi-quarter earnings trends, long-term yield curve movements, or multi-week macroeconomic volatility—information from dozens or hundreds of time steps prior is critical for accurate predictions.

To solve the vanishing gradient barrier, Sepp Hochreiter and Jürgen Schmidhuber introduced Long Short-Term Memory (LSTM) networks in 1997. LSTMs replace standard recurrent cells with sophisticated internal gating mechanisms that regulate information flow, allowing networks to retain long-term memory reliably. This lesson deconstructs the internal architecture of LSTM cells, the mathematics of memory gates, and their application to financial time-series forecasting.

Part 1: The Internal Anatomy of an LSTM Cell

Unlike a standard RNN cell that simply applies a tanh activation to previous hidden states, an LSTM cell maintains a complex internal routing system controlled by three neural “gates” and a dedicated conveyor belt called the Cell State.

1. The Cell State (C_t)

The Cell State acts as a long-term information highway running straight down the entire unrolled chain, with only minor linear interactions. This allows gradients to flow backward uninterrupted across hundreds of time steps, completely eliminating the vanishing gradient problem.

2. The Three Gating Mechanisms

Gates are composed of a sigmoid neural network layer (σ) and a pointwise multiplication operation. The sigmoid layer outputs numbers between 0 and 1, describing how much of each component should be let through:

The Forget Gate (f_t): Decides what information to throw away from the previous cell state. It reads h_{t-1} and x_t, outputting a number between 0 (completely erase) and 1 (completely keep) for each number in the cell state C_{t-1}:

f_t = σ(W_f · [h_{t-1}, x_t] + b_f)

The Input Gate (i_t): Decides what new information we are going to store in the cell state. A tanh layer creates a vector of new candidate values Ĉ_t:

i_t = σ(W_i · [h_{t-1}, x_t] + b_i)

Ĉ_t = tanh(W_C · [h_{t-1}, x_t] + b_C)

The Output Gate (o_t): Determines what parts of the updated cell state will be outputted as the new hidden state h_t:

o_t = σ(W_o · [h_{t-1}, x_t] + b_o)

h_t = o_t * tanh(C_t)

3. Updating the Cell State

The network combines the old cell state multiplied by the forget gate with the new candidate values multiplied by the input gate to form the updated Cell State C_t:

C_t = f_t * C_{t-1} + i_t * Ĉ_t

Part 2: Gated Recurrent Units (GRUs)

While LSTMs are exceptionally powerful, their complex architecture (three gates, separate cell states) requires significant computational resources to train. To optimize performance, Cho et al. introduced the Gated Recurrent Unit (GRU).

1. Simplifying the LSTM Architecture

Combined States: GRUs merge the cell state and hidden state into a single hidden state h_t.

Two Gates: GRUs utilize only two gates—the Reset Gate (determining how to combine new input with previous memory) and the Update Gate (acting similarly to both the forget and input gates of an LSTM).

Advantage: GRUs contain fewer parameters, making them faster to train and less prone to overfitting on noisy, limited financial datasets.

Part 3: Application to Financial Time-Series Forecasting

LSTMs and GRUs are deployed across quantitative finance for sophisticated predictive modeling:

1. Volatility Forecasting and GARCH Comparison

While econometric models like GARCH (Generalized Autoregressive Conditional Heteroskedasticity) model financial variance, LSTM networks capture complex, non-linear volatility clustering across multi-asset portfolios by ingesting high-frequency order book data alongside macroeconomic indicators.

2. Algorithmic Trading Signal Generation

Quantitative trading desks feed multi-variate time-series inputs—historical prices, trading volume, order book imbalances, and sentiment scores—into stacked LSTM networks to forecast directional price movements and generate automated execution signals.

1. LSTM Architecture Mathematical Deep-Dive

Complete LSTM Forward Pass:

class LSTMCell:
    """
    Single LSTM Cell implementation from scratch
    """
    def __init__(self, input_size, hidden_size):
        self.input_size = input_size
        self.hidden_size = hidden_size
        
        # Forget gate weights
        self.W_f = np.random.randn(hidden_size, input_size + hidden_size) * 0.01
        self.b_f = np.zeros((hidden_size, 1))
        
        # Input gate weights
        self.W_i = np.random.randn(hidden_size, input_size + hidden_size) * 0.01
        self.b_i = np.zeros((hidden_size, 1))
        
        # Candidate gate weights
        self.W_C = np.random.randn(hidden_size, input_size + hidden_size) * 0.01
        self.b_C = np.zeros((hidden_size, 1))
        
        # Output gate weights
        self.W_o = np.random.randn(hidden_size, input_size + hidden_size) * 0.01
        self.b_o = np.zeros((hidden_size, 1))
        
        # Store gradients
        self.gradients = {}
    
    def forward(self, x_t, h_prev, C_prev):
        """
        Forward pass through LSTM cell
        """
        # Combine input and previous hidden state
        combined = np.vstack([h_prev, x_t])
        
        # Forget gate
        f_t = self.sigmoid(np.dot(self.W_f, combined) + self.b_f)
        
        # Input gate
        i_t = self.sigmoid(np.dot(self.W_i, combined) + self.b_i)
        
        # Candidate cell state
        C_tilde = np.tanh(np.dot(self.W_C, combined) + self.b_C)
        
        # Update cell state
        C_t = f_t * C_prev + i_t * C_tilde
        
        # Output gate
        o_t = self.sigmoid(np.dot(self.W_o, combined) + self.b_o)
        
        # Hidden state
        h_t = o_t * np.tanh(C_t)
        
        # Store for backprop
        self.cache = (x_t, h_prev, C_prev, f_t, i_t, C_tilde, C_t, o_t, h_t, combined)
        
        return h_t, C_t
    
    def sigmoid(self, x):
        """Sigmoid activation function"""
        return 1 / (1 + np.exp(-np.clip(x, -500, 500)))
    
    def backward(self, dh_next, dC_next):
        """
        Backward pass through LSTM cell (BPTT)
        """
        # Unpack cache
        x_t, h_prev, C_prev, f_t, i_t, C_tilde, C_t, o_t, h_t, combined = self.cache
        
        # Gradient through output gate
        do_t = dh_next * np.tanh(C_t)
        do_t_raw = do_t * o_t * (1 - o_t)
        
        # Gradient through cell state
        dC_t = dh_next * o_t * (1 - np.tanh(C_t)**2) + dC_next
        
        # Gradient through forget gate
        df_t = dC_t * C_prev
        df_t_raw = df_t * f_t * (1 - f_t)
        
        # Gradient through input gate
        di_t = dC_t * C_tilde
        di_t_raw = di_t * i_t * (1 - i_t)
        
        # Gradient through candidate cell
        dC_tilde = dC_t * i_t
        dC_tilde_raw = dC_tilde * (1 - C_tilde**2)
        
        # Combine gradients for weights
        dW_f = np.dot(df_t_raw, combined.T)
        dW_i = np.dot(di_t_raw, combined.T)
        dW_C = np.dot(dC_tilde_raw, combined.T)
        dW_o = np.dot(do_t_raw, combined.T)
        
        db_f = df_t_raw
        db_i = di_t_raw
        db_C = dC_tilde_raw
        db_o = do_t_raw
        
        # Gradient for previous hidden state
        dh_prev = (np.dot(self.W_f[:, :self.hidden_size].T, df_t_raw) +
                   np.dot(self.W_i[:, :self.hidden_size].T, di_t_raw) +
                   np.dot(self.W_C[:, :self.hidden_size].T, dC_tilde_raw) +
                   np.dot(self.W_o[:, :self.hidden_size].T, do_t_raw))
        
        # Gradient for previous cell state
        dC_prev = dC_t * f_t
        
        return dh_prev, dC_prev

2. LSTM Gate Mathematics

Forget Gate Derivative:

f_t = σ(W_f · [h_{t-1}, x_t] + b_f)

∂f_t/∂W_f = f_t(1 - f_t) · [h_{t-1}, x_t]^T

∂L/∂W_f = Σ_{t=1}^{T} ∂L/∂f_t · ∂f_t/∂W_f

Where:
∂L/∂f_t = ∂L/∂C_t · C_{t-1}

Input Gate Derivative:

i_t = σ(W_i · [h_{t-1}, x_t] + b_i)
Ĉ_t = tanh(W_C · [h_{t-1}, x_t] + b_C)

∂L/∂i_t = ∂L/∂C_t · Ĉ_t
∂L/∂Ĉ_t = ∂L/∂C_t · i_t

Output Gate Derivative:

o_t = σ(W_o · [h_{t-1}, x_t] + b_o)
h_t = o_t · tanh(C_t)

∂L/∂o_t = ∂L/∂h_t · tanh(C_t)

3. LSTM vs GRU Comparison

GRU Architecture:

GRU Equations:
1. Update Gate: z_t = σ(W_z · [h_{t-1}, x_t] + b_z)
2. Reset Gate: r_t = σ(W_r · [h_{t-1}, x_t] + b_r)
3. Candidate Hidden: ĥ_t = tanh(W_h · [r_t * h_{t-1}, x_t] + b_h)
4. Hidden State: h_t = (1 - z_t) * h_{t-1} + z_t * ĥ_t

Comparison Table:

Feature	LSTM	GRU
Gates	3 (Forget, Input, Output)	2 (Reset, Update)
Cell State	Separate C_t	Merged with h_t
Parameters	4 × (d_h × (d_x + d_h) + d_h)	3 × (d_h × (d_x + d_h) + d_h)
Memory	Long-term (cell state)	Short-term (hidden state)
Training Speed	Slower	Faster
Performance on Long Sequences	Better	Worse
Overfitting Risk	Higher	Lower

GRU Implementation:

class GRUCell:
    """
    Gated Recurrent Unit implementation
    """
    def __init__(self, input_size, hidden_size):
        self.input_size = input_size
        self.hidden_size = hidden_size
        
        # Update gate weights
        self.W_z = np.random.randn(hidden_size, input_size + hidden_size) * 0.01
        self.b_z = np.zeros((hidden_size, 1))
        
        # Reset gate weights
        self.W_r = np.random.randn(hidden_size, input_size + hidden_size) * 0.01
        self.b_r = np.zeros((hidden_size, 1))
        
        # Candidate hidden weights
        self.W_h = np.random.randn(hidden_size, input_size + hidden_size) * 0.01
        self.b_h = np.zeros((hidden_size, 1))
    
    def forward(self, x_t, h_prev):
        """
        Forward pass through GRU cell
        """
        combined = np.vstack([h_prev, x_t])
        
        # Update gate
        z_t = self.sigmoid(np.dot(self.W_z, combined) + self.b_z)
        
        # Reset gate
        r_t = self.sigmoid(np.dot(self.W_r, combined) + self.b_r)
        
        # Candidate hidden state
        combined_reset = np.vstack([r_t * h_prev, x_t])
        h_tilde = np.tanh(np.dot(self.W_h, combined_reset) + self.b_h)
        
        # Final hidden state
        h_t = (1 - z_t) * h_prev + z_t * h_tilde
        
        return h_t
    
    def sigmoid(self, x):
        return 1 / (1 + np.exp(-np.clip(x, -500, 500)))

4. Stacked LSTM for Financial Forecasting

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Dropout, BatchNormalization

class StackedLSTM:
    """
    Stacked LSTM model for financial forecasting
    """
    def __init__(self, input_shape, hidden_units=[64, 32], output_units=1, dropout_rate=0.2):
        self.model = self.build_model(input_shape, hidden_units, output_units, dropout_rate)
    
    def build_model(self, input_shape, hidden_units, output_units, dropout_rate):
        """
        Build stacked LSTM architecture
        """
        model = Sequential()
        
        # First LSTM layer (return sequences for stacking)
        model.add(LSTM(hidden_units[0], input_shape=input_shape, return_sequences=True))
        model.add(BatchNormalization())
        model.add(Dropout(dropout_rate))
        
        # Second LSTM layer
        if len(hidden_units) > 1:
            for i in range(1, len(hidden_units)-1):
                model.add(LSTM(hidden_units[i], return_sequences=True))
                model.add(BatchNormalization())
                model.add(Dropout(dropout_rate))
            
            # Last LSTM layer (doesn't return sequences)
            model.add(LSTM(hidden_units[-1]))
            model.add(BatchNormalization())
            model.add(Dropout(dropout_rate))
        else:
            model.add(LSTM(hidden_units[0]))
            model.add(BatchNormalization())
            model.add(Dropout(dropout_rate))
        
        # Output layer
        model.add(Dense(output_units, activation='linear'))
        
        # Compile model
        model.compile(optimizer='adam', loss='mse', metrics=['mae'])
        
        return model
    
    def fit(self, X_train, y_train, X_val, y_val, epochs=100, batch_size=32):
        """
        Train the stacked LSTM model
        """
        history = self.model.fit(
            X_train, y_train,
            validation_data=(X_val, y_val),
            epochs=epochs,
            batch_size=batch_size,
            verbose=1
        )
        return history
    
    def predict(self, X_test):
        """
        Make predictions
        """
        return self.model.predict(X_test)

5. Volatility Forecasting with LSTM

class LSTMVolatilityForecaster:
    """
    LSTM-based volatility forecasting
    """
    def __init__(self, lookback=60, hidden_units=[64, 32]):
        self.lookback = lookback
        self.model = self.build_model(hidden_units)
    
    def build_model(self, hidden_units):
        """
        Build LSTM for volatility forecasting
        """
        model = Sequential([
            LSTM(hidden_units[0], input_shape=(self.lookback, 1), return_sequences=True),
            Dropout(0.2),
            LSTM(hidden_units[1], return_sequences=False),
            Dropout(0.2),
            Dense(16, activation='relu'),
            Dense(1, activation='linear')
        ])
        
        model.compile(optimizer='adam', loss='mse', metrics=['mae'])
        return model
    
    def fit(self, returns, epochs=50, batch_size=32):
        """
        Train volatility forecaster
        """
        # Create features (squared returns for volatility)
        vol_features = returns**2
        
        # Create sequences
        X, y = [], []
        for i in range(self.lookback, len(vol_features)):
            X.append(vol_features[i-self.lookback:i])
            y.append(vol_features[i])
        
        X = np.array(X)
        y = np.array(y)
        
        # Train/test split
        split_idx = int(0.8 * len(X))
        X_train, X_test = X[:split_idx], X[split_idx:]
        y_train, y_test = y[:split_idx], y[split_idx:]
        
        # Train model
        history = self.model.fit(
            X_train, y_train,
            validation_data=(X_test, y_test),
            epochs=epochs,
            batch_size=batch_size,
            verbose=1
        )
        
        return history
    
    def predict(self, returns):
        """
        Forecast volatility
        """
        vol_features = returns[-self.lookback:]**2
        X = vol_features.reshape(1, self.lookback, 1)
        return self.model.predict(X)[0][0]

6. Attention in LSTMs

class AttentionLSTM:
    """
    LSTM with Attention Mechanism
    """
    def __init__(self, input_shape, hidden_units=64, output_units=1):
        self.model = self.build_model(input_shape, hidden_units, output_units)
    
    def build_model(self, input_shape, hidden_units, output_units):
        """
        Build LSTM with attention
        """
        inputs = tf.keras.Input(shape=input_shape)
        
        # LSTM layer (return sequences for attention)
        lstm_out = tf.keras.layers.LSTM(hidden_units, return_sequences=True)(inputs)
        
        # Attention mechanism
        attention = tf.keras.layers.Dense(1, activation='tanh')(lstm_out)
        attention = tf.keras.layers.Flatten()(attention)
        attention = tf.keras.layers.Activation('softmax')(attention)
        attention = tf.keras.layers.RepeatVector(hidden_units)(attention)
        attention = tf.keras.layers.Permute([2, 1])(attention)
        
        # Apply attention
        attended = tf.keras.layers.Multiply()([lstm_out, attention])
        attended = tf.keras.layers.GlobalAveragePooling1D()(attended)
        
        # Output layer
        outputs = tf.keras.layers.Dense(output_units, activation='linear')(attended)
        
        model = tf.keras.Model(inputs, outputs)
        model.compile(optimizer='adam', loss='mse', metrics=['mae'])
        
        return model

Certificate in FinTech Lesson 2.6: Long Short-Term Memory (LSTM) Networks and Gated Architectures

Certificate in FinTech