Lesson 5.4: Recurrent Neural Networks – LSTMs, GRUs, and Attention for Financial Time Series

1. Learning Objectives

By the end of this lesson, you will be able to:

Understand the mathematical formulations of RNNs, LSTMs, and GRUs.
Derive the LSTM forward and backward passes from first principles.
Implement LSTMs and GRUs in PyTorch for financial time series forecasting.
Apply sequence models to volatility forecasting, price prediction, and trading signal generation.
Implement bidirectional LSTMs for event studies and sentiment analysis.
Apply attention mechanisms to RNNs for improved long-range dependencies.
Understand the limitations of RNNs and when to use Transformers instead.
Implement sequence-to-sequence models for financial applications.

2. The Vanishing Gradient Problem – Why RNNs Fail

2.1 RNN Mathematical Formulation
h_t = tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)
y_t = W_{hy} h_t + b_y

2.2 The Gradient Problem
The gradient of the loss with respect to the weights at time t involves:
∂h_t/∂h_{t-1}} = W_{hh} * diag(1 - h_t²)

If the eigenvalues of W_{hh} are less than 1, gradients vanish exponentially.
If greater than 1, gradients explode.

Financial Implication: Standard RNNs cannot capture long-term dependencies (e.g., 5-year economic cycles). LSTMs were designed to solve this.

3. Long Short-Term Memory (LSTM) – The Complete Derivation

3.1 LSTM Gates

Forget Gate: f_t = σ(W_f · [h_{t-1}, x_t] + b_f)
Input Gate: i_t = σ(W_i · [h_{t-1}, x_t] + b_i)
Candidate Cell: \tilde{c}_t = tanh(W_c · [h_{t-1}, x_t] + b_c)
Cell State Update: c_t = f_t * c_{t-1} + i_t * \tilde{c}_t
Output Gate: o_t = σ(W_o · [h_{t-1}, x_t] + b_o)
Hidden State: h_t = o_t * tanh(c_t)

3.2 LSTM Implementation from Scratch

import numpy as np

class LSTMCell:
    def __init__(self, input_size, hidden_size):
        self.input_size = input_size
        self.hidden_size = hidden_size

        # Weight matrices
        self.W_f = np.random.randn(hidden_size, input_size + hidden_size) * 0.01
        self.W_i = np.random.randn(hidden_size, input_size + hidden_size) * 0.01
        self.W_c = np.random.randn(hidden_size, input_size + hidden_size) * 0.01
        self.W_o = np.random.randn(hidden_size, input_size + hidden_size) * 0.01

        # Biases
        self.b_f = np.zeros((hidden_size, 1))
        self.b_i = np.zeros((hidden_size, 1))
        self.b_c = np.zeros((hidden_size, 1))
        self.b_o = np.zeros((hidden_size, 1))

    def forward(self, x, h_prev, c_prev):
        # Concatenate input and previous hidden state
        combined = np.vstack([h_prev, x])  # (input_size + hidden_size, 1)

        # Gate calculations
        f = self.sigmoid(np.dot(self.W_f, combined) + self.b_f)
        i = self.sigmoid(np.dot(self.W_i, combined) + self.b_i)
        c_tilde = np.tanh(np.dot(self.W_c, combined) + self.b_c)
        o = self.sigmoid(np.dot(self.W_o, combined) + self.b_o)

        # Cell state update
        c = f * c_prev + i * c_tilde

        # Hidden state
        h = o * np.tanh(c)

        return h, c

    def sigmoid(self, x):
        return 1 / (1 + np.exp(-x))

3.3 LSTM Implementation in PyTorch

import torch
import torch.nn as nn

class LSTMTimeSeries(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim, dropout_rate=0.3):
        super(LSTMTimeSeries, self).__init__()

        self.hidden_dim = hidden_dim
        self.num_layers = num_layers

        self.lstm = nn.LSTM(
            input_size=input_dim,
            hidden_size=hidden_dim,
            num_layers=num_layers,
            batch_first=True,
            dropout=dropout_rate if num_layers > 1 else 0,
            bidirectional=False
        )

        self.fc = nn.Linear(hidden_dim, output_dim)
        self.dropout = nn.Dropout(dropout_rate)

    def forward(self, x):
        # x: (batch_size, seq_len, input_dim)

        # Initialize hidden and cell states
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).to(x.device)
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).to(x.device)

        # Forward pass
        out, _ = self.lstm(x, (h0, c0))

        # Take the last time step
        out = out[:, -1, :]
        out = self.dropout(out)
        out = self.fc(out)

        return out

4. Gated Recurrent Unit (GRU) – The Efficient Alternative

4.1 GRU Mathematical Formulation

Reset Gate: r_t = σ(W_r · [h_{t-1}, x_t] + b_r)
Update Gate: z_t = σ(W_z · [h_{t-1}, x_t] + b_z)
Candidate Hidden: \tilde{h}_t = tanh(W_h · [r_t * h_{t-1}, x_t] + b_h)
Hidden State: h_t = (1 - z_t) * h_{t-1} + z_t * \tilde{h}_t

4.2 GRU Implementation

class GRUTimeSeries(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim, dropout_rate=0.3):
        super(GRUTimeSeries, self).__init__()

        self.gru = nn.GRU(
            input_size=input_dim,
            hidden_size=hidden_dim,
            num_layers=num_layers,
            batch_first=True,
            dropout=dropout_rate if num_layers > 1 else 0,
            bidirectional=False
        )

        self.fc = nn.Linear(hidden_dim, output_dim)
        self.dropout = nn.Dropout(dropout_rate)

    def forward(self, x):
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).to(x.device)
        out, _ = self.gru(x, h0)
        out = out[:, -1, :]
        out = self.dropout(out)
        out = self.fc(out)
        return out

4.3 LSTM vs GRU – Which to Use?

Feature	LSTM	GRU
Parameters	More (4 gates)	Fewer (2 gates)
Training Speed	Slower	Faster
Performance	Better for long sequences	Similar for medium sequences
Memory	More (cell state + hidden)	Less (hidden only)
Best For	Very long dependencies (>100 steps)	Sequences of 10-100 steps

Financial Recommendation: Start with GRU for financial time series (60-90 day sequences). Use LSTM if you need to capture longer-term patterns.

5. Bidirectional LSTMs – Context from Both Directions

Bidirectional LSTMs process the sequence forward and backward, capturing future context.

5.1 Architecture

Input: [x_1, x_2, ..., x_T]

Forward LSTM: [h_1^f, h_2^f, ..., h_T^f]
Backward LSTM: [h_1^b, h_2^b, ..., h_T^b]

Output: [h_1^f ⊕ h_1^b, h_2^f ⊕ h_2^b, ..., h_T^f ⊕ h_T^b]

5.2 Implementation

class BidirectionalLSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim, dropout_rate=0.3):
        super(BidirectionalLSTM, self).__init__()

        self.lstm = nn.LSTM(
            input_size=input_dim,
            hidden_size=hidden_dim,
            num_layers=num_layers,
            batch_first=True,
            dropout=dropout_rate if num_layers > 1 else 0,
            bidirectional=True
        )

        # Hidden dimension is doubled (forward + backward)
        self.fc = nn.Linear(hidden_dim * 2, output_dim)
        self.dropout = nn.Dropout(dropout_rate)

    def forward(self, x):
        out, _ = self.lstm(x)
        out = out[:, -1, :]  # Last time step, both directions
        out = self.dropout(out)
        out = self.fc(out)
        return out

5.3 Financial Application – Sentiment Analysis

def sentiment_analysis(text_sequences):
    """
    Bidirectional LSTM for financial sentiment analysis.
    """
    # Input: sequences of word embeddings
    # Output: sentiment score (-1 to 1)

    class SentimentLSTM(nn.Module):
        def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim=1):
            super(SentimentLSTM, self).__init__()

            self.embedding = nn.Embedding(vocab_size, embedding_dim)
            self.lstm = nn.LSTM(
                input_size=embedding_dim,
                hidden_size=hidden_dim,
                num_layers=2,
                batch_first=True,
                bidirectional=True,
                dropout=0.3
            )
            self.fc = nn.Linear(hidden_dim * 2, output_dim)

        def forward(self, x):
            embedded = self.embedding(x)
            out, _ = self.lstm(embedded)
            out = out[:, -1, :]
            out = torch.tanh(self.fc(out))
            return out

    model = SentimentLSTM(vocab_size=50000, embedding_dim=300, hidden_dim=128)
    return model

6. Attention Mechanisms for RNNs

Attention allows the model to focus on the most relevant parts of the sequence.

6.1 Additive Attention (Bahdanau)
e_{t,j} = v_a^T tanh(W_a h_t + U_a h_j)
α_{t,j} = softmax(e_{t,j})
c_t = Σ_j α_{t,j} h_j

6.2 Multiplicative Attention (Luong)
score(h_t, h_j) = h_t^T W_a h_j
α_{t,j} = softmax(score(h_t, h_j))
c_t = Σ_j α_{t,j} h_j

6.3 Implementation

class AttentionLSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim):
        super(AttentionLSTM, self).__init__()

        self.lstm = nn.LSTM(
            input_size=input_dim,
            hidden_size=hidden_dim,
            num_layers=num_layers,
            batch_first=True,
            bidirectional=False
        )

        # Attention layer
        self.attention = nn.Linear(hidden_dim, 1)
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        # LSTM output
        lstm_out, _ = self.lstm(x)  # (batch, seq_len, hidden_dim)

        # Attention weights
        attention_weights = torch.softmax(self.attention(lstm_out), dim=1)
        attention_weights = attention_weights / (attention_weights.sum(dim=1, keepdim=True) + 1e-8)

        # Context vector (weighted sum)
        context = torch.sum(attention_weights * lstm_out, dim=1)

        # Final prediction
        output = self.fc(context)

        return output, attention_weights

7. Sequence-to-Sequence Models for Financial Applications

7.1 Encoder-Decoder Architecture

class Seq2SeqLSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim, num_layers):
        super(Seq2SeqLSTM, self).__init__()

        # Encoder
        self.encoder = nn.LSTM(
            input_size=input_dim,
            hidden_size=hidden_dim,
            num_layers=num_layers,
            batch_first=True
        )

        # Decoder
        self.decoder = nn.LSTM(
            input_size=hidden_dim,
            hidden_size=hidden_dim,
            num_layers=num_layers,
            batch_first=True
        )

        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        # Encode
        encoder_out, (hidden, cell) = self.encoder(x)

        # Decode (using the last hidden state as initial)
        decoder_input = hidden[-1].unsqueeze(1).repeat(1, x.size(1), 1)
        decoder_out, _ = self.decoder(decoder_input, (hidden, cell))

        # Output
        output = self.fc(decoder_out)
        return output

7.2 Financial Application – Volatility Forecasting

def volatility_forecast(prices, returns, vols):
    """
    Seq2Seq model for volatility forecasting.
    """
    # Input: price returns and other features
    # Output: next N days of volatility forecasts

    class VolatilitySeq2Seq(nn.Module):
        def __init__(self, input_dim=10, hidden_dim=64, output_dim=1):
            super(VolatilitySeq2Seq, self).__init__()

            self.encoder = nn.LSTM(input_dim, hidden_dim, num_layers=2, batch_first=True)
            self.decoder = nn.LSTM(hidden_dim, hidden_dim, num_layers=2, batch_first=True)
            self.fc = nn.Linear(hidden_dim, output_dim)

        def forward(self, x, forecast_steps=5):
            # Encode
            encoder_out, (hidden, cell) = self.encoder(x)

            # Initialize decoder with encoder's final state
            decoder_input = hidden[-1].unsqueeze(1).repeat(1, forecast_steps, 1)

            # Decode
            decoder_out, _ = self.decoder(decoder_input, (hidden, cell))

            # Output forecasts
            forecasts = self.fc(decoder_out)
            return forecasts

    model = VolatilitySeq2Seq()
    return model

8. Summary for the AI Practitioner

RNNs capture temporal dependencies but suffer from vanishing gradients.
LSTMs solve the vanishing gradient problem with gating mechanisms (Forget, Input, Output gates).
GRUs are a simplified version with fewer parameters. They often perform similarly to LSTMs on financial data.
Bidirectional LSTMs capture context from both directions. Useful for sentiment analysis and event studies.
Attention mechanisms allow the model to focus on the most relevant time steps. Essential for long sequences.
Sequence-to-Sequence models are used for multi-step forecasting (e.g., volatility forecasting).
Practical advice: Start with GRU for 60-90 day sequences. Use LSTM for longer dependencies. Add attention if the sequence length exceeds 100.
Training: Use gradient clipping (max_norm=1.0) to prevent exploding gradients. Use dropout (0.2-0.5) to prevent overfitting.

Diploma in Artificial Intelligence in Finance Lesson 5.4: Recurrent Neural Networks – LSTMs, GRUs, and Attention for Financial Time Series

Diploma in Artificial Intelligence in Finance