1. Learning Objectives

By the end of this lesson, you will be able to:

  • Understand the limitations of feedforward networks for sequential financial data.

  • Derive the mathematical formulation of LSTMs and GRUs and implement them in PyTorch.

  • Build a sequence-to-sequence LSTM for financial time series forecasting.

  • Implement and train Transformer models for financial sequences using self-attention.

  • Compare LSTM, GRU, and Transformer performance on financial data.

  • Understand and implement multi-head attention and positional encoding.

  • Apply sequence models to volatility forecasting, price prediction, and trading signal generation.

  • Implement attention visualisation to interpret model decisions.

2. Why Sequence Models for Financial Time Series?

Financial data has temporal dependencies: today’s returns depend on yesterday’s returns (autocorrelation), and volatility clusters (heteroskedasticity). Feedforward networks treat each time step independently, losing the temporal structure. Sequence models (RNNs, LSTMs, Transformers) capture these dependencies.

The Sequence Problem:

text
Input: X = [x_1, x_2, ..., x_T] where x_t is the feature vector at time t.
Output: y = [y_1, y_2, ..., y_T] or y_{T+1} (next-step prediction).

Why RNNs Fail: Vanishing/exploding gradients. Long-term dependencies are lost due to repeated multiplication of weight matrices.

3. Recurrent Neural Networks (RNNs) – The Foundation

3.1 Mathematical Formulation
For a sequence of inputs x_t, the hidden state h_t is updated as:
h_t = tanh( W_{hh} h_{t-1} + W_{xh} x_t + b_h )
y_t = W_{hy} h_t + b_y
where:

  • W_{hh}: Hidden-to-hidden weights (state transition).

  • W_{xh}: Input-to-hidden weights.

  • W_{hy}: Hidden-to-output weights.

3.2 The Vanishing Gradient Problem
The gradient of the loss with respect to the weights at time t involves repeated multiplication of ∂h_t/∂h_{t-1}} = W_{hh} * diag(1 - h_t²). If W_{hh} has eigenvalues < 1, gradients vanish exponentially. If > 1, gradients explode.

Financial Implication: RNNs cannot capture long-term market cycles (e.g., 5-year economic cycles) due to vanishing gradients.

4. Long Short-Term Memory (LSTM) – The Workhorse of Financial Time Series

LSTMs were designed to combat vanishing gradients by introducing a cell state c_t that flows through the network with minimal modification.

4.1 The LSTM Architecture – Complete Derivation

The LSTM has three gates: Forget Gate, Input Gate, and Output Gate. Each gate is a sigmoid function (values in [0, 1]) that controls the flow of information.

Forget Gate: Decides what information to discard from the cell state.
f_t = σ( W_f · [h_{t-1}, x_t] + b_f )

  • σ is the sigmoid function: σ(z) = 1 / (1 + e^{-z}).

  • f_t ∈ [0, 1]. f_t = 0 means “forget everything,” f_t = 1 means “remember everything.”

Input Gate: Decides what new information to store in the cell state.
i_t = σ( W_i · [h_{t-1}, x_t] + b_i )
\tilde{c}_t = tanh( W_c · [h_{t-1}, x_t] + b_c )

  • i_t is the “input gate” (what to write).

  • \tilde{c}_t is the candidate cell state (new information).

Cell State Update: Combine the forget gate and input gate.
c_t = f_t * c_{t-1} + i_t * \tilde{c}_t

  • The * is element-wise multiplication (Hadamard product).

  • This is the key innovation: the cell state is a linear combination of the old state and the new candidate, so gradients flow smoothly.

Output Gate: Decides what to output from the cell state.
o_t = σ( W_o · [h_{t-1}, x_t] + b_o )
h_t = o_t * tanh(c_t)

  • o_t is the output gate.

  • h_t is the hidden state (also the output of the LSTM cell).

4.2 LSTM Implementation in PyTorch

text
class LSTMTimeSeries(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim, dropout_rate=0.3):
        super(LSTMTimeSeries, self).__init__()

        self.hidden_dim = hidden_dim
        self.num_layers = num_layers

        # LSTM layer
        self.lstm = nn.LSTM(
            input_size=input_dim,
            hidden_size=hidden_dim,
            num_layers=num_layers,
            batch_first=True,  # (batch, seq_len, features)
            dropout=dropout_rate if num_layers > 1 else 0,
            bidirectional=False
        )

        # Output layer
        self.fc = nn.Linear(hidden_dim, output_dim)

        # Dropout for output
        self.dropout = nn.Dropout(dropout_rate)

    def forward(self, x):
        # x shape: (batch_size, seq_len, input_dim)

        # Initialise hidden and cell states
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).to(x.device)
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).to(x.device)

        # Forward pass through LSTM
        out, (hn, cn) = self.lstm(x, (h0, c0))

        # Take the last time step output
        out = out[:, -1, :]  # (batch_size, hidden_dim)

        # Output layer
        out = self.dropout(out)
        out = self.fc(out)  # (batch_size, output_dim)

        return out

4.3 Stacked LSTMs for Complex Patterns

text
class StackedLSTM(nn.Module):
    def __init__(self, input_dim, hidden_dims, output_dim, dropout_rate=0.3):
        super(StackedLSTM, self).__init__()

        self.hidden_dims = hidden_dims
        self.num_layers = len(hidden_dims)

        # Build stacked LSTM layers
        lstm_layers = []
        prev_dim = input_dim
        for i, hidden_dim in enumerate(hidden_dims):
            lstm_layers.append(
                nn.LSTM(
                    input_size=prev_dim,
                    hidden_size=hidden_dim,
                    num_layers=1,
                    batch_first=True,
                    dropout=0.0,  # Handled separately
                    bidirectional=False
                )
            )
            prev_dim = hidden_dim

        self.lstm_layers = nn.ModuleList(lstm_layers)
        self.dropout = nn.Dropout(dropout_rate)
        self.fc = nn.Linear(hidden_dims[-1], output_dim)

    def forward(self, x):
        # Pass through each LSTM layer
        for i, lstm in enumerate(self.lstm_layers):
            x, _ = lstm(x)
            if i < self.num_layers - 1:
                x = self.dropout(x)

        # Take last time step
        x = x[:, -1, :]
        x = self.dropout(x)
        x = self.fc(x)
        return x

4.4 Bidirectional LSTMs
Bidirectional LSTMs process the sequence in both forward and backward directions, capturing future context. This is useful for event studies where the label depends on both past and future information.

text
class BidirectionalLSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim, dropout_rate=0.3):
        super(BidirectionalLSTM, self).__init__()

        self.lstm = nn.LSTM(
            input_size=input_dim,
            hidden_size=hidden_dim,
            num_layers=num_layers,
            batch_first=True,
            dropout=dropout_rate if num_layers > 1 else 0,
            bidirectional=True  # Key: bidirectional
        )

        # Hidden dimension is doubled (forward + backward)
        self.fc = nn.Linear(hidden_dim * 2, output_dim)
        self.dropout = nn.Dropout(dropout_rate)

    def forward(self, x):
        out, _ = self.lstm(x)
        out = out[:, -1, :]  # Last time step, both directions
        out = self.dropout(out)
        out = self.fc(out)
        return out

5. Gated Recurrent Units (GRU) – The Efficient Alternative

GRUs are a simplified version of LSTMs with two gates instead of three. They often perform similarly to LSTMs with fewer parameters.

5.1 GRU Mathematical Formulation

Reset Gate: r_t = σ( W_r · [h_{t-1}, x_t] + b_r )

  • Controls how much of the past hidden state to forget.

Update Gate: z_t = σ( W_z · [h_{t-1}, x_t] + b_z )

  • Controls how much of the new state to use.

Candidate Hidden State:
\tilde{h}_t = tanh( W_h · [r_t * h_{t-1}, x_t] + b_h )

Hidden State Update:
h_t = (1 - z_t) * h_{t-1} + z_t * \tilde{h}_t

5.2 GRU Implementation

text
class GRUTimeSeries(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim, dropout_rate=0.3):
        super(GRUTimeSeries, self).__init__()

        self.gru = nn.GRU(
            input_size=input_dim,
            hidden_size=hidden_dim,
            num_layers=num_layers,
            batch_first=True,
            dropout=dropout_rate if num_layers > 1 else 0,
            bidirectional=False
        )

        self.fc = nn.Linear(hidden_dim, output_dim)
        self.dropout = nn.Dropout(dropout_rate)

    def forward(self, x):
        out, _ = self.gru(x)
        out = out[:, -1, :]
        out = self.dropout(out)
        out = self.fc(out)
        return out

5.3 LSTM vs GRU – Which to Use?

  • LSTM: More parameters, better for very long sequences (>100 steps). Captures more complex dependencies.

  • GRU: Fewer parameters, faster training, often comparable performance on medium-length sequences. Good for financial data with 10-50 step sequences.

6. Transformers – The Attention Revolution

Transformers use self-attention to capture dependencies between all positions in the sequence simultaneously. They have completely replaced LSTMs in many financial AI applications (e.g., NLP for earnings calls, multi-asset forecasting).

6.1 Self-Attention – The Core Mechanism

For a sequence of inputs X = [x_1, ..., x_T], we compute:

Query, Key, Value: For each token, compute linear projections:
Q = X W_Q, K = X W_K, V = X W_V

Attention Scores:
Attention(Q, K, V) = softmax( Q K^T / sqrt(d_k) ) V
where d_k is the dimension of the key vectors.

Interpretation: Q K^T computes the similarity between every pair of tokens. The softmax converts these similarities to attention weights. The output is a weighted sum of the values V.

6.2 Multi-Head Attention

Multi-head attention runs multiple attention mechanisms in parallel, allowing the model to focus on different aspects of the sequence.

MultiHead(Q, K, V) = Concat(head_1, ..., head_h) W_O
where head_i = Attention(Q W_i^Q, K W_i^K, V W_i^V).

6.3 Positional Encoding

Since the Transformer has no inherent notion of sequence order, we add positional encodings to the input embeddings:
PE(pos, 2i) = sin(pos / 10000^{2i/d_model})
PE(pos, 2i+1) = cos(pos / 10000^{2i/d_model})
where pos is the position in the sequence and i is the dimension.

6.4 Transformer Implementation for Financial Time Series

text
import math

class PositionalEncoding(nn.Module):
    def __init__(self, d_model, max_len=1000):
        super(PositionalEncoding, self).__init__()

        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))

        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0)  # (1, max_len, d_model)

        self.register_buffer('pe', pe)

    def forward(self, x):
        return x + self.pe[:, :x.size(1), :]

class TransformerTimeSeries(nn.Module):
    def __init__(self, input_dim, d_model, nhead, num_layers, output_dim, dropout_rate=0.1):
        super(TransformerTimeSeries, self).__init__()

        self.input_proj = nn.Linear(input_dim, d_model)
        self.pos_encoder = PositionalEncoding(d_model)

        encoder_layer = nn.TransformerEncoderLayer(
            d_model=d_model,
            nhead=nhead,
            dim_feedforward=d_model * 4,
            dropout=dropout_rate,
            batch_first=True
        )
        self.transformer_encoder = nn.TransformerEncoder(
            encoder_layer, num_layers=num_layers
        )

        self.fc = nn.Linear(d_model, output_dim)
        self.dropout = nn.Dropout(dropout_rate)

    def forward(self, x):
        # x shape: (batch_size, seq_len, input_dim)
        x = self.input_proj(x)  # (batch_size, seq_len, d_model)
        x = self.pos_encoder(x)
        x = self.transformer_encoder(x)
        x = x[:, -1, :]  # Take last time step
        x = self.dropout(x)
        x = self.fc(x)
        return x

7. Sequence Preprocessing for Financial Data

7.1 Creating Sequences

text
def create_sequences(data, seq_length, target_col):
    """
    Convert financial time series into supervised learning sequences.
    """
    X, y = [], []
    for i in range(seq_length, len(data)):
        X.append(data.iloc[i-seq_length:i].values)
        y.append(data.iloc[i][target_col])
    return np.array(X), np.array(y)

# Example
seq_length = 60  # 60 days of history
X_train, y_train = create_sequences(train_data, seq_length, 'Return')
X_val, y_val = create_sequences(val_data, seq_length, 'Return')

7.2 Sequence Normalisation

text
def normalise_sequences(X, mean=None, std=None):
    """
    Normalise sequences using the first point in each sequence.
    This preserves the relative movement within each sequence.
    """
    if mean is None:
        mean = X.mean(axis=1, keepdims=True)
        std = X.std(axis=1, keepdims=True) + 1e-8
    X_norm = (X - mean) / std
    return X_norm, mean, std

# Alternative: Use the first value as a reference point
def normalise_by_first(X):
    X_first = X[:, 0:1, :]  # First time step
    X_norm = X / (X_first + 1e-8)
    return X_norm

8. Attention Visualisation – Interpreting the Model

Visualising attention weights helps understand what the model is focusing on.

text
def plot_attention_weights(attention_weights, labels):
    """
    Visualise attention weights as a heatmap.
    """
    import matplotlib.pyplot as plt
    import seaborn as sns

    plt.figure(figsize=(12, 8))
    sns.heatmap(attention_weights, xticklabels=labels, yticklabels=labels,
                cmap='Blues', annot=True, fmt='.2f')
    plt.title('Attention Weights')
    plt.xlabel('Key Positions')
    plt.ylabel('Query Positions')
    plt.show()

# Extract attention weights from the Transformer
def get_attention_weights(model, X_batch):
    model.eval()
    with torch.no_grad():
        # Forward pass with attention returns
        # This requires modifying the forward method to return attention weights
        pass

9. Hyperparameter Tuning for Sequence Models

 
 
Hyperparameter LSTM/GRU Transformer Financial Recommendation
Sequence Length 20-100 20-100 60 (3 months) for daily data
Hidden Dim 32-256 64-512 128 for most problems
Number of Layers 2-4 2-6 2-3 for LSTM; 3-4 for Transformer
Dropout Rate 0.2-0.5 0.1-0.3 0.3 for noisy data
Learning Rate 1e-4 to 1e-2 1e-5 to 1e-3 1e-3 for LSTM; 1e-4 for Transformer
Batch Size 32-256 32-256 128 for GPU efficiency
Number of Heads N/A 4-16 8 for most problems

10. Summary for the AI Practitioner

  1. LSTMs capture long-term dependencies via the cell state. The three gates (Forget, Input, Output) control information flow.

  2. GRUs are a simplified version with two gates (Reset, Update). They train faster and often perform similarly to LSTMs.

  3. Transformers use self-attention to capture dependencies between all positions simultaneously. They are superior for NLP and multi-asset forecasting.

  4. Sequence creation is critical: seq_length must be chosen carefully. For daily financial data, 60-90 days is common.

  5. Positional encoding is essential for Transformers since they have no inherent sequence order.

  6. Attention visualisation helps interpret model decisions and build trust with stakeholders.

  7. Bidirectional LSTMs are useful when the label depends on future context (e.g., event studies).

  8. LSTM vs Transformer: Use LSTM for smaller datasets (< 100k sequences) and Transformers for larger datasets with complex dependencies.

End of Lessons 3.3 and 3.4

In Lessons 3.5 and 3.6, we will cover Model Deployment and MLOps for Financial AI, focusing on serving models in production, monitoring for data drift, and implementing automated retraining pipelines.

 
 
PreviousNext