Lesson 5.8: Advanced Topics – Attention, Memory, and Hybrid Architectures for Finance

1. Learning Objectives

By the end of this lesson, you will be able to:

Understand the mathematical foundations of the Transformer-XL and its application to long financial sequences.
Implement the Longformer and its sparse attention mechanism for efficient long-sequence processing.
Understand the architecture of Memory-Augmented Neural Networks (MANNs) for financial applications.
Implement Neural Turing Machines (NTMs) and Differentiable Neural Computers (DNCs).
Understand the architecture of the Informer for long-sequence financial forecasting.
Apply hybrid architectures (CNN + LSTM + Attention) to multi-modal financial data.
Understand the limitations and trade-offs of different architectures.
Select the appropriate architecture for different financial tasks.

2. Transformer-XL – Capturing Longer Context

2.1 Limitations of Standard Transformers

Fixed context length: Cannot process sequences longer than the training length.
No cross-sequence information: Each sequence is processed independently.

2.2 Transformer-XL Innovations

Segment-Level Recurrence: Reuses hidden states from previous segments.

For segment τ and τ+1:
H^{(l+1)}_{τ} = Transformer(H^{(l)}_{τ}, H^{(l)}_{τ-1})

The hidden state from previous segment is cached and reused.

Relative Positional Encoding: Instead of absolute positions, use relative positions between tokens.

PE(pos1, pos2) = f(pos1 - pos2)

2.3 Benefits for Finance

Can process entire year of daily data (252 days) in one forward pass.
Captures long-term dependencies (annual patterns, business cycles).

2.4 Implementation (Simplified)

class TransformerXL(nn.Module):
    def __init__(self, d_model, n_heads, n_layers, dropout=0.1, memory_len=100):
        super(TransformerXL, self).__init__()

        self.memory_len = memory_len
        self.layers = nn.ModuleList([
            TransformerBlock(d_model, n_heads, d_model*4, dropout)
            for _ in range(n_layers)
        ])
        self.memory = None

    def forward(self, x):
        # x: (batch, seq_len, d_model)

        # Concatenate with memory
        if self.memory is not None:
            x = torch.cat([self.memory, x], dim=1)

        # Pass through layers
        for layer in self.layers:
            x = layer(x)

        # Update memory
        self.memory = x[:, -self.memory_len:, :].detach()

        return x

3. Longformer – Sparse Attention for Long Sequences

3.1 The Quadratic Complexity Problem
Standard attention: O(T² * d). For T=1000, this is 1 million operations per layer.

3.2 Longformer Attention Patterns

Sliding Window Attention: Each token attends to w tokens on each side.
Dilated Window Attention: Window with gaps, increasing receptive field.
Global Attention: Select tokens attend to all tokens (and vice versa).

3.3 Implementation (Using HuggingFace)

from transformers import LongformerModel, LongformerConfig

class FinancialLongformer(nn.Module):
    def __init__(self, n_features, d_model=256, n_heads=8, n_layers=4, window_size=10):
        super(FinancialLongformer, self).__init__()

        # Project inputs to d_model
        self.input_proj = nn.Linear(n_features, d_model)

        # Longformer configuration
        config = LongformerConfig(
            hidden_size=d_model,
            num_attention_heads=n_heads,
            num_hidden_layers=n_layers,
            attention_window=window_size,
            max_position_embeddings=1000
        )
        self.longformer = LongformerModel(config)

        # Output head
        self.fc = nn.Linear(d_model, 1)

    def forward(self, x):
        # x: (batch, seq_len, n_features)
        x = self.input_proj(x)

        # Longformer expects global attention mask
        global_attention_mask = torch.zeros_like(x[..., 0])
        global_attention_mask[:, 0] = 1  # First token has global attention

        # Forward pass
        outputs = self.longformer(
            x,
            attention_mask=None,
            global_attention_mask=global_attention_mask
        )

        # Pool and output
        pooled = outputs.last_hidden_state.mean(dim=1)
        return self.fc(pooled)

4. Memory-Augmented Neural Networks (MANNs)

4.1 The Memory-Augmented Framework

Controller: Neural network (LSTM, Transformer) that processes input.
Memory Matrix: M ∈ R^{N x D} (N memory slots, D dimensions).
Read Head: Retrieves information from memory.
Write Head: Writes information to memory.

4.2 Reading from Memory

Content-Based Addressing:
k_t = W_k h_t (key vector)
c_t = softmax( k_t^T M_t ) (content weights)
r_t = c_t^T M_t (read vector)

Location-Based Addressing:
Shift weights using a 1D convolution.

4.3 Writing to Memory

Erase:
M_t = M_{t-1} * (1 - w_t^e e_t^T) (erase with erase vector e_t)

Add:
M_t = M_t + w_t^a a_t^T (add with add vector a_t)

4.4 Implementation (Simplified NTM)

class NeuralTuringMachine(nn.Module):
    def __init__(self, input_dim, hidden_dim, memory_size, memory_dim, output_dim):
        super(NeuralTuringMachine, self).__init__()

        self.memory_size = memory_size
        self.memory_dim = memory_dim

        # Memory matrix
        self.memory = nn.Parameter(torch.randn(memory_size, memory_dim) * 0.01)

        # Controller (LSTM)
        self.controller = nn.LSTM(input_dim + memory_dim, hidden_dim, batch_first=True)

        # Read/Write heads
        self.read_head = nn.Linear(hidden_dim, memory_dim)
        self.write_head = nn.Linear(hidden_dim, memory_dim)
        self.erase_head = nn.Linear(hidden_dim, memory_dim)

        # Output
        self.fc = nn.Linear(hidden_dim + memory_dim, output_dim)

    def forward(self, x):
        batch_size = x.size(0)

        # Initial read (zero vector)
        r_t = torch.zeros(batch_size, self.memory_dim).to(x.device)

        outputs = []

        for t in range(x.size(1)):
            # Controller input: concatenate input with read
            c_in = torch.cat([x[:, t, :], r_t], dim=1)
            c_out, _ = self.controller(c_in.unsqueeze(1))

            # Read from memory
            k = self.read_head(c_out.squeeze(1))  # (batch, memory_dim)
            content_weights = F.softmax(k @ self.memory.T, dim=1)  # (batch, memory_size)
            r_t = content_weights @ self.memory  # (batch, memory_dim)

            # Write to memory
            w = F.softmax(self.write_head(c_out.squeeze(1)), dim=1)
            e = torch.sigmoid(self.erase_head(c_out.squeeze(1)))

            # Erase
            erase = w.unsqueeze(-1) * e.unsqueeze(1)  # (batch, memory_size, memory_dim)
            self.memory = self.memory.unsqueeze(0) * (1 - erase)
            self.memory = self.memory.squeeze(0)

            # Add
            add = w.unsqueeze(-1) * r_t.unsqueeze(1)  # (batch, memory_size, memory_dim)
            self.memory = self.memory.unsqueeze(0) + add
            self.memory = self.memory.squeeze(0)

            # Output
            out = self.fc(torch.cat([c_out.squeeze(1), r_t], dim=1))
            outputs.append(out)

        return torch.stack(outputs, dim=1)

4.5 Financial Application – Portfolio Optimisation with Memory

class MemoryPortfolioOptimiser(nn.Module):
    def __init__(self, n_assets, memory_size, memory_dim):
        super(MemoryPortfolioOptimiser, self).__init__()

        self.n_assets = n_assets
        self.memory_size = memory_size
        self.memory_dim = memory_dim

        # Memory
        self.memory = nn.Parameter(torch.randn(memory_size, memory_dim) * 0.01)

        # LSTM controller
        self.lstm = nn.LSTM(n_assets + memory_dim, 128, batch_first=True)

        # Heads
        self.read = nn.Linear(128, memory_dim)
        self.write = nn.Linear(128, memory_dim)
        self.erase = nn.Linear(128, memory_dim)

        # Portfolio weights
        self.fc = nn.Linear(128 + memory_dim, n_assets)

    def forward(self, returns):
        # returns: (batch, seq_len, n_assets)
        batch_size = returns.size(0)

        # Initial read
        r_t = torch.zeros(batch_size, self.memory_dim).to(returns.device)
        portfolio_weights = []

        for t in range(returns.size(1)):
            # Controller
            c_in = torch.cat([returns[:, t, :], r_t], dim=1).unsqueeze(1)
            c_out, _ = self.lstm(c_in)

            # Read
            k = self.read(c_out.squeeze(1))
            weights = F.softmax(k @ self.memory.T, dim=1)
            r_t = weights @ self.memory

            # Write
            w = F.softmax(self.write(c_out.squeeze(1)), dim=1)
            e = torch.sigmoid(self.erase(c_out.squeeze(1)))

            # Update memory (simplified)
            # (In practice, use differentiable operations)

            # Output weights
            combined = torch.cat([c_out.squeeze(1), r_t], dim=1)
            w_portfolio = F.softmax(self.fc(combined), dim=1)
            portfolio_weights.append(w_portfolio)

        return torch.stack(portfolio_weights, dim=1)

5. Informer – Long-Sequence Financial Forecasting

5.1 Architecture Innovations

ProbSparse Attention: Selects the most important query-key pairs.

Self-Attention Distillation: Reduces sequence length in each layer.

Generative Decoding: Generates long forecasts in one forward pass.

5.2 ProbSparse Attention

Instead of computing all T² attention pairs, compute only T * log(T) pairs.

For each query q_i, compute a sparsity score:
M(q_i, K) = max_j (q_i k_j^T / sqrt(d_k)) - mean_j (q_i k_j^T / sqrt(d_k))

Keep only the top T * log(T) queries with the highest scores.

5.3 Implementation (Simplified)

class InformerLayer(nn.Module):
    def __init__(self, d_model, n_heads, d_ff, dropout=0.1, factor=5):
        super(InformerLayer, self).__init__()

        self.factor = factor  # Sparsity factor

        # ProbSparse self-attention (simplified)
        self.attention = MultiHeadAttention(d_model, n_heads, dropout)

        self.ff = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.ReLU(),
            nn.Dropout(dropout),
            nn.Linear(d_ff, d_model)
        )
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        # ProbSparse attention
        attn_out, _ = self.attention(x, x, x)
        x = self.norm1(x + self.dropout(attn_out))

        # FF
        ff_out = self.ff(x)
        x = self.norm2(x + self.dropout(ff_out))

        # Distillation (reduce sequence length by factor 2)
        x = x[:, ::2, :]

        return x

6. Hybrid Architectures – Combining Strengths

6.1 CNN + LSTM + Attention

class HybridFinancialModel(nn.Module):
    def __init__(self, input_dim, seq_len, n_filters=64, hidden_dim=128, n_heads=8):
        super(HybridFinancialModel, self).__init__()

        # CNN for local patterns
        self.conv1 = nn.Conv1d(input_dim, n_filters, kernel_size=3, padding=1)
        self.conv2 = nn.Conv1d(n_filters, n_filters, kernel_size=3, padding=1)

        # LSTM for temporal patterns
        self.lstm = nn.LSTM(n_filters, hidden_dim, num_layers=2, batch_first=True, bidirectional=True)

        # Self-attention for long-range dependencies
        self.attention = MultiHeadAttention(hidden_dim * 2, n_heads)

        # Output
        self.fc = nn.Linear(hidden_dim * 2, 1)

    def forward(self, x):
        # x: (batch, seq_len, input_dim)

        # CNN (1D convolution over time)
        x = x.permute(0, 2, 1)  # (batch, input_dim, seq_len)
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = x.permute(0, 2, 1)  # (batch, seq_len, n_filters)

        # LSTM
        lstm_out, _ = self.lstm(x)  # (batch, seq_len, hidden_dim*2)

        # Attention
        attn_out, _ = self.attention(lstm_out, lstm_out, lstm_out)

        # Pool and output
        pooled = attn_out.mean(dim=1)  # (batch, hidden_dim*2)
        output = self.fc(pooled)

        return output

6.2 Multi-Modal Financial AI

Combines different data types: price data, text data, image data.

class MultiModalFinancialModel(nn.Module):
    def __init__(self, price_dim, text_dim, image_dim, hidden_dim=128):
        super(MultiModalFinancialModel, self).__init__()

        # Price encoder (LSTM)
        self.price_encoder = nn.LSTM(price_dim, hidden_dim, batch_first=True)

        # Text encoder (Transformer)
        self.text_encoder = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d_model=hidden_dim, nhead=8),
            num_layers=3
        )

        # Image encoder (CNN)
        self.image_encoder = nn.Sequential(
            nn.Conv2d(3, 32, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(32, 64, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.AdaptiveAvgPool2d(1)
        )

        # Fusion
        self.fusion = nn.Linear(hidden_dim * 3, hidden_dim)

        # Output
        self.fc = nn.Linear(hidden_dim, 1)

    def forward(self, price_data, text_data, image_data):
        # Price
        price_out, _ = self.price_encoder(price_data)
        price_features = price_out.mean(dim=1)

        # Text
        text_out = self.text_encoder(text_data)
        text_features = text_out.mean(dim=1)

        # Image
        image_out = self.image_encoder(image_data)
        image_features = image_out.reshape(image_out.size(0), -1)

        # Fusion
        combined = torch.cat([price_features, text_features, image_features], dim=1)
        fused = F.relu(self.fusion(combined))

        return self.fc(fused)

7. Architecture Selection Guide for Financial Tasks

Task	Recommended Architecture	Why
Short-term price prediction	LSTM/GRU	Captures temporal dependencies.
Long-term forecasting	Transformer/Informer	Captures long-range dependencies.
Volatility forecasting	GARCH + LSTM	Combines econometrics with deep learning.
Sentiment analysis	Bidirectional LSTM + Attention	Captures context from both directions.
Portfolio optimisation	GNN + Transformer	Captures asset relationships.
Anomaly detection	Autoencoder + LSTM	Reconstruction error identifies anomalies.
Credit scoring	Gradient Boosting	Interpretability and performance.
High-frequency trading	Linear models + SGD	Speed is critical.
Alternative data	CNN + Transformer	Handles images and text.

8. Summary for the AI Practitioner

Transformer-XL extends Transformers with segment-level recurrence and relative positional encoding for long sequences.
Longformer reduces quadratic complexity with sparse attention (sliding window, global attention).
Memory-Augmented Networks (NTMs, DNCs) provide external memory for long-term storage and retrieval.
Informer uses ProbSparse attention for efficient long-sequence forecasting.
Hybrid architectures combine CNN (local), LSTM (temporal), and Attention (long-range) for multi-modal data.
Architecture selection depends on the task: short-term → LSTM, long-term → Transformer, multi-asset → GNN.
Interpretability is a challenge with complex architectures. Use SHAP/LIME for explanations.
Always benchmark against simpler models before deploying complex architectures.

Diploma in Artificial Intelligence in Finance Lesson 5.8: Advanced Topics – Attention, Memory, and Hybrid Architectures for Finance

Diploma in Artificial Intelligence in Finance