lesson4.1:Deep Learning Architectures for Financial Time-Series Forecasting

Lesson Overview

Learning Objectives:

Master the complete deep learning architecture landscape for financial forecasting
Understand the fundamental limitations of linear econometric models
Learn RNN and LSTM architectures for sequential financial data
Master CNNs for order book and volatility surface processing
Understand Transformers and self-attention for global financial context

Part 1: Moving Beyond Linear Time-Series Models

1.1: The Limitations of Classical Linear Models

Classical quantitative finance relied on linear econometric models such as ARIMA and GARCH to forecast asset prices and volatility. While interpretable, these models fail under the complex, non-linear, high-noise dynamics of modern markets. Financial time series exhibit time-varying volatility, non-stationary structural breaks, and multi-horizon dependencies.

Deep Learning architectures overcome these limitations by stacking non-linear layers to learn hierarchical feature representations directly from raw financial data. This lesson covers MLPs, RNNs, LSTMs, CNNs, and Transformer attention mechanisms tailored for financial forecasting.

ARIMA (AutoRegressive Integrated Moving Average) – Complete Analysis:

ARIMA models assume that the time series can be represented as a linear combination of its past values and past forecast errors. The model is specified by three parameters: p (autoregressive order), d (differencing order), and q (moving average order).

The ARIMA Model:

ARIMA(p,d,q):
(1 - Σᵢ₌₁ᵖ φᵢLⁱ)(1-L)ᵈ yₜ = c + (1 + Σⱼ₌₁ᵠ θⱼLʲ)εₜ

Where:
- L is the lag operator (L yₜ = yₜ₋₁)
- φᵢ are the autoregressive parameters
- θⱼ are the moving average parameters
- εₜ is white noise
- c is a constant

Example: ARIMA(1,1,1):
(1 - φ₁L)(1-L)yₜ = c + (1 + θ₁L)εₜ

Key Limitations of ARIMA:

Stationarity Assumptions: ARIMA requires differencing until the series becomes stationary, removing long-term trends and structural information. Financial markets are inherently non-stationary, and differencing can destroy valuable information about trends and regime changes.

Linearity Constraints: The model assumes a linear relationship between past values and current values. Financial markets exhibit non-linear dynamics, including threshold effects, asymmetric responses, and interaction effects.

Fixed Parameter Structure: The model parameters are assumed constant over time. In reality, market dynamics change continuously, rendering fixed-parameter models obsolete.

Inability to Handle High Dimensionality: ARIMA is designed for univariate time series. Modern financial forecasting often requires incorporating hundreds of features.

ARIMA Limitations in Finance:

┌─────────────────────────────────────────────────────────────────────┐
│                    ARIMA Limitations in Finance                   │
│                                                                   │
│  Stationarity Assumption:                                       │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Financial returns are often non-stationary              │   │
│  │  • Differencing removes valuable trend information        │   │
│  │  • Structural breaks violate stationarity                │   │
│  │  • Example: 2008 crisis, COVID-19 pandemic              │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Linearity Constraint:                                         │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Markets exhibit non-linear dynamics                    │   │
│  │  • Asymmetric responses to positive/negative shocks       │   │
│  │  • Threshold effects and regime changes                  │   │
│  │  • Example: Leverage effect (negative shocks more volatile)│   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Fixed Parameters:                                             │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Market dynamics change over time                       │   │
│  │  • Parameters estimated on historical data become obsolete│   │
│  │  • Cannot adapt to regime shifts                         │   │
│  │  • Example: Low volatility regime (2017) vs high (2008)  │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Univariate Limitation:                                       │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Cannot incorporate multiple features                   │   │
│  │  • Ignores cross-asset correlations                      │   │
│  │  • Limited to single time series                         │   │
│  │  • Example: Cannot use macro indicators simultaneously    │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

GARCH (Generalized Autoregressive Conditional Heteroskedasticity) – Complete Analysis:

GARCH models capture volatility clustering but have significant limitations.

The GARCH Model:

Standard GARCH(1,1):
σₜ² = ω + αεₜ₋₁² + βσₜ₋₁²

Where:
- σₜ² is the conditional variance at time t
- εₜ is the residual at time t
- ω, α, β are parameters (α + β < 1 for stationarity)

EGARCH (Exponential GARCH) for Asymmetry:
log(σₜ²) = ω + α|εₜ₋₁|/σₜ₋₁ + γ(εₜ₋₁/σₜ₋₁) + βlog(σₜ₋₁²)

Where γ captures the leverage effect (asymmetric response to negative shocks).

Key Limitations of GARCH:

Linearity in Variance: GARCH models the variance as a linear function of past squared residuals and past variances. Volatility dynamics are often non-linear.

Symmetric Response: Standard GARCH assumes that positive and negative shocks have the same impact on volatility. In reality, negative shocks (market crashes) have a larger impact than positive shocks (leverage effect).

Inability to Capture Jumps: GARCH models smooth volatility and cannot capture sudden jumps caused by news or events.

Limited Forecasting Horizon: GARCH forecasts converge to the unconditional variance, making long-term forecasts unreliable.

The Deep Learning Paradigm Shift:

Deep learning architectures overcome the limitations of classical models through several key advantages:

Non-Linear Representation: Deep networks compose multiple non-linear transformations, enabling them to represent complex, non-linear relationships that linear models cannot capture.
Hierarchical Feature Learning: Each layer learns increasingly abstract features, automatically discovering relevant representations from raw data.
High-Dimensional Capability: Deep networks can process hundreds or thousands of input features, incorporating diverse data sources.
Adaptivity: Models can be retrained or fine-tuned on new data, adapting to changing market conditions.

Linear vs Non-Linear Models:

┌─────────────────────────────────────────────────────────────────────┐
│                    Linear vs Non-Linear Models                    │
│                                                                   │
│  Linear Model (ARIMA):                                          │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  y = β₀ + β₁x₁ + β₂x₂ + ... + βₚxₚ                       │   │
│  │                                                             │   │
│  │  Characteristics:                                          │   │
│  │  • Interpretable                                           │   │
│  │  • Computationally simple                                  │   │
│  │  • Cannot capture non-linear dynamics                     │   │
│  │  • Limited representational capacity                       │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Non-Linear Model (Neural Network):                            │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  y = f₃(f₂(f₁(x₁, x₂, ..., xₚ)))                         │   │
│  │                                                             │   │
│  │  Characteristics:                                          │   │
│  │  • Black box (limited interpretability)                   │   │
│  │  • Computationally intensive                              │   │
│  │  • Can capture arbitrary non-linear relationships          │   │
│  │  • Universal approximation capability                     │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

1.2: The Universal Approximation Theorem

Mathematical Foundation:

Deep learning leverages the theorem stating that a feedforward neural network with one hidden layer and non-linear activations (ReLU, Tanh) can approximate any continuous function on compact subsets of real numbers.

The Universal Approximation Theorem:

Given any continuous function f: K → ℝ on a compact set K ⊂ ℝⁿ,
there exists a neural network with one hidden layer, non-linear activation σ,
and parameters W₁, b₁, W₂, b₂ such that:

||f(x) - (W₂ · σ(W₁ · x + b₁) + b₂)|| < ε

for all x ∈ K, for any ε > 0.

This theorem establishes that feedforward neural networks are universal function approximators.

Mathematical Insight:
- A single hidden layer is sufficient for approximation
- The hidden layer size may need to be arbitrarily large
- The theorem does not guarantee efficient learning
- It applies to continuous functions only

Financial Implications:

Asset prices are believed to be driven by non-linear functions of numerous factors. The Universal Approximation Theorem suggests that neural networks can, in theory, approximate the true pricing function to arbitrary accuracy.

Theoretical vs. Practical Limitations:

The theorem does not guarantee that the network can learn the function from finite data, nor does it specify how many hidden units are required.

The theorem also assumes the function is continuous, which may not hold for financial data with jumps and discontinuities.

The number of parameters required may be impractically large for high-dimensional financial problems.

Why Deep Learning Works in Finance:

Despite these limitations, deep learning has proven successful in finance for several reasons:

Rich Data: Financial markets generate vast amounts of data, providing sufficient samples for training.
Regularization: Techniques like dropout, batch normalization, and weight decay prevent overfitting.
Architectural Priors: CNN and RNN architectures incorporate domain knowledge about the structure of financial data.
Transfer Learning: Models trained on one asset class can be fine-tuned for another.

1.3: The Deep Learning Stack for Finance

The Deep Learning Stack for Finance:

┌─────────────────────────────────────────────────────────────────────┐
│                    The Deep Learning Stack                        │
│                                                                   │
│  Output Layer:                                                  │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Prediction (returns, volatility, classification)        │   │
│  │  • Activation: Linear (regression) or Sigmoid (classification)│   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Dense Layers:                                                 │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Feature transformation and combination                  │   │
│  │  • Non-linear relationships                               │   │
│  │  • Activation: ReLU, Leaky ReLU, ELU                     │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  RNN/LSTM/Transformer:                                        │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Sequential dependencies                                │   │
│  │  • Long-term memory                                      │   │
│  │  • Global context                                        │   │
│  │  • Activation: Tanh (LSTM), ReLU (Transformer)           │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  CNN:                                                         │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Local pattern extraction                              │   │
│  │  • Spatial feature detection                            │   │
│  │  • Activation: ReLU                                     │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Input Layer:                                                  │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Price data, macro indicators, order book, alternative  │   │
│  │    data                                                   │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

Part 2: Recurrent Neural Networks (RNNs) and LSTMs

2.1: The Need for Sequential Processing

Why Sequence Matters:

Financial data is sequential: today’s price depends on yesterday’s close, last week’s news, and last month’s policy. Standard feedforward networks ignore sequence.

The Sequential Nature of Finance:

Asset prices exhibit autocorrelation, meaning today’s price is correlated with yesterday’s.
Volatility clusters, with periods of high volatility followed by high volatility.
Market reactions to news evolve over time, with initial overreactions often correcting.
Trading patterns and investor sentiment develop over time.

The RNN Architecture:

Recurrent Neural Networks (RNNs) process sequences by maintaining a hidden state that evolves over time:

RNN Forward Pass:

For t = 1 to T:
    hₜ = tanh(Wₓₕ xₜ + Wₕₕ hₜ₋₁ + bₕ)
    ŷₜ = Wₕᵧ hₜ + bᵧ

Where:
- xₜ is the input at time t (e.g., price at time t)
- hₜ is the hidden state at time t (memory)
- ŷₜ is the output at time t (prediction)
- Wₓₕ, Wₕₕ, Wₕᵧ are weight matrices
- bₕ, bᵧ are bias vectors

The hidden state hₜ summarizes all information up to time t.

RNN Architecture Diagram:

┌─────────────────────────────────────────────────────────────────────┐
│                    RNN Architecture                                │
│                                                                   │
│  Unrolled RNN (T=3):                                             │
│                                                                   │
│      ŷ₁          ŷ₂          ŷ₃                                 │
│       ▲           ▲           ▲                                  │
│       │           │           │                                  │
│       │           │           │                                  │
│   ┌───┴───┐   ┌───┴───┐   ┌───┴───┐                           │
│   │  h₁   │──►│  h₂   │──►│  h₃   │                           │
│   └───┬───┘   └───┬───┘   └───┬───┘                           │
│       ▲           ▲           ▲                                  │
│       │           │           │                                  │
│       │           │           │                                  │
│   ┌───┴───┐   ┌───┴───┐   ┌───┴───┐                           │
│   │  x₁   │   │  x₂   │   │  x₃   │                           │
│   └───────┘   └───────┘   └───────┘                           │
│                                                                   │
│  Shared weights across time steps:                              │
│  • Wₓₕ: Input to hidden (same for all t)                       │
│  • Wₕₕ: Hidden to hidden (same for all t)                     │
│  • Wₕᵧ: Hidden to output (same for all t)                     │
└─────────────────────────────────────────────────────────────────────┘

2.2: The Vanishing Gradient Problem

Backpropagation Through Time (BPTT):

Training RNNs requires computing gradients with respect to the loss over the entire sequence. This involves backpropagating gradients through time, leading to the well-known vanishing and exploding gradient problems.

Gradient in RNN:

∂L/∂Wₕₕ = Σₜ Σₖ₌₁ᵗ (∂L/∂hₜ) · (∂hₜ/∂hₖ) · (∂hₖ/∂Wₕₕ)

Where:
∂hₜ/∂hₖ = ∏ᵢ₌ₖ₊₁ᵗ (diag(tanh'(hᵢ)) · Wₕₕ)

This product leads to exponential growth or decay of gradients.

Vanishing Gradients:
- If eigenvalues of Wₕₕ < 1: gradients shrink exponentially
- Early time steps contribute almost nothing to the gradient
- Network fails to learn long-term dependencies

Exploding Gradients:
- If eigenvalues of Wₕₕ > 1: gradients grow exponentially
- Leads to numerical instability and NaN values
- Mitigated by gradient clipping

Vanishing Gradient Problem:

┌─────────────────────────────────────────────────────────────────────┐
│                    Vanishing Gradient Problem                     │
│                                                                   │
│  Gradient Magnitude Over Time:                                  │
│                                                                   │
│  Grad │  ●●●●●●●●●●●●●●●●●●●●●●                                │
│  1.0  │  ●                                                    │
│  0.8  │    ●                                                  │
│  0.6  │      ●                                                │
│  0.4  │        ●                                              │
│  0.2  │          ●                                            │
│  0.0  │            ●●●●●●●●●●●●●●●●●●●●●●                     │
│       └─────────────────────────────────────────────────────────┘   │
│       1   2   3   4   5   6   7   8   9   10  11  12  13  14  15  │
│                                                                   │
│  Time Step                                                       │
│                                                                   │
│  Interpretation:                                                │
│  • Gradients vanish exponentially with distance                │
│  • Early time steps receive almost no gradient                │
│  • RNNs have short-term memory (can't learn long-term patterns) │
│  • Example: 2008 crisis → 2023 market behavior               │
└─────────────────────────────────────────────────────────────────────┘

2.3: Long Short-Term Memory (LSTM) Architecture

The LSTM Solution:

LSTMs overcome the vanishing gradient problem by introducing a memory cell that can store information for long periods. The cell state acts as a “conveyor belt” that runs through the network, with only minor linear interactions.

Complete LSTM Mathematical Formulation:

LSTMs introduce memory cells with gates to regulate information flow:

LSTM Cell:

Forget Gate:
fₜ = σ(W_f · [hₜ₋₁, xₜ] + b_f)

The forget gate decides what information to discard from the previous cell state.
- fₜ ∈ [0, 1]
- 0 means "completely forget"
- 1 means "completely remember"

Input Gate:
iₜ = σ(W_i · [hₜ₋₁, xₜ] + b_i)
̃Cₜ = tanh(W_C · [hₜ₋₁, xₜ] + b_C)

The input gate decides what new information to store in the cell state.
- iₜ ∈ [0, 1] (how much to update)
- ̃Cₜ ∈ [-1, 1] (candidate values)

Cell State Update:
Cₜ = fₜ ⊙ Cₜ₋₁ + iₜ ⊙ ̃Cₜ

The cell state is updated by:
1. Multiplying the old state by the forget gate (discard information)
2. Adding new information (input gate × candidate)

Output Gate:
oₜ = σ(W_o · [hₜ₋₁, xₜ] + b_o)
hₜ = oₜ ⊙ tanh(Cₜ)

The output gate decides what information to output.
- oₜ ∈ [0, 1] (how much to output)
- hₜ is the hidden state (output)

Where:
- σ is the sigmoid function: σ(x) = 1/(1+e^(-x))
- tanh is the hyperbolic tangent function
- ⊙ is element-wise multiplication

LSTM Architecture Diagram:

┌─────────────────────────────────────────────────────────────────────┐
│                    LSTM Architecture                              │
│                                                                   │
│      Cₜ₋₁ ────────────────────────────────────────────────► Cₜ   │
│                │           │                        │            │
│                ▼           ▼                        ▼            │
│              Forget      Input                  Output           │
│              Gate        Gate                   Gate             │
│                │           │                        │            │
│                ▼           ▼                        ▼            │
│                fₜ          iₜ                       oₜ           │
│                │           │                        │            │
│                ▼           ▼                        ▼            │
│      ×────────►+◄─────────×                  tanh──×──► hₜ      │
│                │           │                        │            │
│                ▼           ▼                        │            │
│               Cₜ          ̃Cₜ                       │            │
│                           │                        │            │
│                           ▼                        │            │
│                           tanh                    │            │
│                                                                   │
│  Keys:                                                          │
│  1. fₜ = σ(W_f · [hₜ₋₁, xₜ] + b_f)                            │
│  2. iₜ = σ(W_i · [hₜ₋₁, xₜ] + b_i)                            │
│  3. ̃Cₜ = tanh(W_C · [hₜ₋₁, xₜ] + b_C)                         │
│  4. Cₜ = fₜ ⊙ Cₜ₋₁ + iₜ ⊙ ̃Cₜ                                │
│  5. oₜ = σ(W_o · [hₜ₋₁, xₜ] + b_o)                            │
│  6. hₜ = oₜ ⊙ tanh(Cₜ)                                         │
└─────────────────────────────────────────────────────────────────────┘

Why LSTMs Work:

The cell state allows gradients to flow through the network without vanishing. The gradient from the output passes through the cell state with only element-wise multiplications by the forget gate, which are typically near 1.

Gradient Flow in LSTM:

∂L/∂Cₜ = ∂L/∂hₜ · ∂hₜ/∂Cₜ + ∂L/∂Cₜ₊₁ · ∂Cₜ₊₁/∂Cₜ

∂Cₜ₊₁/∂Cₜ = fₜ₊₁ (element-wise multiplication)

This is approximately 1 (if f ≈ 1), so gradients flow freely.

Key insight: The gradient can flow through the cell state without vanishing.

2.4: Gated Recurrent Units (GRUs)

Simplified Architecture:

GRUs are a simplified version of LSTMs with fewer gates and parameters:

GRU Architecture:

1. Reset Gate: rₜ = σ(W_r · [hₜ₋₁, xₜ] + b_r)
   - Decides how much of the past to forget

2. Update Gate: zₜ = σ(W_z · [hₜ₋₁, xₜ] + b_z)
   - Decides how much of the past to keep

3. Candidate Hidden: ̃hₜ = tanh(W_h · [rₜ ⊙ hₜ₋₁, xₜ] + b_h)
   - Candidate new hidden state

4. Hidden Update: hₜ = (1 - zₜ) ⊙ hₜ₋₁ + zₜ ⊙ ̃hₜ
   - Combines old and candidate states

Comparison:

Feature	LSTM	GRU
Gates	3 (Forget, Input, Output)	2 (Reset, Update)
Cell State	Separate (Cₜ)	Merged with hₜ
Parameters	More (4 gate matrices)	Fewer (3 gate matrices)
Performance	Better for long sequences	Comparable for shorter sequences
Speed	Slower	Faster
Memory Usage	Higher	Lower

2.5: Bidirectional LSTMs

Forward and Backward Processing:

Bidirectional LSTMs process sequences in both directions, capturing context from both past and future time steps. This is particularly useful for financial forecasting where both historical and forward-looking information may be relevant.

Bidirectional LSTM:

Forward LSTM: h_forward = LSTM_forward(x₁, ..., x_T)
Backward LSTM: h_backward = LSTM_backward(x_T, ..., x₁)

Final Hidden: hₜ = [h_forward, h_backward]

Applications:
- Earnings call sentiment analysis
- Event-driven trading
- Fraud detection

2.6: LSTM Applications in Finance

Volatility Forecasting:

LSTMs can capture the non-linear dynamics of volatility clustering and leverage effects. Unlike GARCH models, LSTMs can incorporate multiple features and capture complex patterns.

Volatility Forecasting with LSTM:

Input Features:
- Historical returns
- Trading volume
- VIX (implied volatility)
- Macro indicators (interest rates, GDP)

Model Architecture:
- Input Layer: (sequence_length, n_features)
- LSTM(128, return_sequences=True)
- LSTM(64, return_sequences=False)
- Dense(32, activation='relu')
- Dense(1, activation='linear')  # Predicts volatility

Output: Forecasted volatility for next period

Multi-Horizon Return Prediction:

LSTMs can predict returns at multiple horizons by outputting a sequence of predictions. This is useful for trading strategies that need forecasts at different time scales.

Multi-Horizon Forecasting:

Input: Historical data (T steps)
Output: Predictions for T+1, T+2, ..., T+K

Architecture:
- Sequence-to-sequence LSTM
- Encoder: LSTM processes input sequence
- Decoder: LSTM generates output sequence

Applications:
- Trading strategies with different holding periods
- Risk management at multiple horizons
- Portfolio optimization

Event-Driven Forecasting:

LSTMs can process sequences of events (e.g., news, earnings announcements) and their impact on asset prices. The temporal structure of events is crucial for understanding market reactions.

Portfolio Optimization:

LSTM predictions can be used as inputs for portfolio optimization, improving the estimation of expected returns and covariances.

Part 3: Convolutional Neural Networks (CNNs) in Finance

3.1: Why CNNs for Finance?

Local Pattern Extraction:

CNNs, originally for vision, excel at extracting local structural patterns from financial matrices.

Shift Invariance:

CNNs are shift-invariant, meaning they can detect the same pattern regardless of its position in the input. This is useful for detecting patterns that may occur at different times.

Hierarchical Feature Learning:

CNNs learn hierarchical features: low-level patterns (edges, local trends) are combined into higher-level patterns (price patterns, order flow patterns).

3.2: CNN Architecture for Finance

Convolutional Layer:

Convolution Operation:

1D Convolution (Time Series):
Y[i] = Σₖ X[i+k] · W[k]

2D Convolution (Order Book):
Y[i,j] = Σₘ Σₙ X[i+m, j+n] · W[m,n]

Where:
- X is the input tensor
- W is the filter (kernel)
- Y is the output

Pooling Layer:

Pooling reduces dimensionality while preserving important features:

Max Pooling: Takes the maximum value in each window
Average Pooling: Takes the average value in each window

Activation Functions:

ReLU: f(x) = max(0, x) (most common)
Leaky ReLU: f(x) = max(αx, x) (allows small negative values)
ELU: f(x) = x if x > 0, α(e^x – 1) if x ≤ 0

CNN Architecture for Finance:

┌─────────────────────────────────────────────────────────────────────┐
│                    CNN Architecture for Finance                   │
│                                                                   │
│  Input Layer:                                                   │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Order book: (time_steps × price_levels × channels)     │   │
│  │  • Vol surface: (strikes × expirations × 1)             │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Convolutional Layer 1:                                       │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • 32 filters, 3×3 kernel                                │   │
│  │  • Detects local patterns (price movements, order flow)   │   │
│  │  • Activation: ReLU                                       │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Max Pooling:                                                  │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • 2×2 pooling                                           │   │
│  │  • Reduces dimensionality                                │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Convolutional Layer 2:                                       │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • 64 filters, 3×3 kernel                                │   │
│  │  • Detects higher-level patterns                         │   │
│  │  • Activation: ReLU                                       │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Max Pooling:                                                  │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • 2×2 pooling                                           │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Flatten and Dense Layers:                                   │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Flatten: 2D to 1D                                     │   │
│  │  • Dense: 128 neurons (ReLU)                            │   │
│  │  • Dropout: 0.5                                         │   │
│  │  • Dense: 64 neurons (ReLU)                             │   │
│  │  • Output: Prediction                                   │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

3.3: 1D CNNs for Time Series

One-Dimensional Convolutions:

For time series data, 1D convolutions slide a filter along the time dimension.

Financial Applications:

Price trend detection
Anomaly detection in transaction streams
Event detection in high-frequency data
Pattern recognition in technical indicators

3.4: 2D CNNs for Order Books

Order Book Representation:

CNNs excel at extracting local structural patterns from financial matrices. CNNs scan order book matrices and volatility surfaces.

Limit Order Book (LOB) Telemetry:

LOB Tensor Representation: Rows = time ticks, columns = price levels, bid/ask volumes.

Order Book Tensor:

Price Levels → 0.01, 0.02, 0.03, ..., 0.10
             ┌─────────────────────────────────────────┐
Time Step 1  │ 100, 150, 200, 50, 0, 0, 0, 0, 0, 0   │
Time Step 2  │ 120, 180, 220, 60, 0, 0, 0, 0, 0, 0   │
Time Step 3  │ 110, 170, 210, 55, 0, 0, 0, 0, 0, 0   │
...          │ ...                                    │
Time Step T  │ 90, 160, 190, 45, 0, 0, 0, 0, 0, 0    │
             └─────────────────────────────────────────┘

Kernel Filtering:

Convolutional filters detect micro-structural patterns like imbalances, spoofing, and liquidity voids.

Spoofing: A large order placed and then quickly canceled creates a pattern that can be detected by CNNs.
Liquidity Voids: Gaps in order book depth indicate low liquidity and potential price jumps.
Imbalance Patterns: Persistent buying or selling pressure can be detected from the order flow.

3.5: 2D CNNs for Volatility Surfaces

Volatility Surface Representation:

2D CNNs process implied volatility surfaces (strike vs. expiration) to identify anomalies and arbitrage opportunities.

Volatility Surface Tensor:

Strikes → 80, 90, 100, 110, 120
         ┌─────────────────────────────────────────┐
Exp. 1M  │ 25%, 22%, 20%, 23%, 28%               │
Exp. 3M  │ 24%, 21%, 19%, 22%, 27%               │
Exp. 6M  │ 23%, 20%, 18%, 21%, 26%               │
Exp. 1Y  │ 22%, 19%, 17%, 20%, 25%               │
         └─────────────────────────────────────────┘

Pattern Detection:

Smile/Smirk: Detecting the characteristic curve of implied volatility.
Term Structure: Changes in the shape of the surface over time.
Anomalies: Arbitrage opportunities where the surface violates no-arbitrage conditions.

3.6: Temporal Convolutional Networks (TCNs)

Dilated Convolutions:

TCNs use dilated convolutions to increase the receptive field without increasing the number of parameters:

Dilated Convolution:
Y[i] = Σₖ X[i + k · d] · W[k]

Where d is the dilation rate.
- d=1: Standard convolution
- d=2: Skips every other element
- d=4: Skips three elements

Advantages:

Parallel processing (unlike RNNs)
Large receptive field with few parameters
Can handle sequences of any length
Stable gradients (no vanishing/exploding)

Part 4: Transformer Architectures and Self-Attention

4.1: The Limitations of RNNs

Sequential Processing Bottleneck:

Sequential RNN/LSTM processing creates bottlenecks. RNNs process sequences step-by-step, making them slow to train and limited in their ability to capture long-range dependencies.

Information Bottleneck:

The hidden state must compress all information from the entire sequence, leading to information loss.

Inability to Parallelize:

The sequential nature of RNNs prevents parallelization across time steps, limiting training speed.

4.2: The Attention Mechanism

Core Concept:

Transformers process entire histories simultaneously using Self-Attention. Instead of compressing the entire sequence into a fixed-size hidden state, attention allows the model to dynamically look back at all previous time steps and assign weighted importance scores to the most relevant data points.

Queries, Keys, and Values:

Attention Mechanism:

Q: Query (what the model is looking for)
K: Key (what each element in the sequence is)
V: Value (the actual content of each element)

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V

Where:
- Q ∈ ℝ^(n × dₖ)
- K ∈ ℝ^(m × dₖ)
- V ∈ ℝ^(m × dᵥ)
- dₖ is the dimension of keys

4.3: Scaled Dot-Product Attention

The Attention Score:

The attention score measures the relevance of each key to the query:

Scaled Dot-Product Attention:

1. Compute scores: S = Q · Kᵀ
2. Scale: S_scaled = S / √dₖ
3. Softmax: A = softmax(S_scaled)
4. Output: O = A · V

Step 1: Q · Kᵀ = [[q₁·k₁, q₁·k₂, ..., q₁·kₘ],
                   [q₂·k₁, q₂·k₂, ..., q₂·kₘ],
                   ...
                   [qₙ·k₁, qₙ·k₂, ..., qₙ·kₘ]]

Step 2: Scale to prevent large dot products (push softmax into small gradients)

Step 3: Softmax to get attention weights (sum to 1)

Step 4: Weighted sum of values (output)

Why Scaling Matters:

Without scaling, the dot product grows large for high-dimensional vectors, pushing the softmax into regions with very small gradients. Scaling by 1/√dₖ prevents this.

Financial Significance:

Global Context: Links distant events (e.g., January macro news → December price movement). When predicting tomorrow’s asset volatility, the attention mechanism can instantly link today’s market conditions directly to a similar macroeconomic shock that occurred five years ago, bypassing intermediate time steps entirely.

4.4: Multi-Head Attention

Parallel Attention Heads:

Parallel attention heads capture diverse relationships:

Instead of performing a single attention calculation, Multi-Head Attention projects queries, keys, and values into multiple lower-dimensional subspaces and performs attention in parallel.

Multi-Head Attention:

MultiHead(Q, K, V) = Concat(head₁, ..., headₕ) W_O

Where:
headᵢ = Attention(QWᵢ^Q, KWᵢ^K, VWᵢ^V)

Each head focuses on different relationships:
- Head 1: High-frequency order flow imbalances
- Head 2: Macro yield curve movements
- Head 3: Cross-asset correlations
- Head 4: Sentiment indicators
- Head 5: Volatility dynamics
- Head 6: Macroeconomic regime shifts
- Head 7: Sector rotation patterns
- Head 8: Geopolitical risk factors

4.5: Positional Encoding

The Problem:

Transformers process entire sequences simultaneously, lacking inherent awareness of time order.

The Solution:

Positional encodings are added to input embeddings, providing temporal context:

Positional Encoding:

PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Where:
- pos is the position in the sequence
- i is the dimension index
- d_model is the model dimension

Example for d_model=4, pos=0:
PE(0,0) = sin(0) = 0
PE(0,1) = cos(0) = 1
PE(0,2) = sin(0) = 0
PE(0,3) = cos(0) = 1

This creates a unique encoding for each position.

4.6: Transformer Architecture

Encoder-Decoder Structure:

The transformer has an encoder (processes input) and a decoder (generates output). For financial forecasting, the encoder processes historical data and the decoder generates predictions.

Transformer Architecture:

┌─────────────────────────────────────────────────────────────────────┐
│                    Transformer Architecture                       │
│                                                                   │
│  Input Embedding:                                               │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Price data, macro indicators, alternative data         │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Positional Encoding:                                          │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Adds temporal information to embeddings                │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Encoder Layers (N×):                                       │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Multi-Head Self-Attention                              │   │
│  │  • Add & Normalize                                      │   │
│  │  • Feed-Forward Network                                 │   │
│  │  • Add & Normalize                                      │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Decoder Layers (N×):                                       │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Masked Multi-Head Self-Attention                       │   │
│  │  • Cross-Attention                                      │   │
│  │  • Feed-Forward Network                                 │   │
│  │  • Add & Normalize                                      │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Output Layer:                                                 │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Dense layer for prediction                            │   │
│  │  • Returns, volatility, or classification               │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

4.7: Financial Applications of Transformers

Generative Forecasting:

Transformers can generate forecasts for multiple assets simultaneously. This enables generative forecasting and systemic risk surveillance.

Portfolio Risk Surveillance:

Transformers can monitor systemic risk by analyzing cross-asset correlations and contagion patterns.

Multi-Asset Cross-Attention:

Transformer attention mechanisms can ingest thousands of parallel asset price series simultaneously, calculating cross-asset correlations and contagion risks in real-time.

Macro-Financial Modeling:

Transformers can process diverse data types simultaneously:

Macroeconomic indicators (GDP, inflation, unemployment)
Asset prices (equities, bonds, currencies, commodities)
Alternative data (sentiment, satellite, credit card transactions)

Risk Factor Attribution:

Attention weights can be used to identify which factors drive predictions, providing interpretability for risk management and regulatory compliance.

4.8: Transformer Variations for Finance

Temporal Fusion Transformer (TFT):

TFT is specifically designed for time-series forecasting with multiple horizons and interpretability features. It combines LSTM and attention mechanisms.

Informer:

Informer is designed for long-sequence time-series forecasting with reduced complexity.

PatchTST:

PatchTST processes time series by patching sequences, reducing computational complexity while maintaining performance.

Part 5: Comparative Analysis and Practical Considerations

5.1: Architecture Comparison

Feature	RNN/LSTM	CNN	Transformer
Sequential Processing	Yes	No	No
Parallelization	Limited	Yes	Yes
Long-Range Dependencies	Limited	Limited	Excellent
Interpretability	Low	Medium	High
Data Requirements	Moderate	Moderate	Large
Training Speed	Slow	Fast	Fast
Memory Complexity	O(T)	O(T)	O(T²)
Best Use Case	Time series	Local patterns	Global context

5.2: Practical Implementation

Data Preparation:

Normalization: Scale data to [0,1] or standard normal
Sequence Length: Balance between memory and performance
Train/Test Split: Time-based, not random
Validation: Walk-forward validation

Training:

Optimizer: Adam (adaptive learning rate)
Learning Rate: 0.001 (typical starting point)
Batch Size: 32-256 (depending on memory)
Dropout: 0.1-0.5 (to prevent overfitting)
Early Stopping: Monitor validation loss

Evaluation:

Walk-Forward Validation: Simulate real-world deployment
Backtesting: Test on out-of-sample data
Robustness Testing: Stress test on crisis periods

5.3: Common Pitfalls and Solutions

Overfitting:

Use regularization (dropout, weight decay)
Use smaller models
Use early stopping
Use more data (if available)

Data Leakage:

Ensure temporal order in train/test split
Avoid using future data for feature engineering
Use walk-forward validation

Instability:

Use gradient clipping (for RNNs)
Use smaller learning rates
Use batch normalization

Summary

Deep learning architectures revolutionize financial forecasting by moving beyond ARIMA/GARCH limitations:

LSTMs capture sequential dependencies and long-term memory, enabling accurate prediction of returns, volatility, and other financial time series.

CNNs scan order book matrices and volatility surfaces, detecting anomalies, imbalances, and arbitrage opportunities.

Transformers achieve global contextual awareness across portfolios, processing entire histories simultaneously and capturing complex cross-asset correlations.

Together, these architectures empower quantitative researchers to build robust predictive models for complex, non-stationary markets, providing powerful tools for forecasting, risk management, and algorithmic trading.

Key Terminology Glossary

Term	Definition
RNN	Recurrent Neural Network – processes sequences with hidden states
LSTM	Long Short-Term Memory – RNN with memory cells and gates
GRU	Gated Recurrent Unit – simplified LSTM
CNN	Convolutional Neural Network – extracts local patterns
Transformer	Architecture based on self-attention
Attention	Mechanism for weighting importance of elements
Self-Attention	Attention within the same sequence
Multi-Head Attention	Multiple parallel attention heads
Positional Encoding	Adds temporal information to embeddings
Vanishing Gradient	Gradients shrink exponentially in deep networks
Bidirectional RNN	Processes sequences in both directions
Dilated Convolution	Convolution with gaps for larger receptive field
Temporal Fusion Transformer	TFT – specialized transformer for time series
Order Book Tensor	2D representation of limit order book

Certificate in Artificial Intelligence for Finance lesson4.1:Deep Learning Architectures for Financial Time-Series Forecasting

Certificate in Artificial Intelligence for Finance