lesson1.2:Advanced Deep Learning Architectures for Financial Time-Series and Sequential Forecasting

Learning Objectives:

Master Encoder-Decoder (Seq2Seq) LSTM architectures for multi-horizon forecasting
Understand Temporal Convolutional Networks (TCNs) and their advantages
Learn Temporal Fusion Transformers (TFTs) with attention mechanisms
Understand Patch Time-Series Transformers (PatchTST) for efficient long-sequence processing
Apply these architectures to financial time-series prediction problems

Part 1: The Temporal Complexity of Financial Data

1.1: Understanding Financial Time-Series Data

The Nature of Financial Data:

Financial markets generate enormous volumes of sequential data every second. Examples include tick-by-tick asset prices, Limit Order Book (LOB) states, trading volumes, macroeconomic indicators, volatility measures, and interest rate curves.

Unlike ordinary datasets, financial time series possess several challenging characteristics:

Characteristics of Financial Time Series:

┌─────────────────────────────────────────────────────────────────────┐
│                    Financial Time Series Characteristics           │
│                                                                   │
│  Non-Stationarity:                                              │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Statistical properties (mean, variance) change over time│   │
│  │  • Market regimes shift (bull, bear, volatile, calm)      │   │
│  │  • Structural breaks (policy changes, crises)            │   │
│  │  • Challenge: Models must adapt continuously              │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Noise:                                                        │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Random market fluctuations obscure underlying patterns │   │
│  │  • Signal-to-noise ratio is extremely low                 │   │
│  │  • Overfitting risk is high                              │   │
│  │  • Challenge: Distinguish signal from noise              │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Missing Data:                                                 │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Data gaps due to holidays, outages, illiquid markets   │   │
│  │  • Irregular sampling intervals                          │   │
│  │  • Asynchronous data streams                             │   │
│  │  • Challenge: Handle incomplete observations             │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Multi-Horizon Dependencies:                                   │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Current behavior depends on events minutes, days, or    │   │
│  │    months earlier                                         │   │
│  │  • Short-term (momentum) and long-term (trend) effects   │   │
│  │  • Multiple time scales interact                        │   │
│  │  • Challenge: Capture dependencies across scales        │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

Time Series Notation:

A financial time series is represented as:

X = [x₁, x₂, x₃, ..., x_T]

Where:
- xₜ ∈ ℝᵈ is the feature vector at time t
- d is the number of features (price, volume, volatility, etc.)
- T is the sequence length

Target: yₜ ∈ ℝ (or ℝᵐ for multi-output)
- yₜ = f(x₁, x₂, ..., xₜ) + εₜ
- εₜ is random noise

Forecasting:
Given historical observations up to time T,
predict y_{T+1}, y_{T+2}, ..., y_{T+H}
where H is the forecast horizon.

1.2: The Need for Advanced Architectures

Why LSTMs Have Limitations:

Although Long Short-Term Memory (LSTM) networks significantly improved sequential learning by mitigating the vanishing gradient problem, institutional quantitative research has increasingly adopted more advanced architectures capable of modeling long-range dependencies more efficiently.

LSTM Limitations:

┌─────────────────────────────────────────────────────────────────────┐
│                    LSTM Limitations                               │
│                                                                   │
│  Sequential Processing:                                         │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Cannot parallelize across time steps                   │   │
│  │  • Training is slow for long sequences                    │   │
│  │  • GPU utilization is inefficient                         │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Long-Range Dependencies:                                      │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Still struggles with very long sequences (>100 steps)  │   │
│  │  • Information bottleneck in hidden state               │   │
│  │  • Limited capacity to retain very old information       │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Multi-Horizon Forecasting:                                    │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Standard LSTMs predict one step at a time            │   │
│  │  • Error accumulation in autoregressive mode            │   │
│  │  • Training and inference mismatch (teacher forcing)   │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Interpretability:                                             │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Black-box model                                        │   │
│  │  • Difficult to understand why predictions were made     │   │
│  │  • Limited regulatory compliance                         │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

The Advanced Architecture Landscape:

Institutional quantitative research has increasingly adopted:

Encoder-Decoder (Seq2Seq) LSTM architectures for multi-horizon forecasting
Temporal Convolutional Networks (TCNs) for parallel processing
Temporal Fusion Transformers (TFTs) for interpretability and attention
Patch Time-Series Transformers (PatchTST) for efficient long-sequence processing

Part 2: Encoder-Decoder Architectures for Multi-Horizon Forecasting

2.1: Sequence-to-Sequence (Seq2Seq) Architecture

Architecture Overview:

A Sequence-to-Sequence (Seq2Seq) model consists of two neural networks:

Encoder: Processes the historical sequence and compresses it into a latent representation
Decoder: Uses the latent representation to generate forecasts for future time periods

Encoder-Decoder (Seq2Seq) Architecture:

┌─────────────────────────────────────────────────────────────────────┐
│                    Encoder-Decoder Architecture                    │
│                                                                   │
│  Encoder:                                                       │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                                                             │   │
│  │  x₁ → [LSTM] → h₁ ─┐                                       │   │
│  │  x₂ → [LSTM] → h₂ ─┤                                       │   │
│  │  x₃ → [LSTM] → h₃ ─┤ → h_T (Context Vector)              │   │
│  │  ...               ─┤                                       │   │
│  │  x_T → [LSTM] → h_T ─┘                                       │   │
│  │                                                             │   │
│  │  h_T = Encoder(x₁, x₂, ..., x_T)                           │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│                              ▼                                    │
│  Decoder:                                                       │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                                                             │   │
│  │  h_T → [LSTM] → ŷ_{T+1}                                    │   │
│  │         │                                                   │   │
│  │         └──────────────────────────────────────────────────┐│   │
│  │  h_T, ŷ_{T+1} → [LSTM] → ŷ_{T+2}                         ││   │
│  │         │                                                  ││   │
│  │         └──────────────────────────────────────────────────┐│   │
│  │  h_T, ŷ_{T+2} → [LSTM] → ŷ_{T+3}                         ││   │
│  │         │                                                  ││   │
│  │         └──────────────────────────────────────────────────┘│   │
│  │  ...                                                       │   │
│  │  h_T, ŷ_{T+H-1} → [LSTM] → ŷ_{T+H}                       │   │
│  │                                                             │   │
│  │  ŷ_{T+i} = Decoder(h_T, ŷ_{T+i-1})                       │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

The Encoder:

The encoder receives a historical sequence of observations:

x₁, x₂, x₃, ..., x_T

Where:
- xₜ = input feature vector at time t
- T = length of the historical sequence

The encoder compresses the entire sequence into a latent representation called the context vector.

Mathematically:
h_T = Encoder(x₁, x₂, ..., x_T)

Where:
- h_T = latent context vector summarizing all historical information

The encoder may consist of stacked LSTM or GRU layers.

The Decoder:

The decoder uses the context vector to generate forecasts for future time periods.

Forecast Horizon: H
Output sequence: ŷ_{T+1}, ŷ_{T+2}, ..., ŷ_{T+H}

Where:
- ŷ = predicted value
- H = forecast horizon

Mathematically:
ŷ_{T+i} = Decoder(h_T, ŷ_{T+i-1})

For i = 1, 2, ..., H

Financial Applications:

Seq2Seq architectures are particularly useful for:

Stock price forecasting
Volatility prediction
Energy demand forecasting
Exchange-rate prediction
Multi-asset portfolio forecasting

2.2: Teacher Forcing

The Training Challenge:

During training, the decoder needs to learn to generate accurate predictions. However, if the decoder uses its own previous predictions, errors can accumulate and training becomes unstable.

Teacher Forcing Solution:

During training, the decoder receives the true historical value instead of its own prediction.

Teacher Forcing:

Input_{t+1} = y_t

Where:
- y_t = actual observed value

Advantages:
- Faster convergence
- Stable gradient propagation
- Reduced training error

Mathematical Formulation:

Without Teacher Forcing (Autoregressive):
ŷ_{t+1} = Decoder(h_T, ŷ_t)

With Teacher Forcing:
ŷ_{t+1} = Decoder(h_T, y_t)

Where y_t is the actual value, not the predicted value.

2.3: Free Running (Autoregressive Inference)

The Deployment Challenge:

During deployment, future observations are unavailable. The decoder must therefore use its own previous prediction.

Free Running (Autoregressive Inference):

Input_{t+1} = ŷ_t

Where ŷ_t is the model's own prediction.

Issue:
- Small forecasting errors propagate through subsequent predictions
- Error accumulation over long horizons

Error Propagation:

Error Accumulation:

Step 1: ŷ_{T+1} = Decoder(h_T, y_T) + ε₁
Step 2: ŷ_{T+2} = Decoder(h_T, ŷ_{T+1}) + ε₂
Step 3: ŷ_{T+3} = Decoder(h_T, ŷ_{T+2}) + ε₃
...

Each step incorporates the error from the previous step:
ŷ_{T+H} = f(y_T, ε₁, ε₂, ..., ε_H)

Errors accumulate over the forecast horizon.

2.4: Scheduled Sampling

The Solution:

Scheduled Sampling combines Teacher Forcing and Free Running.

Scheduled Sampling:

During training, the next decoder input is selected probabilistically.

Input_{t+1} = {
    y_t    with probability p  (Teacher Forcing)
    ŷ_t    with probability (1-p)  (Free Running)
}

Where:
- p = probability of using Teacher Forcing
- (1-p) = probability of using Free Running

Scheduling:
- Start with p = 1.0 (full Teacher Forcing)
- Gradually decrease p as training progresses
- End with p = 0.0 (full Free Running)

Mathematically:
p(epoch) = max(0, p_start - (p_start / total_epochs) × epoch)

Benefits:

Smooth transition from Teacher Forcing to Free Running
Reduces mismatch between training and inference
Better generalization on unseen data
More stable training

2.5: Attention in Encoder-Decoder

Attention Mechanism:

Instead of compressing the entire sequence into a single context vector, attention allows the decoder to focus on different parts of the encoder output at each step.

Attention in Encoder-Decoder:

For each decoding step i:
1. Compute attention scores: e_{ij} = score(s_i, h_j)
2. Softmax: α_{ij} = exp(e_{ij}) / Σ exp(e_{ik})
3. Context vector: c_i = Σ α_{ij} h_j
4. Update state: s_i = Decoder(s_{i-1}, c_i, y_{i-1})

Where:
- s_i = decoder state at step i
- h_j = encoder state at step j
- score = attention scoring function
- α = attention weights
- c_i = attention context vector

Attention Visualization:

┌─────────────────────────────────────────────────────────────────────┐
│                    Attention Visualization                        │
│                                                                   │
│  Decoder Steps →                                                │
│                                                                   │
│  Step 1: ●●●●●●●●●●●●●○○○○○○○○○○○                               │
│          (Focuses on recent history)                             │
│                                                                   │
│  Step 2: ●●●●●○○○○○○○○○○○○○○○○○○○                               │
│          (Focuses on mid-range history)                          │
│                                                                   │
│  Step 3: ●●○○○○○○○○○○○○○○○○○○●●●●●                               │
│          (Focuses on distant past and recent)                   │
│                                                                   │
│  Step 4: ○○○○○○○○○○○●●●●●●●●●●●●●                                 │
│          (Focuses on future-relevant patterns)                  │
│                                                                   │
│  ● = High attention weight                                       │
│  ○ = Low attention weight                                        │
└─────────────────────────────────────────────────────────────────────┘

Part 3: Temporal Convolutional Networks (TCNs)

3.1: Overview and Advantages

What are TCNs?

Unlike recurrent networks, Temporal Convolutional Networks process the entire sequence simultaneously, making them highly parallelizable.

TCNs combine:

Causal convolutions
Dilated convolutions
Residual connections

TCN Advantages:

┌─────────────────────────────────────────────────────────────────────┐
│                    TCN Advantages                                 │
│                                                                   │
│  Parallel Processing:                                           │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Process all time steps simultaneously                  │   │
│  │  • Efficient GPU utilization                              │   │
│  │  • Faster training and inference                          │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Long-Range Dependencies:                                      │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Dilated convolutions provide exponential receptive field│   │
│  │  • Captures very long-term dependencies efficiently       │   │
│  │  • No vanishing gradient problem                         │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Stable Gradients:                                             │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • No recurrent connections                                │   │
│  │  • Gradients flow directly through the network            │   │
│  │  • No exploding/vanishing gradients                       │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Interpretability:                                             │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Residual connections show skip patterns                │   │
│  │  • Easier to understand important time steps              │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

3.2: Causal Convolutions

The Principle:

Financial forecasting must never use future information. For time t, the prediction depends only on xₜ, xₜ₋₁, xₜ₋₂, …, never on xₜ₊₁.

Causal Convolution:

┌─────────────────────────────────────────────────────────────────────┐
│                    Causal Convolution                             │
│                                                                   │
│  t=1  t=2  t=3  t=4  t=5  t=6  t=7                              │
│  │    │    │    │    │    │    │                                  │
│  ▼    ▼    ▼    ▼    ▼    ▼    ▼                                  │
│  ●    ●    ●    ●    ●    ●    ●                                  │
│  │    │    │    │    │    │    │                                  │
│  └────┼────┼────┼────┼────┼────┘                                │
│       │    │    │    │    │                                       │
│       └────┼────┼────┼────┘                                     │
│            │    │    │                                            │
│            └────┼────┘                                          │
│                 │                                                 │
│                 ▼                                                 │
│              Output                                              │
│                                                                   │
│  Each output depends only on current and past inputs             │
│  No future information used                                      │
└─────────────────────────────────────────────────────────────────────┘

Mathematical Formulation:

Causal Convolution:
y(t) = Σ_{i=0}^{k-1} f(i) · x(t-i)

Where:
- y(t) = convolution output
- f(i) = filter coefficients
- x = input sequence
- k = filter size

Properties:
- y(t) depends only on x(t), x(t-1), ..., x(t-k+1)
- Preserves temporal causality
- Prevents look-ahead bias

3.3: Dilated Convolutions

The Concept:

Dilated convolutions expand the receptive field exponentially without increasing the number of parameters.

Dilated Convolution Visualization:

┌─────────────────────────────────────────────────────────────────────┐
│                    Dilated Convolution Visualization              │
│                                                                   │
│  d=1 (Standard):                                              │
│  x₁  x₂  x₃  x₄  x₅  x₆  x₇                                     │
│   │   │   │                                                    │
│   └───┼───┘                                                    │
│       │                                                         │
│      y(t)                                                       │
│                                                                   │
│  d=2 (Dilated):                                              │
│  x₁  x₂  x₃  x₄  x₅  x₆  x₇                                     │
│   │       │                                                    │
│   └───────┘                                                    │
│       │                                                         │
│      y(t)                                                       │
│                                                                   │
│  d=3 (Dilated):                                              │
│  x₁  x₂  x₃  x₄  x₅  x₆  x₇                                     │
│   │           │                                                 │
│   └───────────┘                                                 │
│       │                                                         │
│      y(t)                                                       │
└─────────────────────────────────────────────────────────────────────┘

Mathematical Formulation:

Dilated Convolution:
y(t) = Σ_{i=0}^{k-1} f(i) · x(t - d·i)

Where:
- y(t) = convolution output
- f(i) = filter coefficient
- x = input sequence
- k = filter size
- d = dilation factor

Special Cases:
- d = 1: Standard convolution
- d > 1: Dilated convolution (skips elements)

Receptive Field:
R = 1 + Σ_{l=1}^{L} (k_l - 1) · d_l

With exponential dilation (d = 2^l):
R = 1 + Σ_{l=1}^{L} (k_l - 1) · 2^l

Receptive Field Expansion:

Receptive Field Growth:

┌─────────────────────────────────────────────────────────────────────┐
│                    Receptive Field Growth                         │
│                                                                   │
│  Layer 1 (d=1, k=3):                                          │
│  Receptive Field = 3                                            │
│  x₁  x₂  x₃  x₄  x₅  x₆  x₇                                     │
│   │   │   │                                                    │
│   └───┼───┘                                                    │
│       │                                                         │
│      y₁                                                         │
│                                                                   │
│  Layer 2 (d=2, k=3):                                          │
│  Receptive Field = 7                                            │
│  x₁  x₂  x₃  x₄  x₅  x₆  x₇                                     │
│   │   │   │   │   │   │   │                                     │
│   └───┼───┘   └───┼───┘   └───┼───┘                           │
│       │           │           │                                 │
│       └───────────┼───────────┘                                 │
│                   │                                             │
│                  y₂                                             │
│                                                                   │
│  Layer 3 (d=4, k=3):                                          │
│  Receptive Field = 15                                           │
│  (Exponential growth)                                           │
└─────────────────────────────────────────────────────────────────────┘

3.4: Residual Connections

The Principle:

Residual learning allows information to bypass intermediate layers, improving gradient flow and enabling very deep networks.

Residual Connection:

┌─────────────────────────────────────────────────────────────────────┐
│                    Residual Connection                           │
│                                                                   │
│  Input ──────────────────────────────────────────────────────────┐ │
│    │                                                             │ │
│    ▼                                                             │ │
│  ┌─────────────────────────────────────────────────────────────┐ │ │
│  │  Convolutional Layer (F)                                  │ │ │
│  └─────────────────────────────────────────────────────────────┘ │ │
│    │                                                             │ │
│    │  +                                                         │ │
│    └───────────────────────────┬─────────────────────────────────┘ │
│                                │                                   │
│                                ▼                                   │
│                          Output                                    │
│                                                                   │
│  y = F(x) + x                                                   │
│                                                                   │
│  Where:                                                         │
│  - F(x) = learned transformation                                │
│  - x = original input                                           │
│                                                                   │
│  Benefits:                                                      │
│  • Reduces vanishing gradients                                  │
│  • Stabilizes optimization                                      │
│  • Enables very deep TCNs                                       │
│  • Allows gradients to bypass layers                            │
└─────────────────────────────────────────────────────────────────────┘

Residual TCN Block:

Residual TCN Block:

┌─────────────────────────────────────────────────────────────────────┐
│                    Residual TCN Block                             │
│                                                                   │
│  Input x                                                         │
│    │                                                              │
│    ▼                                                              │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  Dilated Causal Convolution (d=1)                         │   │
│  │  Weight Normalization                                     │   │
│  │  Activation (ReLU)                                        │   │
│  └─────────────────────────────────────────────────────────────┘   │
│    │                                                              │
│    ▼                                                              │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  Dilated Causal Convolution (d=2)                         │   │
│  │  Weight Normalization                                     │   │
│  └─────────────────────────────────────────────────────────────┘   │
│    │                                                              │
│    │                                                              │
│    │  +                                                          │
│    └─────────────────────────────────────────────────────┐        │
│                                                         │        │
│                                                         ▼        │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  Activation (ReLU)                                        │   │
│  └─────────────────────────────────────────────────────────────┘   │
│    │                                                              │
│    ▼                                                              │
│  Output y = F(x) + x                                            │
└─────────────────────────────────────────────────────────────────────┘

3.5: TCN for Financial Forecasting

TCN Architecture for Finance:

TCN Architecture:

┌─────────────────────────────────────────────────────────────────────┐
│                    TCN Architecture for Finance                   │
│                                                                   │
│  Input: Historical sequence (T steps, D features)                │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  x₁, x₂, ..., x_T (prices, volume, volatility, macros)   │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  ┌───────────────────────────▼─────────────────────────────────┐   │
│  │  Residual TCN Block 1 (d=1)                               │   │
│  │  Receptive Field: 3                                       │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  ┌───────────────────────────▼─────────────────────────────────┐   │
│  │  Residual TCN Block 2 (d=2)                               │   │
│  │  Receptive Field: 7                                       │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  ┌───────────────────────────▼─────────────────────────────────┐   │
│  │  Residual TCN Block 3 (d=4)                               │   │
│  │  Receptive Field: 15                                      │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  ┌───────────────────────────▼─────────────────────────────────┐   │
│  │  Residual TCN Block 4 (d=8)                               │   │
│  │  Receptive Field: 31                                      │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  ┌───────────────────────────▼─────────────────────────────────┐   │
│  │  Global Average Pooling                                  │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  ┌───────────────────────────▼─────────────────────────────────┐   │
│  │  Dense Layers for Prediction                              │   │
│  │  • Hidden layer: 128 neurons (ReLU)                      │   │
│  │  • Output layer: H neurons (forecast horizon)            │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Output: ŷ_{T+1}, ŷ_{T+2}, ..., ŷ_{T+H}                       │
└─────────────────────────────────────────────────────────────────────┘

Part 4: Attention Mechanisms and Temporal Fusion Transformers (TFT)

4.1: Self-Attention Fundamentals

The Core Concept:

Attention mechanisms allow a model to learn which observations are most important rather than treating every historical observation equally.

Scaled Dot-Product Attention:

Scaled Dot-Product Attention:

Attention(Q, K, V) = softmax(QKᵀ / √d_k) · V

Where:
- Q = Query matrix (what we're looking for)
- K = Key matrix (what each element contains)
- V = Value matrix (actual content)
- d_k = dimension of key vectors

Interpretation:
- Each observation assigns importance weights to every other observation
- Large attention scores indicate stronger influence on the prediction
- Softmax ensures weights sum to 1
- Scaling by 1/√d_k prevents vanishing gradients

Attention Weights Visualization:

Attention Weights for Financial Time Series:

┌─────────────────────────────────────────────────────────────────────┐
│                    Attention Weights                              │
│                                                                   │
│  Target: Volatility Forecast                                   │
│                                                                   │
│  Time Step    |  Attention Weight                                │
│  ───────────────────────────────────────────────────────────────  │
│  t-1 (1 day ago)   |  ●●●●●●●●●●●●●●● (0.15)                   │
│  t-2               |  ●●●●●●●●●● (0.10)                        │
│  t-5 (1 week ago)  |  ●●●●●●●●●●●●●●●● (0.16)                  │
│  t-10              |  ●●●●●● (0.06)                            │
│  t-20 (1 month)    |  ●●●●●●●●●●● (0.11)                      │
│  t-30              |  ●●●● (0.04)                              │
│  t-60 (2 months)   |  ●●●●●●●●●●●●●●●●●● (0.18)                │
│  t-90 (3 months)   |  ●●●●● (0.05)                            │
│  t-180 (6 months)  |  ●●●●●●●●●●●●● (0.13)                    │
│  t-365 (1 year)    |  ●●● (0.03)                              │
│                                                                   │
│  Interpretation:                                              │
│  • Recent (1 day) and 1-week horizons are important           │
│  • 2-month ago has high weight (regime change)               │
│  • 1-year ago has low weight (outdated)                      │
└─────────────────────────────────────────────────────────────────────┘

4.2: Multi-Head Attention

The Concept:

Instead of learning one attention pattern, multiple attention mechanisms operate simultaneously.

Multi-Head Attention:

MultiHead(Q, K, V) = Concat(head₁, head₂, ..., headₕ) · Wᴼ

Where:
headᵢ = Attention(QWᵢ^Q, KWᵢ^K, VWᵢ^V)

Benefits:
- Learns multiple market relationships simultaneously
- Captures both short-term and long-term dependencies
- Improves forecasting robustness

Financial Heads Example:

Financial Multi-Head Attention:

┌─────────────────────────────────────────────────────────────────────┐
│                    Financial Multi-Head Attention                 │
│                                                                   │
│  Head 1: High-Frequency Order Flow                             │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Focuses on tick-by-tick price movements               │   │
│  │  • Captures micro-structure patterns                     │   │
│  │  • Identifies order flow imbalances                      │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Head 2: Macroeconomic Yield Curves                           │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Focuses on interest rate movements                     │   │
│  │  • Captures yield curve shifts                            │   │
│  │  • Identifies monetary policy impacts                     │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Head 3: Cross-Asset Correlations                            │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Focuses on relationships between assets                │   │
│  │  • Captures contagion and spillover effects               │   │
│  │  • Identifies risk-off/risk-on regimes                   │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Head 4: Volatility Dynamics                                 │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Focuses on volatility clustering                       │   │
│  │  • Captures leverage effects                             │   │
│  │  • Identifies regime changes in volatility               │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Head 5: Sentiment and News                                   │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Focuses on text-based data                            │   │
│  │  • Captures sentiment shifts                             │   │
│  │  • Identifies event impacts                              │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

4.3: Variable Selection Networks

The Challenge:

Financial datasets often contain hundreds of variables, including interest rates, order book imbalance, trading volume, inflation, GDP growth, and treasury yields. Not all variables are equally important at all times.

Variable Selection Networks:

Variable Selection Networks learn dynamic feature importance, automatically assigning higher weights to the most informative predictors.

Variable Selection Network:

┌─────────────────────────────────────────────────────────────────────┐
│                    Variable Selection Network                     │
│                                                                   │
│  Input Variables:                                               │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  v₁  v₂  v₃  v₄  v₅  v₆  ...  vₙ                        │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  ┌───────────────────────────▼─────────────────────────────────┐   │
│  │  Variable Weights (Learned)                               │   │
│  │  ┌─────────────────────────────────────────────────────┐   │   │
│  │  │  v₁: 0.02  v₂: 0.15  v₃: 0.01  v₄: 0.25         │   │   │
│  │  │  v₅: 0.01  v₆: 0.30  ...  vₙ: 0.05              │   │   │
│  │  └─────────────────────────────────────────────────────┘   │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  ┌───────────────────────────▼─────────────────────────────────┐   │
│  │  Weighted Inputs                                          │   │
│  │  ┌─────────────────────────────────────────────────────┐   │   │
│  │  │  v₁×0.02  v₂×0.15  v₃×0.01  v₄×0.25  ...          │   │   │
│  │  └─────────────────────────────────────────────────────┘   │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  ┌───────────────────────────▼─────────────────────────────────┐   │
│  │  Selected Variables (Top K)                               │   │
│  │  ┌─────────────────────────────────────────────────────┐   │   │
│  │  │  v₆ (0.30), v₄ (0.25), v₂ (0.15), ...            │   │   │
│  │  └─────────────────────────────────────────────────────┘   │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  ┌───────────────────────────▼─────────────────────────────────┐   │
│  │  Temporal Fusion Transformer                              │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

4.4: Temporal Fusion Transformer (TFT) Architecture

Complete TFT Architecture:

Temporal Fusion Transformer Architecture:

┌─────────────────────────────────────────────────────────────────────┐
│                    Temporal Fusion Transformer                    │
│                                                                   │
│  Static Variables (Time-Invariant):                             │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Company sector                                          │   │
│  │  • Geographic region                                      │   │
│  │  • Regulatory environment                                 │   │
│  │  • Business model                                         │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Static Encoder:                                              │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Encodes static features into context vectors           │   │
│  │  • Provides baseline information                          │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Time-Varying Variables:                                      │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Prices, volumes, volatility                           │   │
│  │  • Macro indicators                                      │   │
│  │  • Technical indicators                                  │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Variable Selection Networks:                                 │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Selects most relevant variables                       │   │
│  │  • Learns dynamic feature importance                     │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  LSTM Encoder:                                               │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Processes historical sequence                         │   │
│  │  • Captures sequential dependencies                      │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Multi-Head Attention:                                       │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Learns long-range dependencies                        │   │
│  │  • Captures cross-asset relationships                    │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Decoder + LSTM:                                            │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  • Generates forecasts for future horizons               │   │
│  │  • Uses attention context to inform predictions          │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  Output: ŷ_{T+1}, ŷ_{T+2}, ..., ŷ_{T+H}                       │
└─────────────────────────────────────────────────────────────────────┘

TFT Advantages:

Interpretability: Attention weights show which variables and time steps matter most
Multi-Horizon: Predicts multiple future time steps simultaneously
Static Variables: Incorporates time-invariant features (sector, region)
Variable Selection: Automatically selects most relevant features
Long-Range Dependencies: Captures both short and long-term patterns

Part 5: Patch Time-Series Transformer (PatchTST)

5.1: The Efficiency Problem

Traditional Transformer Limitations:

Traditional transformers process every time step individually. For long financial sequences, this becomes computationally expensive.

Computational Complexity:

Standard Transformer: O(T²)
PatchTST: O(T/P × T/P)

Where:
- T = sequence length
- P = patch size

Example: T=512, P=32
Standard: O(512²) = 262,144 operations
PatchTST: O(16²) = 256 operations
Reduction: 1,000×

5.2: Patching

The Concept:

Instead of individual observations, the model groups consecutive observations into patches.

Patching Visualization:

┌─────────────────────────────────────────────────────────────────────┐
│                    Patching Visualization                         │
│                                                                   │
│  Sequence:                                                      │
│  x₁  x₂  x₃  x₄  x₅  x₆  x₇  x₈  x₉  x₁₀ x₁₁ x₁₂             │
│                                                                   │
│  Patch 1 (P=4):                                               │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  x₁  x₂  x₃  x₄                                           │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Patch 2 (P=4):                                               │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  x₅  x₆  x₇  x₈                                           │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Patch 3 (P=4):                                               │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  x₉  x₁₀ x₁₁ x₁₂                                          │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Number of Patches: 3                                          │
│  Instead of 12 tokens, only 3 patch tokens                    │
│                                                                   │
│  Patch Size P = 4                                               │
│  Sequence Length T = 12                                         │
│  Number of Patches N = T/P = 3                                 │
└─────────────────────────────────────────────────────────────────────┘

Mathematical Formulation:

Patching:

Given sequence x₁, x₂, ..., x_T
Choose patch size P
Number of patches: N = T/P (floor)

Patch i: p_i = [x_{(i-1)P+1}, x_{(i-1)P+2}, ..., x_{iP}]

Each patch is an embedding:
e_i = Linear(p_i)  (projection to d_model dimensions)

Input to Transformer: [e₁, e₂, ..., e_N]
(N tokens instead of T tokens)

5.3: Channel Independence

The Concept:

Rather than mixing all variables, PatchTST processes each feature independently before combining the learned representations.

Channel Independence:

┌─────────────────────────────────────────────────────────────────────┐
│                    Channel Independence                           │
│                                                                   │
│  Traditional (Channel Mixing):                                 │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  All variables processed together                          │   │
│  │  Variable interactions learned jointly                    │   │
│  │  Risk: Interference between unrelated variables           │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  PatchTST (Channel Independence):                              │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  Each variable processed independently                    │   │
│  │                                                             │   │
│  │  Price: [PatchTST for price]                              │   │
│  │  Volume: [PatchTST for volume]                            │   │
│  │  Volatility: [PatchTST for volatility]                    │   │
│  │  ...                                                       │   │
│  │                                                             │   │
│  │  Learned representations combined at the end              │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  Advantages:                                                   │
│  • Less feature interference                                    │   │
│  • Better scalability                                           │   │
│  • Improved generalization                                      │   │
│  • More interpretable                                           │   │
└─────────────────────────────────────────────────────────────────────┘

5.4: PatchTST Architecture

Complete PatchTST Architecture:

PatchTST Architecture:

┌─────────────────────────────────────────────────────────────────────┐
│                    PatchTST Architecture                          │
│                                                                   │
│  Input: Multi-variate time series                               │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  Variable 1: [x₁, x₂, ..., x_T]                          │   │
│  │  Variable 2: [x₁, x₂, ..., x_T]                          │   │
│  │  Variable 3: [x₁, x₂, ..., x_T]                          │   │
│  │  ...                                                       │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  For each variable independently:                            │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                                                             │   │
│  │  Patching:                                                │   │
│  │  ┌─────────────────────────────────────────────────────┐   │   │
│  │  │  Group P consecutive values into patches           │   │   │
│  │  │  Patch size P, stride S                            │   │   │
│  │  └─────────────────────────────────────────────────────┘   │   │
│  │                                                             │   │
│  │  Patch Embedding:                                         │   │
│  │  ┌─────────────────────────────────────────────────────┐   │   │
│  │  │  Linear projection of each patch                  │   │   │
│  │  │  Learnable positional encoding                    │   │   │
│  │  └─────────────────────────────────────────────────────┘   │   │
│  │                                                             │   │
│  │  Transformer Encoder:                                    │   │
│  │  ┌─────────────────────────────────────────────────────┐   │   │
│  │  │  Multi-Head Self-Attention                         │   │   │
│  │  │  Feed-Forward Network                             │   │   │
│  │  │  Layer Normalization                               │   │   │
│  │  └─────────────────────────────────────────────────────┘   │   │
│  │                                                             │   │
│  │  Output: Variable-specific representation                │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  ┌───────────────────────────▼─────────────────────────────────┐   │
│  │  Combine Variable Representations                       │   │
│  │  ┌─────────────────────────────────────────────────────┐   │   │
│  │  │  Average pooling across variables                 │   │   │
│  │  │  Or concatenation + linear projection              │   │   │
│  │  └─────────────────────────────────────────────────────┘   │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                              │                                    │
│  ┌───────────────────────────▼─────────────────────────────────┐   │
│  │  Output: ŷ_{T+1}, ŷ_{T+2}, ..., ŷ_{T+H}                 │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

5.5: PatchTST Implementation

Python Implementation:

import torch
import torch.nn as nn
import torch.nn.functional as F

class PatchTST(nn.Module):
    def __init__(
        self,
        n_variables,      # Number of input variables
        seq_len,          # Input sequence length
        patch_len,        # Length of each patch
        stride,           # Stride between patches
        d_model,          # Model dimension
        n_heads,          # Number of attention heads
        n_layers,         # Number of transformer layers
        d_ff,             # Feed-forward dimension
        output_len,       # Forecast horizon
        dropout=0.1
    ):
        super().__init__()
        self.n_variables = n_variables
        self.patch_len = patch_len
        self.stride = stride
        self.d_model = d_model
        
        # Number of patches
        self.n_patches = (seq_len - patch_len) // stride + 1
        
        # Patch embedding for each variable
        self.patch_embed = nn.Linear(patch_len, d_model)
        
        # Positional encoding for patches
        self.pos_encoding = nn.Parameter(
            torch.randn(1, self.n_patches, d_model) * 0.01
        )
        
        # Transformer encoder layers
        self.encoder_layers = nn.ModuleList([
            TransformerEncoderLayer(d_model, n_heads, d_ff, dropout)
            for _ in range(n_layers)
        ])
        
        # Output projection
        self.output_projection = nn.Linear(d_model * n_variables, output_len)
        
        self.dropout = nn.Dropout(dropout)
        self.norm = nn.LayerNorm(d_model)
    
    def forward(self, x):
        # x: (batch, seq_len, n_variables)
        batch_size = x.shape[0]
        
        # Reshape for channel-independent processing
        # (batch, n_variables, seq_len)
        x = x.permute(0, 2, 1)
        
        # Patching for each variable
        patch_outputs = []
        for var_idx in range(self.n_variables):
            # Extract variable
            var_data = x[:, var_idx, :]  # (batch, seq_len)
            
            # Create patches
            patches = []
            for i in range(self.n_patches):
                start = i * self.stride
                end = start + self.patch_len
                patch = var_data[:, start:end]  # (batch, patch_len)
                patches.append(patch)
            
            # Stack patches
            patches = torch.stack(patches, dim=1)  # (batch, n_patches, patch_len)
            
            # Embed patches
            patch_embeds = self.patch_embed(patches)  # (batch, n_patches, d_model)
            
            # Add positional encoding
            patch_embeds = patch_embeds + self.pos_encoding
            
            # Pass through transformer
            for layer in self.encoder_layers:
                patch_embeds = layer(patch_embeds)
            
            # Pool over patches
            var_rep = patch_embeds.mean(dim=1)  # (batch, d_model)
            patch_outputs.append(var_rep)
        
        # Combine variables
        combined = torch.cat(patch_outputs, dim=1)  # (batch, d_model * n_variables)
        
        # Output projection
        output = self.output_projection(combined)  # (batch, output_len)
        
        return output

class TransformerEncoderLayer(nn.Module):
    def __init__(self, d_model, n_heads, d_ff, dropout=0.1):
        super().__init__()
        self.self_attn = nn.MultiheadAttention(d_model, n_heads, dropout=dropout)
        self.feed_forward = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.ReLU(),
            nn.Dropout(dropout),
            nn.Linear(d_ff, d_model)
        )
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.dropout = nn.Dropout(dropout)
    
    def forward(self, x):
        # Self-attention
        attn_output, _ = self.self_attn(x, x, x)
        x = self.norm1(x + self.dropout(attn_output))
        
        # Feed-forward
        ff_output = self.feed_forward(x)
        x = self.norm2(x + self.dropout(ff_output))
        
        return x

Summary

Modern deep learning architectures significantly outperform traditional recurrent neural networks for financial forecasting by capturing complex temporal dependencies more efficiently.

Key Concepts Covered:

Encoder-Decoder (Seq2Seq) Models: Encode historical observations into a latent context vector and decode future values over multiple forecast horizons, making them suitable for multi-step prediction.

Teacher Forcing and Scheduled Sampling: Improve training stability while reducing the mismatch between training and real-world inference.

Temporal Convolutional Networks (TCNs): Replace recurrent computations with causal and dilated convolutions, enabling parallel processing and efficient learning of long-term dependencies.

Residual Connections: Facilitate the training of deep networks by improving gradient flow and reducing optimization difficulties.

Self-Attention and Temporal Fusion Transformers (TFTs): Dynamically learn which historical observations and input variables are most relevant, providing both improved predictive accuracy and model interpretability.

Patch Time-Series Transformers (PatchTST): Partition long sequences into patches and process channels independently, reducing computational cost while preserving local temporal structure.

Together, these architectures represent the current state of the art in institutional financial forecasting, enabling quantitative trading systems, risk engines, and portfolio management platforms to model complex market dynamics with greater accuracy, scalability, and interpretability.

Key Terminology Glossary

Term	Definition
Seq2Seq	Sequence-to-Sequence – encoder-decoder architecture
Teacher Forcing	Using ground truth as input during training
Free Running	Using predicted values as input during inference
Scheduled Sampling	Gradually transitioning from teacher forcing to free running
TCN	Temporal Convolutional Network – causal dilated convolutions
Causal Convolution	Convolution that uses only past information
Dilated Convolution	Convolution with gaps for larger receptive field
Residual Connection	Skip connection for improved gradient flow
Receptive Field	The range of input observations influencing a prediction
Self-Attention	Attention mechanism within the same sequence
Multi-Head Attention	Multiple parallel attention mechanisms
TFT	Temporal Fusion Transformer – interpretable forecasting
Variable Selection Network	Dynamic feature importance learning
PatchTST	Patch Time-Series Transformer – efficient patching
Channel Independence	Processing each variable separately
Patch Embedding	Projection of a patch to a vector

Certificate in Artificial Intelligence for Finance lesson1.2:Advanced Deep Learning Architectures for Financial Time-Series and Sequential Forecasting

Certificate in Artificial Intelligence for Finance

Learning Objectives:

Part 1: The Temporal Complexity of Financial Data

1.1: Understanding Financial Time-Series Data

1.2: The Need for Advanced Architectures

Part 2: Encoder-Decoder Architectures for Multi-Horizon Forecasting

2.1: Sequence-to-Sequence (Seq2Seq) Architecture

2.2: Teacher Forcing

2.3: Free Running (Autoregressive Inference)

2.4: Scheduled Sampling

2.5: Attention in Encoder-Decoder

Part 3: Temporal Convolutional Networks (TCNs)

3.1: Overview and Advantages

3.2: Causal Convolutions

3.3: Dilated Convolutions

3.4: Residual Connections

3.5: TCN for Financial Forecasting

Part 4: Attention Mechanisms and Temporal Fusion Transformers (TFT)

4.1: Self-Attention Fundamentals

4.2: Multi-Head Attention

4.3: Variable Selection Networks

4.4: Temporal Fusion Transformer (TFT) Architecture

Part 5: Patch Time-Series Transformer (PatchTST)

5.1: The Efficiency Problem

5.2: Patching

5.3: Channel Independence

5.4: PatchTST Architecture

5.5: PatchTST Implementation

Summary

Key Terminology Glossary

Further Reading

International BSF

START YOUR LEARNING JOURNEY