Learning Objectives:
-
Master Encoder-Decoder (Seq2Seq) LSTM architectures for multi-horizon forecasting
-
Understand Temporal Convolutional Networks (TCNs) and their advantages
-
Learn Temporal Fusion Transformers (TFTs) with attention mechanisms
-
Understand Patch Time-Series Transformers (PatchTST) for efficient long-sequence processing
-
Apply these architectures to financial time-series prediction problems
Part 1: The Temporal Complexity of Financial Data
1.1: Understanding Financial Time-Series Data
The Nature of Financial Data:
Financial markets generate enormous volumes of sequential data every second. Examples include tick-by-tick asset prices, Limit Order Book (LOB) states, trading volumes, macroeconomic indicators, volatility measures, and interest rate curves.
Unlike ordinary datasets, financial time series possess several challenging characteristics:
Characteristics of Financial Time Series: ┌─────────────────────────────────────────────────────────────────────┐ │ Financial Time Series Characteristics │ │ │ │ Non-Stationarity: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Statistical properties (mean, variance) change over time│ │ │ │ • Market regimes shift (bull, bear, volatile, calm) │ │ │ │ • Structural breaks (policy changes, crises) │ │ │ │ • Challenge: Models must adapt continuously │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Noise: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Random market fluctuations obscure underlying patterns │ │ │ │ • Signal-to-noise ratio is extremely low │ │ │ │ • Overfitting risk is high │ │ │ │ • Challenge: Distinguish signal from noise │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Missing Data: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Data gaps due to holidays, outages, illiquid markets │ │ │ │ • Irregular sampling intervals │ │ │ │ • Asynchronous data streams │ │ │ │ • Challenge: Handle incomplete observations │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Multi-Horizon Dependencies: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Current behavior depends on events minutes, days, or │ │ │ │ months earlier │ │ │ │ • Short-term (momentum) and long-term (trend) effects │ │ │ │ • Multiple time scales interact │ │ │ │ • Challenge: Capture dependencies across scales │ │ │ └─────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────┘
Time Series Notation:
A financial time series is represented as:
X = [x₁, x₂, x₃, ..., x_T]
Where:
- xₜ ∈ ℝᵈ is the feature vector at time t
- d is the number of features (price, volume, volatility, etc.)
- T is the sequence length
Target: yₜ ∈ ℝ (or ℝᵐ for multi-output)
- yₜ = f(x₁, x₂, ..., xₜ) + εₜ
- εₜ is random noise
Forecasting:
Given historical observations up to time T,
predict y_{T+1}, y_{T+2}, ..., y_{T+H}
where H is the forecast horizon.
1.2: The Need for Advanced Architectures
Why LSTMs Have Limitations:
Although Long Short-Term Memory (LSTM) networks significantly improved sequential learning by mitigating the vanishing gradient problem, institutional quantitative research has increasingly adopted more advanced architectures capable of modeling long-range dependencies more efficiently.
LSTM Limitations: ┌─────────────────────────────────────────────────────────────────────┐ │ LSTM Limitations │ │ │ │ Sequential Processing: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Cannot parallelize across time steps │ │ │ │ • Training is slow for long sequences │ │ │ │ • GPU utilization is inefficient │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Long-Range Dependencies: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Still struggles with very long sequences (>100 steps) │ │ │ │ • Information bottleneck in hidden state │ │ │ │ • Limited capacity to retain very old information │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Multi-Horizon Forecasting: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Standard LSTMs predict one step at a time │ │ │ │ • Error accumulation in autoregressive mode │ │ │ │ • Training and inference mismatch (teacher forcing) │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Interpretability: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Black-box model │ │ │ │ • Difficult to understand why predictions were made │ │ │ │ • Limited regulatory compliance │ │ │ └─────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────┘
The Advanced Architecture Landscape:
Institutional quantitative research has increasingly adopted:
-
Encoder-Decoder (Seq2Seq) LSTM architectures for multi-horizon forecasting
-
Temporal Convolutional Networks (TCNs) for parallel processing
-
Temporal Fusion Transformers (TFTs) for interpretability and attention
-
Patch Time-Series Transformers (PatchTST) for efficient long-sequence processing
Part 2: Encoder-Decoder Architectures for Multi-Horizon Forecasting
2.1: Sequence-to-Sequence (Seq2Seq) Architecture
Architecture Overview:
A Sequence-to-Sequence (Seq2Seq) model consists of two neural networks:
-
Encoder: Processes the historical sequence and compresses it into a latent representation
-
Decoder: Uses the latent representation to generate forecasts for future time periods
Encoder-Decoder (Seq2Seq) Architecture:
┌─────────────────────────────────────────────────────────────────────┐
│ Encoder-Decoder Architecture │
│ │
│ Encoder: │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ x₁ → [LSTM] → h₁ ─┐ │ │
│ │ x₂ → [LSTM] → h₂ ─┤ │ │
│ │ x₃ → [LSTM] → h₃ ─┤ → h_T (Context Vector) │ │
│ │ ... ─┤ │ │
│ │ x_T → [LSTM] → h_T ─┘ │ │
│ │ │ │
│ │ h_T = Encoder(x₁, x₂, ..., x_T) │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Decoder: │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ h_T → [LSTM] → ŷ_{T+1} │ │
│ │ │ │ │
│ │ └──────────────────────────────────────────────────┐│ │
│ │ h_T, ŷ_{T+1} → [LSTM] → ŷ_{T+2} ││ │
│ │ │ ││ │
│ │ └──────────────────────────────────────────────────┐│ │
│ │ h_T, ŷ_{T+2} → [LSTM] → ŷ_{T+3} ││ │
│ │ │ ││ │
│ │ └──────────────────────────────────────────────────┘│ │
│ │ ... │ │
│ │ h_T, ŷ_{T+H-1} → [LSTM] → ŷ_{T+H} │ │
│ │ │ │
│ │ ŷ_{T+i} = Decoder(h_T, ŷ_{T+i-1}) │ │
│ └─────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
The Encoder:
The encoder receives a historical sequence of observations:
x₁, x₂, x₃, ..., x_T Where: - xₜ = input feature vector at time t - T = length of the historical sequence The encoder compresses the entire sequence into a latent representation called the context vector. Mathematically: h_T = Encoder(x₁, x₂, ..., x_T) Where: - h_T = latent context vector summarizing all historical information The encoder may consist of stacked LSTM or GRU layers.
The Decoder:
The decoder uses the context vector to generate forecasts for future time periods.
Forecast Horizon: H
Output sequence: ŷ_{T+1}, ŷ_{T+2}, ..., ŷ_{T+H}
Where:
- ŷ = predicted value
- H = forecast horizon
Mathematically:
ŷ_{T+i} = Decoder(h_T, ŷ_{T+i-1})
For i = 1, 2, ..., H
Financial Applications:
Seq2Seq architectures are particularly useful for:
-
Stock price forecasting
-
Volatility prediction
-
Energy demand forecasting
-
Exchange-rate prediction
-
Multi-asset portfolio forecasting
2.2: Teacher Forcing
The Training Challenge:
During training, the decoder needs to learn to generate accurate predictions. However, if the decoder uses its own previous predictions, errors can accumulate and training becomes unstable.
Teacher Forcing Solution:
During training, the decoder receives the true historical value instead of its own prediction.
Teacher Forcing:
Input_{t+1} = y_t
Where:
- y_t = actual observed value
Advantages:
- Faster convergence
- Stable gradient propagation
- Reduced training error
Mathematical Formulation:
Without Teacher Forcing (Autoregressive):
ŷ_{t+1} = Decoder(h_T, ŷ_t)
With Teacher Forcing:
ŷ_{t+1} = Decoder(h_T, y_t)
Where y_t is the actual value, not the predicted value.
2.3: Free Running (Autoregressive Inference)
The Deployment Challenge:
During deployment, future observations are unavailable. The decoder must therefore use its own previous prediction.
Free Running (Autoregressive Inference):
Input_{t+1} = ŷ_t
Where ŷ_t is the model's own prediction.
Issue:
- Small forecasting errors propagate through subsequent predictions
- Error accumulation over long horizons
Error Propagation:
Error Accumulation:
Step 1: ŷ_{T+1} = Decoder(h_T, y_T) + ε₁
Step 2: ŷ_{T+2} = Decoder(h_T, ŷ_{T+1}) + ε₂
Step 3: ŷ_{T+3} = Decoder(h_T, ŷ_{T+2}) + ε₃
...
Each step incorporates the error from the previous step:
ŷ_{T+H} = f(y_T, ε₁, ε₂, ..., ε_H)
Errors accumulate over the forecast horizon.
2.4: Scheduled Sampling
The Solution:
Scheduled Sampling combines Teacher Forcing and Free Running.
Scheduled Sampling:
During training, the next decoder input is selected probabilistically.
Input_{t+1} = {
y_t with probability p (Teacher Forcing)
ŷ_t with probability (1-p) (Free Running)
}
Where:
- p = probability of using Teacher Forcing
- (1-p) = probability of using Free Running
Scheduling:
- Start with p = 1.0 (full Teacher Forcing)
- Gradually decrease p as training progresses
- End with p = 0.0 (full Free Running)
Mathematically:
p(epoch) = max(0, p_start - (p_start / total_epochs) × epoch)
Benefits:
-
Smooth transition from Teacher Forcing to Free Running
-
Reduces mismatch between training and inference
-
Better generalization on unseen data
-
More stable training
2.5: Attention in Encoder-Decoder
Attention Mechanism:
Instead of compressing the entire sequence into a single context vector, attention allows the decoder to focus on different parts of the encoder output at each step.
Attention in Encoder-Decoder:
For each decoding step i:
1. Compute attention scores: e_{ij} = score(s_i, h_j)
2. Softmax: α_{ij} = exp(e_{ij}) / Σ exp(e_{ik})
3. Context vector: c_i = Σ α_{ij} h_j
4. Update state: s_i = Decoder(s_{i-1}, c_i, y_{i-1})
Where:
- s_i = decoder state at step i
- h_j = encoder state at step j
- score = attention scoring function
- α = attention weights
- c_i = attention context vector
Attention Visualization: ┌─────────────────────────────────────────────────────────────────────┐ │ Attention Visualization │ │ │ │ Decoder Steps → │ │ │ │ Step 1: ●●●●●●●●●●●●●○○○○○○○○○○○ │ │ (Focuses on recent history) │ │ │ │ Step 2: ●●●●●○○○○○○○○○○○○○○○○○○○ │ │ (Focuses on mid-range history) │ │ │ │ Step 3: ●●○○○○○○○○○○○○○○○○○○●●●●● │ │ (Focuses on distant past and recent) │ │ │ │ Step 4: ○○○○○○○○○○○●●●●●●●●●●●●● │ │ (Focuses on future-relevant patterns) │ │ │ │ ● = High attention weight │ │ ○ = Low attention weight │ └─────────────────────────────────────────────────────────────────────┘
Part 3: Temporal Convolutional Networks (TCNs)
3.1: Overview and Advantages
What are TCNs?
Unlike recurrent networks, Temporal Convolutional Networks process the entire sequence simultaneously, making them highly parallelizable.
TCNs combine:
-
Causal convolutions
-
Dilated convolutions
-
Residual connections
TCN Advantages: ┌─────────────────────────────────────────────────────────────────────┐ │ TCN Advantages │ │ │ │ Parallel Processing: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Process all time steps simultaneously │ │ │ │ • Efficient GPU utilization │ │ │ │ • Faster training and inference │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Long-Range Dependencies: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Dilated convolutions provide exponential receptive field│ │ │ │ • Captures very long-term dependencies efficiently │ │ │ │ • No vanishing gradient problem │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Stable Gradients: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • No recurrent connections │ │ │ │ • Gradients flow directly through the network │ │ │ │ • No exploding/vanishing gradients │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Interpretability: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Residual connections show skip patterns │ │ │ │ • Easier to understand important time steps │ │ │ └─────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────┘
3.2: Causal Convolutions
The Principle:
Financial forecasting must never use future information. For time t, the prediction depends only on xₜ, xₜ₋₁, xₜ₋₂, …, never on xₜ₊₁.
Causal Convolution: ┌─────────────────────────────────────────────────────────────────────┐ │ Causal Convolution │ │ │ │ t=1 t=2 t=3 t=4 t=5 t=6 t=7 │ │ │ │ │ │ │ │ │ │ │ ▼ ▼ ▼ ▼ ▼ ▼ ▼ │ │ ● ● ● ● ● ● ● │ │ │ │ │ │ │ │ │ │ │ └────┼────┼────┼────┼────┼────┘ │ │ │ │ │ │ │ │ │ └────┼────┼────┼────┘ │ │ │ │ │ │ │ └────┼────┘ │ │ │ │ │ ▼ │ │ Output │ │ │ │ Each output depends only on current and past inputs │ │ No future information used │ └─────────────────────────────────────────────────────────────────────┘
Mathematical Formulation:
Causal Convolution:
y(t) = Σ_{i=0}^{k-1} f(i) · x(t-i)
Where:
- y(t) = convolution output
- f(i) = filter coefficients
- x = input sequence
- k = filter size
Properties:
- y(t) depends only on x(t), x(t-1), ..., x(t-k+1)
- Preserves temporal causality
- Prevents look-ahead bias
3.3: Dilated Convolutions
The Concept:
Dilated convolutions expand the receptive field exponentially without increasing the number of parameters.
Dilated Convolution Visualization: ┌─────────────────────────────────────────────────────────────────────┐ │ Dilated Convolution Visualization │ │ │ │ d=1 (Standard): │ │ x₁ x₂ x₃ x₄ x₅ x₆ x₇ │ │ │ │ │ │ │ └───┼───┘ │ │ │ │ │ y(t) │ │ │ │ d=2 (Dilated): │ │ x₁ x₂ x₃ x₄ x₅ x₆ x₇ │ │ │ │ │ │ └───────┘ │ │ │ │ │ y(t) │ │ │ │ d=3 (Dilated): │ │ x₁ x₂ x₃ x₄ x₅ x₆ x₇ │ │ │ │ │ │ └───────────┘ │ │ │ │ │ y(t) │ └─────────────────────────────────────────────────────────────────────┘
Mathematical Formulation:
Dilated Convolution:
y(t) = Σ_{i=0}^{k-1} f(i) · x(t - d·i)
Where:
- y(t) = convolution output
- f(i) = filter coefficient
- x = input sequence
- k = filter size
- d = dilation factor
Special Cases:
- d = 1: Standard convolution
- d > 1: Dilated convolution (skips elements)
Receptive Field:
R = 1 + Σ_{l=1}^{L} (k_l - 1) · d_l
With exponential dilation (d = 2^l):
R = 1 + Σ_{l=1}^{L} (k_l - 1) · 2^l
Receptive Field Expansion:
Receptive Field Growth: ┌─────────────────────────────────────────────────────────────────────┐ │ Receptive Field Growth │ │ │ │ Layer 1 (d=1, k=3): │ │ Receptive Field = 3 │ │ x₁ x₂ x₃ x₄ x₅ x₆ x₇ │ │ │ │ │ │ │ └───┼───┘ │ │ │ │ │ y₁ │ │ │ │ Layer 2 (d=2, k=3): │ │ Receptive Field = 7 │ │ x₁ x₂ x₃ x₄ x₅ x₆ x₇ │ │ │ │ │ │ │ │ │ │ │ └───┼───┘ └───┼───┘ └───┼───┘ │ │ │ │ │ │ │ └───────────┼───────────┘ │ │ │ │ │ y₂ │ │ │ │ Layer 3 (d=4, k=3): │ │ Receptive Field = 15 │ │ (Exponential growth) │ └─────────────────────────────────────────────────────────────────────┘
3.4: Residual Connections
The Principle:
Residual learning allows information to bypass intermediate layers, improving gradient flow and enabling very deep networks.
Residual Connection: ┌─────────────────────────────────────────────────────────────────────┐ │ Residual Connection │ │ │ │ Input ──────────────────────────────────────────────────────────┐ │ │ │ │ │ │ ▼ │ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ │ Convolutional Layer (F) │ │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ │ │ │ │ + │ │ │ └───────────────────────────┬─────────────────────────────────┘ │ │ │ │ │ ▼ │ │ Output │ │ │ │ y = F(x) + x │ │ │ │ Where: │ │ - F(x) = learned transformation │ │ - x = original input │ │ │ │ Benefits: │ │ • Reduces vanishing gradients │ │ • Stabilizes optimization │ │ • Enables very deep TCNs │ │ • Allows gradients to bypass layers │ └─────────────────────────────────────────────────────────────────────┘
Residual TCN Block:
Residual TCN Block: ┌─────────────────────────────────────────────────────────────────────┐ │ Residual TCN Block │ │ │ │ Input x │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Dilated Causal Convolution (d=1) │ │ │ │ Weight Normalization │ │ │ │ Activation (ReLU) │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Dilated Causal Convolution (d=2) │ │ │ │ Weight Normalization │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ │ │ │ │ │ + │ │ └─────────────────────────────────────────────────────┐ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Activation (ReLU) │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ Output y = F(x) + x │ └─────────────────────────────────────────────────────────────────────┘
3.5: TCN for Financial Forecasting
TCN Architecture for Finance:
TCN Architecture:
┌─────────────────────────────────────────────────────────────────────┐
│ TCN Architecture for Finance │
│ │
│ Input: Historical sequence (T steps, D features) │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ x₁, x₂, ..., x_T (prices, volume, volatility, macros) │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────▼─────────────────────────────────┐ │
│ │ Residual TCN Block 1 (d=1) │ │
│ │ Receptive Field: 3 │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────▼─────────────────────────────────┐ │
│ │ Residual TCN Block 2 (d=2) │ │
│ │ Receptive Field: 7 │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────▼─────────────────────────────────┐ │
│ │ Residual TCN Block 3 (d=4) │ │
│ │ Receptive Field: 15 │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────▼─────────────────────────────────┐ │
│ │ Residual TCN Block 4 (d=8) │ │
│ │ Receptive Field: 31 │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────▼─────────────────────────────────┐ │
│ │ Global Average Pooling │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────▼─────────────────────────────────┐ │
│ │ Dense Layers for Prediction │ │
│ │ • Hidden layer: 128 neurons (ReLU) │ │
│ │ • Output layer: H neurons (forecast horizon) │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ Output: ŷ_{T+1}, ŷ_{T+2}, ..., ŷ_{T+H} │
└─────────────────────────────────────────────────────────────────────┘
Part 4: Attention Mechanisms and Temporal Fusion Transformers (TFT)
4.1: Self-Attention Fundamentals
The Core Concept:
Attention mechanisms allow a model to learn which observations are most important rather than treating every historical observation equally.
Scaled Dot-Product Attention:
Scaled Dot-Product Attention: Attention(Q, K, V) = softmax(QKᵀ / √d_k) · V Where: - Q = Query matrix (what we're looking for) - K = Key matrix (what each element contains) - V = Value matrix (actual content) - d_k = dimension of key vectors Interpretation: - Each observation assigns importance weights to every other observation - Large attention scores indicate stronger influence on the prediction - Softmax ensures weights sum to 1 - Scaling by 1/√d_k prevents vanishing gradients
Attention Weights Visualization:
Attention Weights for Financial Time Series: ┌─────────────────────────────────────────────────────────────────────┐ │ Attention Weights │ │ │ │ Target: Volatility Forecast │ │ │ │ Time Step | Attention Weight │ │ ─────────────────────────────────────────────────────────────── │ │ t-1 (1 day ago) | ●●●●●●●●●●●●●●● (0.15) │ │ t-2 | ●●●●●●●●●● (0.10) │ │ t-5 (1 week ago) | ●●●●●●●●●●●●●●●● (0.16) │ │ t-10 | ●●●●●● (0.06) │ │ t-20 (1 month) | ●●●●●●●●●●● (0.11) │ │ t-30 | ●●●● (0.04) │ │ t-60 (2 months) | ●●●●●●●●●●●●●●●●●● (0.18) │ │ t-90 (3 months) | ●●●●● (0.05) │ │ t-180 (6 months) | ●●●●●●●●●●●●● (0.13) │ │ t-365 (1 year) | ●●● (0.03) │ │ │ │ Interpretation: │ │ • Recent (1 day) and 1-week horizons are important │ │ • 2-month ago has high weight (regime change) │ │ • 1-year ago has low weight (outdated) │ └─────────────────────────────────────────────────────────────────────┘
4.2: Multi-Head Attention
The Concept:
Instead of learning one attention pattern, multiple attention mechanisms operate simultaneously.
Multi-Head Attention: MultiHead(Q, K, V) = Concat(head₁, head₂, ..., headₕ) · Wᴼ Where: headᵢ = Attention(QWᵢ^Q, KWᵢ^K, VWᵢ^V) Benefits: - Learns multiple market relationships simultaneously - Captures both short-term and long-term dependencies - Improves forecasting robustness
Financial Heads Example:
Financial Multi-Head Attention: ┌─────────────────────────────────────────────────────────────────────┐ │ Financial Multi-Head Attention │ │ │ │ Head 1: High-Frequency Order Flow │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Focuses on tick-by-tick price movements │ │ │ │ • Captures micro-structure patterns │ │ │ │ • Identifies order flow imbalances │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Head 2: Macroeconomic Yield Curves │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Focuses on interest rate movements │ │ │ │ • Captures yield curve shifts │ │ │ │ • Identifies monetary policy impacts │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Head 3: Cross-Asset Correlations │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Focuses on relationships between assets │ │ │ │ • Captures contagion and spillover effects │ │ │ │ • Identifies risk-off/risk-on regimes │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Head 4: Volatility Dynamics │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Focuses on volatility clustering │ │ │ │ • Captures leverage effects │ │ │ │ • Identifies regime changes in volatility │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Head 5: Sentiment and News │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ • Focuses on text-based data │ │ │ │ • Captures sentiment shifts │ │ │ │ • Identifies event impacts │ │ │ └─────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────┘
4.3: Variable Selection Networks
The Challenge:
Financial datasets often contain hundreds of variables, including interest rates, order book imbalance, trading volume, inflation, GDP growth, and treasury yields. Not all variables are equally important at all times.
Variable Selection Networks:
Variable Selection Networks learn dynamic feature importance, automatically assigning higher weights to the most informative predictors.
Variable Selection Network: ┌─────────────────────────────────────────────────────────────────────┐ │ Variable Selection Network │ │ │ │ Input Variables: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ v₁ v₂ v₃ v₄ v₅ v₆ ... vₙ │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌───────────────────────────▼─────────────────────────────────┐ │ │ │ Variable Weights (Learned) │ │ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ │ │ v₁: 0.02 v₂: 0.15 v₃: 0.01 v₄: 0.25 │ │ │ │ │ │ v₅: 0.01 v₆: 0.30 ... vₙ: 0.05 │ │ │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌───────────────────────────▼─────────────────────────────────┐ │ │ │ Weighted Inputs │ │ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ │ │ v₁×0.02 v₂×0.15 v₃×0.01 v₄×0.25 ... │ │ │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌───────────────────────────▼─────────────────────────────────┐ │ │ │ Selected Variables (Top K) │ │ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ │ │ v₆ (0.30), v₄ (0.25), v₂ (0.15), ... │ │ │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌───────────────────────────▼─────────────────────────────────┐ │ │ │ Temporal Fusion Transformer │ │ │ └─────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────┘
4.4: Temporal Fusion Transformer (TFT) Architecture
Complete TFT Architecture:
Temporal Fusion Transformer Architecture:
┌─────────────────────────────────────────────────────────────────────┐
│ Temporal Fusion Transformer │
│ │
│ Static Variables (Time-Invariant): │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ • Company sector │ │
│ │ • Geographic region │ │
│ │ • Regulatory environment │ │
│ │ • Business model │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ Static Encoder: │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ • Encodes static features into context vectors │ │
│ │ • Provides baseline information │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ Time-Varying Variables: │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ • Prices, volumes, volatility │ │
│ │ • Macro indicators │ │
│ │ • Technical indicators │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ Variable Selection Networks: │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ • Selects most relevant variables │ │
│ │ • Learns dynamic feature importance │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ LSTM Encoder: │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ • Processes historical sequence │ │
│ │ • Captures sequential dependencies │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ Multi-Head Attention: │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ • Learns long-range dependencies │ │
│ │ • Captures cross-asset relationships │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ Decoder + LSTM: │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ • Generates forecasts for future horizons │ │
│ │ • Uses attention context to inform predictions │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ Output: ŷ_{T+1}, ŷ_{T+2}, ..., ŷ_{T+H} │
└─────────────────────────────────────────────────────────────────────┘
TFT Advantages:
-
Interpretability: Attention weights show which variables and time steps matter most
-
Multi-Horizon: Predicts multiple future time steps simultaneously
-
Static Variables: Incorporates time-invariant features (sector, region)
-
Variable Selection: Automatically selects most relevant features
-
Long-Range Dependencies: Captures both short and long-term patterns
Part 5: Patch Time-Series Transformer (PatchTST)
5.1: The Efficiency Problem
Traditional Transformer Limitations:
Traditional transformers process every time step individually. For long financial sequences, this becomes computationally expensive.
Computational Complexity: Standard Transformer: O(T²) PatchTST: O(T/P × T/P) Where: - T = sequence length - P = patch size Example: T=512, P=32 Standard: O(512²) = 262,144 operations PatchTST: O(16²) = 256 operations Reduction: 1,000×
5.2: Patching
The Concept:
Instead of individual observations, the model groups consecutive observations into patches.
Patching Visualization: ┌─────────────────────────────────────────────────────────────────────┐ │ Patching Visualization │ │ │ │ Sequence: │ │ x₁ x₂ x₃ x₄ x₅ x₆ x₇ x₈ x₉ x₁₀ x₁₁ x₁₂ │ │ │ │ Patch 1 (P=4): │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ x₁ x₂ x₃ x₄ │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Patch 2 (P=4): │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ x₅ x₆ x₇ x₈ │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Patch 3 (P=4): │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ x₉ x₁₀ x₁₁ x₁₂ │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Number of Patches: 3 │ │ Instead of 12 tokens, only 3 patch tokens │ │ │ │ Patch Size P = 4 │ │ Sequence Length T = 12 │ │ Number of Patches N = T/P = 3 │ └─────────────────────────────────────────────────────────────────────┘
Mathematical Formulation:
Patching:
Given sequence x₁, x₂, ..., x_T
Choose patch size P
Number of patches: N = T/P (floor)
Patch i: p_i = [x_{(i-1)P+1}, x_{(i-1)P+2}, ..., x_{iP}]
Each patch is an embedding:
e_i = Linear(p_i) (projection to d_model dimensions)
Input to Transformer: [e₁, e₂, ..., e_N]
(N tokens instead of T tokens)
5.3: Channel Independence
The Concept:
Rather than mixing all variables, PatchTST processes each feature independently before combining the learned representations.
Channel Independence: ┌─────────────────────────────────────────────────────────────────────┐ │ Channel Independence │ │ │ │ Traditional (Channel Mixing): │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ All variables processed together │ │ │ │ Variable interactions learned jointly │ │ │ │ Risk: Interference between unrelated variables │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ PatchTST (Channel Independence): │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Each variable processed independently │ │ │ │ │ │ │ │ Price: [PatchTST for price] │ │ │ │ Volume: [PatchTST for volume] │ │ │ │ Volatility: [PatchTST for volatility] │ │ │ │ ... │ │ │ │ │ │ │ │ Learned representations combined at the end │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Advantages: │ │ • Less feature interference │ │ │ • Better scalability │ │ │ • Improved generalization │ │ │ • More interpretable │ │ └─────────────────────────────────────────────────────────────────────┘
5.4: PatchTST Architecture
Complete PatchTST Architecture:
PatchTST Architecture:
┌─────────────────────────────────────────────────────────────────────┐
│ PatchTST Architecture │
│ │
│ Input: Multi-variate time series │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Variable 1: [x₁, x₂, ..., x_T] │ │
│ │ Variable 2: [x₁, x₂, ..., x_T] │ │
│ │ Variable 3: [x₁, x₂, ..., x_T] │ │
│ │ ... │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ For each variable independently: │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ Patching: │ │
│ │ ┌─────────────────────────────────────────────────────┐ │ │
│ │ │ Group P consecutive values into patches │ │ │
│ │ │ Patch size P, stride S │ │ │
│ │ └─────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ Patch Embedding: │ │
│ │ ┌─────────────────────────────────────────────────────┐ │ │
│ │ │ Linear projection of each patch │ │ │
│ │ │ Learnable positional encoding │ │ │
│ │ └─────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ Transformer Encoder: │ │
│ │ ┌─────────────────────────────────────────────────────┐ │ │
│ │ │ Multi-Head Self-Attention │ │ │
│ │ │ Feed-Forward Network │ │ │
│ │ │ Layer Normalization │ │ │
│ │ └─────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ Output: Variable-specific representation │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────▼─────────────────────────────────┐ │
│ │ Combine Variable Representations │ │
│ │ ┌─────────────────────────────────────────────────────┐ │ │
│ │ │ Average pooling across variables │ │ │
│ │ │ Or concatenation + linear projection │ │ │
│ │ └─────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────▼─────────────────────────────────┐ │
│ │ Output: ŷ_{T+1}, ŷ_{T+2}, ..., ŷ_{T+H} │ │
│ └─────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
5.5: PatchTST Implementation
Python Implementation:
import torch import torch.nn as nn import torch.nn.functional as F class PatchTST(nn.Module): def __init__( self, n_variables, # Number of input variables seq_len, # Input sequence length patch_len, # Length of each patch stride, # Stride between patches d_model, # Model dimension n_heads, # Number of attention heads n_layers, # Number of transformer layers d_ff, # Feed-forward dimension output_len, # Forecast horizon dropout=0.1 ): super().__init__() self.n_variables = n_variables self.patch_len = patch_len self.stride = stride self.d_model = d_model # Number of patches self.n_patches = (seq_len - patch_len) // stride + 1 # Patch embedding for each variable self.patch_embed = nn.Linear(patch_len, d_model) # Positional encoding for patches self.pos_encoding = nn.Parameter( torch.randn(1, self.n_patches, d_model) * 0.01 ) # Transformer encoder layers self.encoder_layers = nn.ModuleList([ TransformerEncoderLayer(d_model, n_heads, d_ff, dropout) for _ in range(n_layers) ]) # Output projection self.output_projection = nn.Linear(d_model * n_variables, output_len) self.dropout = nn.Dropout(dropout) self.norm = nn.LayerNorm(d_model) def forward(self, x): # x: (batch, seq_len, n_variables) batch_size = x.shape[0] # Reshape for channel-independent processing # (batch, n_variables, seq_len) x = x.permute(0, 2, 1) # Patching for each variable patch_outputs = [] for var_idx in range(self.n_variables): # Extract variable var_data = x[:, var_idx, :] # (batch, seq_len) # Create patches patches = [] for i in range(self.n_patches): start = i * self.stride end = start + self.patch_len patch = var_data[:, start:end] # (batch, patch_len) patches.append(patch) # Stack patches patches = torch.stack(patches, dim=1) # (batch, n_patches, patch_len) # Embed patches patch_embeds = self.patch_embed(patches) # (batch, n_patches, d_model) # Add positional encoding patch_embeds = patch_embeds + self.pos_encoding # Pass through transformer for layer in self.encoder_layers: patch_embeds = layer(patch_embeds) # Pool over patches var_rep = patch_embeds.mean(dim=1) # (batch, d_model) patch_outputs.append(var_rep) # Combine variables combined = torch.cat(patch_outputs, dim=1) # (batch, d_model * n_variables) # Output projection output = self.output_projection(combined) # (batch, output_len) return output class TransformerEncoderLayer(nn.Module): def __init__(self, d_model, n_heads, d_ff, dropout=0.1): super().__init__() self.self_attn = nn.MultiheadAttention(d_model, n_heads, dropout=dropout) self.feed_forward = nn.Sequential( nn.Linear(d_model, d_ff), nn.ReLU(), nn.Dropout(dropout), nn.Linear(d_ff, d_model) ) self.norm1 = nn.LayerNorm(d_model) self.norm2 = nn.LayerNorm(d_model) self.dropout = nn.Dropout(dropout) def forward(self, x): # Self-attention attn_output, _ = self.self_attn(x, x, x) x = self.norm1(x + self.dropout(attn_output)) # Feed-forward ff_output = self.feed_forward(x) x = self.norm2(x + self.dropout(ff_output)) return x
Summary
Modern deep learning architectures significantly outperform traditional recurrent neural networks for financial forecasting by capturing complex temporal dependencies more efficiently.
Key Concepts Covered:
Encoder-Decoder (Seq2Seq) Models: Encode historical observations into a latent context vector and decode future values over multiple forecast horizons, making them suitable for multi-step prediction.
Teacher Forcing and Scheduled Sampling: Improve training stability while reducing the mismatch between training and real-world inference.
Temporal Convolutional Networks (TCNs): Replace recurrent computations with causal and dilated convolutions, enabling parallel processing and efficient learning of long-term dependencies.
Residual Connections: Facilitate the training of deep networks by improving gradient flow and reducing optimization difficulties.
Self-Attention and Temporal Fusion Transformers (TFTs): Dynamically learn which historical observations and input variables are most relevant, providing both improved predictive accuracy and model interpretability.
Patch Time-Series Transformers (PatchTST): Partition long sequences into patches and process channels independently, reducing computational cost while preserving local temporal structure.
Together, these architectures represent the current state of the art in institutional financial forecasting, enabling quantitative trading systems, risk engines, and portfolio management platforms to model complex market dynamics with greater accuracy, scalability, and interpretability.
Key Terminology Glossary
| Term | Definition |
|---|---|
| Seq2Seq | Sequence-to-Sequence – encoder-decoder architecture |
| Teacher Forcing | Using ground truth as input during training |
| Free Running | Using predicted values as input during inference |
| Scheduled Sampling | Gradually transitioning from teacher forcing to free running |
| TCN | Temporal Convolutional Network – causal dilated convolutions |
| Causal Convolution | Convolution that uses only past information |
| Dilated Convolution | Convolution with gaps for larger receptive field |
| Residual Connection | Skip connection for improved gradient flow |
| Receptive Field | The range of input observations influencing a prediction |
| Self-Attention | Attention mechanism within the same sequence |
| Multi-Head Attention | Multiple parallel attention mechanisms |
| TFT | Temporal Fusion Transformer – interpretable forecasting |
| Variable Selection Network | Dynamic feature importance learning |
| PatchTST | Patch Time-Series Transformer – efficient patching |
| Channel Independence | Processing each variable separately |
| Patch Embedding | Projection of a patch to a vector |
Further Reading
-
Sutskever, I., et al. (2014). Sequence to Sequence Learning with Neural Networks. NeurIPS.
-
Bai, S., et al. (2018). An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. arXiv.
-
Lim, B., et al. (2019). Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting. arXiv.
-
Nie, Y., et al. (2023). A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. ICLR.
-
Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS.