Lesson 4.8: Online Learning and Transfer Learning for Financial AI

1. Learning Objectives

By the end of this lesson, you will be able to:

Understand the principles of online learning and its application to financial time series.
Implement Stochastic Gradient Descent (SGD) and its variants for online learning.
Apply online learning to portfolio optimisation and high-frequency trading.
Understand the mathematical foundations of transfer learning and domain adaptation.
Apply fine-tuning to pre-trained models for financial applications.
Implement domain adaptation techniques for regime changes.
Understand the limitations and pitfalls of transfer learning in finance.
Apply meta-learning (learning to learn) for rapid adaptation to new markets.

2. Online Learning – Learning from Streaming Data

2.1 The Online Learning Framework

In online learning, data arrives sequentially. The model makes predictions and receives feedback immediately.

Algorithm:

For t = 1, 2, ..., T:
    1. Receive input x_t
    2. Make prediction ŷ_t = f_t(x_t)
    3. Receive true label y_t
    4. Observe loss l_t(ŷ_t, y_t)
    5. Update model to f_{t+1}

Key Difference from Batch Learning: Online learning must adapt to changing data distributions (non-stationarity).

2.2 Mathematical Formulation

The goal is to minimise cumulative regret:
Regret_T = Σ_{t=1}^{T} l_t(w_t, x_t) - min_{w ∈ W} Σ_{t=1}^{T} l_t(w, x_t)

2.3 Online Gradient Descent (OGD)
w_{t+1} = w_t - η_t ∇l_t(w_t)

Regret Bound: For convex losses with Lipschitz gradients:
Regret_T ≤ O(sqrt(T))

2.4 Online SGD with Financial Data

def online_sgd_portfolio(X, y, learning_rate=0.1, epochs=1):
    """
    Online SGD for portfolio optimisation.
    """
    # Initialise weights
    w = np.ones(X.shape[1]) / X.shape[1]

    for t in range(len(X)):
        # Get current data point
        x_t = X.iloc[t].values
        y_t = y.iloc[t]

        # Make prediction
        y_pred = np.dot(w, x_t)

        # Compute loss (MSE)
        loss = (y_t - y_pred)**2

        # Compute gradient
        gradient = -2 * x_t * (y_t - y_pred)

        # Update weights
        w = w - learning_rate * gradient

        # Project onto simplex (weights sum to 1, w_i >= 0)
        w = project_to_simplex(w)

    return w

2.5 Online Portfolio Optimisation

class OnlinePortfolioOptimiser:
    def __init__(self, n_assets, learning_rate=0.01, method='sd'):
        """
        method: 'sd' (semi-definite), 'eg' (exponentiated gradient)
        """
        self.n_assets = n_assets
        self.lr = learning_rate
        self.method = method

        # Initialise weights
        self.weights = np.ones(n_assets) / n_assets

    def update(self, returns):
        """
        Update portfolio weights using online learning.
        """
        # Current return
        portfolio_return = np.dot(self.weights, returns)

        if self.method == 'sd':
            # Semi-definite online learning
            gradient = -returns / (portfolio_return + 1e-8)
            self.weights = self.weights - self.lr * gradient
            self.weights = self._project_simplex(self.weights)

        elif self.method == 'eg':
            # Exponentiated gradient (for non-negative weights)
            gradient = -returns / (portfolio_return + 1e-8)
            self.weights = self.weights * np.exp(-self.lr * gradient)
            self.weights = self.weights / self.weights.sum()

        # Clip weights to prevent extreme values
        self.weights = np.clip(self.weights, 0, 1)
        self.weights = self.weights / self.weights.sum()

        return self.weights

    def _project_simplex(self, w):
        """
        Project onto the simplex (sum = 1, w_i >= 0).
        """
        w = np.maximum(w, 0)
        w = w / w.sum()
        return w

    def get_weights(self):
        return self.weights

2.6 Online Learning for Market Making

class OnlineMarketMaker:
    def __init__(self, spread=0.001, inventory_limit=1000):
        self.spread = spread
        self.inventory_limit = inventory_limit
        self.inventory = 0
        self.bid = 0
        self.ask = 0

    def update(self, mid_price, order_flow):
        """
        Update quotes based on order flow.
        """
        # Adjust quotes based on inventory
        inventory_signal = -self.inventory / self.inventory_limit

        # Online learning: adapt spread based on volatility
        volatility = np.std(order_flow[-20:])
        dynamic_spread = self.spread + 0.5 * volatility

        # Set quotes
        self.bid = mid_price - dynamic_spread * (1 + inventory_signal * 0.5)
        self.ask = mid_price + dynamic_spread * (1 - inventory_signal * 0.5)

        # Update inventory based on filled orders
        # (simplified)
        filled_orders = self._simulate_fills(order_flow)
        self.inventory += filled_orders

        return self.bid, self.ask

3. Transfer Learning – Leveraging Pre-Trained Models

3.1 What is Transfer Learning?

Transfer learning applies knowledge learned from one task to a different but related task.

Types:

Inductive Transfer: Labels are available in the source domain.
Transductive Transfer: Unlabelled target data is available.
Unsupervised Transfer: No labels in either domain.

3.2 Mathematical Formulation

Given source domain D_S = (X_S, y_S) and target domain D_T = (X_T, y_T), the goal is to learn f: X_T → y_T using knowledge from D_S.

3.3 Transfer Learning in Finance – Scenarios

Source	Target	Application
S&P 500 returns	European equity returns	Cross-market prediction
FX data	Commodity data	Similar market structure
Pre-trained NLP model	Financial text	Sentiment analysis
Pre-trained CNN	Satellite imagery	Alternative data

3.4 Fine-Tuning – The Most Common Approach

import torch
import torch.nn as nn

def fine_tune_model(pretrained_model, target_data, target_labels, num_epochs=10):
    """
    Fine-tune a pre-trained model on target data.
    """
    # Freeze all layers except the last few
    for param in pretrained_model.parameters():
        param.requires_grad = False

    # Replace the last layer for the target task
    pretrained_model.fc = nn.Linear(
        pretrained_model.fc.in_features,
        target_labels.shape[1] if len(target_labels.shape) > 1 else 1
    )

    # Unfreeze the last few layers
    for param in list(pretrained_model.parameters())[-5:]:
        param.requires_grad = True

    # Train on target data
    optimizer = torch.optim.Adam(
        filter(lambda p: p.requires_grad, pretrained_model.parameters()),
        lr=1e-4
    )

    for epoch in range(num_epochs):
        pretrained_model.train()
        for X_batch, y_batch in target_data:
            optimizer.zero_grad()
            y_pred = pretrained_model(X_batch)
            loss = nn.MSELoss()(y_pred, y_batch)
            loss.backward()
            optimizer.step()

    return pretrained_model

3.5 Domain Adaptation – When Distributions Differ

Domain adaptation handles the case where the source and target distributions are different:
P(X_S) ≠ P(X_T) (covariate shift) or P(y_S | X_S) ≠ P(y_T | X_T).

Adversarial Domain Adaptation:

class DomainAdversarialNetwork(nn.Module):
    """
    Domain Adversarial Neural Network (DANN) for domain adaptation.
    """
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(DomainAdversarialNetwork, self).__init__()

        # Feature extractor
        self.feature_extractor = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU()
        )

        # Label predictor
        self.label_predictor = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, output_dim)
        )

        # Domain classifier (adversarial)
        self.domain_classifier = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 2)  # Binary: source or target
        )

    def forward(self, x, alpha=1.0):
        """
        alpha: gradient reversal factor (larger = stronger adaptation)
        """
        features = self.feature_extractor(x)

        # Label prediction
        label = self.label_predictor(features)

        # Domain classification with gradient reversal
        # (Implementation requires custom GradientReversalLayer)
        reversed_features = gradient_reverse(features, alpha)
        domain = self.domain_classifier(reversed_features)

        return label, domain

4. Meta-Learning – Learning to Learn

Meta-learning trains models to quickly adapt to new tasks with limited data.

4.1 MAML (Model-Agnostic Meta-Learning)

Objective: Find initial parameters θ such that one gradient step on a new task leads to good performance.

Inner Loop (Task-Specific):
θ'_i = θ - α ∇θ L_{T_i}(θ)

Outer Loop (Meta):
θ ← θ - β ∇θ Σ_{i=1}^{N} L_{T_i}(θ'_i)

Implementation:

def maml_update(model, tasks, inner_lr=0.01, outer_lr=0.001, inner_steps=5):
    """
    MAML update step.
    """
    # Copy model parameters
    meta_theta = [p.clone() for p in model.parameters()]

    for task in tasks:
        # Clone model for this task
        task_model = copy.deepcopy(model)

        # Inner loop: adapt to task
        for _ in range(inner_steps):
            X, y = task.sample()
            y_pred = task_model(X)
            loss = nn.MSELoss()(y_pred, y)
            gradients = torch.autograd.grad(loss, task_model.parameters())

            # Update task-specific parameters
            for param, grad in zip(task_model.parameters(), gradients):
                param.data = param.data - inner_lr * grad

        # Compute loss on validation set
        X_val, y_val = task.sample_val()
        y_pred_val = task_model(X_val)
        meta_loss = nn.MSELoss()(y_pred_val, y_val)

        # Meta-gradient (gradients of meta-loss wrt original parameters)
        meta_grads = torch.autograd.grad(meta_loss, meta_theta)

        # Update meta-parameters
        for param, grad in zip(meta_theta, meta_grads):
            param.data = param.data - outer_lr * grad

    return meta_theta

4.2 Financial Application – Rapid Adaptation to New Markets

def adapt_to_new_market(pretrained_model, new_market_data, n_epochs=5):
    """
    Use meta-learning to rapidly adapt to a new market.
    """
    # Use MAML or fine-tuning
    # For MAML, we need multiple tasks from the same domain

    # Example: adapt from US to European markets
    # Source tasks: different US market regimes
    # Target task: European market

    # MAML adaptation
    adapted_params = maml_update(pretrained_model, new_market_data)

    return adapted_params

5. Challenges and Pitfalls

Challenge	Description	Solution
Catastrophic Forgetting	Model forgets previous knowledge when learning new tasks.	Elastic Weight Consolidation (EWC), rehearsal.
Negative Transfer	Transfer learning harms performance.	Careful task selection, domain similarity check.
Overfitting to Target	Fine-tuning on small target data leads to overfitting.	Regularisation, smaller learning rate.
Distribution Shift	Source and target distributions differ.	Domain adaptation, adversarial training.
Computational Cost	Online learning can be expensive.	Mini-batch updates, efficient algorithms.

6. Summary for the AI Practitioner

Online learning is essential for streaming financial data (market data, order flow). Regret bounds ensure convergence.
SGD is the workhorse of online learning. Use it with a projection onto the simplex for portfolio weights.
Online portfolio optimisation (Semi-Definite, Exponentiated Gradient) adapts to changing market conditions.
Transfer learning leverages pre-trained models to accelerate training on new tasks.
Fine-tuning is the simplest approach. Freeze early layers and train the last layers on target data.
Domain adaptation handles cases where the input distributions differ. Use adversarial training (DANN) for robust adaptation.
Meta-learning (MAML) enables rapid adaptation to new tasks with limited data. Useful for entering new markets.
Key pitfalls: Catastrophic forgetting, negative transfer, overfitting to target, distribution shift.
Practical rule: Always validate transfer learning on a hold-out set. Monitor for performance degradation.