1. Learning Objectives

By the end of this lesson, you will be able to:

• Understand the bias-variance tradeoff and its implications for financial modelling.

• Derive the Ordinary Least Squares (OLS) estimator from first principles using calculus and linear algebra.

• Interpret regression coefficients in the context of factor models (CAPM, Fama-French).

• Implement and evaluate linear regression for predicting asset returns.

• Derive the logistic regression model from the log-odds formulation.

• Apply logistic regression to credit default prediction and market direction forecasting.

• Understand and implement regularization techniques (Ridge, Lasso, Elastic Net) for financial data.

• Interpret feature coefficients and their economic significance.

2. The Bias-Variance Tradeoff – The Fundamental Tradeoff in Financial ML

The bias-variance tradeoff is the central tension in machine learning. It explains why simple models often outperform complex models on noisy financial data.

Mathematical Formulation:
For a model f trained on dataset D, the expected prediction error at point x_0 is:
E[ (y - f(x_0))² ] = Bias²(f(x_0)) + Var(f(x_0)) + σ²

Where:

• Bias²: Error from approximating a complex true function with a simple model. High bias = underfitting.

• Variance: Error from sensitivity to the training sample. High variance = overfitting.

• σ²: Irreducible error (noise in the data). This cannot be reduced by any model.

Financial Implication:
Financial data has extremely high noise (low signal-to-noise ratio). Therefore:

• Simple models (linear regression) are preferred over complex models (deep learning) unless you have massive datasets.

• Regularisation is mandatory to control variance.

• Cross-validation must be used to find the optimal bias-variance tradeoff.

3. Linear Regression – The Workhorse of Quantitative Finance

Linear regression models the relationship between a dependent variable y and one or more independent variables X.

3.1 Mathematical Formulation
y = Xβ + ε
Where:

• y is an (N x 1) vector of observations.

• X is an (N x p) matrix of features (including a column of ones for the intercept).

• β is a (p x 1) vector of coefficients.

• ε is an (N x 1) vector of errors, assumed to be ε ∼ N(0, σ² I).

3.2 Ordinary Least Squares (OLS) Derivation

Method 1: Calculus (Scalar Form)
Minimise the sum of squared errors:
SSE(β) = Σ_{i=1}^{N} (y_i - x_i^T β)²

Take the derivative with respect to β:
∂SSE/∂β = -2 Σ_{i=1}^{N} x_i (y_i - x_i^T β) = 0

This yields the normal equations:
Σ_{i=1}^{N} x_i y_i = Σ_{i=1}^{N} x_i x_i^T β

Method 2: Linear Algebra (Matrix Form)
SSE(β) = (y - Xβ)^T (y - Xβ) = y^T y - 2β^T X^T y + β^T X^T X β

Take the derivative:
∂SSE/∂β = -2 X^T y + 2 X^T X β = 0

This yields:
X^T X β = X^T y

Assuming X^T X is invertible (no perfect multicollinearity):
β_hat = (X^T X)^{-1} X^T y

3.3 Properties of OLS Estimators

• Unbiased: E[β_hat] = β (provided E[ε] = 0).

• Consistent: As N → ∞, β_hat → β (Law of Large Numbers).

• Efficient: OLS has the minimum variance among all linear unbiased estimators (Gauss-Markov Theorem).

• Variance-Covariance Matrix: Var(β_hat) = σ² (X^T X)^{-1}.

• Standard Errors: SE(β_j) = σ * sqrt( (X^T X)^{-1}_{jj} ).

• σ² Estimator: s² = SSE / (N - p) (unbiased estimator).

3.4 Financial Application – CAPM Beta Estimation
The Capital Asset Pricing Model (CAPM) states:
E[R_i] - R_f = β_i (E[R_m] - R_f)

The regression model is:
R_{i,t} - R_{f,t} = α_i + β_i (R_{m,t} - R_{f,t}) + ε_{i,t}

OLS gives:
β_i = Cov(R_i, R_m) / Var(R_m)
α_i = E[R_i] - β_i E[R_m] (Jensen’s alpha)

Implementation:

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

def compute_capm_beta(stock_returns, market_returns, risk_free=0.02/252):
    """
    Compute CAPM beta using OLS regression.
    """
    # Excess returns
    stock_excess = stock_returns - risk_free
    market_excess = market_returns - risk_free

    # OLS regression
    model = LinearRegression(fit_intercept=True)
    model.fit(market_excess.values.reshape(-1, 1), stock_excess.values)

    beta = model.coef_[0]
    alpha = model.intercept_

    # Annualise
    beta_annual = beta
    alpha_annual = alpha * 252

    return beta_annual, alpha_annual

3.5 Financial Application – Fama-French Three-Factor Model
The Fama-French model extends CAPM:
R_i - R_f = α_i + β_1 (R_m - R_f) + β_2 SMB + β_3 HML + ε_i

Where:

• SMB (Small Minus Big): Size factor.

• HML (High Minus Low): Value factor.

Implementation:

def fama_french_regression(stock_returns, market_returns, smb, hml, risk_free=0.02/252):
    """
    Estimate Fama-French three-factor model coefficients.
    """
    # Construct X matrix
    X = pd.DataFrame({
        'Mkt_RF': market_returns - risk_free,
        'SMB': smb,
        'HML': hml
    })

    # OLS regression
    model = LinearRegression(fit_intercept=True)
    model.fit(X, stock_returns - risk_free)

    beta_mkt = model.coef_[0]
    beta_smb = model.coef_[1]
    beta_hml = model.coef_[2]
    alpha = model.intercept_

    return {'alpha': alpha, 'beta_mkt': beta_mkt, 'beta_smb': beta_smb, 'beta_hml': beta_hml}

4. Logistic Regression – Classification for Financial Decisions

Logistic regression is used when the target variable is binary (e.g., default vs. no default, up vs. down).

4.1 The Logit Model
The probability of the positive class is:
P(y=1 | x) = p(x) = 1 / (1 + e^{-x^T β})

The log-odds (logit) transformation:
ln( p(x) / (1 - p(x)) ) = x^T β

4.2 Maximum Likelihood Estimation

The likelihood function:
L(β) = Π_{i=1}^{N} p(x_i)^{y_i} (1 - p(x_i))^{1 - y_i}

The log-likelihood:
ℓ(β) = Σ_{i=1}^{N} [ y_i ln(p(x_i)) + (1 - y_i) ln(1 - p(x_i)) ]

Substituting p(x) = 1 / (1 + e^{-x^T β}):
ℓ(β) = Σ_{i=1}^{N} [ y_i x_i^T β - ln(1 + e^{x_i^T β}) ]

4.3 Gradient and Hessian

Gradient:
∂ℓ/∂β = Σ_{i=1}^{N} x_i (y_i - p(x_i))

Hessian (for Newton-Raphson):
∂²ℓ/∂β∂β^T = - Σ_{i=1}^{N} p(x_i)(1 - p(x_i)) x_i x_i^T

Since p(x_i)(1 - p(x_i)) > 0, the Hessian is negative definite → the log-likelihood is concave, guaranteeing a unique global maximum.

4.4 Financial Application – Credit Default Prediction

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score, confusion_matrix

def credit_default_model(features, target):
    """
    Logistic regression for credit default prediction.
    """
    # Split data
    from sklearn.model_selection import train_test_split
    X_train, X_test, y_train, y_test = train_test_split(
        features, target, test_size=0.3, random_state=42
    )

    # Train logistic regression
    model = LogisticRegression(
        penalty='l2',
        C=1.0,
        class_weight='balanced',  # Handle imbalanced data
        solver='lbfgs',
        max_iter=1000,
        random_state=42
    )
    model.fit(X_train, y_train)

    # Predictions
    y_pred = model.predict(X_test)
    y_pred_proba = model.predict_proba(X_test)[:, 1]

    # Evaluation
    print(classification_report(y_test, y_pred))
    print(f"AUC: {roc_auc_score(y_test, y_pred_proba):.4f}")

    # Feature importance (coefficients)
    coef_df = pd.DataFrame({
        'feature': features.columns,
        'coefficient': model.coef_[0],
        'odds_ratio': np.exp(model.coef_[0])
    }).sort_values('coefficient', ascending=False)

    return model, coef_df

4.5 Financial Application – Market Direction Forecasting

def market_direction_model(features, target):
    """
    Predict market direction (up/down) using logistic regression.
    """
    from sklearn.preprocessing import StandardScaler
    from sklearn.model_selection import TimeSeriesSplit

    # Standardise features (important for logistic regression)
    scaler = StandardScaler()
    X_scaled = scaler.fit_transform(features)

    # Time series cross-validation
    tscv = TimeSeriesSplit(n_splits=5, test_size=60)

    accuracies = []
    aucs = []

    for train_idx, val_idx in tscv.split(X_scaled):
        X_train, X_val = X_scaled[train_idx], X_scaled[val_idx]
        y_train, y_val = target.iloc[train_idx], target.iloc[val_idx]

        # Train with balanced class weights
        model = LogisticRegression(
            penalty='l1',
            C=0.1,
            class_weight='balanced',
            solver='saga',
            max_iter=1000,
            random_state=42
        )
        model.fit(X_train, y_train)

        y_pred = model.predict(X_val)
        y_pred_proba = model.predict_proba(X_val)[:, 1]

        accuracies.append(np.mean(y_pred == y_val))
        aucs.append(roc_auc_score(y_val, y_pred_proba))

    print(f"Average Accuracy: {np.mean(accuracies):.4f}")
    print(f"Average AUC: {np.mean(aucs):.4f}")

    return model, scaler

5. Regularisation – Controlling Overfitting in Finance

Regularisation adds a penalty term to the loss function to shrink coefficients toward zero.

5.1 Ridge Regression (L2 Regularisation)
Minimises: Σ_{i=1}^{N} (y_i - x_i^T β)² + λ Σ_{j=1}^{p} β_j²

Closed-form solution:
β_ridge = (X^T X + λ I)^{-1} X^T y

Properties:

• Shrinks coefficients toward zero but never exactly zero.

• Handles multicollinearity well (makes X^T X invertible even when singular).

• Suitable when many features have small effects.

5.2 Lasso Regression (L1 Regularisation)
Minimises: Σ_{i=1}^{N} (y_i - x_i^T β)² + λ Σ_{j=1}^{p} |β_j|

Properties:

• Performs feature selection (coefficients can be exactly zero).

• Creates sparse models.

• Suitable when only a few features are important.

5.3 Elastic Net (Combination)
Minimises: Σ_{i=1}^{N} (y_i - x_i^T β)² + λ_1 Σ_{j=1}^{p} |β_j| + λ_2 Σ_{j=1}^{p} β_j²

Properties:

• Combines the strengths of Ridge and Lasso.

• Handles correlated features better than Lasso.

• Suitable for high-dimensional financial data.

5.4 Implementation

from sklearn.linear_model import Ridge, Lasso, ElasticNet
from sklearn.model_selection import GridSearchCV

def regularised_regression(X_train, y_train, X_val, y_val):
    """
    Compare Ridge, Lasso, and Elastic Net on financial data.
    """
    models = {
        'Ridge': Ridge(),
        'Lasso': Lasso(max_iter=10000),
        'ElasticNet': ElasticNet(max_iter=10000)
    }

    # Hyperparameter grids
    param_grids = {
        'Ridge': {'alpha': [0.001, 0.01, 0.1, 1, 10]},
        'Lasso': {'alpha': [0.001, 0.01, 0.1, 1, 10]},
        'ElasticNet': {
            'alpha': [0.001, 0.01, 0.1, 1],
            'l1_ratio': [0.2, 0.5, 0.8]
        }
    }

    results = {}

    for name, model in models.items():
        # Grid search with time series cross-validation
        gs = GridSearchCV(
            model,
            param_grids[name],
            cv=TimeSeriesSplit(n_splits=3, test_size=60),
            scoring='neg_mean_squared_error'
        )
        gs.fit(X_train, y_train)

        # Evaluate on validation set
        y_pred = gs.best_estimator_.predict(X_val)
        mse = np.mean((y_val - y_pred)**2)

        results[name] = {
            'best_params': gs.best_params_,
            'val_mse': mse,
            'model': gs.best_estimator_
        }

    return results

5.5 Interpreting Coefficients in Finance

• Economic Significance: A coefficient of β_j means a one-unit increase in feature j increases the predicted return by β_j units.

• Statistical Significance: Check the p-value (t-statistic). t_j = β_j / SE(β_j). If |t_j| > 2, the coefficient is statistically significant at 95% confidence.

• Regularised Coefficients: Shrunk toward zero. The amount of shrinkage depends on λ. Cross-validation chooses the λ that minimises validation error.

6. Model Evaluation Metrics for Financial Models

6.1 Regression Metrics

6.2 Classification Metrics

7. Summary for the AI Practitioner

1. Bias-Variance Tradeoff: Financial data is noisy → prefer simpler models with strong regularisation.

2. Linear Regression: OLS solution is β = (X^T X)^{-1} X^T y. Used for CAPM beta, Fama-French factors.

3. Logistic Regression: Models probabilities using the logit transformation. Used for credit default, market direction.

4. Regularisation is mandatory: Ridge (L2) shrinks coefficients; Lasso (L1) performs feature selection; Elastic Net combines both.

5. Cross-Validation: Use TimeSeriesSplit (not KFold) for financial data to prevent look-ahead bias.

6. Interpretability: Linear models are highly interpretable. Coefficients have economic meaning.

7. Metrics: Use RMSE/MAE for regression; AUC-ROC for classification (especially with imbalanced data).