1. Learning Objectives

By the end of this lesson, you will be able to:

  • Derive Maximum Likelihood Estimators (MLE) for financial models from first principles, including complete derivations for Normal, Student-t, and GARCH parameters.

  • Compute the Fisher Information matrix and Cramér-Rao Lower Bound for efficient estimators.

  • Apply Bayesian inference with conjugate priors to financial return data.

  • Implement Markov Chain Monte Carlo (MCMC) methods (Metropolis-Hastings, Gibbs Sampling) for posterior sampling.

  • Derive and apply the Expectation-Maximisation (EM) algorithm to latent variable models (regime-switching, mixture models).

  • Conduct hypothesis testing and compute confidence intervals for AI model parameters.

  • Understand the bootstrap and its applications in financial risk estimation.

2. Maximum Likelihood Estimation (MLE) – The Workhorse of Finance

MLE finds the parameter vector θ that maximises the likelihood of the observed data. It is asymptotically efficient (achieves the Cramér-Rao Lower Bound).

2.1 The Likelihood and Log-Likelihood
Given i.i.d. observations {x_1, ..., x_N} with density f(x; θ):
L(θ) = Π_{i=1}^{N} f(x_i; θ).
ℓ(θ) = ln L(θ) = Σ_{i=1}^{N} ln f(x_i; θ).
The MLE is \hat{θ}_MLE = argmax_θ ℓ(θ).

2.2 MLE for the Normal Distribution (Full Derivation)
Assume X_1, ..., X_N ∼ N(μ, σ²).
The log-likelihood is:
ℓ(μ, σ²) = -N/2 ln(2π) - N/2 ln(σ²) - (1/(2σ²)) Σ (x_i - μ)².

Step 1: MLE for μ:
∂ℓ/∂μ = (1/σ²) Σ (x_i - μ) = 0 → Σ x_i - N μ = 0 → \hat{μ} = (1/N) Σ x_i = \bar{x}.

Step 2: MLE for σ²:
Let ν = σ². ∂ℓ/∂ν = -N/(2ν) + (1/(2ν²)) Σ (x_i - μ)² = 0.
Substitute μ = \bar{x} and solve:
-N/(2ν) + (1/(2ν²)) Σ (x_i - \bar{x})² = 0 → Multiply by 2ν²:
-Nν + Σ (x_i - \bar{x})² = 0 → \hat{σ²} = (1/N) Σ (x_i - \bar{x})².
Note: This is biased (divides by N, not N-1). The unbiased estimator uses N-1.

2.3 MLE for the Student-t Distribution (Heavy Tails)
Assume X ∼ t_ν(μ, σ²). The log-likelihood involves the Gamma function Γ:
ℓ(μ, σ², ν) = N [ ln Γ((ν+1)/2) - ln Γ(ν/2) - 0.5 ln(π ν σ²) ] - ( (ν+1)/2 ) Σ ln( 1 + (x_i - μ)² / (ν σ²) ).
There is no closed-form solution. We solve numerically using gradient ascent (e.g., scipy.optimize.minimize). The MLE for ν (degrees of freedom) controls tail heaviness. If \hat{ν} < 4, the distribution has infinite kurtosis.

2.4 MLE for GARCH Parameters
For a GARCH(1,1) model: ε_t = σ_t z_t, z_t ∼ N(0,1), σ_t² = ω + α ε_{t-1}² + β σ_{t-1}².
The log-likelihood is:
ℓ(ω, α, β) = -0.5 Σ [ ln(2π) + ln(σ_t²) + ε_t² / σ_t² ].
There is no closed-form. We use numerical optimisation (e.g., BFGS) with constraints ω > 0, α ≥ 0, β ≥ 0, α + β < 1 (stationarity).

3. Fisher Information and the Cramér-Rao Lower Bound

The Fisher Information measures the amount of information that an observable random variable carries about an unknown parameter.

3.1 Definition
I(θ) = -E[ ∂²ℓ(θ) / ∂θ² ] = E[ (∂ℓ(θ) / ∂θ)² ].
The second equality holds under regularity conditions.

3.2 Cramér-Rao Lower Bound (CRLB)
For any unbiased estimator \hat{θ}, the variance is bounded below by the inverse Fisher Information:
Var(\hat{θ}) ≥ 1 / I(θ).
MLE achieves this bound asymptotically (for large N).

3.3 Example – Normal Distribution
For X ∼ N(μ, σ²) with known σ², the Fisher Information for μ is:
I(μ) = N / σ². Therefore, Var(\hat{μ}) ≥ σ² / N. This is exactly the variance of \bar{x}. MLE achieves the bound.

3.4 Financial Application – Standard Errors
The standard error of the MLE is SE(\hat{θ}) = sqrt( I(\hat{θ})^{-1} ).
For portfolio optimisation, the standard errors of estimated expected returns are σ / sqrt(N). If σ = 20% and N = 120 (10 years of monthly data), the SE is 1.8%. This means a 10% annualised return has a 95% CI of [6.4%, 13.6%].

4. Bayesian Inference – Updating Beliefs with Data

Bayesian statistics treats parameters as random variables with a prior distribution. The posterior is updated using Bayes’ theorem.

4.1 Bayes’ Theorem
p(θ | Data) = p(Data | θ) * p(θ) / p(Data).

  • p(θ) = Prior belief before seeing data.

  • p(Data | θ) = Likelihood (MLE term).

  • p(θ | Data) = Posterior distribution (updated belief).

  • p(Data) = Marginal likelihood (integrates out θ). This is often intractable, leading to MCMC methods.

4.2 Conjugate Priors
A prior is conjugate if the posterior is in the same distributional family as the prior. This allows analytical updates.

Example 1: Normal-Normal Conjugate
Assume X ∼ N(μ, σ²) with known σ². Prior: μ ∼ N(μ_0, τ_0²).
The posterior is:
μ | Data ∼ N( μ_n, τ_n² ), where:
μ_n = ( (μ_0 / τ_0²) + (N \bar{x} / σ²) ) / ( 1/τ_0² + N/σ² ).
τ_n² = 1 / ( 1/τ_0² + N/σ² ).
The posterior mean is a weighted average of the prior mean and the sample mean. As N → ∞, the data dominates and μ_n → \bar{x}.

Example 2: Normal-Inverse-Gamma for (μ, σ²)
Prior: μ | σ² ∼ N(μ_0, σ²/κ_0), σ² ∼ Inv-Gamma(α_0, β_0).
This is the conjugate prior for the full normal distribution.

4.3 Financial Application – Black-Litterman Model
The Black-Litterman model uses Bayesian updating to combine prior equilibrium returns (from the Capital Asset Pricing Model) with subjective views (from a portfolio manager).

  • Prior: μ ∼ N(π, τ Σ), where π is the equilibrium return vector.

  • Likelihood: Views ∼ N(P μ, Ω), where P is a view matrix.

  • Posterior: μ | Views ∼ N( μ_{BL}, Σ_{BL} ).
    This yields a posterior mean that shifts the prior returns toward the expressed views. This is a direct application of the Normal-Normal conjugate formula.

5. Markov Chain Monte Carlo (MCMC) – Sampling from the Posterior

When the posterior is not a known distribution (non-conjugate), we sample from it using MCMC.

5.1 Metropolis-Hastings Algorithm
Algorithm to sample from p(θ | Data):

  1. Initialise θ^{(0)}.

  2. For t = 1, 2, ..., T:
    a. Propose a new state θ^* from a proposal distribution q(θ^* | θ^{(t-1)}).
    b. Compute acceptance probability:
    α = min( 1, [ p(θ^* | Data) * q(θ^{(t-1)} | θ^*) ] / [ p(θ^{(t-1)} | Data) * q(θ^* | θ^{(t-1)}) ] ).
    c. Draw u ∼ Uniform(0,1). If u < α, accept θ^{(t)} = θ^*. Else, reject and set θ^{(t)} = θ^{(t-1)}}.

  3. Discard the first B samples (burn-in). The remaining samples approximate the posterior.

5.2 Gibbs Sampling
A special case of Metropolis-Hastings where we sample from full conditional distributions p(θ_i | θ_{-i}, Data).

  1. Initialise θ^{(0)}.

  2. For t = 1, 2, ..., T:
    a. Sample θ_1^{(t)} ∼ p(θ_1 | θ_2^{(t-1)}, ..., θ_K^{(t-1)}, Data).
    b. Sample θ_2^{(t)} ∼ p(θ_2 | θ_1^{(t)}, θ_3^{(t-1)}, ..., θ_K^{(t-1)}, Data).
    c. Continue for all K parameters.

  3. Burn-in the first B samples.

5.3 Financial Application – Stochastic Volatility Models
Stochastic Volatility (SV) models: r_t = σ_t ε_t, ln(σ_t²) = α + β ln(σ_{t-1}²) + η_t.
The posterior p(α, β, σ_1:T | r_1:T) is high-dimensional and non-standard. We use Gibbs sampling because conditional distributions are known:

  • p(α, β | σ_1:T, r_1:T) is Normal (conjugate).

  • p(σ_t | σ_{-t}, r_1:T) is approximated using Metropolis-Hastings.

5.4 AI Application – Bayesian Neural Networks
Instead of point estimates for weights, we place a prior p(w) ∼ N(0, σ² I) and approximate the posterior p(w | Data) using Variational Inference (which is a deterministic approximation to MCMC, using KL divergence minimisation). This gives uncertainty estimates for predictions, which is crucial for risk management.

6. The Expectation-Maximisation (EM) Algorithm

The EM algorithm finds MLEs when the data has latent (unobserved) variables. It alternates between the E-step (expectation of the log-likelihood) and the M-step (maximisation).

6.1 Setup
Let X be the observed data and Z be the latent variables. The complete data log-likelihood is ln p(X, Z | θ). The observed log-likelihood ln p(X | θ) is intractable because Z is unobserved.

6.2 The EM Algorithm

  1. Initialise θ^{(0)}.

  2. E-Step: Compute the conditional expectation of the complete log-likelihood given X and current parameters:
    Q(θ | θ^{(t)}) = E_{Z | X, θ^{(t)}}[ ln p(X, Z | θ) ].

  3. M-Step: Maximise Q with respect to θ:
    θ^{(t+1)} = argmax_θ Q(θ | θ^{(t)}).

  4. Repeat until convergence.

6.3 Financial Application – Regime-Switching Models
Assume returns are generated by two regimes: Bull (high mean, low variance) and Bear (low mean, high variance). The latent variable Z_t ∈ {Bull, Bear} indicates the regime at time t.

  • E-Step: Compute the probability that the market is in Bull at time t given observed returns and current parameters (this is the Kalman filter / forward-backward algorithm for Markov switching).

  • M-Step: Update the regime parameters (μ_Bull, σ_Bull², μ_Bear, σ_Bear²) and the transition matrix P by maximising the expected log-likelihood.
    The EM algorithm converges to a local maximum.

6.4 Financial Application – Missing Data Imputation
When returns are missing (survivorship bias, holidays), we treat the missing values as latent variables.

  • E-Step: Compute the expected values of the missing data given the observed data and current parameters (using the conditional distribution).

  • M-Step: Update the mean and covariance matrix using the completed data (observed + imputed expected values).
    This is the EM Imputation algorithm, which produces unbiased estimates when data is Missing At Random (MAR).

7. Bootstrap – Non-Parametric Inference

The bootstrap resamples the data to estimate the sampling distribution of a statistic without relying on asymptotic theory.

7.1 The Bootstrap Procedure

  1. Given sample X = {x_1, ..., x_N} of size N.

  2. For b = 1 to B (e.g., B = 10,000):
    a. Draw N observations with replacement from X. Call this X^*_b.
    b. Compute the statistic of interest θ^*_b = f(X^*_b) (e.g., Sharpe ratio, portfolio variance).

  3. The empirical distribution of {θ^*_1, ..., θ^*_B} approximates the sampling distribution of θ.

7.2 Bootstrap Confidence Intervals

  • Percentile Method: The 95% CI is the 2.5th and 97.5th percentiles of the bootstrap distribution.

  • Bias-Corrected Accelerated (BCa): Adjusts for bias and skewness.

7.3 Financial Application – Sharpe Ratio Confidence Intervals
The Sharpe Ratio is SR = \bar{r} / s. Its distribution is not normal for small samples. We use the bootstrap to construct accurate confidence intervals:

  1. Resample returns {r_1, ..., r_T} with replacement.

  2. Compute SR^* = \bar{r}^* / s^*.

  3. Repeat 10,000 times.

  4. Take the 5th and 95th percentiles.
    This is far more accurate than the asymptotic normal approximation SE(SR) = sqrt( (1 + 0.5 SR²) / T ).

7.4 Bootstrapping in AI – Model Stability
If you train a neural network on B bootstrap samples of the training data, you get B different models. The variance of their predictions across bootstrap samples estimates the model’s epistemic uncertainty (uncertainty due to limited data). This is the foundation of Bootstrap Aggregating (Bagging) and Random Forests.

8. Hypothesis Testing and Model Selection

8.1 Likelihood Ratio Test (LRT)
Compare a restricted model (null) to a full model (alternative).
Λ = -2 ( ℓ(θ_0) - ℓ(θ_1) ) ∼ χ²_{df}, where df = dim(θ_1) - dim(θ_0).
Under the null (restricted model is correct), the test statistic follows a Chi-square distribution.
Application: Testing whether adding a factor to a factor model improves fit (e.g., is the Value factor significant?).

8.2 Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC)
These penalise model complexity.
AIC = -2 ℓ(θ) + 2 k, where k is the number of parameters.
BIC = -2 ℓ(θ) + k ln(N).
Lower AIC/BIC indicates better model. BIC penalises complexity more severely than AIC. Used for model selection in ARIMA, GARCH, and factor models.

8.3 Cross-Validation – The AI Standard
The most robust method for model selection in AI is K-Fold Cross-Validation (or Time Series Split for finance). It estimates out-of-sample performance directly, avoiding asymptotic assumptions.

  • Time Series Split: Train on [t_0, t], validate on [t+1, t+H]. Move the window forward. This is the gold standard for financial AI.

9. Summary for the AI Practitioner

  1. MLE is the foundation of loss functions. MSE = Negative Log-Likelihood for Gaussian errors. Cross-Entropy = Negative Log-Likelihood for Bernoulli/Categorical.

  2. Fisher Information gives standard errors. SE(\hat{μ}) = σ / sqrt(N). This quantifies estimation risk.

  3. Bayesian Inference updates prior beliefs with data. The Black-Litterman model is a Bayesian application in portfolio optimisation.

  4. MCMC samples from intractable posteriors. Used for stochastic volatility and Bayesian neural networks.

  5. EM Algorithm handles latent variables. Used for regime-switching models and missing data imputation.

  6. Bootstrap gives non-parametric confidence intervals. Essential for Sharpe ratio inference and AI model uncertainty quantification.

  7. AIC/BIC and Cross-Validation are model selection tools. Time Series Split is mandatory for financial backtesting.

In Lesson 2.5, we will cover Optimisation Theory – Convexity, Gradient Descent variants, Stochastic Gradient Descent, Adam, and Lagrange Multipliers for constrained optimisation.

 
 
 
PreviousNext