LESSON 8: MAXIMUM LIKELIHOOD ESTIMATION (MLE) AND BAYESIAN STATISTICS FOR FINANCIAL MODELLING

SECTION 1: LEARNING OBJECTIVES

By the end of this lesson, you will be able to:

Distinguish between frequentist and Bayesian philosophies in statistical inference.
Derive and apply the Maximum Likelihood Estimation (MLE) method to estimate parameters of financial models.
Compute MLE estimates analytically for Normal and Bernoulli (default) processes.
Understand the concept of a likelihood function and its role in model calibration.
Apply Bayes’ Theorem to update prior beliefs with observed financial data.
Work with conjugate priors (Beta-Binomial, Normal-Normal) to derive closed-form posterior distributions.
Implement MLE and Bayesian updating in Python using scipy.optimize and scipy.stats.
Apply Bayesian methods to practical problems: credit default probability estimation, volatility updating, and the Black-Litterman asset allocation model.
Evaluate the trade-offs between MLE and Bayesian approaches in regulatory and business contexts.

SECTION 2: THE TWO PARADIGMS OF STATISTICAL INFERENCE

Before diving into formulas, we must understand the philosophical divide that shapes all financial modelling.

Aspect	Frequentist (Classical)	Bayesian
View of Parameters	Parameters are fixed, unknown constants. They do not have a distribution.	Parameters are random variables with a probability distribution that reflects our uncertainty.
Data	Data are random. We imagine repeated sampling from the population.	Data are fixed (we observed them). The parameter distribution is updated given the data.
Estimation	MLE, Method of Moments. Produces a point estimate and confidence intervals.	Posterior distribution. Produces a full distribution of plausible parameter values.
Interpretation of 95% Interval	“If we repeated the experiment 100 times, 95 of the confidence intervals would contain the true parameter.”	“There is a 95% probability that the parameter lies in this interval given the observed data.”
Prior Information	Not used. All inference comes from the data.	Explicitly incorporated via a prior distribution.

Why this matters in finance:

Frequentist approaches dominate regulatory reporting (e.g., Basel standardized approaches) and classical risk models.
Bayesian methods are increasingly used for dynamic risk updating, stress testing, and portfolio optimisation (e.g., Black-Litterman) because they allow you to combine subjective views (from experts) with empirical data.

SECTION 3: MAXIMUM LIKELIHOOD ESTIMATION (MLE) – THE WORKHORSE OF FREQUENTIST INFERENCE

MLE is the most widely used parameter estimation method in finance. It asks: “Given this data, which parameter values make the observed data most probable?”

3.1 The Likelihood Function

Suppose we have independent and identically distributed (i.i.d.) observations $x_{1}, x_{2}, \dots, x_{n}$ drawn from a probability distribution with parameter(s) $θ$ . The probability density (or mass) function is $f (x_{i} ∣ θ)$ .

The likelihood function $L (θ)$ is the joint probability of observing the entire sample, viewed as a function of $θ$ :

$L (θ) = i = 1 \prod n f (x_{i} ∣ θ)$

Because products of small numbers can be unstable, we work with the log-likelihood:

$ℓ (θ) = log L (θ) = i = 1 \sum n log f (x_{i} ∣ θ)$

The Maximum Likelihood Estimator (MLE) $θ^MLE$ is the value of $θ$ that maximises $ℓ (θ)$ .

3.2 MLE for the Normal Distribution (Financial Returns)

Assume a set of daily returns $r_{1}, r_{2}, \dots, r_{n}$ are i.i.d. Normal $N (μ, σ^{2})$ . The pdf is:

$f (r_{i} ∣ μ, σ^{2}) = 2 π σ ^{2} 1 exp (- 2 σ ^{2} ( r ^{i} - μ ) ^{2})$

The log-likelihood is:

$ℓ (μ, σ^{2}) = - 2 n log (2 π) - 2 n log (σ^{2}) - 2 σ ^{2} 1 i = 1 \sum n (r_{i} - μ)^{2}$

Derivatives and solutions:

For $μ$ : Set $\partial μ \partialℓ = 0$

$∂ℓ∂μ=1σ2∑i=1n(ri−μ)=0⇒μ^MLE=1n∑i=1nri=rˉ$
For $σ^{2}$ : Set $\partial σ ^{2} \partialℓ = 0$

$σ^MLE2=1n∑i=1n(ri−rˉ)2$

Note: The MLE estimate of variance is biased (it uses $n$ in the denominator). For unbiased estimation, we use $n - 1$ (sample variance). However, MLE is asymptotically unbiased and efficient.

3.3 MLE for the Bernoulli (Default) Process

When modelling loan defaults, each borrower either defaults (1) or does not (0). This is a Bernoulli process with success probability $p$ .

The pmf is: $f (x_{i} ∣ p) = p^{x i} (1 - p)^{1 - x i}$

The likelihood for $n$ observations is:

$L (p) = p^{\sum x i} (1 - p)^{n - \sum x i}$

The log-likelihood:

$ℓ (p) = \sum x_{i} log (p) + (n - \sum x_{i}) log (1 - p)$

Taking the derivative w.r.t $p$ :

$d p d ℓ = p \sum x ^{i} - 1 - p n - \sum x ^{i} = 0$

Solving gives the intuitive result:

$p^MLE=1n∑i=1nxi=default rate$

Business application: A bank calculates its historical default rate as 2% using MLE. This is the point estimate used in regulatory capital calculations (under the IRB approach, adjusted for downturn conditions).

SECTION 4: BAYESIAN STATISTICS – UPDATING BELIEFS WITH DATA

Bayesian inference starts with a prior distribution $P (θ)$ representing our beliefs before seeing data. After observing data $D$ , we update to the posterior distribution $P (θ ∣ D)$ using Bayes’ Theorem:

$P (θ ∣ D) = P ( D ) P ( D ∣ θ ) \cdot P ( θ )$

$P (D ∣ θ)$ is the likelihood (same as in MLE).
$P (θ)$ is the prior.
$P (D) = \int P (D ∣ θ) P (θ) d θ$ is the marginal likelihood (a normalising constant).

In practice, we often ignore $P (D)$ and write:

$P (θ ∣ D) \propto P (D ∣ θ) \cdot P (θ)$

4.1 Conjugate Priors – Analytical Elegance

A prior is conjugate to the likelihood if the posterior belongs to the same family as the prior. This avoids complex numerical integration.

Likelihood	Conjugate Prior	Posterior	Financial Use
Bernoulli (defaults)	Beta distribution	Beta	Updating default probabilities
Normal (known variance)	Normal	Normal	Updating expected returns
Normal (unknown mean & variance)	Normal-Inverse-Gamma	Normal-Inverse-Gamma	Updating volatility

4.2 Beta-Binomial Model – Updating Default Probabilities

Scenario: A bank’s credit risk team has a prior belief that the 1‑year default probability for a certain corporate segment is around 2%. They observe 5 defaults out of 200 new loans. How should they update their belief?

Step 1 – Prior:
Choose a Beta distribution for $p$ : $p \sim Beta (α, β)$
The mean is $α + β α$ . If we want a prior mean of 0.02 and want the prior to represent the equivalent of 50 observations, we set $α + β = 50$ and $α + β α = 0.02$ .
Thus $α = 1$ , $β = 49$ .

Step 2 – Likelihood:
Observe $x = 5$ defaults in $n = 200$ loans.
$P (x ∣ p) \propto p^{5} (1 - p)^{195}$

Step 3 – Posterior:
Because Beta is conjugate to Bernoulli, the posterior is:

$p ∣ x \sim Beta (α + x, β + n - x) = Beta (1 + 5, 49 + 200 - 5) = Beta (6, 244)$

Posterior mean: $6 + 244 6 = 250 6 = 0.024$ (2.4%)

Interpretation: The observed default rate (2.5%) shifted the prior from 2.0% to 2.4%. The posterior mean is a weighted average of the prior mean and the sample mean, weighted by $α + β$ and $n$ respectively.

$Posterior Mean = α + β + n α + x = (α + β + n α + β) \cdot α + β α + (α + β + n n) \cdot n x$

Regulatory relevance: Under IFRS 9 / CECL, expected credit losses must incorporate forward-looking information. Bayesian updating provides a rigorous framework to combine historical default experience with expert macroeconomic forecasts.

4.3 Normal-Normal Model – Updating Expected Returns

Scenario: You estimate the mean return $μ$ of a stock. Your prior: $μ \sim N (μ_{0}, τ_{02})$ . You observe a sample of returns with sample mean $x ˉ$ and known variance $σ^{2}$ (or estimated).

The posterior distribution is:

$μ ∣ x ˉ \sim N (μ_{n}, τ_{n 2})$

where:

$μ_{n} = τ ^{02} 1 + σ ^{2} n τ ^{02} μ ^{0} + σ ^{2} n x ˉ and τ_{n 2} = τ ^{02} 1 + σ ^{2} n 1$

This is a precision-weighted average (precision = 1/variance). More precise information (lower variance) gets more weight.

Business application: Suppose you have a prior expected return of 8% for an equity fund (with uncertainty τ₀ = 4%). After observing a strong quarter with sample mean 12% (n=60, σ=15%), the posterior mean becomes:

Prior precision: 1/16 = 0.0625
Data precision: 60 / 225 = 0.2667
Posterior mean = (0.0625×8% + 0.2667×12%) / (0.0625+0.2667) ≈ 11.3%

Your belief adjusts toward the data, but not fully, because the prior has some weight.

SECTION 5: MAXIMUM A POSTERIORI (MAP) ESTIMATION

The MAP estimator finds the mode of the posterior distribution:

$θ^MAP=arg⁡max⁡θ P(θ∣D)$

Since the marginal likelihood is constant w.r.t $θ$ :

$θ^MAP=arg⁡max⁡θ L(θ)⋅P(θ)$

Taking logs:

$θ^MAP=arg⁡max⁡θ [ℓ(θ)+log⁡P(θ)]$

Key observation: MAP is like MLE with an additional penalty term from the prior. This is equivalent to regularisation in machine learning (L2 regularisation corresponds to a Gaussian prior; L1 corresponds to a Laplace prior).

Financial use: When you have limited data, MAP shrinks extreme estimates towards the prior, reducing overfitting – crucial in credit scoring when developing models with a small number of defaults.

SECTION 6: IMPLEMENTATION IN PYTHON – MLE AND BAYESIAN UPDATING

Let’s build a comprehensive example: estimating the volatility (standard deviation) of a stock return series using MLE and then updating a Bayesian model.

# ===================================================================
# LESSON 8: MLE AND BAYESIAN INFERENCE IN FINANCE
# ===================================================================

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
from scipy.optimize import minimize
from scipy.stats import norm, beta, bernoulli

# Set style
sns.set_style("whitegrid")
np.random.seed(42)

# ----------------------------------------------------------------
# PART A: MLE FOR NORMAL DISTRIBUTION (ESTIMATING VOLATILITY)
# ----------------------------------------------------------------

print("="*70)
print("PART A: MLE ESTIMATION OF MEAN AND VOLATILITY")
print("="*70)

# Generate synthetic daily returns for a stock (250 days)
true_mu = 0.001        # 0.1% daily mean
true_sigma = 0.02      # 2% daily volatility
n_days = 500

returns = np.random.normal(true_mu, true_sigma, n_days)

# Analytical MLE solutions
mu_mle = np.mean(returns)
sigma2_mle = np.var(returns, ddof=0)   # MLE uses n, not n-1
sigma_mle = np.sqrt(sigma2_mle)

print(f"\nTrue Parameters: μ = {true_mu:.4f}, σ = {true_sigma:.4f}")
print(f"MLE Estimates (n={n_days}):")
print(f"  μ̂ = {mu_mle:.6f}")
print(f"  σ̂ = {sigma_mle:.6f}")

# Numerical MLE using scipy.optimize (to show general approach)
def neg_log_likelihood_normal(params, data):
    """Negative log-likelihood for Normal distribution."""
    mu, sigma = params
    if sigma <= 0:
        return 1e10
    n = len(data)
    ll = -0.5 * n * np.log(2 * np.pi) - n * np.log(sigma) - np.sum((data - mu)**2) / (2 * sigma**2)
    return -ll   # we minimize negative log-likelihood

# Initial guess
init_params = [0, 0.01]
result = minimize(neg_log_likelihood_normal, init_params, args=(returns,), method='L-BFGS-B')
mu_num, sigma_num = result.x

print(f"\nNumerical MLE (optimization): μ̂ = {mu_num:.6f}, σ̂ = {sigma_num:.6f}")

# ----------------------------------------------------------------
# PART B: BAYESIAN UPDATING FOR DEFAULT PROBABILITY (Beta-Binomial)
# ----------------------------------------------------------------

print("\n" + "="*70)
print("PART B: BAYESIAN UPDATING – BETA-BINOMIAL")
print("="*70)

# Prior: Beta(α=2, β=98) -> mean 2%, equivalent prior strength = 100
alpha_prior = 2
beta_prior = 98
prior_mean = alpha_prior / (alpha_prior + beta_prior)
print(f"\nPrior: Beta(α={alpha_prior}, β={beta_prior})")
print(f"  Prior Mean = {prior_mean:.4f} ({prior_mean*100:.2f}%)")
print(f"  Prior Strength (α+β) = {alpha_prior + beta_prior}")

# New data: observe defaults in a portfolio
n_obs = 200
x_defaults = 5
observed_rate = x_defaults / n_obs
print(f"\nObserved Data: {x_defaults} defaults out of {n_obs} loans")
print(f"  Sample Default Rate = {observed_rate:.4f} ({observed_rate*100:.2f}%)")

# Posterior: Beta(α + x, β + n - x)
alpha_post = alpha_prior + x_defaults
beta_post = beta_prior + n_obs - x_defaults
post_mean = alpha_post / (alpha_post + beta_post)

print(f"\nPosterior: Beta(α={alpha_post}, β={beta_post})")
print(f"  Posterior Mean = {post_mean:.4f} ({post_mean*100:.2f}%)")

# Credible interval (95% Bayesian credible interval)
lower_bound = beta.ppf(0.025, alpha_post, beta_post)
upper_bound = beta.ppf(0.975, alpha_post, beta_post)
print(f"  95% Credible Interval: [{lower_bound:.4f}, {upper_bound:.4f}]")

# Compare with MLE (which would just be observed_rate)
print(f"\nComparison: MLE point estimate = {observed_rate:.4f}")
print(f"  Bayesian posterior mean = {post_mean:.4f} (shrunk toward prior)")

# Visualize prior, likelihood, and posterior
fig, ax = plt.subplots(figsize=(12, 6))

p_values = np.linspace(0, 0.08, 500)
prior_pdf = beta.pdf(p_values, alpha_prior, beta_prior)
likelihood = beta.pdf(p_values, alpha_prior + x_defaults, beta_prior + n_obs - x_defaults - (alpha_prior+beta_prior)) # scaled approx
# Actually, likelihood is proportional to p^x (1-p)^(n-x). Let's plot it unscaled for visual.
likelihood_unscaled = p_values**x_defaults * (1-p_values)**(n_obs - x_defaults)
likelihood_scaled = likelihood_unscaled / np.max(likelihood_unscaled) * np.max(prior_pdf)  # scale for visualization

posterior_pdf = beta.pdf(p_values, alpha_post, beta_post)

ax.plot(p_values, prior_pdf, 'b--', linewidth=2, label='Prior Beta(2,98)')
ax.plot(p_values, likelihood_scaled, 'g-.', linewidth=2, label='Likelihood (scaled)')
ax.plot(p_values, posterior_pdf, 'r-', linewidth=2, label='Posterior Beta(7,243)')

ax.axvline(prior_mean, color='blue', linestyle=':', alpha=0.7, label=f'Prior Mean: {prior_mean:.3f}')
ax.axvline(observed_rate, color='green', linestyle=':', alpha=0.7, label=f'Data Rate: {observed_rate:.3f}')
ax.axvline(post_mean, color='red', linestyle=':', alpha=0.7, label=f'Posterior Mean: {post_mean:.3f}')

ax.fill_between(p_values, 0, posterior_pdf, where=(p_values>lower_bound)&(p_values<upper_bound), 
                color='red', alpha=0.2, label='95% Credible Interval')

ax.set_title('Bayesian Updating of Default Probability', fontsize=14)
ax.set_xlabel('Default Probability p')
ax.set_ylabel('Density / Scaled Likelihood')
ax.legend()
ax.grid(True, alpha=0.3)

plt.tight_layout()
plt.savefig('bayesian_beta_binomial.png', dpi=300)
plt.show()

# ----------------------------------------------------------------
# PART C: BAYESIAN UPDATING FOR MEAN RETURN (Normal-Normal)
# ----------------------------------------------------------------

print("\n" + "="*70)
print("PART C: BAYESIAN UPDATING FOR MEAN RETURN (NORMAL-NORMAL)")
print("="*70)

# Prior: expected return μ ~ N(μ0=0.08, τ0=0.04)   (8% annual, uncertainty 4%)
mu0 = 0.08
tau0 = 0.04
prior_precision = 1 / tau0**2

# New data: sample of monthly returns
n_months = 24
sample_mean = 0.12   # 12% average over 24 months
sample_std = 0.15    # 15% annualized volatility (assume known sigma)

# Posterior parameters (known sigma, i.e., sample variance known)
sigma = sample_std
data_precision = n_months / sigma**2

# Posterior mean and variance
post_precision = prior_precision + data_precision
post_var = 1 / post_precision
post_sd = np.sqrt(post_var)
post_mean = (prior_precision * mu0 + data_precision * sample_mean) / post_precision

print(f"\nPrior: μ ~ N({mu0:.3f}, {tau0:.3f}^2)")
print(f"Prior Precision: {prior_precision:.2f}")
print(f"\nData: n={n_months}, sample mean={sample_mean:.3f}, σ={sigma:.3f}")
print(f"Data Precision: {data_precision:.2f}")
print(f"\nPosterior: μ ~ N({post_mean:.4f}, {post_sd:.4f}^2)")
print(f"Posterior Mean: {post_mean:.4f} ({post_mean*100:.2f}%)")

# Credible interval
ci_lower = post_mean - 1.96 * post_sd
ci_upper = post_mean + 1.96 * post_sd
print(f"95% Credible Interval: [{ci_lower:.4f}, {ci_upper:.4f}]")

# Weights analysis
weight_prior = prior_precision / (prior_precision + data_precision)
weight_data = data_precision / (prior_precision + data_precision)
print(f"\nWeight on Prior: {weight_prior:.2f}")
print(f"Weight on Data:  {weight_data:.2f}")

# ----------------------------------------------------------------
# PART D: MLE vs BAYESIAN – FINANCIAL DECISION
# ----------------------------------------------------------------

print("\n" + "="*70)
print("PART D: MLE vs BAYESIAN – BUSINESS INTERPRETATION")
print("="*70)

# MLE estimate for mean = sample mean
mle_mean = sample_mean

print(f"\nMLE (Frequentist) Estimate: {mle_mean:.4f} ({mle_mean*100:.2f}%)")
print(f"Bayesian Posterior Mean:   {post_mean:.4f} ({post_mean*100:.2f}%)")
print(f"Difference: {(post_mean - mle_mean)*100:.2f} percentage points")

print("\nInterpretation:")
print("  - MLE gives the sample mean (12%) – simple and objective.")
print("  - Bayesian gives 10.61% – a shrinkage estimator that pulls the extreme")
print("    sample mean toward the prior belief of 8%, reflecting that")
print("    historical returns may be noisy.")
print("\nBusiness Decision:")
print("  If allocating capital, the Bayesian approach is more conservative")
print("  and less likely to over-react to a short-term streak of high returns.")

SECTION 7: ADVANCED TOPIC – THE BLACK‑LITTERMAN MODEL (A BAYESIAN APPROACH TO ASSET ALLOCATION)

The Black‑Litterman model, developed at Goldman Sachs, is a practical application of Bayesian statistics that combines:

Prior: Equilibrium market returns (implied by the Capital Asset Pricing Model).
Views: An investor’s subjective views about the expected returns of certain assets (expressed with uncertainty).

The model uses a Normal-Normal conjugate framework to produce posterior expected returns, which then drive optimal portfolio weights.

Why it matters:

Overcomes the extreme sensitivity of Markowitz mean‑variance optimisation.
Provides a systematic way to include expert judgment in portfolio construction.

Mathematical framework (simplified):

Prior: $μ \sim N (π, τ Σ)$ where $π$ are market‑implied returns and $τ Σ$ is the covariance matrix scaled by uncertainty.
Views: $P μ \sim N (v, Ω)$ , where $P$ is a matrix selecting assets, $v$ are the view returns, and $Ω$ is the view uncertainty.

The posterior mean is:

$μ_{B L} = [(τ Σ)^{-1} + P^{T} Ω^{-1} P]^{-1} [(τ Σ)^{-1} π + P^{T} Ω^{-1} v]$

This is a precision‑weighted average of prior and views.

SECTION 8: CHALLENGES AND BEST PRACTICES

Challenges with MLE in finance:

Small samples: MLE is asymptotically efficient, but in practice, we often have limited data (e.g., only 3‑5 years of credit defaults). This leads to high variance.
Misspecification: If you choose the wrong distribution (e.g., Normal when tails are fat), MLE can be severely biased.

Challenges with Bayesian methods:

Subjectivity: The choice of prior can heavily influence results, especially with limited data. Regulators may be sceptical of “black box” priors.
Computational cost: For complex models, the posterior may not have a closed form (requiring MCMC or variational inference). However, conjugate priors avoid this.

Best practices in banking:

Use weakly informative priors when no strong prior exists (e.g., a Beta(1,1) which is uniform).
Always perform sensitivity analysis – test how the posterior changes with different priors.
Document the rationale for the prior (e.g., based on historical decade‑long averages, industry benchmarks, or expert committee consensus).
For regulatory models (IFRS 9, CECL, CCAR), banks often adopt a hybrid approach: MLE for point estimates, but Bayesian updating for scenario‑based forward‑looking adjustments.

SECTION 9: SUMMARY FOR THE DATA PRACTITIONER

MLE is the gold standard for finding parameters that best explain observed financial data. It is objective and computationally simple in many cases (Normal, Bernoulli).
MLE equations for Normal: $μ^=xˉ$ , $σ^2=1n∑(xi−xˉ)2$ .
Bayesian inference updates prior beliefs using Bayes’ Theorem, resulting in a posterior distribution.
Conjugate priors (Beta-Binomial, Normal-Normal) provide closed‑form solutions, avoiding expensive simulation.
MAP is a point estimate that combines prior and likelihood, equivalent to regularised MLE.
In banking, Bayesian methods are used for default probability estimation, expected return forecasting, and portfolio optimisation (Black‑Litterman).
Always weigh the trade‑offs: MLE is simpler and regulator‑friendly for standardised approaches; Bayesian offers flexibility and dynamic updating, but requires careful prior justification.

SECTION 10: RECOMMENDED NEXT STEPS

Practice deriving MLE for a Gamma distribution (used for modelling operational loss severities).
Implement a Bayesian linear regression model with conjugate priors for a credit scoring dataset.
Explore PyMC or Stan for Bayesian modelling when conjugacy is not available.
Study the Black‑Litterman model in detail and apply it to a small portfolio of ETFs.
Prepare for the next module on Machine Learning for Finance, where regularisation (L1/L2) will be framed as MAP estimation with Laplace/Gaussian priors.

[END OF LESSON 8]

Diploma in Financial Data Analytics LESSON 8: MAXIMUM LIKELIHOOD ESTIMATION (MLE) AND BAYESIAN STATISTICS FOR FINANCIAL MODELLING

Diploma in Financial Data Analytics