LESSON 1: VALUE AT RISK (VaR) AND EXPECTED SHORTFALL (ES) – HISTORICAL SIMULATION AND PARAMETRIC APPROACHES

SECTION 1: LEARNING OBJECTIVES

By the end of this lesson, you will be able to:

Define Value at Risk (VaR) and Expected Shortfall (ES) – the two most widely used risk measures in banking and finance.
Distinguish between the three main VaR methodologies: Historical Simulation, Parametric (Variance-Covariance), and Monte Carlo Simulation.
Implement Historical Simulation VaR using actual historical returns – a non‑parametric, regulator‑accepted approach.
Implement Parametric (Normal) VaR using mean and standard deviation – the simplest and fastest method.
Understand the limitations of VaR – its failure to capture tail risk and lack of sub‑additivity.
Compute Expected Shortfall (ES) – also known as Conditional VaR (CVaR) – which addresses VaR’s shortcomings.
Backtest VaR models using Kupiec’s Proportion of Failures (POF) test and Christoffersen’s conditional coverage test.
Apply these methods to a financial portfolio and interpret results for risk management and regulatory reporting (Basel III, FRTB).
Use Python to compute VaR and ES on real or simulated asset returns, including portfolio aggregation.

SECTION 2: WHAT IS VALUE AT RISK (VaR)?

Value at Risk (VaR) is a statistical measure that quantifies the maximum potential loss of a portfolio over a given time horizon at a given confidence level.

Formal definition:
For a portfolio with value $V_{t}$ , the $1 - α$ VaR is the loss that will not be exceeded with probability $1 - α$ over a specified horizon.

Mathematically, if $L$ is the loss distribution (positive for losses), then:

$P (L \leq VaR_{α}) = 1 - α$

or equivalently, the $α$ -quantile of the loss distribution.

Common conventions in banking:

Confidence levels: 95% (internal risk limits) or 99% (regulatory capital).
Holding period: 1-day (trading desks), 10-day (Basel market risk), or 1-year (credit risk).
For regulatory capital (Basel III), the 99% 10-day VaR is used.

Example: A 1‑day 95% VaR of $10 million means that under normal market conditions, there is a 95% chance that the portfolio will not lose more than $10 million in one day. Equivalently, there is a 5% chance of losing more than $10 million.

SECTION 3: THREE APPROACHES TO COMPUTING VAR

Method	Description	Pros	Cons
Historical Simulation	Use actual historical returns; order them; take the appropriate percentile.	Non‑parametric; does not assume distribution; captures fat tails.	Assumes history repeats; limited by data length; no scenario analysis.
Parametric (Variance-Covariance)	Assume returns follow a normal (or t-) distribution; estimate mean and variance; compute quantile.	Fast; easy to implement; analytical formula.	Underestimates tail risk; relies on normality assumption.
Monte Carlo Simulation	Simulate many future scenarios using a stochastic process; compute quantile from simulated distribution.	Flexible; can incorporate any distribution and complex instruments.	Computationally intensive; model risk; requires calibration.

Regulatory preference:

Basel II/III allows all three, but Historical Simulation is widely used for market risk (due to its simplicity and non‑parametric nature).
For internal models, banks often use Monte Carlo with variance‑covariance for more complex portfolios.
For credit risk, parametric models (e.g., CreditMetrics) are common.

SECTION 4: HISTORICAL SIMULATION VAR – STEP-BY-STEP

Collect historical returns over a lookback period (e.g., 250 trading days for 1‑year).
Sort the returns from worst to best (losses as negative returns).
Select the percentile corresponding to the confidence level.
For 95% VaR, take the 5th percentile of the sorted returns (i.e., the return that is worse than 95% of the observations).
Multiply the portfolio value by this return to get the dollar loss.

Advantages:

No distributional assumptions.
Captures fat tails and skewness.
Intuitive and easy to explain to stakeholders.

Disadvantages:

The VaR estimate depends heavily on the chosen historical period.
Cannot extrapolate beyond historical extremes.
Sensitivity to outliers and periods of high volatility.

SECTION 5: PARAMETRIC (NORMAL) VAR

If we assume daily returns $R$ are normally distributed with mean $μ$ and standard deviation $σ$ , the 1‑day VaR at confidence level $1 - α$ is:

$VaR_{1 - α} = - (μ + z_{α} \cdot σ) \cdot V_{0}$

where $z_{α}$ is the $α$ -quantile of the standard normal distribution (e.g., -1.645 for 95% confidence, -2.326 for 99%).

For a portfolio with weights $w$ , the portfolio variance is $σ_{p 2} = w^{T} Σ w$ , where $Σ$ is the covariance matrix. Then:

$VaR_{1 - α p or t f o l i o} = - (μ_{p} + z_{α} σ_{p}) \cdot V_{0}$

Adjustments for longer horizons:
If returns are i.i.d., VaR scales by the square root of time:
$VaR_{T} = VaR_{1} \times T$ .

Limitations:

Assumes normality – financial returns have fat tails, so normal VaR underestimates risk.
Does not capture autocorrelation or volatility clustering.

SECTION 6: EXPECTED SHORTFALL (ES) – THE COHERENT RISK MEASURE

Expected Shortfall (ES) is the average loss given that the loss exceeds the VaR threshold. It is defined as:

$E S_{α} = E [L ∣ L > VaR_{α}]$

For a normal distribution, ES has a closed‑form formula:

$E S_{α} = - μ + σ \cdot α ϕ ( z ^{α} )$

where $ϕ$ is the standard normal pdf and $z_{α}$ is the $α$ -quantile.

Why ES is preferred by regulators:

Coherent risk measure – it satisfies sub‑additivity (risk of portfolio ≤ sum of risks), unlike VaR.
Captures tail risk – averages losses beyond the VaR threshold.
Basel III/FRTB has replaced VaR with Expected Shortfall for market risk capital calculations (since 2016).

SECTION 7: BACKTESTING VAR – KUPIEC’S POF TEST

Backtesting verifies whether the number of exceptions (days when actual loss exceeds VaR) is consistent with the confidence level.

Kupiec’s Proportion of Failures (POF) test:

Let $N$ be the number of days the loss exceeds VaR out of $T$ days.
Under the null hypothesis that the model is correct, $N \sim Binomial (T, α)$ .
The likelihood ratio test statistic:

$L R = - 2 ln (( 1 - T N ) ^{T - N} ( T N ) ^{N} ( 1 - α ) ^{T - N} α ^{N}) \sim χ^{2} (1)$

If $L R$ exceeds the critical value (e.g., 3.84 for 95% confidence), reject the model.

Christoffersen’s conditional coverage test further checks independence of exceptions (clustering).

SECTION 8: IMPLEMENTATION IN PYTHON – VAR AND ES

We will implement both Historical Simulation and Parametric VaR/ES on a portfolio of two assets (e.g., stocks) using real or simulated returns.

# ===================================================================
# MODULE 5, LESSON 1: VALUE AT RISK AND EXPECTED SHORTFALL
# ===================================================================

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.stats import norm, t
from sklearn.covariance import LedoitWolf  # for robust covariance
import warnings
warnings.filterwarnings('ignore')

# Set style and reproducibility
np.random.seed(42)
sns.set_style("whitegrid")

print("="*70)
print("VALUE AT RISK (VaR) AND EXPECTED SHORTFALL (ES)")
print("="*70)

# ----------------------------------------------------------------
# PART A: GENERATE SYNTHETIC DAILY RETURNS FOR TWO ASSETS
# ----------------------------------------------------------------

# Parameters
n_days = 500
mu1, sigma1 = 0.0005, 0.02   # Asset 1: mean return 0.05%, vol 2%
mu2, sigma2 = 0.0008, 0.03   # Asset 2: mean return 0.08%, vol 3%
rho = 0.4                     # correlation between assets

# Generate correlated returns
cov_matrix = np.array([[sigma1**2, rho*sigma1*sigma2],
                       [rho*sigma1*sigma2, sigma2**2]])
returns = np.random.multivariate_normal([mu1, mu2], cov_matrix, n_days)

# Create DataFrame
df_returns = pd.DataFrame(returns, columns=['Asset1', 'Asset2'])
df_returns['Date'] = pd.date_range(start='2020-01-01', periods=n_days, freq='B')
df_returns.set_index('Date', inplace=True)

# Portfolio weights (equal-weighted)
weights = np.array([0.5, 0.5])
df_returns['Portfolio'] = df_returns.dot(weights)

print("Returns Summary Statistics:")
print(df_returns.describe().round(6))

# ----------------------------------------------------------------
# PART B: HISTORICAL SIMULATION VAR AND ES
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART B: HISTORICAL SIMULATION")
print("-"*60)

# Portfolio returns sorted
port_returns = df_returns['Portfolio'].values
sorted_returns = np.sort(port_returns)

# Confidence levels
confidence_levels = [0.95, 0.99]

# Function to compute historical VaR and ES
def historical_var_es(returns, confidence):
    """
    Historical Simulation VaR and ES.
    Returns: VaR (positive loss) and ES (positive loss).
    """
    alpha = 1 - confidence
    var_idx = int(alpha * len(returns))
    var = -sorted_returns[var_idx]  # positive loss
    # ES: average of losses beyond VaR (returns worse than VaR)
    losses_beyond = -sorted_returns[:var_idx]  # convert losses to positive
    es = np.mean(losses_beyond)
    return var, es

# Compute for both confidence levels
hist_results = {}
for conf in confidence_levels:
    var, es = historical_var_es(port_returns, conf)
    hist_results[conf] = {'VaR': var, 'ES': es}
    print(f"Confidence {conf*100:.0f}%: VaR = {var*100:.4f}%, ES = {es*100:.4f}%")

# Dollar VaR (assuming portfolio value $1,000,000)
portfolio_value = 1_000_000
for conf in confidence_levels:
    var_dollar = hist_results[conf]['VaR'] * portfolio_value
    es_dollar = hist_results[conf]['ES'] * portfolio_value
    print(f"${portfolio_value:,.0f} portfolio, {conf*100:.0f}%: VaR = ${var_dollar:,.2f}, ES = ${es_dollar:,.2f}")

# ----------------------------------------------------------------
# PART C: PARAMETRIC (NORMAL) VAR AND ES
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART C: PARAMETRIC (NORMAL) VaR AND ES")
print("-"*60)

# Estimate mean and std of portfolio returns
mu_p = np.mean(port_returns)
sigma_p = np.std(port_returns, ddof=1)

print(f"Portfolio mean return: {mu_p*100:.4f}%")
print(f"Portfolio std deviation: {sigma_p*100:.4f}%")

# Function for normal VaR and ES
def normal_var_es(mu, sigma, confidence, portfolio_value=1):
    """
    Parametric VaR and ES under normal distribution.
    Returns VaR and ES as positive losses.
    """
    alpha = 1 - confidence
    z = norm.ppf(alpha)  # negative quantile
    var = -(mu + z * sigma)  # positive loss
    # ES formula: -mu + sigma * pdf(z)/alpha
    es = -mu + sigma * norm.pdf(z) / alpha
    return var, es

# Compute
normal_results = {}
for conf in confidence_levels:
    var, es = normal_var_es(mu_p, sigma_p, conf, portfolio_value)
    normal_results[conf] = {'VaR': var, 'ES': es}
    print(f"Confidence {conf*100:.0f}%: VaR = {var*100:.4f}%, ES = {es*100:.4f}%")

# Dollar amounts
for conf in confidence_levels:
    var_dollar = normal_results[conf]['VaR'] * portfolio_value
    es_dollar = normal_results[conf]['ES'] * portfolio_value
    print(f"${portfolio_value:,.0f} portfolio, {conf*100:.0f}%: VaR = ${var_dollar:,.2f}, ES = ${es_dollar:,.2f}")

# ----------------------------------------------------------------
# PART D: PORTFOLIO VAR – USING COVARIANCE MATRIX
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART D: PORTFOLIO VaR (Variance-Covariance Approach)")
print("-"*60)

# Estimate mean vector and covariance matrix
mu_vec = np.mean(df_returns[['Asset1', 'Asset2']].values, axis=0)
cov_mat = np.cov(df_returns[['Asset1', 'Asset2']].values.T)

# Portfolio variance
sigma_p_cov = np.sqrt(weights.T @ cov_mat @ weights)

print(f"Portfolio std from covariance: {sigma_p_cov*100:.4f}%")

# Compute VaR using the covariance matrix
for conf in confidence_levels:
    var_cov, es_cov = normal_var_es(mu_p, sigma_p_cov, conf, portfolio_value)
    print(f"Confidence {conf*100:.0f}% (cov matrix): VaR = {var_cov*100:.4f}%, ES = {es_cov*100:.4f}%")

# ----------------------------------------------------------------
# PART E: BACKTESTING – KUPIEC’S PROPORTION OF FAILURES
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART E: BACKTESTING HISTORICAL SIMULATION VaR")
print("-"*60)

# We need a longer history for backtesting: use the full dataset but split into estimation and test periods.
# For demonstration, we use the same dataset and treat it as test (but normally we would use out-of-sample).

# Use historical VaR at 95% and count exceptions
alpha_test = 0.05  # for 95% VaR
var_95_hist = hist_results[0.95]['VaR']  # positive loss
# Exceptions when loss exceeds VaR (i.e., portfolio return < -var_95_hist)
exceptions = (port_returns < -var_95_hist).sum()
T = len(port_returns)
print(f"Number of exceptions (loss > VaR): {exceptions} out of {T} days")
print(f"Expected exceptions: {alpha_test * T:.0f} days")

# Kupiec's POF test
def kupiec_pof(exceptions, T, alpha):
    """Kupiec's Proportion of Failures test statistic."""
    p_hat = exceptions / T
    if p_hat == 0 or p_hat == 1:
        return np.inf  # degenerate case
    lr = -2 * ( (T-exceptions)*np.log((1-alpha)/(1-p_hat)) + exceptions*np.log(alpha/p_hat) )
    return lr

lr_stat = kupiec_pof(exceptions, T, alpha_test)
print(f"Kupiec's LR statistic: {lr_stat:.4f}")
print(f"Critical value (95%): 3.841")
if lr_stat > 3.841:
    print("Result: REJECT null hypothesis – model is not accurate.")
else:
    print("Result: FAIL TO REJECT null – model is acceptable.")

# ----------------------------------------------------------------
# PART F: VISUALISATION – VAR AND ES ON DISTRIBUTION
# ----------------------------------------------------------------

fig, axes = plt.subplots(1, 2, figsize=(16, 6))

# Left: Historical VaR and ES
ax = axes[0]
n, bins, patches = ax.hist(port_returns, bins=50, density=True, alpha=0.7, color='lightblue', edgecolor='black')
ax.axvline(x=-var_95_hist, color='red', linestyle='-', linewidth=2, label=f'95% VaR = {var_95_hist*100:.2f}%')
ax.axvline(x=-hist_results[0.99]['VaR'], color='darkred', linestyle='--', linewidth=2, label=f'99% VaR = {hist_results[0.99]["VaR"]*100:.2f}%')
# Shade tail for ES
tail_threshold = -var_95_hist
tail_returns = port_returns[port_returns < tail_threshold]
ax.hist(tail_returns, bins=20, density=True, alpha=0.5, color='red', label='Tail (beyond VaR)')
ax.set_title('Historical VaR and ES (95%)', fontsize=12)
ax.set_xlabel('Daily Return')
ax.set_ylabel('Density')
ax.legend()
ax.grid(True, alpha=0.3)

# Right: Normal VaR and ES
ax = axes[1]
x = np.linspace(-0.15, 0.15, 500)
y = norm.pdf(x, mu_p, sigma_p)
ax.plot(x, y, 'b-', linewidth=2, label='Normal fit')
ax.hist(port_returns, bins=50, density=True, alpha=0.4, color='lightblue', edgecolor='black')
ax.axvline(x=-normal_results[0.95]['VaR'], color='red', linestyle='-', linewidth=2, label=f'95% VaR = {normal_results[0.95]["VaR"]*100:.2f}%')
ax.axvline(x=-normal_results[0.99]['VaR'], color='darkred', linestyle='--', linewidth=2, label=f'99% VaR = {normal_results[0.99]["VaR"]*100:.2f}%')
ax.fill_between(x, 0, y, where=(x < -normal_results[0.95]['VaR']), color='red', alpha=0.3, label='Tail region')
ax.set_title('Parametric (Normal) VaR and ES', fontsize=12)
ax.set_xlabel('Daily Return')
ax.set_ylabel('Density')
ax.legend()
ax.grid(True, alpha=0.3)

plt.tight_layout()
plt.savefig('var_es_distribution.png', dpi=300)
plt.show()

# ----------------------------------------------------------------
# PART G: COMPARISON OF METHODS
# ----------------------------------------------------------------

print("\n" + "="*70)
print("PART G: COMPARISON OF VaR AND ES ESTIMATES")
print("="*70)

comparison_df = pd.DataFrame({
    'Method': ['Historical', 'Parametric (Normal)'],
    '95% VaR (%)': [hist_results[0.95]['VaR']*100, normal_results[0.95]['VaR']*100],
    '95% ES (%)': [hist_results[0.95]['ES']*100, normal_results[0.95]['ES']*100],
    '99% VaR (%)': [hist_results[0.99]['VaR']*100, normal_results[0.99]['VaR']*100],
    '99% ES (%)': [hist_results[0.99]['ES']*100, normal_results[0.99]['ES']*100]
})
print(comparison_df.to_string(index=False))

print("\nInterpretation:")
print("  • Historical VaR is typically larger than normal VaR due to fat tails.")
print("  • ES is always greater than VaR (as it averages tail losses).")
print("  • For regulatory capital, use ES at 97.5% (FRTB) instead of VaR at 99%.")

# ----------------------------------------------------------------
# PART H: REGULATORY CONTEXT – BASEL III / FRTB
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART H: REGULATORY CONTEXT – BASEL III / FRTB")
print("-"*60)

print("""
Key Regulatory Requirements:
  • Basel III revised market risk framework (FRTB – Fundamental Review of the Trading Book) replaces VaR with Expected Shortfall (ES) at 97.5% confidence, 10-day horizon.
  • Banks must use a 10-day ES, scaled by sqrt(10) from 1-day ES.
  • ES is considered a coherent risk measure and better captures tail risk.
  • Backtesting still uses VaR (because it's easier to test), but capital is based on ES.
  • Historical Simulation is the most commonly used method for regulatory ES, often with a 1-year (250-day) lookback period.
  • Firms must also conduct stress testing and scenario analysis.
""")

SECTION 9: KEY REGULATORY REQUIREMENTS

Regulation	Requirement	Implication for VaR/ES
Basel III / FRTB	Use Expected Shortfall (ES) at 97.5% for market risk capital.	Replace VaR with ES for capital calculations.
SR 11-7 (US)	Model validation must include backtesting and stress testing.	Document VaR methodology; perform backtesting; explain exceptions.
Solvency II (EU)	Insurers must calculate 99.5% VaR (or internal model) over 1-year horizon.	Similar to banking but longer horizon.
FAS 157 / IFRS 13	Fair value measurement requires consideration of market risk.	VaR may be used for risk disclosures.

SECTION 10: SUMMARY FOR THE DATA PRACTITIONER

Value at Risk (VaR) quantifies the maximum loss at a given confidence level over a given horizon.
Historical Simulation is intuitive and non‑parametric; Parametric (Normal) is fast but assumes normality.
Expected Shortfall (ES) is the average loss in the tail beyond VaR – a coherent risk measure now required by regulators.
Backtesting (Kupiec’s POF) is essential to validate VaR models.
In practice, banks use a combination of methods: Historical for market risk, Monte Carlo for complex portfolios, and parametric for quick estimates.
Regulatory trend: ES over VaR, FRTB, and increased emphasis on tail risk.

SECTION 11: RECOMMENDED NEXT STEPS

Apply VaR/ES to a real portfolio using actual stock data (e.g., from Yahoo Finance).
Compute 10‑day VaR/ES by scaling with sqrt(10) (assuming i.i.d. returns).
Implement Monte Carlo simulation for VaR/ES with a GARCH process (next lesson).
Explore Extreme Value Theory (EVT) for modelling tail distributions.
Study the FRTB standardised approach vs. internal models approach.

[END OF LESSON 1 – MODULE 5]

Diploma in Financial Data Analytics LESSON 1: VALUE AT RISK (VaR) AND EXPECTED SHORTFALL (ES) – HISTORICAL SIMULATION AND PARAMETRIC APPROACHES

Diploma in Financial Data Analytics