1. Learning Objectives

By the end of this lesson, you will be able to:

  • Rigorously define Shannon entropy and its properties (non-negativity, maximum entropy, chain rule).

  • Derive and interpret the Kullback-Leibler (KL) Divergence and its relationship to Maximum Likelihood Estimation.

  • Apply the Principle of Maximum Entropy to construct probability distributions from partial information (e.g., risk-neutral densities).

  • Use KL Divergence as a loss function in Variational Autoencoders (VAEs) and for model selection in financial AI.

  • Understand Mutual Information and its application to feature selection and discovering non-linear dependencies.

  • Apply Information-Theoretic Risk Measures (Entropic Value at Risk, Entropic Risk Measures) for portfolio optimisation.

  • Understand the relationship between KL Divergence and Bayesian inference (ELBO).

2. Shannon Entropy – The Measure of Uncertainty

Entropy quantifies the average amount of information or surprise contained in a random variable.

2.1 Definition for Discrete Random Variables
For a discrete random variable X with probability mass function p(x), the entropy is:
H(X) = - Σ_{x ∈ X} p(x) log p(x).
The base of the logarithm determines the unit (base 2 → bits, base e → nats).
Interpretation: Entropy is the average number of bits needed to encode the outcomes of X. Higher entropy means more uncertainty (uniform distribution has maximum entropy).

2.2 Definition for Continuous Random Variables (Differential Entropy)
For a continuous random variable X with density p(x):
h(X) = - ∫ p(x) ln p(x) dx.
Warning: Differential entropy can be negative (unlike discrete entropy) and is not invariant under transformations.

2.3 Properties of Entropy

  • Non-negativity: H(X) ≥ 0 for discrete variables.

  • Maximum Entropy: For a discrete variable with n outcomes, entropy is maximised at ln n when p(x) = 1/n (uniform distribution).

  • Chain Rule: H(X, Y) = H(X) + H(Y | X).

  • Independence: H(X, Y) = H(X) + H(Y) if X and Y are independent.

2.4 Financial Application – Maximum Entropy Principle
Suppose we know only the first two moments of a distribution (mean μ and variance σ²). The maximum entropy distribution subject to these constraints is the Normal distribution.
p(x) = (1 / sqrt(2π σ²)) exp( -(x - μ)² / (2σ²) ).
This justifies the use of the normal distribution when only mean and variance are known. It is the “least informative” or “most conservative” distribution given the constraints.

2.5 Financial Application – Risk-Neutral Density Estimation
In option pricing, we observe market prices and want to extract the risk-neutral density q(S_T).
The maximum entropy principle chooses the density q that maximises entropy subject to:

  1. ∫ q(S_T) dS_T = 1.

  2. ∫ q(S_T) * Payoff_i(S_T) dS_T = MarketPrice_i (for all traded options).
    The resulting density is the most conservative distribution consistent with market prices. This is widely used in practice to estimate the implied distribution of future asset prices.

3. Kullback-Leibler (KL) Divergence – Measuring the Distance Between Distributions

KL divergence measures how one probability distribution p diverges from a reference distribution q.

3.1 Definition
For discrete distributions:
D_{KL}(p || q) = Σ_{x} p(x) ln( p(x) / q(x) ).
For continuous distributions:
D_{KL}(p || q) = ∫ p(x) ln( p(x) / q(x) ) dx.

3.2 Properties

  • Non-negativity: D_{KL}(p || q) ≥ 0, with equality if and only if p = q almost everywhere (Gibbs inequality).

  • Not Symmetric: D_{KL}(p || q) ≠ D_{KL}(q || p). This is why it is called “divergence,” not “distance.”

  • Chain Rule: D_{KL}(p(x, y) || q(x, y)) = D_{KL}(p(x) || q(x)) + D_{KL}(p(y | x) || q(y | x)).

3.3 Relationship to Maximum Likelihood Estimation
Minimising D_{KL}(p_{data} || p_{model}) is equivalent to maximising the log-likelihood. Let p_{data} be the empirical distribution and p_{model}(x; θ) be the model. The KL divergence is:
D_{KL}(p_{data} || p_{model}) = -E_{p_{data}}[ ln p_{model}(x; θ) ] + Constant.
Thus, minimising KL divergence = maximising the expected log-likelihood. This is the theoretical foundation for training AI models with MLE loss functions.

3.4 Financial Application – Model Selection
To compare two models M_1 and M_2, we can compute:
D_{KL}(p_{data} || p_{M_1}) vs. D_{KL}(p_{data} || p_{M_2}).
The model with smaller KL divergence better fits the data. This is equivalent to comparing AIC/BIC in large samples.

3.5 Financial Application – Regime Switching
In a regime-switching model, we have two regimes (bull/bear). KL divergence can quantify how distinct the two regimes are:
D_{KL}(p_{Bull} || p_{Bear}).
A high KL divergence means the regimes are well-separated, and the switching model is useful. A low divergence means the regimes are similar and the model is overfitting.

4. Mutual Information – Measuring Dependence

Mutual information (MI) measures the amount of information that one random variable contains about another. Unlike correlation, it captures non-linear dependencies.

4.1 Definition
For two random variables X and Y:
I(X; Y) = Σ_{x} Σ_{y} p(x, y) ln( p(x, y) / (p(x) p(y)) ).
Equivalently:
I(X; Y) = H(X) - H(X | Y) = H(Y) - H(Y | X) = H(X) + H(Y) - H(X, Y).

4.2 Properties

  • Non-negative: I(X; Y) ≥ 0, equality if and only if X and Y are independent.

  • Symmetric: I(X; Y) = I(Y; X).

  • Relationship to Correlation: If (X, Y) are jointly normal with correlation ρ, then:
    I(X; Y) = -0.5 ln(1 - ρ²).
    Mutual information generalises correlation to non-linear dependencies.

4.3 Financial Application – Feature Selection
In an AI model with many features, we want to select the most informative features. Compute I(Feature_i; Target) for each feature. Features with high mutual information are the most predictive.
Algorithm – Maximum Relevance Minimum Redundancy (mRMR):

  1. Select features with maximum relevance to the target: max I(Feature_i; Target).

  2. Remove redundant features: min I(Feature_i; Feature_j) for selected features.
    This is used in high-frequency trading to select the most predictive order book features.

4.4 Financial Application – Discovering Non-Linear Dependencies
Correlation only captures linear relationships. Mutual information can detect non-linear relationships, such as threshold effects (e.g., the stock moves only when volatility exceeds a certain level). Use MI to uncover hidden dependencies that an AI model can exploit.

5. Cross-Entropy – The Loss Function for Classification

Cross-entropy is the average number of bits needed to encode data from distribution p using a code designed for distribution q.

5.1 Definition
H(p, q) = -Σ_{x} p(x) ln q(x).
For binary classification (logistic regression):
L = - (1/N) Σ_{i=1}^N [ y_i ln( \hat{y}_i ) + (1 - y_i) ln(1 - \hat{y}_i) ].
This is exactly the cross-entropy loss.

5.2 Relationship to KL Divergence
H(p, q) = H(p) + D_{KL}(p || q).
Minimising cross-entropy is equivalent to minimising KL divergence when H(p) is constant (which it is for fixed data).

5.3 Financial Application – Credit Default Prediction
In credit scoring, the target y_i ∈ {0, 1} indicates default. Cross-entropy loss is the standard loss function. The model outputs \hat{y}_i = P(default_i = 1 | features). Minimising cross-entropy yields calibrated probabilities of default, which are essential for regulatory capital calculations (Basel III).

6. Variational Inference – Approximating Intractable Posteriors

In Bayesian inference, the posterior p(θ | Data) = p(Data | θ) p(θ) / p(Data) is often intractable because the marginal likelihood p(Data) is hard to compute. Variational inference approximates the posterior with a simpler distribution q(θ) by minimising KL divergence.

6.1 The Evidence Lower Bound (ELBO)
ln p(Data) = D_{KL}(q(θ) || p(θ | Data)) + ELBO(q).
Since D_{KL} ≥ 0, the ELBO is a lower bound on the log-likelihood.
ELBO(q) = E_{q(θ)}[ ln p(Data | θ) ] - D_{KL}(q(θ) || p(θ)).
Maximising the ELBO with respect to q is equivalent to minimising D_{KL}(q(θ) || p(θ | Data)).

6.2 Financial Application – Bayesian Neural Networks for Risk
In a Bayesian neural network, we place a prior p(w) on the weights and approximate the posterior q(w) using variational inference. The ELBO loss is:
L = E_{q(w)}[ Loss(w; Data) ] + D_{KL}(q(w) || p(w)).
The second term acts as a regulariser (equivalent to L2 regularisation under a Gaussian prior). The first term is the expected loss under the weight distribution.
Benefits: The model outputs a predictive distribution, not a point estimate. This gives:

  • Aleatoric Uncertainty: Var(y | x, w) (data noise).

  • Epistemic Uncertainty: Var_w( E[y | x, w] ) (model uncertainty).
    Both are crucial for risk management and regulatory reporting.

6.3 Financial Application – Black-Litterman Model (Bayesian View Combination)
The Black-Litterman model is a special case of Bayesian inference. The posterior q(μ) is obtained by combining the prior p(μ) ∼ N(π, τ Σ) and the likelihood (views) p(Views | μ) ∼ N(P μ, Ω). The posterior is Normal, and the KL divergence between prior and posterior quantifies the information gained from the views.

7. Entropic Risk Measures – Information-Theoretic Risk

Traditional risk measures (VaR, Expected Shortfall) are based on quantiles. Entropic risk measures incorporate risk aversion through exponential utilities.

7.1 Entropic Value at Risk (EVaR)
EVaR is a coherent risk measure that bounds the probability of losses below a threshold:
EVaR_α(X) = inf_{θ > 0} (1/θ) ln( E[ e^{θ X} ] / (1 - α) ).
This is a tight upper bound on VaR and is more sensitive to tail risk.

7.2 Entropic Risk Measure (Exponential Utility)
For a risk-averse agent with exponential utility U(x) = -e^{-γ x}, the certainty equivalent is:
CE(X) = - (1/γ) ln( E[ e^{-γ X} ] ).
This is an entropic risk measure. The parameter γ is the coefficient of risk aversion.
AI Application: In reinforcement learning for portfolio optimisation, the reward function can use entropic risk instead of expected return. The agent learns to maximise the certainty equivalent, which penalises tail risk.

7.3 Information-Theoretic Portfolio Optimisation
Instead of mean-variance optimisation, we can use entropy to construct a diversified portfolio. The Maximum Entropy Portfolio maximises entropy of the weights subject to a target return and risk constraint:
max_w H(w) subject to w^T μ ≥ R_target, w^T Σ w ≤ σ²_target.
This yields a more diversified portfolio than the minimum variance portfolio (which tends to concentrate weights).

8. KL Divergence in Generative AI – GANs and VAEs

8.1 Variational Autoencoders (VAEs)
VAEs use KL divergence as a regulariser. The loss function is:
L = Reconstruction_Loss + D_{KL}( q(z | x) || p(z) ).
The KL term forces the latent distribution q(z | x) to be close to the prior p(z) ∼ N(0, I). This ensures smooth interpolation in the latent space.
Financial Application: Generate synthetic financial time series (returns, prices) that preserve statistical properties. VAE-generated data can augment small datasets and improve the robustness of AI models. Synthetic data can also be used for stress testing.

8.2 Generative Adversarial Networks (GANs)
GANs implicitly minimise a divergence between the real data distribution p_{data} and the generated distribution p_g. The original GAN minimises the Jensen-Shannon divergence (which is a symmetrised version of KL divergence).
Financial Application: GANs generate realistic financial time series for backtesting. A TimeGAN specifically generates time-series data that preserves temporal dependencies.

8.3 Distributionally Robust Optimisation (DRO)
DRO minimises the worst-case expected loss over a set of distributions close to the empirical distribution:
min_θ sup_{Q: D_{KL}(Q || P_data) ≤ ρ} E_Q[ Loss(θ; X) ].
The KL divergence constraint defines an “uncertainty set” around the empirical distribution. This leads to a robust estimator that performs well even under distributional shifts (regime changes). This is the theoretical foundation for adversarial training in finance.

9. Summary for the AI Practitioner

  1. Entropy measures uncertainty. The Maximum Entropy Principle justifies the normal distribution when only mean and variance are known. It is also used to infer risk-neutral densities from option prices.

  2. KL Divergence measures the distance between distributions. Minimising KL divergence is equivalent to Maximum Likelihood Estimation. It is the foundation of cross-entropy loss and variational inference.

  3. Mutual Information captures non-linear dependencies. Use it for feature selection (mRMR) and to discover hidden relationships in financial data.

  4. Cross-Entropy is the loss function for classification. It gives calibrated probabilities (e.g., probability of default).

  5. Variational Inference (ELBO) provides a tractable way to do Bayesian inference. It yields epistemic uncertainty, which is essential for risk management.

  6. Entropic Risk Measures (EVaR, exponential utility) incorporate risk aversion explicitly. Use them in reinforcement learning for portfolio optimisation.

  7. VAEs and GANs use KL divergence and related divergences to generate synthetic financial data. This augments small datasets and enables stress testing.

  8. Distributionally Robust Optimisation uses KL divergence to define an uncertainty set. This protects against regime changes and model misspecification.

End of Module 2 – Complete

Module 2 is now fully populated with all 8 lessons. The student now has a rigorous mathematical foundation in Linear Algebra, Multivariate Calculus, Probability Theory, Statistical Inference, Optimisation Theory, Stochastic Calculus, Numerical Linear Algebra, and Information Theory – all with direct financial applications.

In Module 3, we will apply this mathematics to Programming for Financial AI (Python) , covering NumPy, Pandas, PyTorch/TensorFlow, data pipelines, and production-grade AI implementation.

 
 
PreviousNext