1. LEARNING OBJECTIVES

By the end of this lesson, you will be able to:

  • Define and compute partial derivatives, gradients, and directional derivatives of scalar and vector-valued functions.

  • Construct and interpret the Jacobian matrix for vector-valued functions.

  • Construct and interpret the Hessian matrix and use it to determine convexity and local extrema.

  • Derive and apply the multivariate Taylor expansion to approximate complex financial functions.

  • Apply the Chain Rule in vectorised form to derive gradients for neural networks (backpropagation).

  • Perform constrained optimisation using Lagrange multipliers and KKT conditions.

  • Understand the relationship between gradients and the efficient frontier in Markowitz portfolio optimisation.

2. PARTIAL DERIVATIVES AND GRADIENTS – THE DIRECTION OF STEEPEST ASCENT

For a scalar-valued function f: R^n → R, the gradient is a vector of partial derivatives:

∇f(x) = [ ∂f/∂x_1, ∂f/∂x_2, …, ∂f/∂x_n ]^T

2.1 Interpretation of the Gradient

  • The gradient points in the direction of the steepest increase of f.

  • The negative gradient -∇f(x) points in the direction of the steepest decrease (used in Gradient Descent).

  • The magnitude ||∇f(x)|| is the rate of increase in the steepest direction.

2.2 Directional Derivative

The rate of change of f in the direction of a unit vector v is:

D_v f(x) = ∇f(x) · v = v^T ∇f(x)

In finance, if f(w) is the Sharpe ratio of a portfolio and v is a shift in portfolio weights, the directional derivative tells you how much the Sharpe ratio changes.

Example (Computing a Gradient):

Let f(x, y) = 3x^2 + 4xy + 2y^2. The partial derivatives are:

∂f/∂x = 6x + 4y
∂f/∂y = 4x + 4y

The gradient vector is:

∇f(x, y) = [6x + 4y, 4x + 4y]^T

2.3 Financial Application – Markowitz Optimisation

The objective function (to minimise) is the portfolio variance:

σ_p²(w) = w^T Σ w

Subject to w^T 1 = 1 (weights sum to 1).

The gradient of the portfolio variance with respect to the weights is:

∇σ_p²(w) = 2 Σ w

Setting this to zero gives the unconstrained minimum variance portfolio: Σ w = 0 → w = 0, which is invalid (sums to zero). The constrained solution is found using Lagrange multipliers (see Section 7).

3. THE JACOBIAN MATRIX – DERIVATIVES OF VECTOR-VALUED FUNCTIONS

For a vector-valued function f: R^n → R^m, where f(x) = [f_1(x), f_2(x), …, f_m(x)]^T, the Jacobian is an (m x n) matrix:

J_f(x) = ∂f_i / ∂x_j, for i = 1..m, j = 1..n

Explicit Form:

J_f(x) =
[ ∂f_1/∂x_1 ∂f_1/∂x_2 … ∂f_1/∂x_n ]
[ ∂f_2/∂x_1 ∂f_2/∂x_2 … ∂f_2/∂x_n ]
[ … … … … ]
[ ∂f_m/∂x_1 ∂f_m/∂x_2 … ∂f_m/∂x_n ]

3.1 Interpretation

The Jacobian is the linear approximation of f at a point x_0. It maps perturbations dx in the input space to perturbations dy in the output space:

dy ≈ J_f(x_0) dx

3.2 Financial Application – Risk Mapping

Consider a portfolio with N assets and M risk factors. The mapping from risk factors x ∈ R^M to portfolio returns y ∈ R^N is y = f(x). The Jacobian J_f is the sensitivity matrix (deltas, betas). The covariance matrix of returns is:

Σ_y = J_f Σ_x J_f^T

This is the Delta-Normal approach to Value at Risk (VaR).

Example (Computing the Jacobian):

Let f: R^2 → R^2 be defined as:

f_1(x, y) = x^2 + y^2
f_2(x, y) = 2xy

The Jacobian is:

J_f(x, y) = [ ∂f_1/∂x ∂f_1/∂y ]
[ ∂f_2/∂x ∂f_2/∂y ]
= [ 2x 2y ]
[ 2y 2x ]

3.3 The Chain Rule in Vectorised Form (Critical for AI)

If y = f(x) and L = g(y) is a scalar loss, then the gradient of L with respect to x is:

∂L/∂x = (∂y/∂x)^T (∂L/∂y) = J_f^T ∇_y L

In neural networks, this is the backpropagation algorithm. For a layer with weights W, output y = W x, the gradient with respect to W is:

∂L/∂W = (∂L/∂y) x^T

This is an outer product and is the foundation of gradient descent training for neural networks.

Proof of the Chain Rule (Vectorised Form):

We have L = g(f(x)). The partial derivative of L with respect to x_i is:

∂L/∂x_i = Σ_{j=1}^m (∂L/∂y_j) * (∂y_j/∂x_i)

In matrix form, this is:

∇_x L = (J_f)^T ∇_y L

Where J_f is the Jacobian of f with respect to x, and ∇_y L is the gradient of L with respect to y.

4. THE HESSIAN MATRIX – CURVATURE AND CONVEXITY

For a scalar-valued function f: R^n → R, the Hessian is an (n x n) symmetric matrix of second-order partial derivatives:

H_f(x) = [ ∂²f / ∂x_i ∂x_j ]

If f is twice continuously differentiable, H_f is symmetric (∂²f/∂x_i∂x_j = ∂²f/∂x_j∂x_i).

Explicit Form:

H_f(x) =
[ ∂²f/∂x_1² ∂²f/∂x_1∂x_2 … ∂²f/∂x_1∂x_n ]
[ ∂²f/∂x_2∂x_1 ∂²f/∂x_2² … ∂²f/∂x_2∂x_n ]
[ … … … … ]
[ ∂²f/∂x_n∂x_1 ∂²f/∂x_n∂x_2 … ∂²f/∂x_n² ]

4.1 Convexity and the Hessian

  • f is convex if H_f(x) is Positive Semi-Definite (PSD) for all x.

  • f is strictly convex if H_f(x) is Positive Definite (PD).

  • f is concave if H_f(x) is Negative Semi-Definite (NSD).

  • f is strictly concave if H_f(x) is Negative Definite (ND).

Testing for Definiteness:

  • A symmetric matrix A is Positive Definite if all eigenvalues are > 0.

  • A symmetric matrix A is Positive Semi-Definite if all eigenvalues are ≥ 0.

  • A symmetric matrix A is Negative Definite if all eigenvalues are < 0.

  • A symmetric matrix A is Indefinite if it has both positive and negative eigenvalues.

4.2 Financial Application – Loss Functions

Mean Squared Error (MSE) Loss:

L(w) = (1/N) Σ_{i=1}^N (y_i – w^T x_i)^2

The Hessian is:

H_L = (2/N) X^T X

Where X is the (N x n) design matrix. Since X^T X is PSD, MSE is convex, and its global minimum can be found analytically.

Cross-Entropy Loss (Logistic Regression):

L(w) = – (1/N) Σ_{i=1}^N [ y_i ln(p_i) + (1 – y_i) ln(1 – p_i) ]

Where p_i = 1 / (1 + e^(-w^T x_i)). The Hessian is:

H_L = (1/N) X^T D X

Where D is a diagonal matrix with entries p_i(1 – p_i) > 0. Since D is positive definite (all diagonal entries > 0), X^T D X is PSD. Therefore, Cross-Entropy Loss is convex.

Neural Networks:

The loss surface is non-convex (the Hessian is indefinite). This is why we get local minima and saddle points. However, in high dimensions, saddle points dominate local minima, and Stochastic Gradient Descent (SGD) escapes them efficiently.

4.3 Newton’s Method

Newton’s method uses the Hessian to find the minimum of f:

x_{k+1} = x_k – H_f(x_k)^{-1} ∇f(x_k)

This converges quadratically (much faster than gradient descent) but is computationally expensive (O(n^3) to invert the Hessian). Used in second-order optimisers like L-BFGS for small-scale AI models.

Example (Computing the Hessian):

Let f(x, y) = x^2 + 3xy + y^2. The first derivatives are:

∂f/∂x = 2x + 3y
∂f/∂y = 3x + 2y

The Hessian is:

H_f = [ ∂²f/∂x² ∂²f/∂x∂y ]
[ ∂²f/∂y∂x ∂²f/∂y² ]
= [ 2 3 ]
[ 3 2 ]

The eigenvalues of H are: λ_1 = 5, λ_2 = -1. Since one eigenvalue is negative, H is indefinite, and the function is non-convex (it has a saddle point at (0,0)).

5. TAYLOR SERIES – APPROXIMATING NON-LINEAR FUNCTIONS

The Taylor expansion allows us to approximate a function near a point x_0 using its derivatives. This is essential for deriving option pricing formulas and for understanding the local behaviour of neural networks.

5.1 Univariate Taylor Expansion

For a scalar f: R → R:

f(x) = f(x_0) + f'(x_0)(x – x_0) + (1/2) f”(x_0)(x – x_0)^2 + (1/6) f”'(x_0)(x – x_0)^3 + …

The remainder (error) for the nth-order expansion is O((x – x_0)^(n+1)).

Example (Expanding e^x around x_0 = 0):

e^x = 1 + x + x^2/2 + x^3/6 + x^4/24 + …

5.2 Multivariate Taylor Expansion

For f: R^n → R:

f(x) = f(x_0) + ∇f(x_0)^T (x – x_0) + (1/2) (x – x_0)^T H_f(x_0) (x – x_0) + O(||x – x_0||^3)

5.3 Financial Application – Option Pricing Approximation (Delta-Gamma)

The change in the value of an option V(S) when the underlying price S changes by ΔS is approximated by:

ΔV ≈ Δ * ΔS + (1/2) Γ * (ΔS)^2

Where:

  • Δ = ∂V/∂S (Delta, first derivative).

  • Γ = ∂²V/∂S² (Gamma, second derivative).

This is the Delta-Gamma approximation for VaR of options portfolios.

Derivation of Delta-Gamma:

The Taylor expansion of V(S) around S_0 is:

V(S) = V(S_0) + V'(S_0)(S – S_0) + (1/2) V”(S_0)(S – S_0)^2 + …

Substituting Δ = V'(S_0) and Γ = V”(S_0):

V(S) = V(S_0) + Δ * (S – S_0) + (1/2) Γ * (S – S_0)^2

Rearranging for the change in V:

ΔV = V(S) – V(S_0) = Δ * (S – S_0) + (1/2) Γ * (S – S_0)^2

5.4 Financial Application – Portfolio Return Approximation

The return of a portfolio P(w) is a function of weights w. The Taylor expansion around w_0:

P(w) ≈ P(w_0) + ∇P(w_0)^T (w – w_0) + (1/2) (w – w_0)^T H_P(w_0) (w – w_0)

In a neural network portfolio optimiser, the loss function is expanded to approximate the local gradient.

6. THE GRADIENT IN HIGH DIMENSIONS – THE SADDLE POINT PROBLEM

For a differentiable function f: R^n → R, a critical point x* satisfies ∇f(x*) = 0. This can be:

  • Local Minimum: H_f(x*) is Positive Definite (all eigenvalues > 0).

  • Local Maximum: H_f(x*) is Negative Definite (all eigenvalues < 0).

  • Saddle Point: H_f(x*) has both positive and negative eigenvalues.

6.1 Why Saddle Points Matter in AI

In high-dimensional neural networks, most critical points are saddle points, not local minima. Gradient descent gets stuck at saddle points because the gradient is zero in some directions. However, random perturbations (noise) and momentum-based optimisers (Adam) help escape saddle points.

6.2 The Hessian at Saddle Points

If the Hessian has a negative eigenvalue, there exists a direction v where v^T H v < 0. Moving slightly in that direction decreases the loss. This is the basis for Saddle-Free Newton methods.

6.3 The Ratio of Saddle Points to Local Minima

In high dimensions, the probability that a critical point is a local minimum is exponentially small. For a function with random Hessian, the fraction of critical points that are local minima is approximately 2^(-n), where n is the dimension. This explains why deep learning optimisation is challenging.

7. CONSTRAINED OPTIMISATION – LAGRANGE MULTIPLIERS AND KKT CONDITIONS

In finance, constraints are everywhere:

  • w^T 1 = 1 (weights sum to 1).

  • w_i ≥ 0 (no short-selling).

  • w^T μ ≥ μ_target (target return).

  • w^T Σ w ≤ σ²_max (maximum risk).

We use Lagrange multipliers to handle equality constraints and KKT conditions for inequality constraints.

7.1 Lagrange Multipliers (Equality Constraints)

Minimise f(x) subject to g(x) = 0. The Lagrangian is:

L(x, λ) = f(x) + λ g(x)

The necessary conditions for optimality (first-order) are:

  1. ∂L/∂x = ∇f(x) + λ ∇g(x) = 0

  2. ∂L/∂λ = g(x) = 0

7.2 Financial Application – The Minimum Variance Portfolio

Minimise w^T Σ w subject to w^T 1 = 1.

Step 1: Form the Lagrangian:

L(w, λ) = w^T Σ w + λ(w^T 1 – 1)

Step 2: Take the gradient with respect to w:

∂L/∂w = 2 Σ w + λ 1 = 0
2 Σ w = -λ 1
w = – (λ/2) Σ^{-1} 1

Step 3: Apply the constraint w^T 1 = 1:

w^T 1 = -(λ/2) 1^T Σ^{-1} 1 = 1
-λ/2 = 1 / (1^T Σ^{-1} 1)

Step 4: Substitute back:

w_{MVP} = Σ^{-1} 1 / (1^T Σ^{-1} 1)

This is the closed-form solution for the minimum variance portfolio.

7.3 Karush-Kuhn-Tucker (KKT) Conditions (Inequality Constraints)

For minimising f(x) subject to g_i(x) ≤ 0 and h_j(x) = 0:

  1. Stationarity:

∇f(x) + Σ_i μ_i ∇g_i(x) + Σ_j λ_j ∇h_j(x*) = 0

  1. Primal Feasibility:

g_i(x) ≤ 0, h_j(x) = 0

  1. Dual Feasibility:

μ_i ≥ 0

  1. Complementary Slackness:

μ_i g_i(x*) = 0 for all i

7.4 Financial Application – Portfolio with No Short-Selling

Add constraint w_i ≥ 0. The KKT conditions yield the solution where many weights are zero (the active constraints). This is solved using quadratic programming (QP) solvers like quadprog or cvxopt.

Example (Two-Asset Portfolio with No Short-Selling):

Minimise σ_p² = w_1² σ_1² + w_2² σ_2² + 2 w_1 w_2 σ_1 σ_2 ρ
Subject to:

  • w_1 + w_2 = 1

  • w_1 ≥ 0, w_2 ≥ 0

The solution will lie on the boundary (w_1 = 0 or w_2 = 0) unless the unconstrained solution satisfies the non-negativity constraints.

8. THE CHAIN RULE AND BACKPROPAGATION – THE ENGINE OF AI

The chain rule in vectorised form is the mathematical core of deep learning.

8.1 Forward Pass

For a neural network with layers:

z^{(1)} = W^{(1)} x + b^{(1)}
a^{(1)} = σ(z^{(1)})
z^{(2)} = W^{(2)} a^{(1)} + b^{(2)}
a^{(2)} = σ(z^{(2)})
…
y = a^{(L)} = σ(z^{(L)})
L = Loss(y, y_target)

8.2 Backward Pass (Vectorised Chain Rule)

For a layer l, we have:

∂L/∂W^{(l)} = (∂L/∂z^{(l)}) (a^{(l-1)})^T
∂L/∂b^{(l)} = ∂L/∂z^{(l)}

The gradient flows backward from the output to the input. This is implemented in PyTorch and TensorFlow using automatic differentiation (autograd).

Derivation of the Backward Pass:

Let δ^{(l)} = ∂L/∂z^{(l)} (the gradient of the loss with respect to the pre-activation at layer l).

For the output layer (layer L):

δ^{(L)} = ∂L/∂a^{(L)} * σ'(z^{(L)})

For a hidden layer l:

δ^{(l)} = (W^{(l+1)})^T δ^{(l+1)} * σ'(z^{(l)})

The gradient of the loss with respect to the weights at layer l is:

∂L/∂W^{(l)} = δ^{(l)} (a^{(l-1)})^T

The gradient of the loss with respect to the biases at layer l is:

∂L/∂b^{(l)} = δ^{(l)}

8.3 The Jacobian-Vector Product (JVP) and Vector-Jacobian Product (VJP)

In deep learning, we never explicitly compute the full Jacobian (which is huge). Instead, we compute the VJP:

v^T J_f

Where v is the gradient from the layer above. This is what autograd does: it efficiently computes ∂L/∂x = (∂L/∂y)^T J_f.

The VJP Computation:

Given y = f(x) and v = ∂L/∂y, we want to compute ∂L/∂x.

∂L/∂x = v^T J_f

Where J_f is the Jacobian of f with respect to x.

Example (Linear Layer):

y = W x. The Jacobian of y with respect to x is J_f = W^T (the transpose of W). The VJP is:

∂L/∂x = v^T W^T = (W v)^T

So the gradient with respect to x is W^T v.

9. SUMMARY FOR THE AI PRACTITIONER

  • Gradients point uphill; negative gradients point downhill (Gradient Descent). The gradient of portfolio variance is 2Σw.

  • The Jacobian maps input perturbations to output perturbations. Used for risk mapping (Delta-Normal VaR).

  • The Hessian determines curvature. Convex loss functions (MSE, Cross-Entropy) have PSD Hessians. Neural networks are non-convex (saddle points dominate).

  • Taylor expansions approximate non-linear functions. Delta-Gamma VaR is a Taylor expansion of an option portfolio.

  • Lagrange multipliers solve constrained optimisation. The minimum variance portfolio has a closed-form solution using Lagrange multipliers.

  • Backpropagation is the vectorised chain rule. The gradient of a layer’s weights is an outer product of the layer input and the incoming gradient.

10. PRACTICAL IMPLEMENTATION IN PYTHON

A. Computing Gradients with NumPy:

python
import numpy as np

def f(x):
    return x[0]**2 + 3*x[0]*x[1] + x[1]**2

def gradient_f(x):
    dx0 = 2*x[0] + 3*x[1]
    dx1 = 3*x[0] + 2*x[1]
    return np.array([dx0, dx1])

x = np.array([1.0, 2.0])
grad = gradient_f(x)
print(grad)  # Output: [8. 7.]

B. Computing the Hessian with NumPy:

python
def hessian_f(x):
    return np.array([[2.0, 3.0], [3.0, 2.0]])

H = hessian_f(x)
eigenvalues, eigenvectors = np.linalg.eig(H)
print(eigenvalues)  # Output: [5., -1.]

C. Gradient Descent Implementation:

python
def gradient_descent(f, grad_f, x0, learning_rate=0.01, epochs=1000):
    x = x0.copy()
    for i in range(epochs):
        grad = grad_f(x)
        x = x - learning_rate * grad
    return x

x0 = np.array([10.0, 10.0])
x_min = gradient_descent(f, gradient_f, x0)
print(x_min)  # Approximately [0., 0.]

D. Newton’s Method Implementation:

python
def newtons_method(f, grad_f, hess_f, x0, epochs=10):
    x = x0.copy()
    for i in range(epochs):
        grad = grad_f(x)
        H = hess_f(x)
        x = x - np.linalg.solve(H, grad)
    return x

x_min_newton = newtons_method(f, gradient_f, hessian_f, x0)
print(x_min_newton)  # Approximately [0., 0.]

E. Markowitz Minimum Variance Portfolio with Lagrange Multipliers:

python
def min_variance_portfolio(cov_matrix):
    n = cov_matrix.shape[0]
    ones = np.ones(n)
    inv_cov = np.linalg.inv(cov_matrix)
    weights = inv_cov @ ones / (ones.T @ inv_cov @ ones)
    return weights

# Example with 3 assets
cov = np.array([[0.04, 0.02, 0.01],
                [0.02, 0.06, 0.03],
                [0.01, 0.03, 0.08]])
weights = min_variance_portfolio(cov)
print(weights)  # Optimal weights summing to 1

 

PreviousNext