1. Learning Objectives

By the end of this lesson, you will be able to:

  • Define and compute partial derivatives, gradients, and directional derivatives of scalar and vector-valued functions.

  • Construct and interpret the Jacobian matrix for vector-valued functions.

  • Construct and interpret the Hessian matrix and use it to determine convexity and local extrema.

  • Derive and apply the multivariate Taylor expansion to approximate complex financial functions.

  • Apply the Chain Rule in vectorised form to derive gradients for neural networks (backpropagation).

  • Perform constrained optimisation using Lagrange multipliers and KKT conditions.

  • Understand the relationship between gradients and the efficient frontier in Markowitz portfolio optimisation.

2. Partial Derivatives and Gradients – The Direction of Steepest Ascent

For a scalar-valued function f: R^n → R, the gradient is a vector of partial derivatives:
∇f(x) = [ ∂f/∂x_1, ∂f/∂x_2, ..., ∂f/∂x_n ]^T.

2.1 Interpretation of the Gradient

  • The gradient points in the direction of the steepest increase of f.

  • The negative gradient -∇f(x) points in the direction of the steepest decrease (used in Gradient Descent).

  • The magnitude ||∇f(x)|| is the rate of increase.

2.2 Directional Derivative
The rate of change of f in the direction of a unit vector v is:
D_v f(x) = ∇f(x) · v = v^T ∇f(x).
In finance, if f(w) is the Sharpe ratio of a portfolio and v is a shift in portfolio weights, the directional derivative tells you how much the Sharpe ratio changes.

2.3 Financial Application – Markowitz Optimisation
The objective function (to minimise) is the portfolio variance:
σ_p²(w) = w^T Σ w, subject to w^T 1 = 1 (weights sum to 1).
The gradient is:
∇σ_p²(w) = 2 Σ w.
Setting this to zero gives the minimum variance portfolio: Σ w = 0 → w = 0, which is invalid (sums to zero). The constrained solution is found using Lagrange multipliers (see Section 7).

3. The Jacobian Matrix – Derivatives of Vector-Valued Functions

For a vector-valued function f: R^n → R^m, where f(x) = [f_1(x), f_2(x), ..., f_m(x)]^T, the Jacobian is an (m x n) matrix:
J_f(x) = ∂f_i / ∂x_j, for i = 1..m, j = 1..n.

3.1 Interpretation
The Jacobian is the linear approximation of f at a point x_0. It maps perturbations dx in the input space to perturbations dy in the output space:
dy ≈ J_f(x_0) dx.

3.2 Financial Application – Risk Mapping
Consider a portfolio with N assets and M risk factors. The mapping from risk factors x ∈ R^M to portfolio returns y ∈ R^N is y = f(x). The Jacobian J_f is the sensitivity matrix (deltas, betas). The covariance matrix of returns is:
Σ_y = J_f Σ_x J_f^T.
This is the Delta-Normal approach to VaR.

3.3 The Chain Rule in Vectorised Form (Critical for AI)
If y = f(x) and L = g(y) is a scalar loss, then the gradient of L w.r.t x is:
∂L/∂x = (∂y/∂x)^T (∂L/∂y) = J_f^T ∇_y L.
In neural networks, this is the backpropagation algorithm. For a layer with weights W, output y = W x, the gradient w.r.t W is:
∂L/∂W = (∂L/∂y) x^T (an outer product). This is the foundation of gradient descent training.

4. The Hessian Matrix – Curvature and Convexity

For a scalar-valued function f: R^n → R, the Hessian is an (n x n) symmetric matrix of second-order partial derivatives:
H_f(x) = [ ∂²f / ∂x_i ∂x_j ].
If f is twice continuously differentiable, H_f is symmetric (∂²f/∂x_i∂x_j = ∂²f/∂x_j∂x_i).

4.1 Convexity and the Hessian

  • f is convex if H_f(x) is Positive Semi-Definite (PSD) for all x.

  • f is strictly convex if H_f(x) is Positive Definite (PD).

  • f is concave if H_f(x) is Negative Semi-Definite.

4.2 Financial Application – Loss Functions

  • MSE Loss: L(w) = (1/N) Σ (y_i - w^T x_i)². The Hessian is (2/N) X^T X, which is PSD. Therefore, MSE is convex, and its global minimum can be found analytically.

  • Cross-Entropy Loss: For logistic regression, the Hessian is (1/N) X^T D X, where D is a diagonal matrix of p_i(1-p_i) > 0. PSD → Convex.

  • Neural Networks: The loss surface is non-convex (the Hessian is indefinite). This is why we get local minima and saddle points. However, in high dimensions, saddle points dominate local minima, and SGD escapes them efficiently.

4.3 Newton’s Method
Newton’s method uses the Hessian to find the minimum of f:
x_{k+1} = x_k - H_f(x_k)^{-1} ∇f(x_k).
This converges quadratically (much faster than gradient descent) but is computationally expensive (O(n³) to invert the Hessian). Used in second-order optimisers like L-BFGS for small-scale AI models.

5. Taylor Series – Approximating Non-Linear Functions

The Taylor expansion allows us to approximate a function near a point x_0 using its derivatives. This is essential for deriving option pricing formulas and for understanding the local behaviour of neural networks.

5.1 Univariate Taylor Expansion
For a scalar f: R → R:
f(x) = f(x_0) + f'(x_0)(x - x_0) + (1/2) f''(x_0)(x - x_0)² + (1/6) f'''(x_0)(x - x_0)³ + ...
The remainder (error) for the nth-order expansion is O((x - x_0)^{n+1}).

5.2 Multivariate Taylor Expansion
For f: R^n → R:
f(x) = f(x_0) + ∇f(x_0)^T (x - x_0) + (1/2) (x - x_0)^T H_f(x_0) (x - x_0) + O(||x - x_0||³).

5.3 Financial Application – Option Pricing Approximation (Delta-Gamma)
The change in the value of an option V(S) when the underlying price S changes by ΔS is approximated by:
ΔV ≈ Δ * ΔS + (1/2) Γ * (ΔS)², where:

  • Δ = ∂V/∂S (Delta, first derivative).

  • Γ = ∂²V/∂S² (Gamma, second derivative).
    This is the Delta-Gamma approximation for VaR of options portfolios.

5.4 Financial Application – Portfolio Return Approximation
The return of a portfolio P(w) is a function of weights w. The Taylor expansion around w_0:
P(w) ≈ P(w_0) + ∇P(w_0)^T (w - w_0) + (1/2) (w - w_0)^T H_P(w_0) (w - w_0).
In a neural network portfolio optimiser, the loss function is expanded to approximate the local gradient.

6. The Gradient in High Dimensions – The Saddle Point Problem

For a differentiable function f: R^n → R, a critical point x^* satisfies ∇f(x^*) = 0. This can be:

  • Local Minimum: H_f(x^*) is Positive Definite (all eigenvalues > 0).

  • Local Maximum: H_f(x^*) is Negative Definite (all eigenvalues < 0).

  • Saddle Point: H_f(x^*) has both positive and negative eigenvalues.

6.1 Why Saddle Points Matter in AI
In high-dimensional neural networks, most critical points are saddle points, not local minima. Gradient descent gets stuck at saddle points because the gradient is zero in some directions. However, random perturbations (noise) and momentum-based optimisers (Adam) help escape saddle points.

6.2 The Hessian at Saddle Points
If the Hessian has a negative eigenvalue, there exists a direction v where v^T H v < 0. Moving slightly in that direction decreases the loss. This is the basis for Saddle-Free Newton methods.

7. Constrained Optimisation – Lagrange Multipliers and KKT Conditions

In finance, constraints are everywhere: w^T 1 = 1 (weights sum to 1), w_i ≥ 0 (no short-selling), or w^T μ ≥ μ_target (target return). We use Lagrange multipliers to handle equality constraints.

7.1 Lagrange Multipliers (Equality Constraints)
Minimise f(x) subject to g(x) = 0. The Lagrangian is:
L(x, λ) = f(x) + λ g(x).
The necessary conditions for optimality (first-order) are:

  1. ∂L/∂x = ∇f(x) + λ ∇g(x) = 0.

  2. ∂L/∂λ = g(x) = 0.

7.2 Financial Application – The Minimum Variance Portfolio
Minimise w^T Σ w subject to w^T 1 = 1.
Lagrangian: L(w, λ) = w^T Σ w + λ(w^T 1 - 1).
Gradient w.r.t w: 2 Σ w + λ 1 = 0 → w = - (λ/2) Σ^{-1} 1.
Constraint: w^T 1 = 1 → -(λ/2) 1^T Σ^{-1} 1 = 1 → -λ/2 = 1 / (1^T Σ^{-1} 1).
Therefore:
w_{MVP} = Σ^{-1} 1 / (1^T Σ^{-1} 1).
This is the closed-form solution for the minimum variance portfolio.

7.3 Karush-Kuhn-Tucker (KKT) Conditions (Inequality Constraints)
For minimising f(x) subject to g_i(x) ≤ 0 and h_j(x) = 0:

  1. Stationarity: ∇f(x*) + Σ_i μ_i ∇g_i(x*) + Σ_j λ_j ∇h_j(x*) = 0.

  2. Primal Feasibility: g_i(x*) ≤ 0, h_j(x*) = 0.

  3. Dual Feasibility: μ_i ≥ 0.

  4. Complementary Slackness: μ_i g_i(x*) = 0 for all i.

7.4 Financial Application – Portfolio with No Short-Selling
Add constraint w_i ≥ 0. The KKT conditions yield the solution where many weights are zero (the active constraints). This is solved using quadratic programming (QP) solvers like quadprog or cvxopt.

8. The Chain Rule and Backpropagation – The Engine of AI

The chain rule in vectorised form is the mathematical core of deep learning.

8.1 Forward Pass
For a neural network with layers:
z^{(1)} = W^{(1)} x + b^{(1)}
a^{(1)} = σ(z^{(1)})
…
y = a^{(L)} = σ(z^{(L)})
L = Loss(y, y_target).

8.2 Backward Pass (Vectorised Chain Rule)
∂L/∂W^{(l)} = (∂L/∂z^{(l)}) (a^{(l-1)})^T.
∂L/∂b^{(l)} = ∂L/∂z^{(l)}.
The gradient flows backward from the output to the input. This is implemented in PyTorch and TensorFlow using automatic differentiation (autograd).

8.3 The Jacobian-Vector Product (JVP) and Vector-Jacobian Product (VJP)
In deep learning, we never explicitly compute the full Jacobian (which is huge). Instead, we compute the VJP:
v^T J_f, where v is the gradient from the layer above. This is what autograd does: it efficiently computes ∂L/∂x = (∂L/∂y)^T J_f.

9. Summary for the AI Practitioner

  1. Gradients point uphill; negative gradients point downhill (Gradient Descent). The gradient of portfolio variance is 2Σw.

  2. The Jacobian maps input perturbations to output perturbations. Used for risk mapping (Delta-Normal VaR).

  3. The Hessian determines curvature. Convex loss functions (MSE, Cross-Entropy) have PSD Hessians. Neural networks are non-convex (saddle points dominate).

  4. Taylor expansions approximate non-linear functions. Delta-Gamma VaR is a Taylor expansion of an option portfolio.

  5. Lagrange multipliers solve constrained optimisation. The minimum variance portfolio has a closed-form solution using Lagrange multipliers.

  6. Backpropagation is the vectorised chain rule. The gradient of a layer’s weights is an outer product of the layer input and the incoming gradient.

In Lesson 2.3, we will move to Advanced Probability Theory, covering Conditional Expectation, Martingales, the Radon-Nikodym derivative, and the Girsanov Theorem – all of which are foundational for derivative pricing and risk-neutral valuation.

 
 
PreviousNext