1. Learning Objectives

By the end of this lesson, you will be able to:

• Rigorously define convex sets, convex functions, and prove the conditions for convexity using first-order and second-order characterisations.

• Derive the gradient descent algorithm from a Taylor series approximation and prove its convergence under strong convexity.

• Implement and compare Batch Gradient Descent, Stochastic Gradient Descent (SGD), Mini-Batch SGD, and advanced optimisers (Momentum, RMSprop, Adam).

• Apply Lagrange multipliers and Karush-Kuhn-Tucker (KKT) conditions to solve constrained optimisation problems in finance.

• Derive the closed-form solution for the minimum variance portfolio and the tangency portfolio.

• Understand the duality between primal and dual problems and its application in Support Vector Machines and robust portfolio optimisation.

• Analyse the convergence properties of optimisation algorithms and their practical implications for training financial AI models.

---

2. Convex Sets and Convex Functions – The Foundation of Tractable Optimisation

Convex optimisation problems have a unique global minimum and can be solved efficiently. Most financial optimisation problems (portfolio optimisation, linear regression) are convex. Neural network training is non-convex.

2.1 Convex Sets
A set C ⊆ R^n is convex if for any x, y ∈ C and any λ ∈ [0, 1]:
λx + (1-λ)y ∈ C.
Interpretation: The line segment connecting any two points in the set lies entirely within the set.
Financial Example: The set of all portfolio weights {w : w^T 1 = 1, w_i ≥ 0} (no short-selling constraint) is convex.

2.2 Convex Functions
A function f: R^n → R is convex if its domain is convex and for all x, y in the domain and λ ∈ [0, 1]:
f(λx + (1-λ)y) ≤ λ f(x) + (1-λ) f(y).
Interpretation: The function lies below the chord connecting any two points on its graph.

2.3 First-Order Condition for Convexity
If f is differentiable, f is convex if and only if for all x, y:
f(y) ≥ f(x) + ∇f(x)^T (y - x).
Interpretation: The tangent line at any point lies below the function. This is the basis for gradient descent: the negative gradient provides a descent direction.

2.4 Second-Order Condition for Convexity
If f is twice differentiable, f is convex if and only if its Hessian H_f(x) is Positive Semi-Definite (PSD) for all x:
H_f(x) ⪰ 0 (i.e., v^T H_f(x) v ≥ 0 for all v).
Financial Examples of Convex Loss Functions:

• MSE Loss: L(w) = (1/N) ||y - Xw||². Hessian = (2/N) X^T X which is PSD.

• Cross-Entropy Loss (Logistic Regression): Hessian = (1/N) X^T D X where D is a diagonal matrix of p_i(1-p_i) > 0. PSD.

• L1 Regularised Loss (Lasso): L(w) = MSE + λ ||w||_1. The L1 norm is convex (though not differentiable at zero).

• Portfolio Variance: σ_p²(w) = w^T Σ w. Hessian = 2Σ which is PSD.

2.5 Strict Convexity and Strong Convexity

• Strictly Convex: f(λx + (1-λ)y) < λ f(x) + (1-λ) f(y) for x ≠ y. Guarantees a unique global minimum.

• Strongly Convex: There exists m > 0 such that H_f(x) ⪰ m I for all x. This guarantees linear or quadratic convergence of gradient descent. The portfolio variance with positive definite Σ is strongly convex.

---

3. Gradient Descent – The Workhorse of AI

Gradient descent (GD) iteratively moves in the direction of the negative gradient to minimise f(w).

3.1 The Update Rule
w_{t+1} = w_t - η_t ∇f(w_t)
where η_t is the learning rate (step size) at iteration t.

3.2 Derivation from Taylor Expansion
From the first-order Taylor expansion:
f(w_t + d) ≈ f(w_t) + ∇f(w_t)^T d.
To minimise the linear approximation subject to ||d|| ≤ ε (a trust region), choose d = -η ∇f(w_t). The negative gradient is the steepest descent direction.

3.3 Convergence of Gradient Descent
Assume f is L-smooth (Lipschitz continuous gradient): ||∇f(x) - ∇f(y)|| ≤ L ||x - y||.
With a fixed step size η = 1/L, gradient descent satisfies:
f(w_t) - f(w^*) ≤ ( L ||w_0 - w^*||² ) / (2t).
This is a sublinear convergence rate O(1/t).

3.4 Convergence under Strong Convexity
If f is m-strongly convex and L-smooth, with η = 1/L:
||w_t - w^*||² ≤ (1 - m/L)^t ||w_0 - w^*||².
This is linear convergence (exponential decay). The ratio κ = L/m is the condition number. If κ is large (ill-conditioned), convergence is slow.

---

4. Stochastic Gradient Descent (SGD) – Scaling to Big Data

For large datasets, computing the full gradient ∇f(w) = (1/N) Σ_{i=1}^N ∇f_i(w) is expensive. SGD uses a single random sample (or mini-batch) to approximate the gradient.

4.1 The SGD Update Rule
w_{t+1} = w_t - η_t ∇f_{i_t}(w_t)
where i_t is randomly sampled from {1, ..., N}.
The expected value of the stochastic gradient equals the full gradient:
E[ ∇f_{i_t}(w_t) ] = ∇f(w_t).
So SGD is an unbiased estimator of the true gradient.

4.2 Convergence of SGD
SGD does not converge to the exact minimum; it oscillates around it. With a diminishing step size η_t = η_0 / sqrt(t), SGD converges to a neighbourhood of the minimum:
E[ f(w_t) - f(w^*) ] ≤ O(1/sqrt(t)).
This is slower than full GD but much faster per iteration for large datasets.

4.3 Mini-Batch SGD
Instead of one sample, use a batch of B samples:
w_{t+1} = w_t - η_t * (1/B) Σ_{i ∈ Batch} ∇f_i(w_t).

• B = 1: Pure SGD (high variance, fast).

• B = N: Full Batch GD (low variance, slow for large N).

• B = 32, 64, 128, 256: Typical choices for deep learning. This balances noise and computational efficiency. Mini-batch gradients can be computed in parallel on GPUs.

4.4 Financial Application – Online Portfolio Optimisation
In high-frequency trading, data arrives continuously. We can use SGD to update portfolio weights in real-time:
w_{t+1} = w_t - η_t ∇L(w_t; r_t).
This allows the portfolio to adapt to new market conditions without retraining on the entire history.

---

5. Advanced Optimisers – Momentum and Adaptive Learning Rates

5.1 Momentum (Heavy Ball Method)
Momentum accelerates gradient descent by accumulating a velocity term.
v_{t+1} = β v_t + (1-β) ∇f(w_t)
w_{t+1} = w_t - η_t v_{t+1}
where β ∈ (0,1) (typically 0.9).
Interpretation: The velocity v_t is a moving average of past gradients. It dampens oscillations and accelerates convergence in directions of low curvature. In financial terms, momentum smooths out noisy gradient estimates from volatile asset returns.

5.2 Nesterov Accelerated Gradient (NAG)
NAG is a look-ahead version of momentum. It computes the gradient at the look-ahead position w_t - β v_t:
v_{t+1} = β v_t + (1-β) ∇f(w_t - β v_t).
w_{t+1} = w_t - η_t v_{t+1}.
NAG has better convergence guarantees (improves from O(1/t) to O(1/t²) for smooth convex functions).

5.3 AdaGrad (Adaptive Gradient)
AdaGrad adapts the learning rate for each parameter independently using the sum of squared past gradients:
G_t = Σ_{s=1}^t (∇f(w_s))² (element-wise square).
w_{t+1} = w_t - (η / sqrt(G_t + ε)) ⊙ ∇f(w_t).
Parameters with large gradients get smaller learning rates; parameters with small gradients get larger learning rates.
Financial Application: If some asset features have high variance (e.g., small-cap stocks), AdaGrad automatically reduces their learning rate, preventing the model from overreacting to their noise.

5.4 RMSprop
RMSprop is a variant of AdaGrad that uses an exponentially weighted moving average instead of accumulating all past squared gradients:
E[g²]_t = β E[g²]_{t-1} + (1-β) (∇f(w_t))².
w_{t+1} = w_t - (η / sqrt(E[g²]_t + ε)) ⊙ ∇f(w_t).
This prevents the learning rate from decaying to zero, which is an issue with AdaGrad.

5.5 Adam (Adaptive Moment Estimation) – The Current Standard
Adam combines momentum and RMSprop. It maintains both a moving average of the gradient (m_t, first moment) and a moving average of the squared gradient (v_t, second moment).
m_t = β_1 m_{t-1} + (1-β_1) ∇f(w_t).
v_t = β_2 v_{t-1} + (1-β_2) (∇f(w_t))².
Bias correction (to account for zero initialisation):
\hat{m}_t = m_t / (1 - β_1^t).
\hat{v}_t = v_t / (1 - β_2^t).
Update:
w_{t+1} = w_t - η * \hat{m}_t / (sqrt(\hat{v}_t) + ε).
Default parameters: β_1 = 0.9, β_2 = 0.999, ε = 10^{-8}.
Adam is robust, works well on noisy financial data, and is the default optimiser for most deep learning frameworks (PyTorch, TensorFlow).

---

6. Constrained Optimisation – Lagrange Multipliers

Most financial problems have constraints: weights sum to 1, no short-selling, target return constraints.

6.1 Equality Constraints – Lagrange Multipliers
Minimise f(x) subject to g_i(x) = 0 for i = 1, ..., m.
The Lagrangian is:
L(x, λ) = f(x) + Σ_{i=1}^m λ_i g_i(x).
The necessary conditions for optimality (first-order) are:

1. ∂L/∂x_j = ∂f/∂x_j + Σ_i λ_i ∂g_i/∂x_j = 0 for all j.

2. ∂L/∂λ_i = g_i(x) = 0 for all i.

6.2 Financial Application – Minimum Variance Portfolio (Full Derivation)
Minimise f(w) = w^T Σ w subject to g(w) = w^T 1 - 1 = 0.
Lagrangian: L(w, λ) = w^T Σ w + λ(w^T 1 - 1).
Gradient w.r.t w: 2 Σ w + λ 1 = 0 → w = -(λ/2) Σ^{-1} 1.
Constraint: w^T 1 = 1 → -(λ/2) 1^T Σ^{-1} 1 = 1 → -λ/2 = 1 / (1^T Σ^{-1} 1).
Therefore:
w_{MVP} = Σ^{-1} 1 / (1^T Σ^{-1} 1).
This is the closed-form solution for the minimum variance portfolio.

6.3 Financial Application – Tangency Portfolio (Maximum Sharpe Ratio)
Maximise SR(w) = (w^T μ - r_f) / sqrt(w^T Σ w).
This is equivalent to minimising w^T Σ w subject to w^T (μ - r_f 1) = 1.
Let α = μ - r_f 1.
Lagrangian: L(w, λ) = w^T Σ w + λ(w^T α - 1).
Gradient: 2 Σ w + λ α = 0 → w = -(λ/2) Σ^{-1} α.
Constraint: w^T α = 1 → -(λ/2) α^T Σ^{-1} α = 1 → -λ/2 = 1 / (α^T Σ^{-1} α).
Therefore:
w_{Tangency} = Σ^{-1} (μ - r_f 1) / ( (μ - r_f 1)^T Σ^{-1} (μ - r_f 1) ).
This is the optimal risky portfolio according to the Capital Asset Pricing Model (CAPM).

---

7. Inequality Constraints – Karush-Kuhn-Tucker (KKT) Conditions

For problems with inequality constraints g_i(x) ≤ 0 and equality constraints h_j(x) = 0, the KKT conditions are necessary for optimality (and sufficient under convexity).

7.1 KKT Conditions
For minimising f(x):

1. Stationarity: ∇f(x*) + Σ_i μ_i ∇g_i(x*) + Σ_j λ_j ∇h_j(x*) = 0.

2. Primal Feasibility: g_i(x*) ≤ 0 for all i; h_j(x*) = 0 for all j.

3. Dual Feasibility: μ_i ≥ 0 for all i.

4. Complementary Slackness: μ_i g_i(x*) = 0 for all i.

7.2 Interpretation

• If g_i(x*) < 0 (constraint is inactive), then μ_i = 0 (the constraint does not affect the optimum).

• If g_i(x*) = 0 (constraint is active), then μ_i ≥ 0 (the constraint affects the optimum).

7.3 Financial Application – No Short-Selling Constraint
Minimise w^T Σ w subject to w^T 1 = 1 and w_i ≥ 0 for all i.
The KKT conditions yield:
2 Σ w + λ 1 - μ = 0, where μ_i ≥ 0 and μ_i w_i = 0.
If w_i > 0 (asset is held), then μ_i = 0, and the asset satisfies the unconstrained condition. If μ_i > 0, then w_i = 0 (asset is not held). This is solved using Quadratic Programming (QP) solvers. The solution yields a sparse portfolio where many weights are exactly zero.

7.4 Financial Application – Target Return Constraint
Minimise w^T Σ w subject to w^T 1 = 1 and w^T μ ≥ μ_target.
If the target return constraint is active (w^T μ = μ_target), the KKT conditions yield:
w = Σ^{-1} (λ_1 1 + λ_2 μ), where λ_1 and λ_2 are chosen to satisfy the two equality constraints. This is the Efficient Frontier.

---

8. Duality – The Primal and Dual Problems

Duality transforms a constrained primal problem (often difficult) into an unconstrained dual problem (often easier).

8.1 Lagrangian Dual
Given primal problem: minimise f(x) subject to g_i(x) ≤ 0.
The Lagrangian is L(x, μ) = f(x) + Σ_i μ_i g_i(x).
The dual function is D(μ) = min_x L(x, μ).
The dual problem is: maximise D(μ) subject to μ_i ≥ 0.
Weak duality: Dual ≤ Primal always holds.
Strong duality (Slater’s condition): if the primal is convex and strictly feasible, then Dual = Primal.

8.2 Financial Application – Support Vector Machines (SVM) in Finance
SVM is used for classification (e.g., predicting market direction). The primal problem is:
Minimise ||w||² subject to y_i (w^T x_i - b) ≥ 1 for all i.
The dual problem is:
Maximise Σ_i α_i - (1/2) Σ_i Σ_j α_i α_j y_i y_j x_i^T x_j subject to Σ_i α_i y_i = 0 and α_i ≥ 0.
The dual is a convex quadratic program (QP) that is easier to solve. The decision boundary depends only on support vectors (α_i > 0). This is the basis for kernel methods in financial classification tasks.

8.3 Financial Application – Robust Portfolio Optimisation
In robust optimisation, we assume the expected return vector μ is uncertain and lies in an ellipsoidal uncertainty set U = {μ : (μ - \hat{μ})^T Σ_μ^{-1} (μ - \hat{μ}) ≤ κ²}.
The primal robust problem is:
Maximise min_{μ ∈ U} (w^T μ) subject to w^T 1 = 1.
The robust counterpart is:
Maximise w^T \hat{μ} - κ sqrt(w^T Σ_μ w) subject to w^T 1 = 1.
This is a convex optimisation problem. The dual formulation yields a solution that is robust to estimation errors in μ.

---

9. Numerical Optimisation in Practice

9.1 Constraint Handling in AI
In PyTorch/TensorFlow, constraints (e.g., w_i ≥ 0) are often enforced by reparameterisation:

• w = softmax(θ) to enforce w_i ≥ 0 and Σ w_i = 1. This is differentiable and gradient descent can be applied directly to θ.

• w = σ(θ) to enforce w_i ∈ [0, 1].

9.2 Equality Constraints via Penalty Methods
Instead of hard constraints, add a penalty term to the loss:
L_penalty(w) = L(w) + λ * (w^T 1 - 1)².
As λ → ∞, the constraint is satisfied exactly. This is the basis for Augmented Lagrangian and Alternating Direction Method of Multipliers (ADMM).

9.3 ADMM in Finance – Large-Scale Portfolio Optimisation
ADMM decomposes large optimisation problems into smaller subproblems. For portfolio optimisation:
min_w (w^T Σ w) subject to w^T 1 = 1, w_i ≥ 0.
ADMM introduces an auxiliary variable z = w and solves:
min_w (w^T Σ w) + (ρ/2) ||w - z||² subject to z^T 1 = 1, z_i ≥ 0.
The w-update is a quadratic problem; the z-update is a projection onto the simplex. ADMM is highly efficient for thousands of assets.

---

10. Summary for the AI Practitioner

1. Convexity ensures a unique global minimum. Portfolio variance (w^T Σ w) is convex. MSE loss is convex. Use convex optimisation for reliable results.

2. Gradient Descent is the foundation of AI training. Full GD is slow; SGD and Mini-Batch SGD are fast and scalable.

3. Momentum and Adam are essential for training deep networks on noisy financial data. Adam is the default choice.

4. Lagrange multipliers give closed-form solutions for mean-variance optimisation. The Minimum Variance Portfolio is w_{MVP} = Σ^{-1} 1 / (1^T Σ^{-1} 1).

5. KKT conditions handle inequality constraints (e.g., no short-selling). The solution is sparse (many weights zero).

6. Duality transforms hard primal problems into easier dual problems. Used in SVM and robust portfolio optimisation.

7. Reparameterisation (softmax, sigmoid) makes constrained optimisation differentiable for neural networks.

8. ADMM solves large-scale problems efficiently by decomposing them.