Lesson 2.1: Linear Algebra – Vectors, Matrices, Eigen-Decomposition, and Singular Value Decomposition (SVD)

1. Learning Objectives

By the end of this lesson, you will be able to:

Perform rigorous vector and matrix operations including inner products, outer products, and tensor products.
Define and prove the key properties of matrix norms (Frobenius, L1, L2, Nuclear) and their financial interpretations.
Compute and interpret the determinant, trace, and eigenvalues of a covariance matrix.
Derive the Eigen-Decomposition of a symmetric positive-definite matrix and apply it to Principal Component Analysis (PCA) for factor modelling.
Derive the Singular Value Decomposition (SVD) and prove its existence for any matrix.
Apply SVD to the low-rank approximation of financial correlation matrices and denoising.
Understand the Matrix Cookbook for derivatives (used extensively in neural network backpropagation).

2. Vectors – The Fundamental Data Structure

A vector is an ordered list of numbers. In finance, a vector represents a portfolio of assets w ∈ R^N or a vector of returns r ∈ R^N.

2.1 Vector Operations and Properties

Addition: (u + v)_i = u_i + v_i. Commutative (u+v = v+u) and Associative ((u+v)+w = u+(v+w)).
Scalar Multiplication: (α u)_i = α * u_i. Distributive: α(u+v) = αu + αv.
Inner Product (Dot Product): u · v = u^T v = Σ_{i=1}^N u_i * v_i. This is the projection of u onto v. In finance, the expected return of a portfolio is E[R_p] = w^T μ, where μ is the vector of expected returns.
Outer Product: u v^T is an (N x N) matrix with entries (u v^T)_{ij} = u_i * v_j. This forms the basis for constructing covariance matrices from factor exposures.
Norm (Euclidean / L2): ||u||_2 = sqrt(Σ u_i²) = sqrt(u^T u). The length of a vector. A portfolio’s volatility is σ_p = sqrt(w^T Σ w), which is exactly an L2 norm weighted by the covariance matrix.
L1 Norm: ||u||_1 = Σ |u_i|. Used in Lasso regularisation (λ ||w||_1) to enforce sparsity in portfolio weights, forcing many weights to zero.
L∞ Norm: ||u||_∞ = max_i |u_i|. Used in robust optimisation to limit the maximum exposure to any single asset.

2.2 Linear Independence and Basis
A set of vectors {v_1, ..., v_k} is linearly independent if Σ_{i=1}^k c_i v_i = 0 implies c_i = 0 for all i. The dimension of a vector space is the maximum number of linearly independent vectors. In finance, factor models (Fama-French) represent asset returns as a linear combination of a small set of independent factors (Market, Size, Value).

3. Matrices – The Transformation Engine

A matrix A ∈ R^{m x n} has m rows and n columns. In finance, a matrix can represent a covariance matrix (N x N), a returns matrix (T x N), or a factor loading matrix (N x K).

3.1 Matrix Operations

Addition: (A+B)_{ij} = A_{ij} + B_{ij}. Only if same dimensions.
Scalar Multiplication: (αA)_{ij} = α A_{ij}.
Matrix Multiplication: (AB)_{ij} = Σ_{k=1}^p A_{ik} B_{kj}. A must be (m x p), B must be (p x n). This is the composition of linear transformations.
Transpose: (A^T)_{ij} = A_{ji}. If A is (m x n), then A^T is (n x m).
- Properties: (A^T)^T = A, (AB)^T = B^T A^T, (A+B)^T = A^T + B^T.
Trace: Tr(A) = Σ_{i=1}^n A_{ii}. The sum of diagonal elements. In finance, the trace of a covariance matrix is the total variance across all assets. Tr(Σ) = Σ_i σ_i².
- Properties: Tr(A+B) = Tr(A) + Tr(B), Tr(AB) = Tr(BA) (cyclic property). This property is critical for deriving derivatives in neural networks.

3.2 Matrix Norms

Frobenius Norm: ||A||_F = sqrt(Σ_i Σ_j A_{ij}²) = sqrt(Tr(A^T A)). This measures the total magnitude of all elements. Used in regularising neural network weights (weight decay).
Spectral Norm (L2 Norm for Matrices): ||A||_2 = max_{x ≠ 0} (||Ax||_2 / ||x||_2) = sqrt(λ_max(A^T A)). This is the maximum amplification factor. Used in controlling the Lipschitz constant of neural networks (spectral normalisation for GANs).
Nuclear Norm: ||A||_* = Σ_i σ_i(A), where σ_i are the singular values. This is the sum of singular values. Used in matrix completion problems (e.g., imputing missing covariance matrix entries) because it promotes low-rank solutions.

3.3 Determinant
The determinant det(A) is a scalar that measures the volume scaling factor of the linear transformation. It is non-zero if and only if A is invertible.

For 2×2: det( [[a, b], [c, d]] ) = ad - bc.
Properties: det(AB) = det(A)det(B), det(A^T) = det(A), det(A^{-1}) = 1/det(A).
Finance Application: The determinant of a covariance matrix det(Σ) is the Generalised Variance. If det(Σ) = 0, the matrix is singular (some assets are perfectly collinear). In high-dimensional finance, covariance matrices are often nearly singular, requiring shrinkage (Ledoit-Wolf) to make them invertible.

4. Eigenvalues and Eigenvectors – The Anatomy of Covariance

For a square matrix A ∈ R^{n x n}, a scalar λ and a non-zero vector v satisfying Av = λv are called an eigenvalue and eigenvector, respectively.

4.1 Characteristic Equation
det(A - λI) = 0. This polynomial in λ has degree n. Its roots are the eigenvalues.

4.2 Properties of Eigenvalues

The trace is the sum of eigenvalues: Tr(A) = Σ_i λ_i.
The determinant is the product of eigenvalues: det(A) = Π_i λ_i.
If A is symmetric (A = A^T), all eigenvalues are real, and the eigenvectors are orthogonal.
If A is positive definite (x^T A x > 0 for all x ≠ 0), all eigenvalues are positive. Covariance matrices are positive semi-definite (eigenvalues ≥ 0).

4.3 Eigen-Decomposition (Spectral Decomposition)
If A is symmetric, it can be written as:
A = Q Λ Q^T
where:

Q is an orthogonal matrix (Q^T Q = I) whose columns are the eigenvectors.
Λ is a diagonal matrix containing the eigenvalues λ_1 ≥ λ_2 ≥ ... ≥ λ_n.

4.4 Financial Application – Principal Component Analysis (PCA)
PCA is used to reduce the dimensionality of a financial return matrix R ∈ R^{T x N}.

Compute the covariance matrix Σ = (1/(T-1)) R_c^T R_c, where R_c is the centered return matrix.
Eigen-decompose Σ = Q Λ Q^T.
The principal components (factors) are F = R_c Q. Each column of F is a portfolio (the eigenvectors). The first principal component explains the largest fraction of total variance: λ_1 / Σ_i λ_i.
PCA in AI: Instead of feeding N=500 asset returns into a neural network, feed the first K=10 principal components. This drastically reduces input dimensionality, stabilises training, and acts as a natural denoiser (the last components are mostly noise).

4.5 Matrix Powers
If A = Q Λ Q^T, then A^p = Q Λ^p Q^T. This is used to compute the square root of a covariance matrix:
Σ^{1/2} = Q Λ^{1/2} Q^T. This is essential for generating correlated random returns in Monte Carlo simulations (Cholesky decomposition is a special case).

5. Singular Value Decomposition (SVD) – The Master Decomposition

While eigen-decomposition works only for square matrices, SVD works for any matrix A ∈ R^{m x n}.

5.1 The SVD Theorem
Any matrix A can be factorised as:
A = U Σ V^T
where:

U ∈ R^{m x m} is an orthogonal matrix (U^T U = I). Its columns are the left singular vectors.
V ∈ R^{n x n} is an orthogonal matrix (V^T V = I). Its columns are the right singular vectors.
Σ ∈ R^{m x n} is a diagonal matrix (with the same dimensions as A) containing the singular values σ_1 ≥ σ_2 ≥ ... ≥ σ_r > 0, where r = rank(A).

5.2 Relationship to Eigen-Decomposition

The singular values of A are the square roots of the eigenvalues of A^T A (or A A^T).
σ_i = sqrt(λ_i(A^T A)).

5.3 Low-Rank Approximation (Eckart-Young Theorem)
The best rank-k approximation of A (in the Frobenius norm sense) is:
A_k = U_k Σ_k V_k^T, where U_k and V_k are the first k columns of U and V, and Σ_k is the diagonal matrix of the top k singular values.
The approximation error is:
||A - A_k||_F = sqrt( Σ_{i=k+1}^{r} σ_i² ).

5.4 Financial Applications of SVD

Application 1: Denoising Covariance Matrices
A sample covariance matrix Σ_sample is noisy. We can denoise it using SVD:

Compute the returns matrix R ∈ R^{T x N}.
Perform SVD: R = U Σ V^T.
The covariance matrix is Σ = R^T R / (T-1) = V (Σ^T Σ / (T-1)) V^T.
Replace small singular values (below a threshold) with zero. This removes noise components.
Reconstruct a denoised covariance matrix. This is known as Random Matrix Theory (RMT) denoising and is widely used in portfolio optimisation.

Application 2: Matrix Completion (Imputing Missing Returns)
If some asset returns are missing, we can formulate the problem as:
minimize_X rank(X) subject to X_{ij} = R_{ij} for observed entries.
The convex relaxation is the nuclear norm minimisation:
minimize_X ||X||_* subject to X_{ij} = R_{ij}.
This is solved using iterative SVD (soft-thresholding of singular values). Used in constructing survivorship-bias-free datasets.

Application 3: Collaborative Filtering for Asset Selection
Given a user-returns matrix (users are strategies, assets are stocks), SVD finds latent factors that explain which strategies work on which assets. This is the foundation of Style Investing.

Application 4: Factor Model Identification (SVD vs PCA)
PCA on the covariance matrix is equivalent to SVD on the centered return matrix R_c:
R_c = U Σ V^T. The loadings matrix is V (eigenvectors of R_c^T R_c), and the factor scores are F = U Σ.

6. The Matrix Cookbook – Essential Derivatives for AI

When training neural networks, we compute gradients of a scalar loss L with respect to weight matrices W. The following identities are non-negotiable.

6.1 Derivative of a Quadratic Form
Given symmetric matrix A:
∂/∂x (x^T A x) = 2 A x.

Proof: x^T A x = Σ_i Σ_j A_{ij} x_i x_j. Derivative w.r.t x_k gives Σ_j A_{kj} x_j + Σ_i A_{ik} x_i = 2 Σ_i A_{ki} x_i.
Finance Application: The portfolio variance is σ_p² = w^T Σ w. The gradient of variance w.r.t portfolio weights is ∂σ_p²/∂w = 2 Σ w. This is used in gradient-based portfolio optimisation.

6.2 Derivative of a Linear Form
∂/∂x (a^T x) = a.

6.3 Derivative of a Matrix Trace
∂/∂A Tr(AB) = B^T.
∂/∂A Tr(A^T B) = B.
These identities are used in deriving the gradient for neural network layers.

6.4 Derivative of the Log-Determinant
∂/∂A ln det(A) = A^{-1} (for symmetric positive definite A).

Finance Application: In estimating covariance matrices using Maximum Likelihood, the log-likelihood includes ln det(Σ). The MLE solution sets the derivative to zero, yielding the sample covariance matrix.

7. Special Matrix Types in Finance

7.1 Positive Definite (PD) and Positive Semi-Definite (PSD)

A matrix A is PSD if x^T A x ≥ 0 for all x. Covariance matrices are PSD.
A matrix is PD if x^T A x > 0 for all x ≠ 0. This means the covariance matrix is invertible (no assets are perfectly correlated).

7.2 Correlation Matrix
A correlation matrix C has diagonal entries C_{ii} = 1 and off-diagonal entries C_{ij} = ρ_{ij} ∈ [-1, 1]. It must be PSD.

7.3 Toeplitz and Hankel Matrices

Toeplitz: Constant along diagonals. A_{ij} = f(i-j). Used in modelling autocorrelation structures.
Hankel: Constant along anti-diagonals. Used in time-series embeddings.

7.4 Structured Covariance Matrices (Factor Models)
The covariance matrix of N assets under a K-factor model (K << N):
Σ = B Λ B^T + Ψ
where:

B is the (N x K) factor loading matrix.
Λ is the (K x K) factor covariance matrix.
Ψ is a diagonal matrix of idiosyncratic variances.
This structure allows efficient inversion using the Woodbury Matrix Identity:
(B Λ B^T + Ψ)^{-1} = Ψ^{-1} - Ψ^{-1} B (Λ^{-1} + B^T Ψ^{-1} B)^{-1} B^T Ψ^{-1}.
This is used to compute the inverse of large covariance matrices in seconds rather than hours.

8. Summary for the AI Practitioner

Vectors and Matrices are the language of finance. A portfolio is a vector; a covariance matrix is a matrix.
Norms matter: L2 for volatility, L1 for sparsity (Lasso), Frobenius for weight decay, Nuclear for low-rank matrix completion.
Eigen-Decomposition is PCA. It gives you the directions of maximum variance (the principal factors). Use it to reduce input dimensionality for your neural network.
SVD is the master decomposition. It denoises covariance matrices, imputes missing data, and identifies latent factors. Every AI engineer must know SVD.
Matrix derivatives are non-negotiable. You must know ∂(x^T A x)/∂x = 2Ax and the trace derivatives. These are the foundation of backpropagation.
Factor Models + Woodbury Identity make large-scale covariance inversion tractable. Use this in portfolio optimisation within AI models.

Diploma in Artificial Intelligence in Finance Lesson 2.1: Linear Algebra – Vectors, Matrices, Eigen-Decomposition, and Singular Value Decomposition (SVD)

Diploma in Artificial Intelligence in Finance