1. Learning Objectives

By the end of this lesson, you will be able to:

• Understand the biological and mathematical foundations of artificial neural networks.

• Derive the universal approximation theorem and its implications for financial modelling.

• Implement feedforward neural networks from scratch in PyTorch.

• Understand and select appropriate activation functions for financial problems (ReLU, Leaky ReLU, ELU, Swish, GELU).

• Derive the backpropagation algorithm using the chain rule in vectorised form.

• Implement custom loss functions for financial objectives (Sharpe Ratio loss, Quantile loss).

• Understand the vanishing and exploding gradient problems and their solutions.

• Apply neural networks to financial regression and classification tasks.

2. The Biological and Mathematical Foundations of Neural Networks

2.1 The Biological Inspiration

• Neurons receive signals through dendrites.

• Signals are integrated at the cell body (soma).

• If the integrated signal exceeds a threshold, the neuron fires an action potential through the axon.

• Synapses modulate the strength of connections between neurons.

2.2 The Mathematical Neuron (Perceptron)

Input: x = [x_1, x_2, ..., x_n]
Weights: w = [w_1, w_2, ..., w_n]
Bias: b
Weighted Sum: z = Σ_{i=1}^n w_i x_i + b = w^T x + b
Activation: a = f(z)
Output: y = a

2.3 The Universal Approximation Theorem (UAT)
The UAT states that a feedforward neural network with a single hidden layer containing a finite number of neurons can approximate any continuous function on a compact subset of R^n, provided the activation function is non-constant, bounded, and continuous.

Mathematical Statement:
For any continuous function f: K → R (where K ⊆ R^n is compact) and any ε > 0, there exists a neural network with one hidden layer such that:
| f(x) - Σ_{i=1}^{N} α_i σ(w_i^T x + b_i) | < ε for all x ∈ K.

Financial Implication: Neural networks can approximate any pricing function, risk metric, or trading strategy. The challenge is not representational power but generalisation.

3. Feedforward Neural Network Architecture

3.1 Layer Structure

Input Layer:    d_0 neurons (features)
Hidden Layer 1: d_1 neurons
Hidden Layer 2: d_2 neurons
...
Hidden Layer L: d_L neurons
Output Layer:   d_{L+1} neurons (predictions)

Each layer: z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}
           a^{(l)} = σ(z^{(l)})

3.2 Weight Matrices and Biases

• W^{(l)} ∈ R^{d_l x d_{l-1}}

• b^{(l)} ∈ R^{d_l}

• a^{(0)} = x (input)

• y = a^{(L+1)} (output)

3.3 Architecture Design Principles for Finance

4. Activation Functions – Mathematical Formulations

4.1 Sigmoid (Logistic)
σ(x) = 1 / (1 + e^{-x})

Domain: (-∞, ∞) → Range: (0, 1)

Derivative: σ'(x) = σ(x) * (1 - σ(x))

Pros: Smooth, probabilistic interpretation.
Cons: Vanishing gradients (derivative → 0 for large |x|). Not zero-centered.

Financial Use: Output layer for binary classification (probability of default).

4.2 Tanh (Hyperbolic Tangent)
tanh(x) = (e^x - e^{-x}) / (e^x + e^{-x})

Domain: (-∞, ∞) → Range: (-1, 1)

Derivative: tanh'(x) = 1 - tanh²(x)

Pros: Zero-centered (better for gradient descent).
Cons: Vanishing gradients.

Financial Use: Hidden layers in smaller networks.

4.3 ReLU (Rectified Linear Unit)
ReLU(x) = max(0, x)

Domain: (-∞, ∞) → Range: [0, ∞)

Derivative:
ReLU'(x) = 1 if x > 0, 0 if x < 0, undefined at x = 0.

Pros: Computationally efficient. No vanishing gradient for positive inputs. Sparse activation.
Cons: Dead ReLU problem (neurons can die and never activate). Not zero-centered.

Financial Use: Default for hidden layers in most financial neural networks.

4.4 Leaky ReLU
LeakyReLU(x) = max(αx, x) where α ∈ (0, 1) (typically 0.01)

Derivative:
LeakyReLU'(x) = 1 if x > 0, α if x < 0.

Pros: Prevents dead ReLU neurons. Small negative slope maintains gradient flow.
Cons: Extra hyperparameter (α).

4.5 ELU (Exponential Linear Unit)
ELU(x) = x if x > 0, α(e^x - 1) if x ≤ 0

Derivative:
ELU'(x) = 1 if x > 0, ELU(x) + α if x < 0.

Pros: Smooth, zero-centered, robust to outliers.
Cons: Exponential computation (slower).

4.6 Swish / SiLU
Swish(x) = x * σ(x) = x / (1 + e^{-x})

Derivative: Swish'(x) = σ(x) + x * σ(x) * (1 - σ(x))

Pros: Smooth, non-monotonic, can improve performance.
Cons: More computationally expensive.

4.7 GELU (Gaussian Error Linear Unit)
GELU(x) = x * Φ(x) where Φ is the standard normal CDF.

Pros: Used in Transformers (BERT, GPT). Smooth, probabilistic.
Cons: Complex, requires approximation.

4.8 Implementation

import torch
import torch.nn as nn

class ActivationFunctions:
    @staticmethod
    def sigmoid(x):
        return torch.sigmoid(x)

    @staticmethod
    def tanh(x):
        return torch.tanh(x)

    @staticmethod
    def relu(x):
        return torch.relu(x)

    @staticmethod
    def leaky_relu(x, alpha=0.01):
        return torch.nn.functional.leaky_relu(x, alpha)

    @staticmethod
    def elu(x, alpha=1.0):
        return torch.nn.functional.elu(x, alpha)

    @staticmethod
    def swish(x):
        return x * torch.sigmoid(x)

    @staticmethod
    def gelu(x):
        return torch.nn.functional.gelu(x)

4.9 Choosing Activation Functions for Finance

5. Backpropagation – The Chain Rule in Vectorised Form

Backpropagation computes the gradient of the loss function with respect to all weights and biases using the chain rule.

5.1 Forward Pass (for a single sample)

a^{(0)} = x
z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}  for l = 1, ..., L
a^{(l)} = σ(z^{(l)})                    for l = 1, ..., L
y_hat = a^{(L)}
L = Loss(y, y_hat)

5.2 Backward Pass (Gradient Computation)

Step 1: Gradient at output layer
δ^{(L)} = ∂L/∂z^{(L)} = ∂L/∂a^{(L)} * σ'(z^{(L)})

Step 2: Gradient for hidden layers (recursive)
δ^{(l)} = (W^{(l+1)})^T δ^{(l+1)} * σ'(z^{(l)})

Step 3: Gradient of weights and biases
∂L/∂W^{(l)} = δ^{(l)} (a^{(l-1)})^T
∂L/∂b^{(l)} = δ^{(l)}

5.3 Vectorised Form for Mini-Batches

For a mini-batch of size m:
A^{(0)} = X  (m x d_0)
Z^{(l)} = A^{(l-1)} (W^{(l)})^T + b^{(l)}
A^{(l)} = σ(Z^{(l)})

Backward:
D^{(L)} = ∂L/∂A^{(L)} * σ'(Z^{(L)})
D^{(l)} = D^{(l+1)} W^{(l+1)} * σ'(Z^{(l)})

Gradients:
∂L/∂W^{(l)} = (D^{(l)})^T A^{(l-1)} / m
∂L/∂b^{(l)} = sum(D^{(l)}, axis=0) / m

5.4 Implementation of Backpropagation from Scratch

class NeuralNetwork:
    def __init__(self, layer_dims, activation='relu'):
        self.layer_dims = layer_dims
        self.activation = activation
        self.weights = []
        self.biases = []

        # Xavier initialisation
        for i in range(1, len(layer_dims)):
            w = np.random.randn(layer_dims[i-1], layer_dims[i]) * np.sqrt(2 / layer_dims[i-1])
            b = np.zeros((1, layer_dims[i]))
            self.weights.append(w)
            self.biases.append(b)

    def _activation(self, z):
        if self.activation == 'relu':
            return np.maximum(0, z)
        elif self.activation == 'tanh':
            return np.tanh(z)
        elif self.activation == 'sigmoid':
            return 1 / (1 + np.exp(-z))
        else:
            return z

    def _activation_derivative(self, z):
        if self.activation == 'relu':
            return (z > 0).astype(float)
        elif self.activation == 'tanh':
            return 1 - np.tanh(z)**2
        elif self.activation == 'sigmoid':
            s = 1 / (1 + np.exp(-z))
            return s * (1 - s)
        else:
            return np.ones_like(z)

    def forward(self, X):
        self.activations = [X]
        self.z_values = []

        for w, b in zip(self.weights, self.biases):
            z = self.activations[-1] @ w + b
            self.z_values.append(z)
            a = self._activation(z)
            self.activations.append(a)

        return self.activations[-1]

    def backward(self, y, y_hat, learning_rate=0.01):
        m = y.shape[0]

        # Gradient of loss (MSE) w.r.t output
        dA = (y_hat - y) / m

        # Backpropagate through layers
        for l in range(len(self.weights) - 1, -1, -1):
            dZ = dA * self._activation_derivative(self.z_values[l])
            dW = self.activations[l].T @ dZ
            db = np.sum(dZ, axis=0, keepdims=True)

            # Update weights and biases
            self.weights[l] -= learning_rate * dW
            self.biases[l] -= learning_rate * db

            # Compute dA for next layer (if not first layer)
            if l > 0:
                dA = dZ @ self.weights[l].T

5.5 Backpropagation in PyTorch (Autograd)

# PyTorch handles backpropagation automatically
import torch
import torch.nn as nn
import torch.optim as optim

# Define model
model = nn.Sequential(
    nn.Linear(50, 128),
    nn.ReLU(),
    nn.Linear(128, 64),
    nn.ReLU(),
    nn.Linear(64, 1)
)

# Loss and optimizer
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Training loop
for epoch in range(100):
    # Forward pass
    y_pred = model(X_train)
    loss = criterion(y_pred, y_train)

    # Backward pass (autograd)
    optimizer.zero_grad()
    loss.backward()

    # Update weights
    optimizer.step()

6. Custom Loss Functions for Financial AI

6.1 Sharpe Ratio Loss

def sharpe_loss(y_pred, y_true, risk_free=0.0):
    """
    Maximise Sharpe Ratio (minimise negative Sharpe).
    """
    returns = y_pred * y_true  # Position * actual return
    mean_return = returns.mean()
    std_return = returns.std(unbiased=False) + 1e-8
    sharpe = (mean_return - risk_free) / std_return
    return -sharpe  # Minimise negative Sharpe

6.2 Quantile Loss (Pinball Loss)

def quantile_loss(y_pred, y_true, quantile=0.95):
    """
    Pinball loss for quantile regression.
    """
    error = y_true - y_pred
    return torch.max(quantile * error, (quantile - 1) * error).mean()

6.3 Huber Loss (Combines MSE and MAE)

def huber_loss(y_pred, y_true, delta=1.0):
    """
    Huber loss is robust to outliers.
    """
    error = y_true - y_pred
    is_small = torch.abs(error) <= delta
    squared_loss = 0.5 * error**2
    linear_loss = delta * (torch.abs(error) - 0.5 * delta)
    return torch.where(is_small, squared_loss, linear_loss).mean()

6.4 Risk-Adjusted Loss

def risk_adjusted_loss(y_pred, y_true, risk_penalty=1.0):
    """
    Loss = -Return + risk_penalty * Risk (standard deviation)
    """
    returns = y_pred * y_true
    mean_return = returns.mean()
    std_return = returns.std(unbiased=False)
    return -(mean_return - risk_penalty * std_return)

7. Vanishing and Exploding Gradients – The Deep Network Problem

7.1 Vanishing Gradients

• Gradients become exponentially small as they backpropagate.

• Early layers learn very slowly.

• Common causes: Sigmoid/Tanh activations (derivatives < 1), deep networks.

7.2 Exploding Gradients

• Gradients become exponentially large.

• Weights become NaN.

• Common causes: Large weights, deep networks, unstable initialisation.

7.3 Solutions

7.4 Implementation of Gradient Clipping

def train_with_gradient_clipping(model, optimizer, X, y, max_norm=1.0):
    """
    Train with gradient clipping to prevent exploding gradients.
    """
    for epoch in range(epochs):
        optimizer.zero_grad()
        y_pred = model(X)
        loss = criterion(y_pred, y)
        loss.backward()

        # Gradient clipping
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm)

        optimizer.step()

8. Vanishing Gradient Problem – Mathematical Analysis

For a network with L layers, the gradient of the loss with respect to the first layer is:
∂L/∂W^{(1)} = ∂L/∂a^{(L)} * Π_{l=2}^{L} (∂a^{(l)}/∂z^{(l)} * ∂z^{(l)}/∂a^{(l-1)}) * ∂a^{(1)}/∂W^{(1)}

If |σ'(z^{(l)}) * W^{(l)}| < 1 for all layers, the product decays exponentially.

For Sigmoid: σ'(z) ≤ 0.25 → gradients vanish.
For ReLU: σ'(z) = 1 for z > 0 → no vanishing.

Financial Implication: Use ReLU activations for hidden layers to avoid vanishing gradients in deep financial networks.

9. Summary for the AI Practitioner

1. Neural networks are universal approximators – they can approximate any continuous function. The challenge is generalisation, not representation.

2. Activation functions are critical. Use ReLU/Leaky ReLU for hidden layers, linear for regression output, sigmoid/softmax for classification.

3. Backpropagation is the chain rule in vectorised form. PyTorch handles it automatically via autograd.

4. Vanishing gradients are a major issue in deep networks. Use ReLU, batch normalisation, and proper initialisation.

5. Custom loss functions allow direct optimisation of financial objectives (Sharpe Ratio, Quantile Loss).

6. Regularisation (dropout, weight decay) is mandatory for financial neural networks to prevent overfitting.

7. Shallow networks (2-3 hidden layers) often outperform deep networks on financial data due to the low signal-to-noise ratio.