1. Learning Objectives

By the end of this lesson, you will be able to:

• Understand the mathematical foundations of generative models: maximum likelihood estimation, latent variable models, and the ELBO.

• Derive and implement Variational Autoencoders (VAEs) for generating synthetic financial data (returns, scenarios, LOB states).

• Derive and implement Generative Adversarial Networks (GANs) for realistic financial time series generation.

• Understand the theory of diffusion models and their application to financial data generation.

• Apply generative models to data augmentation, scenario generation, and privacy-preserving data sharing.

• Evaluate generative models using quantitative metrics (Inception Score, FID, log-likelihood) and financial-specific metrics (distributional matching, correlation preservation).

2. The Statistical Foundations of Generative Modeling

2.1 The Generative Problem

Given a dataset X = {x₁, x₂, …, x_N} drawn from an unknown distribution p_data(x), we want to learn a model p_θ(x) that approximates p_data(x). Once learned, we can sample new data from p_θ(x).

This is fundamentally a density estimation problem. The quality of the generative model depends on how well it captures the true data distribution.

2.2 Maximum Likelihood Estimation (MLE)

The standard approach is to maximize the log-likelihood of the data under the model:

L(θ) = (1/N) * ∑_{i=1}^{N} log p_θ(x_i)

As N → ∞, maximizing the log-likelihood is equivalent to minimizing the KL divergence between the true data distribution and the model distribution:

KL(p_data || p_θ) = E_{x ~ p_data} [ log(p_data(x) / p_θ(x)) ]

This provides a theoretical foundation for generative modeling.

2.3 The Latent Variable Perspective

Many complex data distributions (e.g., financial returns, images) can be represented more compactly using latent variables z. The generative process is:

1. Sample z ~ p(z) (the prior, often a standard normal).

2. Sample x ~ p_θ(x | z) (the decoder).

The model likelihood is:

p_θ(x) = ∫ p_θ(x | z) p(z) dz

This integral is often intractable. This leads to two main families of models: VAEs (which approximate the posterior) and GANs (which avoid likelihood estimation entirely).

3. Variational Autoencoders (VAEs)

3.1 The Intractability Problem

In VAEs, we use a neural network to parameterize p_θ(x | z). The likelihood p_θ(x) = ∫ p_θ(x|z) p(z) dz is intractable because we cannot integrate over all z. We also cannot easily compute the posterior p(z | x) for inference.

3.2 The Evidence Lower Bound (ELBO)

The VAE introduces a variational distribution q_φ(z | x) (the encoder) to approximate the true posterior. The log-likelihood can be decomposed as:

log p_θ(x) = E_{q_φ(z|x)} [ log p_θ(x|z) ] - KL(q_φ(z|x) || p(z)) + KL(q_φ(z|x) || p(z|x))

The first two terms constitute the ELBO (Evidence Lower Bound):

ELBO(x) = E_{q_φ(z|x)} [ log p_θ(x|z) ] - KL(q_φ(z|x) || p(z))

Since KL(q_φ(z|x) || p(z|x)) ≥ 0, we have:

log p_θ(x) ≥ ELBO(x)

Thus, maximizing the ELBO is a lower-bound maximization.

ELBO Derivation:

log p_θ(x) = log ∫ p_θ(x|z) p(z) dz
= log ∫ p_θ(x|z) p(z) * (q_φ(z|x) / q_φ(z|x)) dz
= log E_{q_φ(z|x)} [ p_θ(x|z) p(z) / q_φ(z|x) ]
≥ E_{q_φ(z|x)} [ log p_θ(x|z) + log(p(z) / q_φ(z|x)) ]
= E_{q_φ(z|x)} [ log p_θ(x|z) ] - KL(q_φ(z|x) || p(z))

3.3 The Reparameterization Trick

The ELBO involves an expectation over q_φ(z|x). To perform gradient descent, we need to backpropagate through the sampling of z. The reparameterization trick expresses z as a deterministic function of x, φ, and a noise variable ε:

z = μ_φ(x) + σ_φ(x) * ε, where ε ~ N(0, I)

Then the expectation is over ε, which is independent of φ, allowing gradient flow.

3.4 VAE Loss Function

L(θ, φ) = -E_{ε ~ N(0,I)} [ log p_θ(x | μ_φ(x) + σ_φ(x) * ε) ] + KL(N(μ_φ(x), σ_φ(x)^2) || N(0, I))

The first term is the reconstruction loss (e.g., MSE for real-valued data, cross-entropy for binary). The KL term acts as a regularizer, pushing the encoder to produce distributions close to the prior.

For a standard normal prior and a diagonal Gaussian posterior:

KL(q_φ(z|x) || p(z)) = -0.5 * ∑_{j=1}^{d} [ 1 + log(σ_{φ,j}^2) - μ_{φ,j}^2 - σ_{φ,j}^2 ]

3.5 VAEs for Financial Time Series Generation

Architecture modifications for time series:

• Use LSTM or Transformer encoders and decoders to handle sequential data.

• The latent space can capture the underlying market regime (e.g., high volatility vs. low volatility).

Generation process:

1. Sample z from the prior (standard normal).

2. Decode z to generate a sequence of returns (e.g., 252 daily returns).

3. The generated sequence can be used for scenario generation or data augmentation.

Conditional VAE: Condition the generation on a label (e.g., market regime) or on past data to generate future scenarios.

4. Generative Adversarial Networks (GANs)

4.1 The Game-Theoretic Framework

GANs consist of two networks: a generator G and a discriminator D. The generator maps a noise vector z (from a simple distribution) to a data sample. The discriminator tries to distinguish between real data and generated data. The game is a minimax problem:

min_G max_D V(D, G) = E_{x ~ p_data} [ log D(x) ] + E_{z ~ p_z} [ log(1 - D(G(z))) ]

Training objective:

• The discriminator maximizes its ability to distinguish real from fake.

• The generator minimizes the discriminator’s ability to distinguish.

4.2 The Optimal Discriminator

For a fixed generator G, the optimal discriminator is:

D*(x) = p_data(x) / (p_data(x) + p_g(x))

This is derived by taking the derivative of the value function with respect to D.

Substituting D* into the value function gives the Jensen-Shannon divergence between p_data and p_g:

V(G, D*) = 2 * JSD(p_data || p_g) - log(4)

Thus, minimizing the GAN objective is equivalent to minimizing the JSD between the true and generated distributions.

4.3 Training Challenges and Solutions

Mode collapse: The generator produces only a few types of samples (collapsing to a few modes). Solutions:

• Wasserstein GAN (WGAN): Replaces the JSD with the Earth Mover (Wasserstein) distance. The discriminator (critic) is trained to approximate the Wasserstein distance. The loss is:

  min_G max_{D ∈ Lipschitz} E_{x ~ p_data}[D(x)] - E_{z ~ p_z}[D(G(z))]

  This requires weight clipping or gradient penalty (WGAN-GP).

• Unrolled GANs: Update the discriminator several times before updating the generator, reducing mode collapse.

• Mini-batch discrimination: Allow the discriminator to compare samples within a batch, making it harder to fool.

Vanishing gradients: If the discriminator becomes too good, the generator’s gradients vanish (because log(1-D(G(z))) approaches 0). Use LSGAN (least squares GAN) which has a smoother loss.

4.4 GANs for Financial Data

Applications:

• Synthetic LOB generation: Generate realistic order book states.

• Price path generation: Generate realistic stock price paths for backtesting.

• Data augmentation: Augment limited datasets (e.g., fraud detection, credit default) with synthetic but realistic examples.

Implementation:

class FinancialGAN:
    def __init__(self, seq_len, n_features, latent_dim):
        self.generator = self._build_generator(seq_len, n_features, latent_dim)
        self.discriminator = self._build_discriminator(seq_len, n_features)
        self.gan = self._build_gan()
    
    def _build_generator(self, seq_len, n_features, latent_dim):
        model = Sequential([
            Dense(128, input_dim=latent_dim),
            LeakyReLU(0.2),
            Dense(256),
            LeakyReLU(0.2),
            Dense(seq_len * n_features, activation='tanh'),
            Reshape((seq_len, n_features))
        ])
        return model

Evaluation for financial GANs:

• Distributional metrics: Compare the mean, variance, skewness, and kurtosis of generated returns with real returns.

• Autocorrelation: Check if the generated series preserves the autocorrelation structure.

• Correlation: For multivariate generation, check if the correlation matrix is preserved.

• Discriminative score: Train a classifier to distinguish real from generated; if it cannot, the GAN is successful.

5. Diffusion Models

Diffusion models (e.g., Denoising Diffusion Probabilistic Models – DDPMs) have recently achieved state-of-the-art results in image generation. They are based on a forward and reverse diffusion process.

5.1 Forward Diffusion Process

We start with a data point x₀. We gradually add Gaussian noise over T steps:

q(x_t | x_{t-1}) = N(x_t; sqrt(1 - β_t) * x_{t-1}, β_t * I)

where β_t is a variance schedule (small for t=1, increasing). After T steps (large, e.g., 1000), x_T approximates pure noise.

A nice property: we can directly sample x_t from x₀:

x_t = sqrt(ᾱ_t) * x₀ + sqrt(1 - ᾱ_t) * ε, where ε ~ N(0, I)

where α_t = 1 - β_t and ᾱ_t = ∏_{s=1}^{t} α_s.

5.2 Reverse Diffusion Process

The reverse process learns to denoise:

p_θ(x_{t-1} | x_t) = N(x_{t-1}; μ_θ(x_t, t), Σ_θ(x_t, t))

We train a neural network to predict the noise ε that was added at each step. The loss is:

L = E_{t, x₀, ε} [ || ε - ε_θ( sqrt(ᾱ_t) * x₀ + sqrt(1 - ᾱ_t) * ε, t ) ||^2 ]

This is a surprisingly simple objective: predict the noise added at each step.

5.3 Sampling

To generate a new sample:

1. Sample x_T ~ N(0, I).

2. For t = T, T-1, …, 1:

   • Sample z ~ N(0, I) (if t > 1, else z = 0).

   • Compute x_{t-1} = (1/sqrt(α_t)) * (x_t - (1-α_t)/sqrt(1-ᾱ_t) * ε_θ(x_t, t)) + sqrt(β_t) * z

5.4 Diffusion Models in Finance

Advantages:

• Stable training (no adversarial games like GANs).

• Generate high-quality, diverse samples.

• Can be conditioned on past data or other variables.

Applications:

• Generate realistic financial time series.

• Scenario generation for stress testing.

• Imputation of missing data.

Computational cost: Sampling requires T steps (e.g., 1000), making it slower than GANs. However, models can be distilled to fewer steps.

6. Evaluating Generative Models

7. Summary for the AI Practitioner

• Generative models (VAEs, GANs, diffusion) can create synthetic financial data for augmentation, scenario analysis, and privacy-preserving sharing.

• VAEs provide a probabilistic framework with a tractable ELBO; the reparameterization trick enables gradient-based training.

• GANs use a game-theoretic approach; training requires careful balancing (WGAN-GP is recommended).

• Diffusion models are stable and produce high-quality samples but are computationally more expensive.

• For financial applications, evaluate using distributional metrics (mean, variance, correlations) in addition to standard generative metrics.