Lesson 5.6: Autoencoders and Generative Models for Financial Data

1. Learning Objectives

By the end of this lesson, you will be able to:

Understand the mathematical foundations of autoencoders and their applications in finance.
Implement vanilla autoencoders for denoising and dimensionality reduction.
Derive the Variational Autoencoder (VAE) from first principles using the Evidence Lower Bound (ELBO).
Apply VAEs to synthetic financial data generation and anomaly detection.
Understand the mathematical formulation of Generative Adversarial Networks (GANs).
Implement GANs for generating realistic financial time series.
Apply TimeGAN for synthetic time series generation.
Understand the ethical and regulatory considerations of synthetic financial data.

2. Autoencoders – The Basics

2.1 Mathematical Formulation
An autoencoder consists of an encoder and a decoder:

Encoder: z = f_θ(x) (maps input to latent space)
Decoder: x_hat = g_φ(z) (reconstructs input from latent space)

Objective: Minimise reconstruction error:
L(θ, φ) = E_{x ~ p_data} [ ||x - g_φ(f_θ(x))||² ]

2.2 Architecture

Input: x ∈ R^d
Encoder: z = σ(W_e x + b_e)  (z ∈ R^m, m < d)
Decoder: x_hat = σ(W_d z + b_d)  (x_hat ∈ R^d)
Loss: ||x - x_hat||²

2.3 Implementation

class Autoencoder(nn.Module):
    def __init__(self, input_dim, latent_dim=16, hidden_dims=[64, 32]):
        super(Autoencoder, self).__init__()

        # Encoder
        encoder_layers = []
        prev_dim = input_dim
        for h_dim in hidden_dims:
            encoder_layers.append(nn.Linear(prev_dim, h_dim))
            encoder_layers.append(nn.ReLU())
            prev_dim = h_dim
        encoder_layers.append(nn.Linear(prev_dim, latent_dim))
        self.encoder = nn.Sequential(*encoder_layers)

        # Decoder
        decoder_layers = []
        prev_dim = latent_dim
        for h_dim in reversed(hidden_dims):
            decoder_layers.append(nn.Linear(prev_dim, h_dim))
            decoder_layers.append(nn.ReLU())
            prev_dim = h_dim
        decoder_layers.append(nn.Linear(prev_dim, input_dim))
        self.decoder = nn.Sequential(*decoder_layers)

    def forward(self, x):
        z = self.encoder(x)
        x_hat = self.decoder(z)
        return x_hat, z

2.4 Financial Application – Denoising Returns

class DenoisingAutoencoder(nn.Module):
    def __init__(self, input_dim, latent_dim=16):
        super(DenoisingAutoencoder, self).__init__()

        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 64),
            nn.ReLU(),
            nn.Linear(64, 32),
            nn.ReLU(),
            nn.Linear(32, latent_dim)
        )

        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 32),
            nn.ReLU(),
            nn.Linear(32, 64),
            nn.ReLU(),
            nn.Linear(64, input_dim)
        )

    def forward(self, x_noisy):
        # Add noise during training
        z = self.encoder(x_noisy)
        x_clean = self.decoder(z)
        return x_clean

def denoise_returns(returns, noise_factor=0.1):
    """
    Denoise financial returns using autoencoder.
    """
    # Add noise to input
    noise = torch.randn_like(returns) * noise_factor * returns.std()
    returns_noisy = returns + noise

    model = DenoisingAutoencoder(returns.shape[1])
    # Train on noisy inputs with clean targets

    with torch.no_grad():
        returns_denoised = model(returns_noisy)

    return returns_denoised

3. Variational Autoencoders (VAE) – Probabilistic Generation

3.1 The Bayesian Framework

Instead of learning a deterministic latent representation, VAEs learn a distribution over the latent space.

Generative Model:

Prior: p(z) = N(0, I)
Likelihood: p(x | z) = N(μ(z), σ²(z))

Inference Model (Encoder):
q(z | x) = N(μ_φ(x), σ_φ²(x))

3.2 The Evidence Lower Bound (ELBO)

The marginal likelihood is:
log p(x) = log ∫ p(x|z) p(z) dz

This is intractable. Instead, we maximise the ELBO:
ELBO = E_{q(z|x)}[log p(x|z)] - KL(q(z|x) || p(z))

Derivation:
log p(x) = KL(q(z|x) || p(z|x)) + ELBO
Since KL ≥ 0, maximising the ELBO maximises a lower bound on log p(x).

3.3 The Reparameterisation Trick

To backpropagate through the sampling step, we use:
z = μ + σ * ε, where ε ~ N(0, I)

3.4 Implementation

class VAE(nn.Module):
    def __init__(self, input_dim, latent_dim=16, hidden_dims=[64, 32]):
        super(VAE, self).__init__()

        self.latent_dim = latent_dim

        # Encoder
        encoder_layers = []
        prev_dim = input_dim
        for h_dim in hidden_dims:
            encoder_layers.append(nn.Linear(prev_dim, h_dim))
            encoder_layers.append(nn.ReLU())
            prev_dim = h_dim

        self.encoder = nn.Sequential(*encoder_layers)

        # Mean and log-variance
        self.fc_mean = nn.Linear(prev_dim, latent_dim)
        self.fc_logvar = nn.Linear(prev_dim, latent_dim)

        # Decoder
        decoder_layers = []
        prev_dim = latent_dim
        for h_dim in reversed(hidden_dims):
            decoder_layers.append(nn.Linear(prev_dim, h_dim))
            decoder_layers.append(nn.ReLU())
            prev_dim = h_dim
        decoder_layers.append(nn.Linear(prev_dim, input_dim))

        self.decoder = nn.Sequential(*decoder_layers)

    def encode(self, x):
        h = self.encoder(x)
        mu = self.fc_mean(h)
        logvar = self.fc_logvar(h)
        return mu, logvar

    def reparameterise(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std

    def decode(self, z):
        return self.decoder(z)

    def forward(self, x):
        mu, logvar = self.encode(x)
        z = self.reparameterise(mu, logvar)
        x_recon = self.decode(z)
        return x_recon, mu, logvar

    def loss(self, x, x_recon, mu, logvar):
        # Reconstruction loss (MSE)
        recon_loss = F.mse_loss(x_recon, x, reduction='sum')

        # KL divergence
        kl_loss = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())

        return (recon_loss + kl_loss) / x.size(0)

3.5 Financial Application – Synthetic Returns Generation

def generate_synthetic_returns(vae, n_samples, features_dim):
    """
    Generate synthetic returns using trained VAE.
    """
    vae.eval()
    with torch.no_grad():
        # Sample from prior
        z = torch.randn(n_samples, vae.latent_dim)
        synthetic_returns = vae.decode(z)

    return synthetic_returns

4. Generative Adversarial Networks (GANs)

4.1 The Game Theory Framework

GANs consist of two networks:

Generator (G): Maps noise z to fake data x_fake = G(z).
Discriminator (D): Distinguishes real data x_real from fake data x_fake.

Objective (Minimax Game):
min_G max_D V(D, G) = E_{x ~ p_data}[log D(x)] + E_{z ~ p_z}[log(1 - D(G(z)))]

4.2 The Loss Functions

Discriminator Loss:
L_D = -E_{x ~ p_data}[log D(x)] - E_{z ~ p_z}[log(1 - D(G(z)))]

Generator Loss:
L_G = -E_{z ~ p_z}[log D(G(z))]

4.3 Implementation

class Generator(nn.Module):
    def __init__(self, latent_dim=100, output_dim=50, hidden_dims=[256, 128]):
        super(Generator, self).__init__()

        layers = []
        prev_dim = latent_dim
        for h_dim in hidden_dims:
            layers.append(nn.Linear(prev_dim, h_dim))
            layers.append(nn.BatchNorm1d(h_dim))
            layers.append(nn.ReLU())
            prev_dim = h_dim

        layers.append(nn.Linear(prev_dim, output_dim))
        self.network = nn.Sequential(*layers)

    def forward(self, z):
        return self.network(z)

class Discriminator(nn.Module):
    def __init__(self, input_dim=50, hidden_dims=[128, 256]):
        super(Discriminator, self).__init__()

        layers = []
        prev_dim = input_dim
        for h_dim in hidden_dims:
            layers.append(nn.Linear(prev_dim, h_dim))
            layers.append(nn.LeakyReLU(0.2))
            layers.append(nn.Dropout(0.3))
            prev_dim = h_dim

        layers.append(nn.Linear(prev_dim, 1))
        self.network = nn.Sequential(*layers)

    def forward(self, x):
        return torch.sigmoid(self.network(x))

class GAN(nn.Module):
    def __init__(self, latent_dim, output_dim, device='cpu'):
        super(GAN, self).__init__()

        self.latent_dim = latent_dim
        self.device = device

        self.generator = Generator(latent_dim, output_dim).to(device)
        self.discriminator = Discriminator(output_dim).to(device)

    def train_gan(self, data_loader, n_epochs=100, lr=0.0002):
        g_optimizer = torch.optim.Adam(self.generator.parameters(), lr=lr, betas=(0.5, 0.999))
        d_optimizer = torch.optim.Adam(self.discriminator.parameters(), lr=lr, betas=(0.5, 0.999))

        for epoch in range(n_epochs):
            for real_data in data_loader:
                batch_size = real_data.size(0)
                real_data = real_data.to(self.device)

                # Train discriminator
                d_optimizer.zero_grad()

                # Real data
                real_output = self.discriminator(real_data)
                real_loss = -torch.log(real_output + 1e-8).mean()

                # Fake data
                z = torch.randn(batch_size, self.latent_dim).to(self.device)
                fake_data = self.generator(z)
                fake_output = self.discriminator(fake_data.detach())
                fake_loss = -torch.log(1 - fake_output + 1e-8).mean()

                d_loss = real_loss + fake_loss
                d_loss.backward()
                d_optimizer.step()

                # Train generator
                g_optimizer.zero_grad()

                z = torch.randn(batch_size, self.latent_dim).to(self.device)
                fake_data = self.generator(z)
                fake_output = self.discriminator(fake_data)
                g_loss = -torch.log(fake_output + 1e-8).mean()

                g_loss.backward()
                g_optimizer.step()

            if epoch % 10 == 0:
                print(f"Epoch {epoch}: D_loss={d_loss.item():.4f}, G_loss={g_loss.item():.4f}")

5. TimeGAN – Synthetic Financial Time Series

5.1 Architecture

TimeGAN combines GANs with autoencoders to generate realistic time series.

Components:

Encoder: Maps time series to latent space.
Decoder: Maps latent space back to time series.
Generator: Generates synthetic latent sequences.
Discriminator: Distinguishes real from synthetic sequences.

5.2 Implementation (Simplified)

class TimeGAN(nn.Module):
    def __init__(self, input_dim, hidden_dim, latent_dim, seq_len):
        super(TimeGAN, self).__init__()

        self.seq_len = seq_len

        # Autoencoder
        self.encoder = nn.LSTM(input_dim, hidden_dim, batch_first=True)
        self.encoder_fc = nn.Linear(hidden_dim, latent_dim)

        self.decoder = nn.LSTM(latent_dim, hidden_dim, batch_first=True)
        self.decoder_fc = nn.Linear(hidden_dim, input_dim)

        # Generator (LSTM)
        self.generator = nn.LSTM(latent_dim, hidden_dim, batch_first=True)
        self.generator_fc = nn.Linear(hidden_dim, latent_dim)

        # Discriminator
        self.discriminator = nn.LSTM(latent_dim, hidden_dim, batch_first=True)
        self.discriminator_fc = nn.Linear(hidden_dim, 1)

    def encode(self, x):
        out, _ = self.encoder(x)
        z = self.encoder_fc(out)
        return z

    def decode(self, z):
        out, _ = self.decoder(z)
        x_hat = self.decoder_fc(out)
        return x_hat

    def generate(self, z):
        out, _ = self.generator(z)
        z_gen = self.generator_fc(out)
        return z_gen

    def discriminate(self, z):
        out, _ = self.discriminator(z)
        d = self.discriminator_fc(out)
        return torch.sigmoid(d)

    def forward(self, x):
        # Encode
        z = self.encode(x)

        # Reconstruct
        x_hat = self.decode(z)

        # Generate
        z_noise = torch.randn_like(z)
        z_gen = self.generate(z_noise)

        # Discriminate
        d_real = self.discriminate(z)
        d_fake = self.discriminate(z_gen)

        return x_hat, z_gen, d_real, d_fake

6. Financial Applications of Generative Models

6.1 Synthetic Data Generation for Model Training

Generate additional training data for asset pricing models.
Create scenarios for stress testing.
Augment small datasets to prevent overfitting.

6.2 Anomaly Detection

def anomaly_detection_vae(vae, data, threshold=3.0):
    """
    Detect anomalies using VAE reconstruction error.
    """
    vae.eval()
    with torch.no_grad():
        x_recon, mu, logvar = vae(data)
        recon_error = torch.mean((data - x_recon)**2, dim=1)

    anomaly_mask = recon_error > (recon_error.mean() + threshold * recon_error.std())
    return anomaly_mask.numpy(), recon_error.numpy()

6.3 Data Imputation

def impute_missing_data(vae, data, missing_mask):
    """
    Impute missing values using VAE.
    """
    # Iterative imputation: alternately impute and reconstruct
    data_imputed = data.clone()
    for _ in range(10):
        with torch.no_grad():
            x_recon, _, _ = vae(data_imputed)
            data_imputed[missing_mask] = x_recon[missing_mask]

    return data_imputed

7. Ethical and Regulatory Considerations

Consideration	Implication
Data Privacy	Synthetic data must preserve privacy (differential privacy).
Regulatory Compliance	Synthetic data must be documented and validated for use in regulatory submissions.
Model Risk	Synthetic data may not capture tail risks. Stress test all models on real data.
Bias Amplification	Generative models can amplify biases in training data.

8. Summary for the AI Practitioner

Autoencoders learn compressed representations. Use them for denoising and dimensionality reduction.
VAEs learn probabilistic latent representations. The ELBO is E_q[log p(x|z)] - KL(q(z|x) || p(z)).
The reparameterisation trick makes VAEs trainable via backpropagation: z = μ + σ * ε.
GANs use a minimax game between generator and discriminator. They generate realistic but can be unstable.
TimeGAN generates realistic financial time series by combining GANs with autoencoders.
Synthetic data augments small datasets, enables stress testing, and preserves privacy.
Anomaly detection with VAEs/GANs identifies unusual market behaviour.
Ethical considerations: Synthetic data must be validated and may amplify biases.

Diploma in Artificial Intelligence in Finance Lesson 5.6: Autoencoders and Generative Models for Financial Data

Diploma in Artificial Intelligence in Finance