Lesson 4.2: Unsupervised Learning and Ensemble Methods for Finance

1. Learning Objectives

By the end of this lesson, you will be able to:

Apply Principal Component Analysis (PCA) for factor extraction and dimensionality reduction.
Implement K-means and hierarchical clustering for market regime identification.
Use anomaly detection for fraud detection and risk monitoring.
Understand the mathematical foundations of ensemble methods (Bagging, Boosting, Stacking).
Implement Random Forests for financial prediction with feature importance.
Implement Gradient Boosting Machines (XGBoost, LightGBM, CatBoost) for high-performance financial modelling.
Apply ensemble methods to volatility forecasting and credit scoring.
Understand the trade-offs between different ensemble methods in finance.

2. Principal Component Analysis (PCA) – Factor Extraction in Finance

PCA is the most important unsupervised learning technique in finance. It extracts the latent factors driving asset returns.

2.1 Mathematical Formulation

Given centered data matrix X ∈ R^{N x p} (N observations, p features):

Compute the covariance matrix: Σ = (1/(N-1)) X^T X.
Eigen-decompose Σ = V Λ V^T, where:
- V is the matrix of eigenvectors (loadings).
- Λ is the diagonal matrix of eigenvalues (variances).
The principal components (PCs) are: Z = X V.
The first k PCs explain (Σ_{i=1}^k λ_i) / (Σ_{i=1}^p λ_i) of the total variance.

2.2 Financial Application – Factor Extraction

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

def extract_factors(returns, n_factors=5):
    """
    Extract principal components as factors.
    """
    # Standardise returns (PCA is scale-sensitive)
    scaler = StandardScaler()
    returns_scaled = scaler.fit_transform(returns)

    # PCA
    pca = PCA(n_components=n_factors)
    factors = pca.fit_transform(returns_scaled)

    # Explained variance
    explained_variance = pca.explained_variance_ratio_

    # Loadings (V matrix)
    loadings = pca.components_.T

    # Create factor DataFrame
    factor_df = pd.DataFrame(
        factors,
        index=returns.index,
        columns=[f'Factor_{i+1}' for i in range(n_factors)]
    )

    return factor_df, loadings, explained_variance, pca

2.3 Interpreting PCA in Finance

The first principal component typically captures the market factor (all assets move together).
The second principal component often captures a sector/industry factor.
The remaining components capture style factors (value, growth, momentum).

2.4 Random Matrix Theory (RMT) Denoising
RMT provides a threshold for separating signal from noise in the eigenvalue spectrum.

Marchenko-Pastur Threshold:
λ_max = σ² (1 + sqrt(p/N))² (for white noise)

If N = 500 (assets) and T = 1000 (days), then p/N = 0.5. Assuming σ² = 1, the noise threshold is (1 + sqrt(0.5))² ≈ 2.91.

Procedure:

Compute eigenvalues of the correlation matrix.
Compare eigenvalues to the Marchenko-Pastur threshold.
Only eigenvalues above the threshold are considered signal.
Keep the corresponding eigenvectors for the denoised covariance matrix.

def rmt_denoise(correlation_matrix, N_assets, T_days):
    """
    Denoise correlation matrix using Random Matrix Theory.
    """
    p, N = N_assets, T_days
    gamma = p / N
    threshold = (1 + np.sqrt(gamma))**2

    # Eigen-decompose
    eigenvalues, eigenvectors = np.linalg.eigh(correlation_matrix)

    # Identify signal eigenvalues
    signal_mask = eigenvalues > threshold
    signal_values = eigenvalues[signal_mask]
    signal_vectors = eigenvectors[:, signal_mask]

    # Reconstruct denoised matrix
    denoised = signal_vectors @ np.diag(signal_values) @ signal_vectors.T

    # Normalise to unit diagonal (correlation matrix)
    diag = np.sqrt(np.diag(denoised))
    denoised = denoised / np.outer(diag, diag)

    return denoised

3. Clustering – Market Regime Identification

Clustering groups similar observations. In finance, it identifies market regimes (bull, bear, range-bound).

3.1 K-Means Clustering

Algorithm:

Initialise K centroids randomly.
Assign each point to the nearest centroid.
Update centroids as the mean of assigned points.
Repeat steps 2-3 until convergence.

Objective Function:
minimize_{C} Σ_{k=1}^{K} Σ_{x ∈ C_k} ||x - μ_k||²

Financial Application – Regime Detection:

from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler

def detect_market_regimes(returns, n_regimes=3):
    """
    Identify market regimes using K-means clustering.
    """
    # Extract features for clustering (volatility, skewness, kurtosis)
    rolling_window = 60  # 3 months

    features = pd.DataFrame(index=returns.index)
    features['Volatility'] = returns.rolling(rolling_window).std()
    features['Skewness'] = returns.rolling(rolling_window).skew()
    features['Kurtosis'] = returns.rolling(rolling_window).kurt()

    features = features.dropna()

    # Standardise
    scaler = StandardScaler()
    features_scaled = scaler.fit_transform(features)

    # K-means clustering
    kmeans = KMeans(n_clusters=n_regimes, random_state=42, n_init=10)
    regimes = kmeans.fit_predict(features_scaled)

    # Map regimes to interpretable labels (e.g., Bull, Bear, Range)
    regime_df = pd.DataFrame({
        'regime': regimes
    }, index=features.index)

    # Analyse each regime
    regime_analysis = {}
    for regime in range(n_regimes):
        regime_returns = returns.loc[regime_df[regime_df['regime'] == regime].index]
        regime_analysis[regime] = {
            'mean_return': regime_returns.mean() * 252,
            'volatility': regime_returns.std() * np.sqrt(252),
            'sharpe': regime_returns.mean() / regime_returns.std() * np.sqrt(252),
            'count': len(regime_returns)
        }

    return regime_df, regime_analysis, kmeans

3.2 Hierarchical Clustering – Asset Classification

Hierarchical clustering builds a tree (dendrogram) of similar assets. It is useful for sector identification and portfolio diversification.

from scipy.cluster.hierarchy import dendrogram, linkage, fcluster

def cluster_assets(returns, method='ward', n_clusters=None, threshold=None):
    """
    Cluster assets using hierarchical clustering.
    """
    # Compute correlation matrix
    corr_matrix = returns.corr()

    # Convert to distance matrix (1 - correlation)
    distance_matrix = 1 - corr_matrix

    # Perform hierarchical clustering
    linkage_matrix = linkage(distance_matrix, method=method)

    # Extract clusters
    if n_clusters is not None:
        clusters = fcluster(linkage_matrix, n_clusters, criterion='maxclust')
    elif threshold is not None:
        clusters = fcluster(linkage_matrix, threshold, criterion='distance')

    # Create cluster mapping
    cluster_df = pd.DataFrame({
        'asset': returns.columns,
        'cluster': clusters
    })

    # Plot dendrogram
    plt.figure(figsize=(12, 8))
    dendrogram(linkage_matrix, labels=returns.columns, leaf_rotation=90)
    plt.tight_layout()
    plt.show()

    return cluster_df, linkage_matrix

4. Anomaly Detection – Fraud and Outlier Detection

Anomaly detection identifies unusual observations that deviate from normal patterns. Used for fraud detection, market manipulation, and data quality checks.

4.1 Isolation Forest

Isolation Forest isolates anomalies by randomly partitioning the data space. Anomalies are easier to isolate (shorter paths).

Algorithm:

Build an ensemble of isolation trees.
Each tree randomly selects a feature and a split value.
Anomaly score is the average path length across trees.
Shorter path lengths indicate anomalies.

Implementation:

from sklearn.ensemble import IsolationForest

def detect_fraud(transaction_data, contamination=0.01):
    """
    Detect fraudulent transactions using Isolation Forest.
    """
    # Features: amount, time, location, frequency, etc.
    features = transaction_data[['amount', 'hour', 'day_of_week', 'frequency']]

    # Standardise
    scaler = StandardScaler()
    features_scaled = scaler.fit_transform(features)

    # Isolation Forest
    iso_forest = IsolationForest(
        contamination=contamination,
        random_state=42,
        n_estimators=100
    )
    predictions = iso_forest.fit_predict(features_scaled)

    # 1 = normal, -1 = anomaly
    transaction_data['anomaly'] = predictions
    transaction_data['anomaly_score'] = iso_forest.score_samples(features_scaled)

    return transaction_data

4.2 Local Outlier Factor (LOF)

LOF computes the local density deviation of a data point relative to its neighbours.

Mathematical Formulation:
For a data point x, define:

k-distance: distance to the k-th nearest neighbour.
reachability distance: max(k-distance(y), distance(x, y))
local reachability density (LRD): inverse of average reachability distance.
LOF(x) = (average LRD of neighbours) / LRD(x)

LOF > 1 indicates an anomaly.

from sklearn.neighbors import LocalOutlierFactor

def detect_anomalies_lof(returns, n_neighbors=20, contamination=0.01):
    """
    Detect anomalous returns using LOF.
    """
    lof = LocalOutlierFactor(
        n_neighbors=n_neighbors,
        contamination=contamination,
        novelty=False
    )
    predictions = lof.fit_predict(returns)

    # Get anomaly scores
    anomaly_scores = -lof.negative_outlier_factor_

    return predictions, anomaly_scores

5. Ensemble Methods – Combining Multiple Models

Ensemble methods combine multiple base models to improve predictive performance and robustness.

5.1 Bagging (Bootstrap Aggregating)

Bagging reduces variance by training multiple models on bootstrap samples and averaging their predictions.

Algorithm:

Generate B bootstrap samples from the training data.
Train a model on each bootstrap sample.
Average predictions (regression) or majority vote (classification).

Random Forest: Bagging with decision trees, with random feature selection at each split.

from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor

def random_forest_model(X_train, y_train, X_val, y_val, model_type='classification'):
    """
    Random Forest for financial prediction.
    """
    if model_type == 'classification':
        model = RandomForestClassifier(
            n_estimators=100,
            max_depth=10,
            min_samples_split=20,
            min_samples_leaf=10,
            max_features='sqrt',
            class_weight='balanced',
            random_state=42,
            n_jobs=-1
        )
    else:
        model = RandomForestRegressor(
            n_estimators=100,
            max_depth=10,
            min_samples_split=20,
            min_samples_leaf=10,
            max_features='sqrt',
            random_state=42,
            n_jobs=-1
        )

    model.fit(X_train, y_train)

    # Feature importance
    importance_df = pd.DataFrame({
        'feature': X_train.columns,
        'importance': model.feature_importances_
    }).sort_values('importance', ascending=False)

    # Predictions
    y_pred = model.predict(X_val)

    # For classification, get probabilities
    if model_type == 'classification':
        y_pred_proba = model.predict_proba(X_val)

    return model, importance_df, y_pred

5.2 Boosting

Boosting reduces bias by sequentially training models, where each model corrects the errors of previous models.

AdaBoost:

Assigns higher weights to misclassified samples.
Each model focuses on hard-to-classify samples.

Gradient Boosting:

Trains models sequentially on the residuals of previous models.
Uses gradient descent to minimise the loss function.

from sklearn.ensemble import GradientBoostingClassifier, GradientBoostingRegressor

def gradient_boosting_model(X_train, y_train, X_val, y_val, model_type='classification'):
    """
    Gradient Boosting for financial prediction.
    """
    if model_type == 'classification':
        model = GradientBoostingClassifier(
            n_estimators=100,
            learning_rate=0.1,
            max_depth=5,
            min_samples_split=20,
            min_samples_leaf=10,
            subsample=0.8,
            random_state=42
        )
    else:
        model = GradientBoostingRegressor(
            n_estimators=100,
            learning_rate=0.1,
            max_depth=5,
            min_samples_split=20,
            min_samples_leaf=10,
            subsample=0.8,
            random_state=42
        )

    model.fit(X_train, y_train)

    # Feature importance
    importance_df = pd.DataFrame({
        'feature': X_train.columns,
        'importance': model.feature_importances_
    }).sort_values('importance', ascending=False)

    return model, importance_df

5.3 XGBoost – Extreme Gradient Boosting

XGBoost is the industry standard for financial machine learning due to its speed, accuracy, and regularisation.

Key Features:

Regularisation: L1 and L2 penalties prevent overfitting.
Handling Missing Values: Automatically learns the best direction for missing values.
Tree Pruning: Grows trees and prunes them to prevent overfitting.
Parallel Processing: Faster training.

Mathematical Formulation:
XGBoost minimises:
L(θ) = Σ_i l(y_i, ŷ_i) + Σ_k Ω(f_k)
where Ω(f) = γ T + (1/2) λ ||w||² (T is the number of leaves, w is leaf weights).

Implementation:

import xgboost as xgb

def xgboost_model(X_train, y_train, X_val, y_val, model_type='classification'):
    """
    XGBoost for financial prediction.
    """
    if model_type == 'classification':
        model = xgb.XGBClassifier(
            n_estimators=100,
            learning_rate=0.1,
            max_depth=6,
            min_child_weight=1,
            subsample=0.8,
            colsample_bytree=0.8,
            reg_alpha=0.1,  # L1 regularisation
            reg_lambda=1.0,  # L2 regularisation
            scale_pos_weight=1,  # For imbalanced data
            objective='binary:logistic',
            random_state=42,
            n_jobs=-1
        )
    else:
        model = xgb.XGBRegressor(
            n_estimators=100,
            learning_rate=0.1,
            max_depth=6,
            min_child_weight=1,
            subsample=0.8,
            colsample_bytree=0.8,
            reg_alpha=0.1,
            reg_lambda=1.0,
            objective='reg:squarederror',
            random_state=42,
            n_jobs=-1
        )

    # Early stopping
    eval_set = [(X_train, y_train), (X_val, y_val)]
    model.fit(
        X_train, y_train,
        eval_set=eval_set,
        eval_metric='logloss' if model_type == 'classification' else 'rmse',
        early_stopping_rounds=20,
        verbose=False
    )

    # Feature importance
    importance_df = pd.DataFrame({
        'feature': X_train.columns,
        'importance': model.feature_importances_
    }).sort_values('importance', ascending=False)

    return model, importance_df

5.4 LightGBM – Lightweight Gradient Boosting

LightGBM is faster and more memory-efficient than XGBoost, especially for large datasets.

Key Features:

Gradient-based One-Side Sampling (GOSS): Samples high-gradient instances more frequently.
Exclusive Feature Bundling (EFB): Bundles mutually exclusive features to reduce dimensions.
Leaf-wise Tree Growth: Grows tree leaf-wise (not level-wise), leading to faster convergence.

import lightgbm as lgb

def lightgbm_model(X_train, y_train, X_val, y_val, model_type='classification'):
    """
    LightGBM for financial prediction.
    """
    if model_type == 'classification':
        model = lgb.LGBMClassifier(
            n_estimators=100,
            learning_rate=0.1,
            max_depth=6,
            num_leaves=31,
            min_child_samples=20,
            subsample=0.8,
            colsample_bytree=0.8,
            reg_alpha=0.1,
            reg_lambda=1.0,
            class_weight='balanced',
            random_state=42,
            n_jobs=-1
        )
    else:
        model = lgb.LGBMRegressor(
            n_estimators=100,
            learning_rate=0.1,
            max_depth=6,
            num_leaves=31,
            min_child_samples=20,
            subsample=0.8,
            colsample_bytree=0.8,
            reg_alpha=0.1,
            reg_lambda=1.0,
            random_state=42,
            n_jobs=-1
        )

    # Early stopping
    eval_set = [(X_val, y_val)]
    model.fit(
        X_train, y_train,
        eval_set=eval_set,
        eval_metric='logloss' if model_type == 'classification' else 'rmse',
        early_stopping_rounds=20,
        verbose=False
    )

    return model

5.5 Stacking – Combining Diverse Models

Stacking trains a meta-model on the predictions of multiple base models.

from sklearn.ensemble import StackingClassifier, StackingRegressor
from sklearn.linear_model import LogisticRegression, Ridge

def stacking_model(X_train, y_train, X_val, y_val, model_type='classification'):
    """
    Stacking ensemble for financial prediction.
    """
    if model_type == 'classification':
        base_models = [
            ('rf', RandomForestClassifier(n_estimators=100, max_depth=10, random_state=42)),
            ('xgb', xgb.XGBClassifier(n_estimators=100, learning_rate=0.1, random_state=42)),
            ('lgb', lgb.LGBMClassifier(n_estimators=100, learning_rate=0.1, random_state=42))
        ]

        meta_model = LogisticRegression()

        model = StackingClassifier(
            estimators=base_models,
            final_estimator=meta_model,
            cv=5,
            stack_method='predict_proba'
        )
    else:
        base_models = [
            ('rf', RandomForestRegressor(n_estimators=100, max_depth=10, random_state=42)),
            ('xgb', xgb.XGBRegressor(n_estimators=100, learning_rate=0.1, random_state=42)),
            ('lgb', lgb.LGBMRegressor(n_estimators=100, learning_rate=0.1, random_state=42))
        ]

        meta_model = Ridge(alpha=1.0)

        model = StackingRegressor(
            estimators=base_models,
            final_estimator=meta_model,
            cv=5
        )

    model.fit(X_train, y_train)
    return model

6. Ensemble Model Selection – Which to Use?

Method	Strengths	Weaknesses	Financial Use Case
Random Forest	Robust, handles non-linearity, feature importance	Prone to overfitting, slower inference	Credit scoring, anomaly detection
Gradient Boosting	High accuracy, flexible	Prone to overfitting, slower training	Volatility forecasting, alpha generation
XGBoost	Regularised, fast, handles missing values	Complex hyperparameters	Market prediction, fraud detection
LightGBM	Extremely fast, memory efficient	Can overfit on small datasets	High-frequency prediction, large-scale data
Stacking	Combines strengths of multiple models	Computationally expensive	Portfolio construction, ensemble alpha

7. Summary for the AI Practitioner

PCA extracts latent factors from returns. The first PC is the market factor. Use RMT to denoise correlation matrices.
K-means clustering identifies market regimes (bull, bear, range). Use rolling features (volatility, skewness) for regime detection.
Isolation Forest and LOF detect anomalies. Essential for fraud detection and data quality monitoring.
Random Forest reduces variance via bagging. Provides feature importance for interpretability.
XGBoost/LightGBM are the industry standards. They are fast, accurate, and handle financial data well.
Stacking combines diverse models for superior performance. Use it when you have multiple strong base models.
Ensemble methods are preferred over single models in finance due to their robustness and superior out-of-sample performance.
Feature importance from tree-based models is valuable for explaining predictions to stakeholders.

Diploma in Artificial Intelligence in Finance Lesson 4.2: Unsupervised Learning and Ensemble Methods for Finance

Diploma in Artificial Intelligence in Finance

1. Learning Objectives

2. Principal Component Analysis (PCA) – Factor Extraction in Finance

3. Clustering – Market Regime Identification

4. Anomaly Detection – Fraud and Outlier Detection

5. Ensemble Methods – Combining Multiple Models

6. Ensemble Model Selection – Which to Use?

7. Summary for the AI Practitioner