1. Learning Objectives

By the end of this lesson, you will be able to:

  • Understand the curse of dimensionality and its impact on financial models.

  • Apply PCA, Kernel PCA, and t-SNE for dimensionality reduction.

  • Implement feature selection methods (filter, wrapper, embedded) for financial datasets.

  • Use mutual information and correlation analysis for feature selection.

  • Apply Lasso regression for automatic feature selection.

  • Implement Recursive Feature Elimination (RFE) for model-based feature selection.

  • Understand the tradeoff between interpretability and performance in feature selection.

2. The Curse of Dimensionality – Why Feature Reduction Matters

In high-dimensional spaces, data becomes sparse, and traditional statistical methods fail.

2.1 Key Issues:

  1. Data Sparsity: The number of data points needed grows exponentially with dimensionality.

  2. Distance Concentration: In high dimensions, all pairwise distances become similar, making nearest-neighbour methods ineffective.

  3. Overfitting: With many features, models can fit the noise rather than the signal.

  4. Computational Cost: Training time increases with the number of features.

Financial Implication: Financial datasets often have thousands of features (e.g., all S&P 500 returns, technical indicators). Dimensionality reduction is mandatory.

2.2 The Practical Rule
If you have N observations, you should have at most N/10 features. With 10 years of daily data (N ≈ 2500), you should use at most 250 features.

3. Linear Dimensionality Reduction – PCA Revisited

PCA finds the directions of maximum variance in the data.

3.1 Explained Variance Ratio
The proportion of variance explained by the first k components:
EV_k = (Σ_{i=1}^{k} λ_i) / (Σ_{i=1}^{p} λ_i)

Rule of Thumb: Keep enough components to explain 80-95% of the variance.

text
def pca_feature_reduction(X, variance_threshold=0.95):
    """
    Reduce dimensionality using PCA with variance threshold.
    """
    from sklearn.decomposition import PCA
    from sklearn.preprocessing import StandardScaler

    # Standardise
    scaler = StandardScaler()
    X_scaled = scaler.fit_transform(X)

    # PCA
    pca = PCA(n_components=variance_threshold)  # n_components can be a float for variance threshold
    X_pca = pca.fit_transform(X_scaled)

    print(f"Original features: {X.shape[1]}")
    print(f"Reduced features: {X_pca.shape[1]}")
    print(f"Explained variance: {pca.explained_variance_ratio_.sum():.4f}")

    return X_pca, pca, scaler

3.2 Factor Model Construction with PCA

text
def factor_model_pca(returns, n_factors=5):
    """
    Build a factor model using PCA.
    """
    # PCA
    pca = PCA(n_components=n_factors)
    factors = pca.fit_transform(returns)

    # Loadings
    loadings = pca.components_.T

    # Explained variance
    evr = pca.explained_variance_ratio_

    # Create results
    factor_df = pd.DataFrame(
        factors,
        index=returns.index,
        columns=[f'F_{i+1}' for i in range(n_factors)]
    )

    loading_df = pd.DataFrame(
        loadings,
        index=returns.columns,
        columns=[f'Loading_{i+1}' for i in range(n_factors)]
    )

    print(f"Total explained variance: {evr.sum():.4f}")

    return factor_df, loading_df, pca

4. Non-Linear Dimensionality Reduction

4.1 t-SNE (t-Distributed Stochastic Neighbor Embedding)

t-SNE is a non-linear technique that preserves local structure. It is excellent for visualisation but cannot be used for live predictions (transductive).

Mathematical Formulation:

  1. Compute pairwise similarities in high-dimensional space using a Gaussian kernel:
    p_{j|i} = exp(-||x_i - x_j||² / (2σ_i²)) / Σ_{k≠i} exp(-||x_i - x_k||² / (2σ_i²))

  2. Compute pairwise similarities in low-dimensional space using a Student-t kernel:
    q_{ij} = (1 + ||y_i - y_j||²)^{-1} / Σ_{k≠l} (1 + ||y_k - y_l||²)^{-1}

  3. Minimise the KL divergence between p and q.

text
from sklearn.manifold import TSNE

def tsne_visualisation(X, n_components=2, perplexity=30, random_state=42):
    """
    Visualise high-dimensional data using t-SNE.
    """
    # Standardise
    scaler = StandardScaler()
    X_scaled = scaler.fit_transform(X)

    # t-SNE
    tsne = TSNE(
        n_components=n_components,
        perplexity=perplexity,
        learning_rate='auto',
        init='pca',
        random_state=random_state
    )
    X_tsne = tsne.fit_transform(X_scaled)

    return X_tsne, tsne

4.2 UMAP (Uniform Manifold Approximation and Projection)

UMAP is faster than t-SNE and preserves global structure better. It is also more stable and can be used for inductive learning (with a trained transformer).

text
import umap

def umap_visualisation(X, n_components=2, n_neighbors=15, min_dist=0.1):
    """
    Visualise high-dimensional data using UMAP.
    """
    # Standardise
    scaler = StandardScaler()
    X_scaled = scaler.fit_transform(X)

    # UMAP
    reducer = umap.UMAP(
        n_components=n_components,
        n_neighbors=n_neighbors,
        min_dist=min_dist,
        random_state=42
    )
    X_umap = reducer.fit_transform(X_scaled)

    return X_umap, reducer, scaler

4.3 Financial Application – Market Regime Visualisation

text
def regime_visualisation(returns, regime_labels):
    """
    Visualise market regimes using t-SNE or UMAP.
    """
    # Extract rolling features
    features = pd.DataFrame(index=returns.index)
    features['Volatility'] = returns.rolling(60).std()
    features['Skewness'] = returns.rolling(60).skew()
    features['Kurtosis'] = returns.rolling(60).kurt()
    features['Mean_Return'] = returns.rolling(60).mean()
    features = features.dropna()

    # Align with regime labels
    regime_labels_aligned = regime_labels[regime_labels.index.isin(features.index)]

    # UMAP
    X_umap, reducer, scaler = umap_visualisation(features)

    # Plot
    import matplotlib.pyplot as plt
    plt.figure(figsize=(12, 8))
    scatter = plt.scatter(
        X_umap[:, 0], X_umap[:, 1],
        c=regime_labels_aligned.values,
        cmap='viridis',
        alpha=0.6
    )
    plt.colorbar(scatter, label='Regime')
    plt.title('Market Regime Visualisation with UMAP')
    plt.xlabel('UMAP Component 1')
    plt.ylabel('UMAP Component 2')
    plt.grid(True, alpha=0.3)
    plt.show()

    return X_umap

5. Feature Selection Methods

5.1 Filter Methods (Statistical)
Select features based on statistical measures independent of the model.

Correlation-Based Selection:

text
def correlation_feature_selection(X, y, threshold=0.3):
    """
    Select features with high correlation to the target.
    """
    correlations = pd.DataFrame({
        'feature': X.columns,
        'correlation': [np.corrcoef(X[col], y)[0, 1] for col in X.columns]
    })

    selected = correlations[np.abs(correlations['correlation']) > threshold]
    return selected.sort_values('correlation', ascending=False)

Mutual Information Selection:

text
from sklearn.feature_selection import mutual_info_classif, mutual_info_regression

def mutual_information_selection(X, y, model_type='classification', k=20):
    """
    Select top k features using mutual information.
    """
    if model_type == 'classification':
        mi = mutual_info_classif(X, y)
    else:
        mi = mutual_info_regression(X, y)

    mi_df = pd.DataFrame({
        'feature': X.columns,
        'mi_score': mi
    }).sort_values('mi_score', ascending=False)

    selected_features = mi_df.head(k)['feature'].tolist()
    return selected_features, mi_df

5.2 Wrapper Methods (Model-Based)

Recursive Feature Elimination (RFE):
RFE recursively removes the least important features based on the model’s coefficients or feature importance.

text
from sklearn.feature_selection import RFE, RFECV

def recursive_feature_elimination(model, X, y, n_features_to_select=20):
    """
    Select features using Recursive Feature Elimination.
    """
    rfe = RFE(
        estimator=model,
        n_features_to_select=n_features_to_select,
        step=1,
        verbose=1
    )
    rfe.fit(X, y)

    # Get selected features
    selected_indices = rfe.get_support(indices=True)
    selected_features = X.columns[selected_indices].tolist()

    # Feature ranking
    ranking_df = pd.DataFrame({
        'feature': X.columns,
        'ranking': rfe.ranking_,
        'selected': rfe.support_
    }).sort_values('ranking')

    return selected_features, ranking_df, rfe

5.3 Embedded Methods (Regularisation)

Lasso Regression for Feature Selection:
Lasso has L1 regularisation, which drives coefficients to exactly zero.

text
from sklearn.linear_model import LassoCV

def lasso_feature_selection(X, y, cv=5):
    """
    Select features using Lasso regression with cross-validation.
    """
    # Lasso with cross-validation
    lasso = LassoCV(
        cv=cv,
        random_state=42,
        n_alphas=100,
        max_iter=10000
    )
    lasso.fit(X, y)

    # Get coefficients
    coef_df = pd.DataFrame({
        'feature': X.columns,
        'coefficient': lasso.coef_
    })

    # Features with non-zero coefficients
    selected_features = coef_df[np.abs(coef_df['coefficient']) > 0]['feature'].tolist()

    print(f"Selected {len(selected_features)} features out of {X.shape[1]}")

    return selected_features, coef_df, lasso

5.4 Tree-Based Feature Importance
Random Forest and XGBoost provide built-in feature importance.

text
def tree_feature_importance(model, X, feature_names, top_k=20):
    """
    Extract feature importance from tree-based model.
    """
    importance_df = pd.DataFrame({
        'feature': feature_names,
        'importance': model.feature_importances_
    }).sort_values('importance', ascending=False)

    selected_features = importance_df.head(top_k)['feature'].tolist()

    return selected_features, importance_df

6. Feature Engineering – Creating Predictive Features

6.1 Interaction Features
Financial relationships are often non-linear and interactive.

text
def create_interaction_features(X, top_k=5):
    """
    Create interaction features for top features.
    """
    # Select top features by variance
    variances = X.var()
    top_features = variances.nlargest(top_k).index.tolist()

    # Create interactions
    for i, f1 in enumerate(top_features):
        for f2 in top_features[i+1:]:
            X[f'{f1}_x_{f2}'] = X[f1] * X[f2]

    return X

6.2 Polynomial Features

text
from sklearn.preprocessing import PolynomialFeatures

def create_polynomial_features(X, degree=2, interaction_only=True):
    """
    Create polynomial features.
    """
    poly = PolynomialFeatures(
        degree=degree,
        interaction_only=interaction_only,
        include_bias=False
    )
    X_poly = poly.fit_transform(X)

    # Get feature names
    feature_names = poly.get_feature_names_out(X.columns)

    return X_poly, feature_names, poly

6.3 Lagged Features for Time Series

text
def create_lagged_features(X, lags=[1, 2, 3, 5, 10]):
    """
    Create lagged features for time series.
    """
    X_lagged = X.copy()
    for lag in lags:
        for col in X.columns:
            X_lagged[f'{col}_lag_{lag}'] = X[col].shift(lag)
    return X_lagged

7. Summary for the AI Practitioner

  1. Curse of Dimensionality: More features is not better. Use feature reduction to avoid overfitting.

  2. PCA is the workhorse for linear dimensionality reduction. Use it to extract latent factors from returns.

  3. t-SNE and UMAP are for visualisation only. They cannot be used for live predictions.

  4. Mutual information captures non-linear dependencies. Use it for feature selection.

  5. Lasso performs automatic feature selection through L1 regularisation.

  6. RFE selects features based on model importance. Works with any model that provides feature importance or coefficients.

  7. Tree-based importance is fast and interpretable. Use it with Random Forest or XGBoost.

  8. Feature engineering (interactions, polynomials, lags) can significantly improve model performance.

 

 
 
PreviousNext