1. Learning Objectives

By the end of this lesson, you will be able to:

• Understand the mathematical formulation of Support Vector Machines (SVM) for classification.

• Derive the primal and dual formulations of SVM and interpret the KKT conditions.

• Apply SVM to credit scoring, market direction prediction, and fraud detection.

• Understand the kernel trick and implement non-linear classification with RBF, polynomial, and sigmoid kernels.

• Interpret support vectors and their role in the decision boundary.

• Tune SVM hyperparameters (C, gamma) using cross-validation for financial data.

• Implement one-class SVM for anomaly detection in financial time series.

2. Support Vector Machines – The Mathematical Foundation

SVM finds the hyperplane that maximises the margin between two classes. It is a maximum-margin classifier.

2.1 The Separable Case (Hard Margin)

Given training data {(x_i, y_i)} for i = 1, ..., N with y_i ∈ {-1, +1}.

Decision Boundary: w^T x + b = 0

Classification Rule: y_i (w^T x_i + b) ≥ 1 for all i (perfect separation).

Margin: The distance from the decision boundary to the closest points:
margin = 2 / ||w||

Optimisation Problem:
minimize_{w, b} (1/2) ||w||²
subject to y_i (w^T x_i + b) ≥ 1 for i = 1, ..., N

2.2 The Non-Separable Case (Soft Margin)

Introduce slack variables ξ_i ≥ 0 to allow misclassifications.

Optimisation Problem:
minimize_{w, b, ξ} (1/2) ||w||² + C Σ_{i=1}^{N} ξ_i
subject to y_i (w^T x_i + b) ≥ 1 - ξ_i, ξ_i ≥ 0

Interpretation of C:

• C → ∞: Hard margin (no misclassifications allowed). Prone to overfitting.

• C → 0: Very soft margin. Prone to underfitting.

• In finance, C is often small to prevent overfitting to noisy data.

2.3 The Lagrangian and Dual Formulation

Lagrangian:
L(w, b, ξ, α, β) = (1/2) ||w||² + C Σ ξ_i - Σ α_i [y_i (w^T x_i + b) - 1 + ξ_i] - Σ β_i ξ_i

KKT Conditions:

1. ∂L/∂w = 0 → w = Σ α_i y_i x_i

2. ∂L/∂b = 0 → Σ α_i y_i = 0

3. ∂L/∂ξ_i = 0 → C - α_i - β_i = 0 → 0 ≤ α_i ≤ C

Dual Problem:
maximize_α Σ_{i=1}^{N} α_i - (1/2) Σ_{i=1}^{N} Σ_{j=1}^{N} α_i α_j y_i y_j x_i^T x_j
subject to Σ_{i=1}^{N} α_i y_i = 0, 0 ≤ α_i ≤ C

Decision Function:
f(x) = sign( Σ_{i=1}^{N} α_i y_i x_i^T x + b )

Support Vectors: Data points with α_i > 0. Only these points define the decision boundary.

3. The Kernel Trick – Non-Linear Classification

The kernel trick allows SVM to find non-linear decision boundaries without explicitly computing the feature transformation.

3.1 Mathematical Formulation

Instead of computing x_i^T x_j, we compute K(x_i, x_j) = φ(x_i)^T φ(x_j) where φ is a feature map to a higher-dimensional space.

3.2 Common Kernels

3.3 The RBF Kernel in Detail
K(x_i, x_j) = exp( - ||x_i - x_j||² / (2σ²) )

Interpretation:

• γ = 1/(2σ²). Larger γ → smaller σ → tighter RBF → more complex decision boundary (higher variance).

• Smaller γ → larger σ → smoother decision boundary (higher bias).

Financial Implication: In finance, the RBF kernel is preferred because market data has complex, non-linear relationships. However, γ must be carefully tuned to avoid overfitting.

3.4 Kernel PCA for Non-Linear Factor Extraction

Kernel PCA extends PCA to non-linear feature spaces.

Algorithm:

1. Compute the kernel matrix K where K_{ij} = K(x_i, x_j).

2. Center the kernel matrix: K_c = K - 1_N K - K 1_N + 1_N K 1_N.

3. Eigen-decompose K_c to get the principal components in the feature space.

from sklearn.decomposition import KernelPCA

def kernel_pca_factors(returns, kernel='rbf', n_components=5, gamma=0.1):
    """
    Extract non-linear factors using Kernel PCA.
    """
    kpca = KernelPCA(
        n_components=n_components,
        kernel=kernel,
        gamma=gamma,
        fit_inverse_transform=False,
        random_state=42
    )

    factors = kpca.fit_transform(returns)
    factor_df = pd.DataFrame(
        factors,
        index=returns.index,
        columns=[f'KernelFactor_{i+1}' for i in range(n_components)]
    )

    return factor_df, kpca

4. SVM Implementation for Financial Applications

4.1 Credit Scoring with SVM

from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

def credit_scoring_svm(features, target, kernel='rbf'):
    """
    SVM for credit default prediction.
    """
    # Train-test split
    X_train, X_test, y_train, y_test = train_test_split(
        features, target, test_size=0.3, random_state=42, stratify=target
    )

    # Standardise features (SVM is scale-sensitive)
    scaler = StandardScaler()
    X_train_scaled = scaler.fit_transform(X_train)
    X_test_scaled = scaler.transform(X_test)

    # SVM with class weighting
    model = SVC(
        kernel=kernel,
        C=1.0,
        gamma='scale' if kernel == 'rbf' else 'auto',
        class_weight='balanced',
        probability=True,
        random_state=42
    )

    # Grid search for hyperparameters
    from sklearn.model_selection import GridSearchCV

    if kernel == 'rbf':
        param_grid = {
            'C': [0.1, 1.0, 10.0],
            'gamma': ['scale', 'auto', 0.1, 0.01, 0.001]
        }
    elif kernel == 'linear':
        param_grid = {'C': [0.1, 1.0, 10.0]}
    elif kernel == 'poly':
        param_grid = {
            'C': [0.1, 1.0, 10.0],
            'degree': [2, 3, 4],
            'coef0': [0, 0.5, 1.0]
        }

    gs = GridSearchCV(
        model, param_grid, cv=5, scoring='roc_auc', n_jobs=-1
    )
    gs.fit(X_train_scaled, y_train)

    print(f"Best parameters: {gs.best_params_}")
    print(f"Best AUC: {gs.best_score_:.4f}")

    # Evaluate on test set
    y_pred = gs.predict(X_test_scaled)
    y_pred_proba = gs.predict_proba(X_test_scaled)[:, 1]

    from sklearn.metrics import classification_report, roc_auc_score, confusion_matrix
    print(classification_report(y_test, y_pred))
    print(f"Test AUC: {roc_auc_score(y_test, y_pred_proba):.4f}")

    # Identify support vectors
    n_support = gs.best_estimator_.n_support_
    print(f"Support vectors per class: {n_support}")

    return gs.best_estimator_, scaler

4.2 Market Direction Prediction with SVM

def market_direction_svm(features, target, lookback=60):
    """
    SVM for market direction prediction.
    """
    from sklearn.model_selection import TimeSeriesSplit

    # Standardise
    scaler = StandardScaler()
    X_scaled = scaler.fit_transform(features)

    # Time series cross-validation
    tscv = TimeSeriesSplit(n_splits=5, test_size=60)

    aucs = []
    accuracies = []

    for train_idx, val_idx in tscv.split(X_scaled):
        X_train, X_val = X_scaled[train_idx], X_scaled[val_idx]
        y_train, y_val = target.iloc[train_idx], target.iloc[val_idx]

        model = SVC(
            kernel='rbf',
            C=1.0,
            gamma='scale',
            class_weight='balanced',
            probability=True,
            random_state=42
        )
        model.fit(X_train, y_train)

        y_pred = model.predict(X_val)
        y_pred_proba = model.predict_proba(X_val)[:, 1]

        aucs.append(roc_auc_score(y_val, y_pred_proba))
        accuracies.append(np.mean(y_pred == y_val))

    print(f"Average Accuracy: {np.mean(accuracies):.4f} ± {np.std(accuracies):.4f}")
    print(f"Average AUC: {np.mean(aucs):.4f} ± {np.std(aucs):.4f}")

    return model, scaler

4.3 One-Class SVM for Anomaly Detection

One-class SVM identifies the boundary of the normal data distribution. It is used for fraud detection and outlier detection.

Mathematical Formulation:
minimize_{w, ξ, ρ} (1/2) ||w||² - ρ + (1/(νN)) Σ ξ_i
subject to w^T φ(x_i) ≥ ρ - ξ_i, ξ_i ≥ 0

ν ∈ (0, 1] controls the fraction of outliers expected.

from sklearn.svm import OneClassSVM

def one_class_svm_anomaly(returns, nu=0.05, gamma='scale'):
    """
    One-Class SVM for anomaly detection in returns.
    """
    # Standardise
    scaler = StandardScaler()
    returns_scaled = scaler.fit_transform(returns)

    # One-class SVM
    model = OneClassSVM(
        nu=nu,  # Expected fraction of outliers
        kernel='rbf',
        gamma=gamma,
        random_state=42
    )

    predictions = model.fit_predict(returns_scaled)

    # 1 = normal, -1 = anomaly
    anomaly_indices = np.where(predictions == -1)[0]
    anomaly_scores = model.decision_function(returns_scaled)

    # Create results
    result_df = pd.DataFrame({
        'returns': returns.iloc[:, 0],
        'anomaly': predictions == -1,
        'anomaly_score': anomaly_scores
    }, index=returns.index)

    print(f"Detected {len(anomaly_indices)} anomalies ({len(anomaly_indices)/len(returns):.2%})")

    return result_df, model, scaler

5. Hyperparameter Tuning for SVM in Finance

5.1 C (Regularisation Parameter)

• Low C: Large margin, more misclassifications allowed. Lower variance, higher bias.

• High C: Small margin, fewer misclassifications allowed. Higher variance, lower bias.

• Financial Rule: Start with C = 1.0. Increase if underfitting, decrease if overfitting.

5.2 Gamma (RBF Kernel Parameter)

• Low γ: Broad RBF, smooth decision boundary. Lower variance, higher bias.

• High γ: Tight RBF, wiggly decision boundary. Higher variance, lower bias.

• Financial Rule: Use gamma = 'scale' (default in sklearn) which uses 1/(n_features * X.var()).

5.3 Class Weight

• In finance, classes are often imbalanced (e.g., defaults are rare).

• Use class_weight='balanced' to automatically weight classes inversely to their frequency.

5.4 Cross-Validation Strategy

• For financial time series, use TimeSeriesSplit (not KFold).

• Ensure the test set is always chronologically after the training set.

6. Interpretability of SVM Models

6.1 Support Vectors
Support vectors are the data points that define the decision boundary. In finance, they represent the most “informative” or “critical” observations.

6.2 Feature Importance (for Linear SVM)
For linear SVM, feature importance is |w_j| (the absolute value of the coefficient).

def linear_svm_feature_importance(model, feature_names):
    """
    Extract feature importance from linear SVM.
    """
    if model.kernel != 'linear':
        raise ValueError("Feature importance only for linear SVM")

    coefficients = model.coef_.flatten()
    importance_df = pd.DataFrame({
        'feature': feature_names,
        'coefficient': coefficients,
        'importance': np.abs(coefficients)
    }).sort_values('importance', ascending=False)

    return importance_df

6.3 SHAP for Non-Linear SVM
For non-linear SVM (RBF kernel), use SHAP to explain predictions.

def svm_shap_explanation(model, X_data, feature_names):
    """
    Explain SVM predictions using SHAP.
    """
    import shap

    # Use KernelExplainer for any black-box model
    explainer = shap.KernelExplainer(model.predict_proba, X_data[:100])
    shap_values = explainer.shap_values(X_data[:100])

    # Summary plot
    shap.summary_plot(shap_values, X_data[:100], feature_names=feature_names)

    return shap_values

7. Summary for the AI Practitioner

1. SVM finds the maximum-margin hyperplane. It is effective for high-dimensional financial data.

2. The kernel trick enables non-linear classification without explicitly transforming features. RBF is the most common kernel in finance.

3. C controls the tradeoff between margin and misclassification. Use cross-validation to tune it.

4. Gamma controls the RBF kernel width. Smaller values smooth the decision boundary; larger values capture complex patterns (but risk overfitting).

5. Support vectors are the most informative data points. Only a fraction of data points define the decision boundary.

6. One-class SVM detects anomalies without labeled data. Useful for fraud detection and market surveillance.

7. SVM is scale-sensitive: Always standardise features before training.