LESSON 3: SUPPORT VECTOR MACHINES (SVM) – CLASSIFICATION WITH MARGINS IN FINANCIAL DATA

SECTION 1: LEARNING OBJECTIVES

By the end of this lesson, you will be able to:

Understand the geometric intuition behind Support Vector Machines – maximising the margin between classes.
Distinguish between hard-margin and soft-margin SVM and explain when each is appropriate.
Apply the kernel trick to handle non-linear relationships in financial data without explicitly creating polynomial features.
Interpret SVM outputs – support vectors, margin width, and the role of the regularisation parameter C.
Evaluate SVM performance on financial classification tasks (e.g., fraud detection, credit scoring, market direction prediction).
Compare SVM with logistic regression and Random Forest in terms of interpretability, computational cost, and predictive power.
Implement SVM in Python using scikit-learn with linear, polynomial, and RBF kernels.
Understand the business applications of SVM in banking and finance – including alternative data classification and anomaly detection.

SECTION 2: THE GEOMETRIC INTUITION – MAXIMISING THE MARGIN

A Support Vector Machine (SVM) is a maximum margin classifier. Its goal is to find the hyperplane that separates two classes while maximising the distance (margin) between the hyperplane and the nearest data points from each class.

Why margin matters:

A wider margin generally leads to better generalisation (lower test error) because the decision boundary is more robust to small changes in the data.
SVMs focus only on the support vectors – the data points closest to the decision boundary – making the model memory-efficient.

Linear Separability:
In its simplest form, SVM assumes the data are linearly separable. The decision boundary is:

$w \cdot x + b = 0$

where:

$w$ is the weight vector (normal to the hyperplane)
$b$ is the bias (intercept)

Classification rule: Predict $y = + 1$ if $w \cdot x + b > 0$ , and $y = - 1$ otherwise.

Margin: The distance from the hyperplane to the closest point of each class is $∥ w ∥ 1$ . The total margin is $∥ w ∥ 2$ .

Objective: Maximise the margin, equivalent to minimising $∥ w ∥$ , subject to:

$y_{i} (w \cdot x_{i} + b) \geq 1, \forall i$

This is the hard-margin SVM – it assumes perfect separation.

SECTION 3: SOFT-MARGIN SVM – HANDLING REAL-WORLD DATA

In practice, financial data is rarely perfectly separable. Soft-margin SVM introduces slack variables $ξ_{i} \geq 0$ that allow some misclassifications:

$y_{i} (w \cdot x_{i} + b) \geq 1 - ξ_{i}, ξ_{i} \geq 0$

The optimisation objective becomes:

$w, b, ξ min 21 ∥ w ∥^{2} + C i = 1 \sum n ξ_{i}$

The regularisation parameter $C$ controls the trade-off:

Large $C$ : Penalises misclassifications heavily → narrower margin, potentially overfitting.
Small $C$ : Allows more misclassifications → wider margin, potentially underfitting.

Financial interpretation:

In credit scoring, a high $C$ might be appropriate when default classification is critical (minimise Type II error).
In fraud detection, $C$ is often tuned to balance false positives (annoying customers) and false negatives (financial losses).

SECTION 4: THE KERNEL TRICK – NON-LINEAR DECISION BOUNDARIES

Many financial relationships are non-linear. The kernel trick allows SVM to implicitly map data into a higher-dimensional space where a linear hyperplane can separate them, without explicitly computing the transformation.

Common kernels in finance:

Kernel	Formula	Use Case
Linear	$K (x_{i}, x_{j}) = x_{i} \cdot x_{j}$	Baseline, interpretable, high-dimensional text data
Polynomial	$K (x_{i}, x_{j}) = (γ x_{i} \cdot x_{j} + r)^{d}$	Captures interactions; used for non-linear but smooth relationships
Radial Basis Function (RBF)	$K (x_{i}, x_{j}) = exp (- γ ∥ x_{i} - x_{j} ∥^{2})$	Most popular; handles complex non-linear boundaries; requires tuning of $γ$ and $C$
Sigmoid	$K (x_{i}, x_{j}) = tanh (γ x_{i} \cdot x_{j} + r)$	Neural network-like; less common in finance

The RBF kernel is the default choice for most financial datasets because:

It can approximate any continuous function (universal approximator).
It only has one additional parameter $γ$ to tune.

Interpretation of $γ$ :

Small $γ$ : Each point has a wide influence → smoother decision boundary, underfitting.
Large $γ$ : Each point has a narrow influence → wiggly boundary, overfitting.

SECTION 5: SVM IN FINANCE – KEY APPLICATIONS

Application	SVM Role	Why SVM?
Credit Scoring	Classify default vs. non-default	Handles non-linearities; performs well with small-to-medium datasets; support vectors can be audited.
Fraud Detection	Classify fraudulent vs. legitimate transactions	RBF kernel captures complex patterns; can be combined with anomaly detection techniques.
Market Direction Prediction	Predict up/down movement of stock/index	Captures non-linear relationships between technical indicators and price movement.
Sentiment Analysis	Classify news sentiment (positive/negative) for algorithmic trading	Linear SVM is highly effective for text classification (high-dimensional sparse data).
Credit Card Approval	Approve/deny based on applicant profile	Margin maximisation provides robust decisions; interpretable with linear kernel.
Corporate Bond Rating	Predict rating class (investment grade vs. speculative)	Handles multi-class extensions; performs well with financial ratios.

Regulatory considerations:

Linear SVM (with linear kernel) is generally acceptable for regulatory models because it can be expressed as a linear combination of features (similar to logistic regression).
RBF kernels are considered “black-box” and require additional explanation (SHAP, LIME) for SR 11‑7 compliance.
Always document the choice of kernel, hyperparameters, and validation results.

SECTION 6: IMPLEMENTATION IN PYTHON

We’ll apply SVM to our credit default dataset using linear, polynomial, and RBF kernels, comparing their performance.

# ===================================================================
# MODULE 4, LESSON 3: SUPPORT VECTOR MACHINES FOR CREDIT SCORING
# ===================================================================

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (roc_auc_score, classification_report, 
                             confusion_matrix, roc_curve)
from sklearn.pipeline import Pipeline
import time

# Reuse dataset from Lesson 1 (already generated)
# X_train_scaled, X_test_scaled, y_train, y_test available

print("="*70)
print("SUPPORT VECTOR MACHINES FOR CREDIT DEFAULT PREDICTION")
print("="*70)

# ----------------------------------------------------------------
# PART A: SVM WITH LINEAR KERNEL (BASELINE)
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART A: LINEAR SVM (Interpretable, similar to logistic regression)")
print("-"*60)

start_time = time.time()
svm_linear = SVC(kernel='linear', C=1.0, probability=True, random_state=42)
svm_linear.fit(X_train_scaled, y_train)
linear_time = time.time() - start_time

train_auc_lin = roc_auc_score(y_train, svm_linear.predict_proba(X_train_scaled)[:, 1])
test_auc_lin = roc_auc_score(y_test, svm_linear.predict_proba(X_test_scaled)[:, 1])

print(f"Training AUC: {train_auc_lin:.4f}")
print(f"Test AUC:     {test_auc_lin:.4f}")
print(f"Training time: {linear_time:.2f} seconds")

# Extract coefficients (weights) for interpretation
coef = svm_linear.coef_[0]
feature_names = ['income', 'age', 'dti', 'credit_score', 'loan_amount']
coef_df = pd.DataFrame({'feature': feature_names, 'coef': coef}).sort_values('coef', ascending=False)

print("\nFeature Weights (Linear SVM):")
print(coef_df.to_string(index=False))
print("\nInterpretation: Positive weights increase default probability; negative weights decrease it.")

# ----------------------------------------------------------------
# PART B: SVM WITH RBF KERNEL (DEFAULT, NON-LINEAR)
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART B: RBF KERNEL SVM (Default, captures non-linearities)")
print("-"*60)

start_time = time.time()
svm_rbf = SVC(kernel='rbf', C=1.0, gamma='scale', probability=True, random_state=42)
svm_rbf.fit(X_train_scaled, y_train)
rbf_time = time.time() - start_time

train_auc_rbf = roc_auc_score(y_train, svm_rbf.predict_proba(X_train_scaled)[:, 1])
test_auc_rbf = roc_auc_score(y_test, svm_rbf.predict_proba(X_test_scaled)[:, 1])

print(f"Training AUC: {train_auc_rbf:.4f}")
print(f"Test AUC:     {test_auc_rbf:.4f}")
print(f"Training time: {rbf_time:.2f} seconds")

# Number of support vectors (indicates model complexity)
n_sv = len(svm_rbf.support_vectors_)
print(f"Number of Support Vectors: {n_sv} out of {len(X_train_scaled)} training samples")
print(f"Support vector ratio: {n_sv/len(X_train_scaled):.2%}")

# ----------------------------------------------------------------
# PART C: HYPERPARAMETER TUNING WITH GRID SEARCH
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART C: HYPERPARAMETER TUNING (GridSearchCV for RBF SVM)")
print("-"*60)

# Define parameter grid
param_grid = {
    'C': [0.1, 1.0, 10.0, 100.0],
    'gamma': ['scale', 'auto', 0.001, 0.01, 0.1, 1.0]
}

# Use a reduced grid for demonstration speed
# In practice, use all combinations, but this can be computationally expensive
param_grid_reduced = {
    'C': [0.1, 1.0, 10.0],
    'gamma': ['scale', 0.01, 0.1]
}

grid_search = GridSearchCV(
    SVC(kernel='rbf', probability=True, random_state=42),
    param_grid_reduced,
    cv=5,
    scoring='roc_auc',
    n_jobs=-1,
    verbose=1
)
grid_search.fit(X_train_scaled, y_train)

print(f"\nBest parameters: {grid_search.best_params_}")
print(f"Best cross-validated AUC: {grid_search.best_score_:.4f}")

best_svm = grid_search.best_estimator_
test_auc_best = roc_auc_score(y_test, best_svm.predict_proba(X_test_scaled)[:, 1])
print(f"Test AUC with best model: {test_auc_best:.4f}")

# ----------------------------------------------------------------
# PART D: POLYNOMIAL KERNEL (FOR INTERACTIONS)
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART D: POLYNOMIAL KERNEL SVM (degree=2)")
print("-"*60)

svm_poly = SVC(kernel='poly', degree=2, C=1.0, gamma='scale', probability=True, random_state=42)
svm_poly.fit(X_train_scaled, y_train)

train_auc_poly = roc_auc_score(y_train, svm_poly.predict_proba(X_train_scaled)[:, 1])
test_auc_poly = roc_auc_score(y_test, svm_poly.predict_proba(X_test_scaled)[:, 1])

print(f"Training AUC: {train_auc_poly:.4f}")
print(f"Test AUC:     {test_auc_poly:.4f}")

# ----------------------------------------------------------------
# PART E: MODEL COMPARISON AND VISUALISATION
# ----------------------------------------------------------------

print("\n" + "="*70)
print("PART E: MODEL COMPARISON")
print("="*70)

comparison = pd.DataFrame({
    'Model': ['Linear SVM', 'RBF SVM (default)', 'RBF SVM (tuned)', 'Polynomial SVM (deg=2)'],
    'Test AUC': [test_auc_lin, test_auc_rbf, test_auc_best, test_auc_poly],
    'Interpretability': ['High (linear weights)', 'Low (black-box)', 'Low (black-box)', 'Moderate (polynomial)'],
    'Training Time (s)': [linear_time, rbf_time, grid_search.cv_results_['mean_fit_time'].mean(), 0]
})
print(comparison.to_string(index=False))

# Visualisation: ROC Curves
fig, ax = plt.subplots(figsize=(10, 7))

# Plot ROC for each model
for model, name, color in [
    (svm_linear, 'Linear SVM', 'blue'),
    (svm_rbf, 'RBF SVM (default)', 'green'),
    (best_svm, 'RBF SVM (tuned)', 'red'),
    (svm_poly, 'Polynomial SVM', 'orange')
]:
    y_prob = model.predict_proba(X_test_scaled)[:, 1]
    fpr, tpr, _ = roc_curve(y_test, y_prob)
    auc = roc_auc_score(y_test, y_prob)
    ax.plot(fpr, tpr, color=color, linewidth=2, label=f'{name} (AUC={auc:.3f})')

ax.plot([0, 1], [0, 1], 'k--', linewidth=1, label='Random (AUC=0.5)')
ax.set_xlabel('False Positive Rate (1 - Specificity)')
ax.set_ylabel('True Positive Rate (Recall)')
ax.set_title('ROC Curve Comparison – SVM Kernels', fontsize=14)
ax.legend(loc='lower right')
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('svm_roc_comparison.png', dpi=300)
plt.show()

# ----------------------------------------------------------------
# PART F: CONFUSION MATRIX FOR BEST MODEL
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART F: CONFUSION MATRIX – BEST RBF SVM")
print("-"*60)

y_pred_best = best_svm.predict(X_test_scaled)
cm = confusion_matrix(y_test, y_pred_best)

fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', 
            xticklabels=['No Default', 'Default'],
            yticklabels=['No Default', 'Default'],
            ax=ax)
ax.set_title('Confusion Matrix – Tuned RBF SVM', fontsize=14)
ax.set_ylabel('Actual')
ax.set_xlabel('Predicted')
plt.tight_layout()
plt.savefig('svm_confusion_matrix.png', dpi=300)
plt.show()

# Classification report
print("\nClassification Report (Best RBF SVM):")
print(classification_report(y_test, y_pred_best, target_names=['No Default', 'Default']))

# ----------------------------------------------------------------
# PART G: BUSINESS INTERPRETATION OF SUPPORT VECTORS
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART G: BUSINESS INTERPRETATION")
print("-"*60)

# Extract support vectors and their indices
support_indices = best_svm.support_
support_labels = y_train.iloc[support_indices]

print(f"Number of support vectors: {len(support_indices)}")
print(f"Support vector composition: {np.sum(support_labels)} defaults, {len(support_indices) - np.sum(support_labels)} non-defaults")

# Show some characteristics of support vectors vs. non-support vectors
support_df = pd.DataFrame(X_train_scaled[support_indices], columns=feature_names)
support_df['default'] = support_labels.values
non_support_df = pd.DataFrame(X_train_scaled[~np.isin(np.arange(len(X_train_scaled)), support_indices)], 
                              columns=feature_names)
non_support_df['default'] = y_train.iloc[~np.isin(np.arange(len(X_train_scaled)), support_indices)]

print("\nMean feature values: Support Vectors vs Non-Support Vectors")
comparison_sv = pd.DataFrame({
    'feature': feature_names,
    'support_vec_mean': support_df[feature_names].mean(),
    'non_support_mean': non_support_df[feature_names].mean(),
    'difference': support_df[feature_names].mean() - non_support_df[feature_names].mean()
})
print(comparison_sv.to_string(index=False))

print("\nBusiness Insight:")
print("  - Support vectors represent the most 'difficult' or 'borderline' cases.")
print("  - These are the loans that are closest to the decision boundary.")
print("  - Monitoring support vector characteristics over time helps detect drift.")

SECTION 7: ADVANTAGES AND DISADVANTAGES IN FINANCE

Aspect	Pro	Con
Interpretability	Linear SVM is highly interpretable (weights).	RBF/Poly kernels are black‑box; require SHAP/LIME.
Performance	Excellent with high‑dimensional data; robust to overfitting (with proper C).	Slower training than logistic regression; not ideal for >100K samples.
Regulatory	Linear SVM can be validated like logistic regression.	RBF requires extensive validation and explanation.
Handling non‑linearity	Kernel trick captures complex patterns.	Kernel choice and hyperparameters require expertise.
Memory	Only support vectors are stored – memory‑efficient.	Number of support vectors grows with data size.

SECTION 8: SUMMARY FOR THE DATA PRACTITIONER

SVM finds the maximum margin hyperplane that separates classes.
Soft‑margin with slack variables handles non‑separable data.
Kernels (linear, polynomial, RBF) allow non‑linear decision boundaries.
Linear SVM is a strong competitor to logistic regression – similar performance, but often more robust.
RBF SVM is powerful for complex patterns but is considered a “black‑box” model.
Hyperparameter tuning (C, γ) is critical for optimal performance.
In banking, use linear SVM for regulatory models and RBF SVM for internal risk ranking and fraud detection (with additional explainability).

[END OF LESSON 3 – MODULE 4]

Diploma in Financial Data Analytics LESSON 3: SUPPORT VECTOR MACHINES (SVM) – CLASSIFICATION WITH MARGINS IN FINANCIAL DATA

Diploma in Financial Data Analytics