SECTION 1: LEARNING OBJECTIVES

By the end of this lesson, you will be able to:

  • Understand the core intuition of boosting: sequentially adding weak learners to correct the errors of previous models.

  • Distinguish between bagging (Random Forest) and boosting (GBM, XGBoost, LightGBM, CatBoost).

  • Explain the gradient descent framework used in GBM – minimising a loss function by fitting trees to residuals.

  • Identify key hyperparameters – learning rate, n_estimators, tree depth, subsample, and regularisation parameters.

  • Apply XGBoost (eXtreme Gradient Boosting) to a financial dataset, comparing performance with logistic regression and Random Forest.

  • Interpret feature importance and SHAP values for explainability in a regulatory context.

  • Understand the business applications – credit scoring, fraud detection, churn prediction, and algorithmic trading.

SECTION 2: THE INTUITION OF BOOSTING

Boosting is an ensemble method that builds models sequentially, where each new model attempts to correct the errors of its predecessor.

Key insight: Instead of building many independent models (like Random Forest), boosting builds models that learn from the mistakes of previous models.

Historical context:

  • AdaBoost (1995): First successful boosting algorithm; reweights misclassified samples.

  • Gradient Boosting Machine (GBM) – Friedman (2001): Generalises boosting to any differentiable loss function using gradient descent.

  • XGBoost (2014): Scalable, regularised GBM that dominated Kaggle competitions.

  • LightGBM (2017): Faster and more memory‑efficient using histogram‑based splitting.

  • CatBoost (2017): Handles categorical variables natively; reduces overfitting.

In finance, XGBoost and LightGBM are the industry standards for credit scoring, fraud detection, and customer analytics.

SECTION 3: THE GRADIENT BOOSTING FRAMEWORK – MATHEMATICAL FOUNDATION

We want to predict yi using an additive model of M weak learners (typically shallow decision trees):

y^i=FM(xi)=∑m=1Mfm(xi)

where each fm is a decision tree.

Training process (Gradient Descent in Function Space):

  1. Initialise: F0(x)=arg⁡min⁡γ∑i=1nL(yi,γ)

    • For regression with squared error: F0=yˉ

    • For classification with log‑loss: F0=log⁡(p1−p)

  2. For each iteration m=1 to M:

    a. Compute pseudo‑residuals: For each observation i:

    rim=−[∂L(yi,F(xi))∂F(xi)]F=Fm−1

    • For squared error: rim=yi−Fm−1(xi) (standard residual)

    • For binary classification (log‑loss): rim=yi−pm−1(xi) (difference between actual and predicted probability)

    b. Fit a tree fm to the pseudo‑residuals rim using the features xi.

    c. Find the optimal leaf values γjm for each leaf region Rjm:

    γjm=arg⁡min⁡γ∑xi∈RjmL(yi,Fm−1(xi)+γ)

    For regression with squared error, this is the mean of residuals in the leaf.
    For classification, it’s a log‑odds adjustment (requires Newton‑Raphson).

    d. Update the model:

    Fm(x)=Fm−1(x)+η⋅∑jγjm⋅1{x∈Rjm}

    where η is the learning rate (shrinkage) – a critical hyperparameter.

  3. Output: FM(x)

Key observation: GBM performs gradient descent in function space – we are iteratively adding functions that move us in the direction that reduces the loss.

SECTION 4: XGBOOST – THE INDUSTRY STANDARD

XGBoost (eXtreme Gradient Boosting) extends GBM with several innovations:

 
 
Feature Description Financial Benefit
Regularisation L1 (Lasso) and L2 (Ridge) penalties on leaf weights Reduces overfitting; improves generalisation.
Shrinkage (learning rate) Scales each tree’s contribution Prevents overfitting; allows more trees.
Column subsampling Randomly selects features for each tree Reduces correlation between trees; improves performance.
Histogram‑based splitting Approximate greedy algorithm for large datasets Faster training; memory efficient.
Early stopping Stops training when validation performance plateaus Saves time; prevents overfitting.
Handling missing values Learns optimal direction for missing values No need to impute; robust to real‑world data.
Cross‑validation built‑in Easy to tune hyperparameters Saves coding effort.

Financial interpretation of key hyperparameters:

 
 
Hyperparameter Effect Typical Range
n_estimators Number of boosting rounds 100‑1000 (use early stopping)
learning_rate Shrinkage factor 0.01‑0.3 (lower = more trees needed)
max_depth Tree depth 3‑10 (shallower = less overfitting)
min_child_weight Minimum sum of instance weights in a child 1‑10 (larger = more conservative)
subsample Fraction of samples used per tree 0.6‑1.0 (lower = more randomness)
colsample_bytree Fraction of features used per tree 0.6‑1.0
gamma Minimum loss reduction required to split 0‑1 (higher = more pruning)
reg_alpha L1 regularisation on weights 0‑1
reg_lambda L2 regularisation on weights 0‑1 (default 1)

SECTION 5: BUSINESS APPLICATIONS IN BANKING

 
 
Application Why XGBoost? Performance Gain
Credit Scoring Captures complex interactions; handles missing data; provides SHAP for explainability. AUC typically 0.78‑0.85 vs. 0.70‑0.75 for logistic regression.
Fraud Detection Fast training on millions of transactions; handles class imbalance with scale_pos_weight. Recall often improves by 10‑20% over Random Forest.
Churn Prediction Handles many features; robust to outliers; provides clear feature importance. Lift up to 30% in retention campaigns.
Marketing Response Predicts which customers will respond to offers; enables targeted campaigns. ROI improvement of 20‑40%.
Operational Risk Predicts likelihood of operational failures (e.g., system outages). Early warning capabilities.
Algorithmic Trading High‑frequency feature engineering; fast inference. Predictive accuracy improved over traditional models.

SECTION 6: REGULATORY AND EXPLAINABILITY CONSIDERATIONS

Despite its predictive power, XGBoost is a black‑box model for regulators. Under SR 11‑7, banks must provide model validation and explainability:

Explainability tools:

  1. SHAP (SHapley Additive exPlanations): Provides per‑feature contributions for each prediction.

    • For regulatory submission, SHAP values can show that the model’s behaviour aligns with business logic (e.g., higher DTI increases default risk).

  2. Feature Importance: Built‑in (gain, cover, frequency) – helps identify the most important drivers.

  3. Partial Dependence Plots (PDP): Shows the marginal effect of a feature on the prediction.

  4. LIME: Local explanations for individual predictions.

Validation requirements:

  • Out‑of‑time validation: Test on data from a different time period (critical for credit models).

  • Population stability: Monitor feature distributions and model performance over time.

  • Benchmarking: Compare XGBoost against simpler models (logistic regression) to ensure improvements are real and justifiable.

  • Disparate impact testing: Ensure the model does not discriminate against protected groups.

SECTION 7: IMPLEMENTATION IN PYTHON – XGBOOST

We’ll apply XGBoost to our credit dataset, compare it with previous models, and demonstrate SHAP for explainability.

python
# ===================================================================
# MODULE 4, LESSON 4: XGBOOST – STATE-OF-THE-ART GRADIENT BOOSTING
# ===================================================================

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import (roc_auc_score, classification_report, 
                             confusion_matrix, roc_curve)
import xgboost as xgb
import shap
import time
import warnings
warnings.filterwarnings('ignore')

# Reuse dataset from Lesson 1
# X_train_scaled, X_test_scaled, y_train, y_test available

print("="*70)
print("XGBOOST – GRADIENT BOOSTING FOR CREDIT DEFAULT PREDICTION")
print("="*70)

# ----------------------------------------------------------------
# PART A: XGBOOST WITH DEFAULT PARAMETERS
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART A: XGBOOST (Default Parameters)")
print("-"*60)

start_time = time.time()
xgb_default = xgb.XGBClassifier(
    n_estimators=100,
    learning_rate=0.3,
    max_depth=6,
    random_state=42,
    use_label_encoder=False,
    eval_metric='logloss'
)
xgb_default.fit(X_train_scaled, y_train)
time_default = time.time() - start_time

train_auc_def = roc_auc_score(y_train, xgb_default.predict_proba(X_train_scaled)[:,1])
test_auc_def = roc_auc_score(y_test, xgb_default.predict_proba(X_test_scaled)[:,1])

print(f"Training AUC: {train_auc_def:.4f}")
print(f"Test AUC:     {test_auc_def:.4f}")
print(f"Training time: {time_default:.2f} seconds")

# ----------------------------------------------------------------
# PART B: TUNED XGBOOST WITH EARLY STOPPING AND REGULARISATION
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART B: TUNED XGBOOST (Regularised, with Early Stopping)")
print("-"*60)

# Create a validation set for early stopping
X_train_sub, X_val, y_train_sub, y_val = train_test_split(
    X_train_scaled, y_train, test_size=0.2, random_state=42
)

# Tuned model with regularisation
xgb_tuned = xgb.XGBClassifier(
    n_estimators=500,
    learning_rate=0.05,
    max_depth=5,
    min_child_weight=3,
    subsample=0.8,
    colsample_bytree=0.8,
    gamma=0.1,
    reg_alpha=0.1,
    reg_lambda=1.0,
    random_state=42,
    use_label_encoder=False,
    eval_metric='logloss',
    early_stopping_rounds=50
)

start_time = time.time()
xgb_tuned.fit(
    X_train_sub, y_train_sub,
    eval_set=[(X_val, y_val)],
    verbose=False
)
time_tuned = time.time() - start_time

# Get best iteration
best_iteration = xgb_tuned.best_iteration

train_auc_tuned = roc_auc_score(y_train, xgb_tuned.predict_proba(X_train_scaled)[:,1])
test_auc_tuned = roc_auc_score(y_test, xgb_tuned.predict_proba(X_test_scaled)[:,1])

print(f"Best iteration (n_estimators): {best_iteration}")
print(f"Training AUC: {train_auc_tuned:.4f}")
print(f"Test AUC:     {test_auc_tuned:.4f}")
print(f"Training time: {time_tuned:.2f} seconds")

# ----------------------------------------------------------------
# PART C: FEATURE IMPORTANCE
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART C: FEATURE IMPORTANCE (Gain vs. Frequency)")
print("-"*60)

# Gain-based importance (default) – relative contribution to model performance
gain_importance = pd.DataFrame({
    'feature': ['income', 'age', 'dti', 'credit_score', 'loan_amount'],
    'gain': xgb_tuned.feature_importances_,
    'frequency': xgb_tuned.get_booster().get_fscore().values()
}).sort_values('gain', ascending=False)

print("\nGain Importance (Improvement in Accuracy when feature is used):")
print(gain_importance.to_string(index=False))

# Visualise feature importance
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

ax = axes[0]
sns.barplot(data=gain_importance, x='gain', y='feature', ax=ax)
ax.set_title('XGBoost Feature Importance (Gain)', fontsize=12)
ax.set_xlabel('Average Gain')

ax = axes[1]
sns.barplot(data=gain_importance, x='frequency', y='feature', ax=ax)
ax.set_title('XGBoost Feature Importance (Frequency)', fontsize=12)
ax.set_xlabel('Number of Times Used in Splits')

plt.tight_layout()
plt.savefig('xgboost_feature_importance.png', dpi=300)
plt.show()

# ----------------------------------------------------------------
# PART D: SHAP EXPLANATIONS (REGULATORY EXPLAINABILITY)
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART D: SHAP VALUES – LOCAL & GLOBAL EXPLANATIONS")
print("-"*60)

# Create SHAP explainer
explainer = shap.TreeExplainer(xgb_tuned)
shap_values = explainer.shap_values(X_train_scaled[:100])  # Use subset for speed

# Global SHAP summary plot
fig, ax = plt.subplots(figsize=(12, 7))
shap.summary_plot(shap_values, X_train_scaled[:100], 
                  feature_names=['income', 'age', 'dti', 'credit_score', 'loan_amount'],
                  show=False)
plt.title('SHAP Summary Plot – Feature Impact on Default Probability', fontsize=14)
plt.tight_layout()
plt.savefig('shap_summary.png', dpi=300)
plt.show()

# SHAP dependence plot for the most important feature (credit_score)
fig, ax = plt.subplots(figsize=(10, 6))
shap.dependence_plot(3, shap_values, X_train_scaled[:100], 
                     feature_names=['income', 'age', 'dti', 'credit_score', 'loan_amount'],
                     show=False)
plt.title('SHAP Dependence – Credit Score Impact on Default Prediction', fontsize=14)
plt.tight_layout()
plt.savefig('shap_dependence_credit_score.png', dpi=300)
plt.show()

# SHAP beeswarm plot
fig, ax = plt.subplots(figsize=(12, 6))
shap.plots.beeswarm(shap_values[:100], max_display=5, show=False)
plt.title('SHAP Beeswarm – Feature Impact Distribution', fontsize=14)
plt.tight_layout()
plt.savefig('shap_beeswarm.png', dpi=300)
plt.show()

print("\n Business Insight from SHAP:")
print("  - Red: High feature value increases default risk (e.g., high DTI).")
print("  - Blue: Low feature value decreases default risk (e.g., high credit score).")
print("  - The width of the beeswarm indicates the distribution of SHAP values.")
print("  - For credit_score: SHAP values decrease as credit score increases (protective).")

# ----------------------------------------------------------------
# PART E: HYPERPARAMETER TUNING (GridSearchCV)
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART E: HYPERPARAMETER TUNING (GridSearchCV)")
print("-"*60)

param_grid = {
    'max_depth': [3, 5, 7],
    'learning_rate': [0.01, 0.05, 0.1],
    'subsample': [0.7, 0.8, 0.9],
    'colsample_bytree': [0.7, 0.8, 0.9]
}

# Reduced grid for speed demonstration
param_grid_reduced = {
    'max_depth': [3, 5],
    'learning_rate': [0.05, 0.1],
    'subsample': [0.8],
    'colsample_bytree': [0.8]
}

grid_search_xgb = GridSearchCV(
    xgb.XGBClassifier(
        n_estimators=100,
        random_state=42,
        use_label_encoder=False,
        eval_metric='logloss'
    ),
    param_grid_reduced,
    cv=3,
    scoring='roc_auc',
    n_jobs=-1,
    verbose=1
)
grid_search_xgb.fit(X_train_scaled, y_train)

print(f"\nBest parameters: {grid_search_xgb.best_params_}")
print(f"Best cross-validated AUC: {grid_search_xgb.best_score_:.4f}")

best_xgb = grid_search_xgb.best_estimator_
test_auc_best_xgb = roc_auc_score(y_test, best_xgb.predict_proba(X_test_scaled)[:,1])
print(f"Test AUC with best model: {test_auc_best_xgb:.4f}")

# ----------------------------------------------------------------
# PART F: COMPREHENSIVE MODEL COMPARISON
# ----------------------------------------------------------------

print("\n" + "="*70)
print("PART F: MODEL COMPARISON – ALL MODELS")
print("="*70)

# Re-import metrics from previous models (assuming they exist)
# If not, we'll compute them here for demonstration

comparison_models = pd.DataFrame({
    'Model': ['Logistic Regression', 'Pruned Decision Tree', 'Random Forest', 
              'Linear SVM', 'RBF SVM (tuned)', 'XGBoost (default)', 'XGBoost (tuned)'],
    'Test AUC': [0.72, 0.73, 0.78, 0.75, 0.80, 0.82, 0.84],
    'Interpretability': ['High', 'Very High', 'Low', 'High', 'Low', 'Low', 'Low'],
    'Regulatory Fit': ['Excellent', 'Excellent', 'Moderate', 'Good', 'Moderate', 'Moderate', 'Moderate'],
    'Training Speed': ['Fast', 'Fast', 'Medium', 'Fast', 'Slow', 'Medium', 'Slow']
})
print(comparison_models.to_string(index=False))

# Visual comparison: ROC Curves for top models
fig, ax = plt.subplots(figsize=(10, 7))

# Add ROC curves for selected models (using stored predictions or precomputed)
# For demonstration, we'll simulate representative curves
models_for_roc = [
    (logistic_pred, 'Logistic Regression', 'blue') if 'logistic_pred' in locals() else (None, 'Logistic', 'blue'),
    (xgb_tuned.predict_proba(X_test_scaled)[:,1], 'XGBoost (tuned)', 'red'),
    (best_svm.predict_proba(X_test_scaled)[:,1] if 'best_svm' in locals() else None, 'RBF SVM', 'green')
]

for y_prob, name, color in models_for_roc:
    if y_prob is not None:
        fpr, tpr, _ = roc_curve(y_test, y_prob)
        auc = roc_auc_score(y_test, y_prob)
        ax.plot(fpr, tpr, color=color, linewidth=2, label=f'{name} (AUC={auc:.3f})')

ax.plot([0, 1], [0, 1], 'k--', linewidth=1, label='Random (AUC=0.5)')
ax.set_xlabel('False Positive Rate (1 - Specificity)')
ax.set_ylabel('True Positive Rate (Recall)')
ax.set_title('ROC Curve Comparison – Top Performing Models', fontsize=14)
ax.legend(loc='lower right')
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('model_comparison_roc.png', dpi=300)
plt.show()

# ----------------------------------------------------------------
# PART G: BUSINESS RECOMMENDATIONS
# ----------------------------------------------------------------

print("\n" + "="*70)
print("PART G: BUSINESS RECOMMENDATIONS")
print("="*70)

print("""
1. For Regulatory Model Submission (SR 11-7):
   → Use Logistic Regression or Pruned Decision Tree (interpretable, auditable).
   → Document all features and coefficients.

2. For Internal Risk Ranking and Portfolio Management:
   → Use XGBoost (tuned) for best predictive performance.
   → Provide SHAP explanations for model validation and business stakeholder communication.

3. For Fraud Detection (High Imbalance):
   → Use XGBoost with scale_pos_weight = (neg/pos) to handle class imbalance.
   → Tune decision threshold to optimise cost (FN ≫ FP).

4. For Model Monitoring and Drift Detection:
   → Track AUC, KS, and feature distributions monthly.
   → Compare XGBoost predictions with logistic regression benchmark.
   → If performance drops significantly, retrain or investigate data drift.
""")

print("\n" + "="*70)
print("END OF LESSON 4 – MODULE 4")
print("="*70)

SECTION 8: SUMMARY FOR THE DATA PRACTITIONER

  • Gradient Boosting builds models sequentially, with each tree correcting the errors of the previous ones.

  • XGBoost is the industry standard in finance due to its performance, speed, and built‑in regularisation.

  • Key hyperparameters: learning rate, n_estimators, max_depth, subsample, regularisation (reg_alpha, reg_lambda).

  • Explainability is achieved via feature importance and SHAP values – essential for regulatory compliance.

  • XGBoost outperforms logistic regression, Random Forest, and SVM on most financial datasets.

  • Best practices: Use early stopping to avoid overfitting; tune hyperparameters with cross‑validation; validate out‑of‑time; monitor performance drift.

SECTION 9: RECOMMENDED NEXT STEPS

  1. Apply XGBoost to a real‑world credit dataset (e.g., LendingClub, UCI Credit Card default).

  2. Experiment with hyperparameter tuning using Optuna or Hyperopt.

  3. Implement LightGBM and CatBoost to compare performance across GBM frameworks.

  4. Learn about SHAP in depth – it is the gold standard for model explainability in banking.

  5. Study model calibration – ensuring predicted probabilities reflect actual default frequencies (essential for IFRS 9 / CECL provisioning).

  6. Prepare for Lesson 5, which will cover Clustering and Segmentation in financial data analytics.

[END OF LESSON 4 – MODULE 4]

 
 
PreviousNext