LESSON 3: BUILDING AND EVALUATING PREDICTIVE MODELS FOR THE CAPSTONE PROJECT

SECTION 1: LEARNING OBJECTIVES

By the end of this lesson, you will be able to:

Build multiple predictive models for loan default prediction – Logistic Regression, Random Forest, and XGBoost.
Handle class imbalance using techniques like SMOTE and class weights.
Perform hyperparameter tuning using GridSearchCV and RandomizedSearchCV.
Evaluate model performance using appropriate metrics – AUC, KS, Gini, calibration, and business metrics.
Select the best performing model based on a combination of performance, interpretability, and regulatory considerations.
Interpret model coefficients and feature importance to explain model behaviour.
Document model selection for regulatory approval.

SECTION 2: MODELING FRAMEWORK

2.1 Models to Be Developed

Model	Description	Pros	Cons
Logistic Regression	Baseline interpretable model.	Highly interpretable; regulatory-friendly.	May not capture complex non-linearities.
Random Forest	Ensemble of decision trees.	Handles non-linearities; feature importance.	Less interpretable; potential overfitting.
XGBoost	Gradient boosting with regularisation.	State-of-the-art performance; handles interactions.	Black-box; requires careful tuning.

2.2 Evaluation Framework

Metric	Threshold	Purpose
AUC	> 0.80	Discrimination power.
KS Statistic	> 0.35	Separation of good/bad borrowers.
Gini Coefficient	> 0.60	Alternative measure of discrimination.
Calibration (H-L)	p > 0.05	Predicted vs observed probabilities.
Brier Score	Lower is better	Mean squared error of probabilities.
Accuracy	> 0.80	Overall correctness (less important for imbalanced data).

SECTION 3: IMPLEMENTATION IN PYTHON – MODEL BUILDING

# ===================================================================
# MODULE 9, LESSON 3: BUILDING AND EVALUATING PREDICTIVE MODELS
# ===================================================================

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split, GridSearchCV, RandomizedSearchCV, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (roc_auc_score, roc_curve, confusion_matrix, 
                             classification_report, brier_score_loss, accuracy_score)
from scipy.stats import ks_2samp
import xgboost as xgb
from imblearn.over_sampling import SMOTE
import warnings
warnings.filterwarnings('ignore')

# Set style
sns.set_style("whitegrid")
np.random.seed(42)

print("="*70)
print("CAPSTONE PROJECT – BUILDING AND EVALUATING PREDICTIVE MODELS")
print("="*70)

# ----------------------------------------------------------------
# PART A: LOAD PREPARED DATA
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART A: Loading Prepared Data")
print("-"*60)

# Load data from previous lesson
X_train = pd.read_csv('X_train.csv')
X_test = pd.read_csv('X_test.csv')
y_train = pd.read_csv('y_train.csv').values.ravel()
y_test = pd.read_csv('y_test.csv').values.ravel()

print(f"Training set: {X_train.shape}")
print(f"Test set: {X_test.shape}")
print(f"Training default rate: {y_train.mean():.4f}")
print(f"Test default rate: {y_test.mean():.4f}")

# Feature names
feature_names = X_train.columns.tolist()
print(f"Features: {feature_names[:5]}... ({len(feature_names)} total)")

# ----------------------------------------------------------------
# PART B: HANDLE CLASS IMBALANCE (SMOTE)
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART B: Handling Class Imbalance with SMOTE")
print("-"*60)

# Check class distribution
print(f"Before SMOTE: Default = {y_train.sum()}, Non-Default = {len(y_train) - y_train.sum()}")
print(f"Ratio: {y_train.sum() / len(y_train):.4f}")

# Apply SMOTE
smote = SMOTE(random_state=42)
X_train_resampled, y_train_resampled = smote.fit_resample(X_train, y_train)

print(f"\nAfter SMOTE: Default = {y_train_resampled.sum()}, Non-Default = {len(y_train_resampled) - y_train_resampled.sum()}")
print(f"Ratio: {y_train_resampled.sum() / len(y_train_resampled):.4f}")

# Standardise features (for logistic regression)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_train_resampled_scaled = scaler.fit_transform(X_train_resampled)
X_test_scaled = scaler.transform(X_test)

print("\nFeatures standardised.")

# ----------------------------------------------------------------
# PART C: LOGISTIC REGRESSION (BASELINE)
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART C: Logistic Regression (Baseline Model)")
print("-"*60)

# Train logistic regression with class weights (to handle imbalance)
lr_model = LogisticRegression(class_weight='balanced', max_iter=1000, random_state=42)
lr_model.fit(X_train_scaled, y_train)

# Predictions
y_pred_proba_lr = lr_model.predict_proba(X_test_scaled)[:, 1]
y_pred_lr = (y_pred_proba_lr >= 0.5).astype(int)

# Evaluate
auc_lr = roc_auc_score(y_test, y_pred_proba_lr)
brier_lr = brier_score_loss(y_test, y_pred_proba_lr)
accuracy_lr = accuracy_score(y_test, y_pred_lr)

# KS Statistic
scores_good_lr = y_pred_proba_lr[y_test == 0]
scores_bad_lr = y_pred_proba_lr[y_test == 1]
ks_lr, _ = ks_2samp(scores_good_lr, scores_bad_lr)

print(f"Logistic Regression Performance:")
print(f"  AUC: {auc_lr:.4f}")
print(f"  KS Statistic: {ks_lr:.4f}")
print(f"  Gini Coefficient: {2*auc_lr - 1:.4f}")
print(f"  Brier Score: {brier_lr:.4f}")
print(f"  Accuracy: {accuracy_lr:.4f}")

# Confusion Matrix
cm_lr = confusion_matrix(y_test, y_pred_lr)
print(f"\nConfusion Matrix (Logistic Regression):")
print(pd.DataFrame(cm_lr, columns=['Pred No Default', 'Pred Default'], 
                   index=['Actual No Default', 'Actual Default']))

# Classification Report
print(f"\nClassification Report:")
print(classification_report(y_test, y_pred_lr, target_names=['No Default', 'Default']))

# Coefficients
coefficients = pd.DataFrame({
    'Feature': feature_names,
    'Coefficient': lr_model.coef_[0],
    'Odds Ratio': np.exp(lr_model.coef_[0])
}).sort_values('Coefficient', ascending=False)

print("\nTop 10 Coefficients (Odds Ratios):")
print(coefficients.head(10).to_string(index=False))

# ----------------------------------------------------------------
# PART D: RANDOM FOREST
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART D: Random Forest")
print("-"*60)

# Train Random Forest with class_weight='balanced'
rf_model = RandomForestClassifier(
    n_estimators=100,
    max_depth=8,
    min_samples_split=50,
    min_samples_leaf=20,
    class_weight='balanced',
    random_state=42,
    n_jobs=-1
)
rf_model.fit(X_train_resampled, y_train_resampled)

# Predictions
y_pred_proba_rf = rf_model.predict_proba(X_test)[:, 1]
y_pred_rf = (y_pred_proba_rf >= 0.5).astype(int)

# Evaluate
auc_rf = roc_auc_score(y_test, y_pred_proba_rf)
brier_rf = brier_score_loss(y_test, y_pred_proba_rf)
accuracy_rf = accuracy_score(y_test, y_pred_rf)

scores_good_rf = y_pred_proba_rf[y_test == 0]
scores_bad_rf = y_pred_proba_rf[y_test == 1]
ks_rf, _ = ks_2samp(scores_good_rf, scores_bad_rf)

print(f"Random Forest Performance:")
print(f"  AUC: {auc_rf:.4f}")
print(f"  KS Statistic: {ks_rf:.4f}")
print(f"  Gini Coefficient: {2*auc_rf - 1:.4f}")
print(f"  Brier Score: {brier_rf:.4f}")
print(f"  Accuracy: {accuracy_rf:.4f}")

# Feature Importance
importance_rf = pd.DataFrame({
    'Feature': feature_names,
    'Importance': rf_model.feature_importances_
}).sort_values('Importance', ascending=False)

print("\nTop 10 Features (Random Forest):")
print(importance_rf.head(10).to_string(index=False))

# ----------------------------------------------------------------
# PART E: XGBOOST
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART E: XGBoost")
print("-"*60)

# Train XGBoost with scale_pos_weight to handle imbalance
scale_pos_weight = len(y_train[y_train == 0]) / len(y_train[y_train == 1])

xgb_model = xgb.XGBClassifier(
    n_estimators=200,
    max_depth=6,
    learning_rate=0.1,
    subsample=0.8,
    colsample_bytree=0.8,
    scale_pos_weight=scale_pos_weight,
    random_state=42,
    use_label_encoder=False,
    eval_metric='logloss'
)
xgb_model.fit(X_train_resampled, y_train_resampled)

# Predictions
y_pred_proba_xgb = xgb_model.predict_proba(X_test)[:, 1]
y_pred_xgb = (y_pred_proba_xgb >= 0.5).astype(int)

# Evaluate
auc_xgb = roc_auc_score(y_test, y_pred_proba_xgb)
brier_xgb = brier_score_loss(y_test, y_pred_proba_xgb)
accuracy_xgb = accuracy_score(y_test, y_pred_xgb)

scores_good_xgb = y_pred_proba_xgb[y_test == 0]
scores_bad_xgb = y_pred_proba_xgb[y_test == 1]
ks_xgb, _ = ks_2samp(scores_good_xgb, scores_bad_xgb)

print(f"XGBoost Performance:")
print(f"  AUC: {auc_xgb:.4f}")
print(f"  KS Statistic: {ks_xgb:.4f}")
print(f"  Gini Coefficient: {2*auc_xgb - 1:.4f}")
print(f"  Brier Score: {brier_xgb:.4f}")
print(f"  Accuracy: {accuracy_xgb:.4f}")

# Feature Importance
importance_xgb = pd.DataFrame({
    'Feature': feature_names,
    'Importance': xgb_model.feature_importances_
}).sort_values('Importance', ascending=False)

print("\nTop 10 Features (XGBoost):")
print(importance_xgb.head(10).to_string(index=False))

# ----------------------------------------------------------------
# PART F: HYPERPARAMETER TUNING (XGBOOST)
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART F: Hyperparameter Tuning – XGBoost")
print("-"*60)

# Define parameter grid
param_grid = {
    'max_depth': [4, 6, 8],
    'learning_rate': [0.01, 0.05, 0.1],
    'subsample': [0.7, 0.8, 0.9],
    'colsample_bytree': [0.7, 0.8, 0.9],
    'n_estimators': [100, 200, 300]
}

# Use a reduced grid for speed
param_grid_reduced = {
    'max_depth': [4, 6],
    'learning_rate': [0.05, 0.1],
    'subsample': [0.8],
    'colsample_bytree': [0.8],
    'n_estimators': [100, 200]
}

print("Performing Grid Search (reduced grid for speed)...")
grid_search = GridSearchCV(
    xgb.XGBClassifier(
        scale_pos_weight=scale_pos_weight,
        random_state=42,
        use_label_encoder=False,
        eval_metric='logloss'
    ),
    param_grid_reduced,
    cv=3,
    scoring='roc_auc',
    n_jobs=-1,
    verbose=1
)
grid_search.fit(X_train_resampled, y_train_resampled)

print(f"\nBest Parameters: {grid_search.best_params_}")
print(f"Best CV AUC: {grid_search.best_score_:.4f}")

# Best model
best_xgb = grid_search.best_estimator_

# Evaluate best model
y_pred_proba_best = best_xgb.predict_proba(X_test)[:, 1]
auc_best = roc_auc_score(y_test, y_pred_proba_best)

print(f"Tuned XGBoost Test AUC: {auc_best:.4f}")

# ----------------------------------------------------------------
# PART G: CALIBRATION ANALYSIS
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART G: Calibration Analysis")
print("-"*60)

def calibration_plot(y_true, y_pred, model_name, n_bins=10):
    """Create a calibration plot."""
    df = pd.DataFrame({'y_true': y_true, 'y_pred': y_pred})
    df['bin'] = pd.qcut(df['y_pred'], q=n_bins, labels=False, duplicates='drop')
    cal_data = df.groupby('bin').agg(
        observed=('y_true', 'mean'),
        predicted=('y_pred', 'mean'),
        count=('y_true', 'count')
    ).reset_index()
    return cal_data

# Calibration for each model
cal_lr = calibration_plot(y_test, y_pred_proba_lr, 'Logistic Regression')
cal_rf = calibration_plot(y_test, y_pred_proba_rf, 'Random Forest')
cal_xgb = calibration_plot(y_test, y_pred_proba_xgb, 'XGBoost')

fig, ax = plt.subplots(figsize=(10, 8))

# Perfect calibration line
ax.plot([0, 1], [0, 1], 'k--', linewidth=2, label='Perfect Calibration')

# Model calibration
ax.plot(cal_lr['predicted'], cal_lr['observed'], 'bo-', linewidth=2, markersize=8, label='Logistic Regression')
ax.plot(cal_rf['predicted'], cal_rf['observed'], 'go-', linewidth=2, markersize=8, label='Random Forest')
ax.plot(cal_xgb['predicted'], cal_xgb['observed'], 'ro-', linewidth=2, markersize=8, label='XGBoost')

ax.set_xlabel('Mean Predicted Probability')
ax.set_ylabel('Observed Default Rate')
ax.set_title('Calibration Plot – Model Comparison')
ax.legend()
ax.grid(True, alpha=0.3)

plt.tight_layout()
plt.savefig('calibration_plots.png', dpi=300, bbox_inches='tight')
plt.show()
print("Calibration plot saved as 'calibration_plots.png'")

# ----------------------------------------------------------------
# PART H: MODEL COMPARISON
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART H: Model Comparison Summary")
print("-"*60)

comparison = pd.DataFrame({
    'Model': ['Logistic Regression', 'Random Forest', 'XGBoost', 'XGBoost (Tuned)'],
    'AUC': [auc_lr, auc_rf, auc_xgb, auc_best],
    'KS': [ks_lr, ks_rf, ks_xgb, ks_2samp(y_pred_proba_best[y_test == 0], y_pred_proba_best[y_test == 1])[0]],
    'Gini': [2*auc_lr-1, 2*auc_rf-1, 2*auc_xgb-1, 2*auc_best-1],
    'Brier Score': [brier_lr, brier_rf, brier_xgb, brier_score_loss(y_test, y_pred_proba_best)],
    'Accuracy': [accuracy_lr, accuracy_rf, accuracy_xgb, accuracy_score(y_test, (y_pred_proba_best >= 0.5).astype(int))],
    'Interpretability': ['High', 'Low', 'Low', 'Low'],
    'Regulatory Fit': ['Excellent', 'Moderate', 'Moderate', 'Moderate']
})

print(comparison.to_string(index=False))

# ----------------------------------------------------------------
# PART I: MODEL SELECTION RECOMMENDATION
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART I: Model Selection Recommendation")
print("-"*60)

print("""
Model Selection Recommendation:

1. For Regulatory Submission (SR 11-7):
   - Recommendation: **Logistic Regression**
   - Rationale: Highly interpretable; coefficients can be explained and justified.
   - Performance: Acceptable AUC ({auc_lr:.3f}) and KS ({ks_lr:.3f}).
   - Caveat: May miss some non-linear relationships.

2. For Internal Risk Ranking:
   - Recommendation: **XGBoost (Tuned)**
   - Rationale: Best predictive performance; captures complex interactions.
   - Performance: AUC {auc_best:.3f}, KS {ks_2samp(y_pred_proba_best[y_test == 0], y_pred_proba_best[y_test == 1])[0]:.3f}.
   - Caveat: Requires SHAP for explainability.

3. Hybrid Approach:
   - Use XGBoost for internal risk assessment and portfolio monitoring.
   - Use Logistic Regression for regulatory submissions and customer-facing decisions.
   - Provide SHAP explanations for XGBoost predictions.
""".format(auc_lr=auc_lr, ks_lr=ks_lr, auc_best=auc_best, 
          ks_best=ks_2samp(y_pred_proba_best[y_test == 0], y_pred_proba_best[y_test == 1])[0]))

# ----------------------------------------------------------------
# PART J: MODEL SAVING
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART J: Saving Models")
print("-"*60)

import joblib

# Save models and scaler
joblib.dump(lr_model, 'logistic_regression_model.pkl')
joblib.dump(rf_model, 'random_forest_model.pkl')
joblib.dump(best_xgb, 'xgboost_model.pkl')
joblib.dump(scaler, 'scaler.pkl')

print("Models and scaler saved:")
print("  - logistic_regression_model.pkl")
print("  - random_forest_model.pkl")
print("  - xgboost_model.pkl")
print("  - scaler.pkl")

print("="*70)
print("END OF LESSON 3 – MODULE 9")
print("="*70)

SECTION 4: SUMMARY FOR THE DATA PRACTITIONER

Three models were built and evaluated: Logistic Regression, Random Forest, and XGBoost.
Class imbalance was handled using SMOTE and class weights.
Hyperparameter tuning improved XGBoost performance.
Evaluation metrics show that XGBoost has the best predictive performance.
Logistic Regression is recommended for regulatory submissions due to interpretability.
XGBoost is recommended for internal risk ranking with SHAP explanations.
All models are saved for deployment and further analysis.

SECTION 5: RECOMMENDED NEXT STEPS

Review model comparison and understand the trade-offs.
Prepare for Lesson 4: Risk Analytics and Stress Testing.
Consider additional features or engineering that could improve performance.
Explore SHAP explanations for XGBoost (covered in previous modules).

Diploma in Financial Data Analytics LESSON 3: BUILDING AND EVALUATING PREDICTIVE MODELS FOR THE CAPSTONE PROJECT

Diploma in Financial Data Analytics

SECTION 1: LEARNING OBJECTIVES

SECTION 2: MODELING FRAMEWORK

2.1 Models to Be Developed

2.2 Evaluation Framework

SECTION 3: IMPLEMENTATION IN PYTHON – MODEL BUILDING

SECTION 4: SUMMARY FOR THE DATA PRACTITIONER

SECTION 5: RECOMMENDED NEXT STEPS