LESSON 2: DECISION TREES AND RANDOM FORESTS – INTERPRETABLE ENSEMBLE METHODS FOR FINANCE

SECTION 1: LEARNING OBJECTIVES

By the end of this lesson, you will be able to:

Understand the structure of a decision tree – nodes, branches, and leaves – and how it partitions the feature space.
Apply splitting criteria – Gini Impurity, Entropy, and Information Gain – to build optimal trees.
Interpret a decision tree visually as a set of if‑then rules, which is highly valued in regulatory reviews.
Explain the problem of overfitting in trees and use pruning, max depth, and min samples split to mitigate it.
Understand the ensemble concept – why Random Forests combine many trees to improve predictive performance and robustness.
Distinguish between bagging (Bootstrap Aggregating) and boosting, and explain why Random Forests use bagging.
Measure feature importance from a Random Forest and use it for model interpretation and variable selection.
Apply Random Forests to a financial problem – fraud detection or credit scoring – and compare performance against logistic regression.
Recognise the trade‑offs between interpretability (single tree) and predictive power (Random Forest) in regulatory contexts.

SECTION 2: THE DECISION TREE – AN INTUITIVE RULE‑BASED MODEL

A decision tree is a flowchart‑like structure where each internal node represents a test on a feature (e.g., “Is credit score < 650?”), each branch represents the outcome of the test, and each leaf node represents a prediction (e.g., probability of default).

Advantages:

Highly interpretable: A non‑technical stakeholder can follow the path from root to leaf.
Non‑parametric: No assumptions about the distribution of predictors.
Handles non‑linearities and interactions naturally (without needing to create interaction terms).
Robust to outliers (splits are based on thresholds, not means).

Disadvantages:

High variance: Small changes in data can produce a completely different tree (instability).
Overfitting: Deep trees memorise noise.
Bias toward features with many levels – e.g., a continuous variable like income may dominate if not handled carefully.

SECTION 3: SPLITTING CRITERIA – HOW TREES ARE BUILT

At each node, the algorithm searches over all features and all possible split points to find the split that maximises purity (i.e., minimises impurity) in the child nodes.

3.1 Gini Impurity

For a node with class probabilities $p_{k}$ (proportion of class k in the node):

$Gini (t) = 1 - k = 1 \sum K p_{k 2}$

Gini = 0 when all observations belong to one class (pure).
Gini = 0.5 for a balanced two‑class node.

The reduction in Gini impurity is used to select the best split.

3.2 Entropy / Information Gain

$Entropy (t) = - k = 1 \sum K p_{k} log_{2} (p_{k})$

Entropy is also 0 when pure, and maximum (1 for binary) when balanced.

Information Gain is the reduction in entropy before and after the split. The split with the highest gain is chosen.

In practice: Gini and Entropy yield similar trees; Gini is slightly faster to compute.

3.3 Regression Trees (for continuous targets)

If predicting a continuous variable (e.g., loan amount), we use variance reduction – choose the split that minimises the sum of squared errors within child nodes.

SECTION 4: CONTROLLING OVERFITTING – PRUNING AND HYPERPARAMETERS

To build a robust tree, we must constrain its complexity:

Parameter	Effect	Typical Value (sklearn)
`max_depth`	Limits how deep the tree can grow.	3‑10 (deeper captures more interactions)
`min_samples_split`	Minimum samples required to split an internal node.	20‑50 (prevents splits on small groups)
`min_samples_leaf`	Minimum samples in a leaf node.	10‑20 (ensures stable predictions)
`max_features`	Number of features considered at each split (important for Random Forests).	`sqrt(n_features)` for classification
Pruning (Cost‑Complexity)	Prune branches that add little predictive power after a penalty `ccp_alpha`.	Tuned via cross‑validation

Business rule of thumb: In credit scoring, trees deeper than 6‑8 layers are often overfitted; regulatory models prefer shallower trees that are easy to explain.

SECTION 5: RANDOM FORESTS – THE POWER OF ENSEMBLES

A Random Forest builds many decision trees (e.g., 100‑500) and averages their predictions. This addresses the high variance of a single tree.

How it works:

Bootstrap sampling: For each tree, draw a random sample (with replacement) of size n from the training data. About 63% of the original data is used per tree (out‑of‑bag (OOB) samples remain for validation).
Random feature selection: At each node, instead of searching all features, we search a random subset of max_features features. This decorrelates the trees.
Grow each tree fully (no pruning) or with minimal constraints.
Aggregate predictions:
- Classification: Majority vote of all trees.
- Regression: Average of all tree predictions.

Why it works:

Bagging reduces variance without increasing bias.
Random feature selection ensures trees are uncorrelated; the average of uncorrelated estimators has much lower variance than any single estimator.

Financial advantages:

High predictive power – often outperforms logistic regression on complex datasets with interactions.
Feature importance – can rank variables, aiding in feature selection for simpler models.
Handles missing data gracefully (can use surrogate splits).

Regulatory drawbacks:

Black‑box nature: Even with feature importance, it’s harder to explain a specific prediction than a logistic regression or a single tree.
SR 11‑7 expectation: If using a Random Forest, banks must provide additional explainability tools (e.g., SHAP, LIME) to satisfy model validation.

SECTION 6: FEATURE IMPORTANCE – INTERPRETING THE FOREST

Random Forest provides two types of importance:

Gini Importance (Mean Decrease in Impurity): The total reduction in impurity (Gini or Entropy) attributed to each feature, averaged over all trees. Higher means the feature is more important for splitting.
Permutation Importance: Shuffle the values of a feature and measure the drop in model performance. A large drop indicates high importance. This is preferred for regulatory validation because it is less biased toward high‑cardinality features.

Use case: A bank can use Random Forest importance to trim the feature set before building a simpler logistic regression model – improving interpretability without sacrificing too much predictive power.

SECTION 7: IMPLEMENTATION IN PYTHON

We extend our credit dataset and compare a single Decision Tree, a pruned tree, and a Random Forest.

# ===================================================================
# MODULE 4, LESSON 2: DECISION TREES AND RANDOM FORESTS
# ===================================================================

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, GridSearchCV
from sklearn.metrics import roc_auc_score, classification_report
from sklearn.inspection import permutation_importance

# Reuse the dataset from Lesson 1 (X_train, X_test, y_train, y_test, X_train_scaled, X_test_scaled)
# But for trees, we use raw (unscaled) data because trees are scale-invariant.
# We'll use the original X_train and X_test (not scaled).

print("="*70)
print("DECISION TREES AND RANDOM FORESTS FOR CREDIT RISK")
print("="*70)

# ----------------------------------------------------------------
# PART A: SINGLE DECISION TREE (UNPRUNED) – HIGH OVERFITTING
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART A: UNPRUNED DECISION TREE")
print("-"*60)

tree_unpruned = DecisionTreeClassifier(random_state=42)
tree_unpruned.fit(X_train, y_train)

train_auc = roc_auc_score(y_train, tree_unpruned.predict_proba(X_train)[:,1])
test_auc = roc_auc_score(y_test, tree_unpruned.predict_proba(X_test)[:,1])

print(f"Training AUC: {train_auc:.4f}")
print(f"Test AUC:     {test_auc:.4f}")
print(f"Overfitting gap: {train_auc - test_auc:.4f} (large gap indicates severe overfitting)")

# ----------------------------------------------------------------
# PART B: PRUNED DECISION TREE (REGULATORY-FRIENDLY)
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART B: PRUNED DECISION TREE (max_depth=4, min_samples_leaf=50)")
print("-"*60)

tree_pruned = DecisionTreeClassifier(
    max_depth=4, 
    min_samples_leaf=50,
    min_samples_split=100,
    random_state=42
)
tree_pruned.fit(X_train, y_train)

train_auc_pr = roc_auc_score(y_train, tree_pruned.predict_proba(X_train)[:,1])
test_auc_pr = roc_auc_score(y_test, tree_pruned.predict_proba(X_test)[:,1])

print(f"Training AUC: {train_auc_pr:.4f}")
print(f"Test AUC:     {test_auc_pr:.4f}")
print(f"Overfitting gap: {train_auc_pr - test_auc_pr:.4f}")

# Visualize the pruned tree
plt.figure(figsize=(20, 10))
plot_tree(tree_pruned, feature_names=X_train.columns, class_names=['No Default', 'Default'], 
          filled=True, rounded=True, fontsize=10, max_depth=4)
plt.title("Pruned Decision Tree (Depth=4) – Interpretable Rules", fontsize=14)
plt.savefig('pruned_decision_tree.png', dpi=300, bbox_inches='tight')
plt.show()

# Extract rules from the tree (for regulatory documentation)
def extract_rules(tree, feature_names, max_depth=3):
    """Extract decision rules from the tree up to a given depth."""
    tree_ = tree.tree_
    feature_name = [feature_names[i] if i != -1 else "undefined!" for i in tree_.feature]
    
    def recurse(node, depth, rule):
        if depth > max_depth:
            return
        if tree_.feature[node] != -1:
            name = feature_name[node]
            threshold = tree_.threshold[node]
            # Left child: <= threshold; Right child: > threshold
            recurse(tree_.children_left[node], depth+1, rule + [f"{name} ≤ {threshold:.2f}"])
            recurse(tree_.children_right[node], depth+1, rule + [f"{name} > {threshold:.2f}"])
        else:
            # Leaf node: print the rule path and the predicted class/probability
            proba = tree_.value[node] / tree_.value[node].sum()
            print(f"  {' -> '.join(rule)}  → Default Prob: {proba[0][1]:.3f}")
    
    print("\nDecision Rules (Top 3 levels):")
    recurse(0, 0, [])
    return

extract_rules(tree_pruned, X_train.columns, max_depth=3)

# ----------------------------------------------------------------
# PART C: RANDOM FOREST – ENSEMBLE OF TREES
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART C: RANDOM FOREST (n_estimators=200, max_depth=10)")
print("-"*60)

rf = RandomForestClassifier(
    n_estimators=200,
    max_depth=10,
    min_samples_split=50,
    min_samples_leaf=20,
    max_features='sqrt',
    oob_score=True,
    random_state=42,
    n_jobs=-1
)
rf.fit(X_train, y_train)

train_auc_rf = roc_auc_score(y_train, rf.predict_proba(X_train)[:,1])
test_auc_rf = roc_auc_score(y_test, rf.predict_proba(X_test)[:,1])
oob_auc = roc_auc_score(y_train, rf.oob_decision_function_[:,1])  # OOB predictions

print(f"Training AUC:   {train_auc_rf:.4f}")
print(f"OOB AUC:        {oob_auc:.4f} (out-of-bag estimate)")
print(f"Test AUC:       {test_auc_rf:.4f}")
print(f"Overfitting gap: {train_auc_rf - test_auc_rf:.4f}")

# Classification report on test set
y_pred_rf = rf.predict(X_test)
print("\nClassification Report (Random Forest):")
print(classification_report(y_test, y_pred_rf, target_names=['No Default', 'Default']))

# ----------------------------------------------------------------
# PART D: FEATURE IMPORTANCE ANALYSIS (GINI vs PERMUTATION)
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART D: FEATURE IMPORTANCE")
print("-"*60)

# Gini importance (built-in)
gini_importance = pd.DataFrame({
    'feature': X_train.columns,
    'gini_importance': rf.feature_importances_
}).sort_values('gini_importance', ascending=False)

print("\nGini Importance (Mean Decrease in Impurity):")
print(gini_importance.to_string(index=False))

# Permutation importance (more reliable for regulatory)
perm_importance = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=42)
perm_importance_df = pd.DataFrame({
    'feature': X_train.columns,
    'perm_importance': perm_importance.importances_mean,
    'std': perm_importance.importances_std
}).sort_values('perm_importance', ascending=False)

print("\nPermutation Importance (Drop in AUC when feature is shuffled):")
print(perm_importance_df.to_string(index=False))

# Visual comparison
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

ax = axes[0]
sns.barplot(data=gini_importance, x='gini_importance', y='feature', ax=ax)
ax.set_title('Gini Importance', fontsize=12)
ax.set_xlabel('Mean Decrease in Impurity')

ax = axes[1]
sns.barplot(data=perm_importance_df, x='perm_importance', y='feature', ax=ax)
ax.set_title('Permutation Importance (AUC drop)', fontsize=12)
ax.set_xlabel('Mean Decrease in AUC')

plt.tight_layout()
plt.savefig('feature_importance_comparison.png', dpi=300)
plt.show()

# ----------------------------------------------------------------
# PART E: HYPERPARAMETER TUNING (Grid Search)
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART E: HYPERPARAMETER TUNING (GridSearchCV)")
print("-"*60)

param_grid = {
    'n_estimators': [100, 200],
    'max_depth': [8, 12, 16],
    'min_samples_split': [20, 50, 100]
}

# Use a smaller grid for speed in demonstration
grid_search = GridSearchCV(
    RandomForestClassifier(random_state=42, n_jobs=-1),
    param_grid,
    cv=3,
    scoring='roc_auc',
    n_jobs=-1,
    verbose=1
)
grid_search.fit(X_train, y_train)

print(f"\nBest parameters: {grid_search.best_params_}")
print(f"Best cross-validated AUC: {grid_search.best_score_:.4f}")

best_rf = grid_search.best_estimator_
test_auc_best = roc_auc_score(y_test, best_rf.predict_proba(X_test)[:,1])
print(f"Test AUC with best model: {test_auc_best:.4f}")

# ----------------------------------------------------------------
# PART F: COMPARISON ACROSS MODELS
# ----------------------------------------------------------------

print("\n" + "="*70)
print("PART F: MODEL COMPARISON SUMMARY")
print("="*70)

comparison = pd.DataFrame({
    'Model': ['Unpruned Tree', 'Pruned Tree (Depth=4)', 'Random Forest (Default)', 'Random Forest (Tuned)'],
    'Test AUC': [test_auc, test_auc_pr, test_auc_rf, test_auc_best],
    'Interpretability': ['High (but overfit)', 'Very High (simple rules)', 'Low (ensemble)', 'Low (ensemble)'],
    'Regulatory Fit': ['Poor (overfit)', 'Excellent (transparent)', 'Moderate (requires SHAP)', 'Moderate']
})
print(comparison.to_string(index=False))

print("\n Business Recommendation:")
print("  - For initial model development / regulatory submission: Use the pruned tree or logistic regression.")
print("  - For internal risk ranking and decision support: Use the tuned Random Forest.")
print("  - Always provide feature importance and partial dependence plots for ensemble models.")

SECTION 8: REGULATORY PERSPECTIVE ON TREES AND FORESTS

Single Decision Tree: Accepted by regulators if well‑pruned and documented. The rule‑based nature satisfies the “right to explanation” under GDPR and Fair Lending.
Random Forest: Requires additional explainability techniques:
- SHAP (SHapley Additive exPlanations): Provides per‑instance feature contributions.
- Partial Dependence Plots (PDP): Shows the marginal effect of a feature on the prediction.
- LIME: Local interpretable model‑agnostic explanations.
Model Validation (SR 11‑7): Banks must demonstrate that the ensemble does not produce disparate impact and that its decisions are consistent with business logic. This is often achieved by comparing Random Forest predictions against a simpler benchmark (like logistic regression) and monitoring drift.

SECTION 9: SUMMARY FOR THE DATA PRACTITIONER

Decision trees create interpretable rules but overfit unless constrained.
Splitting criteria (Gini, Entropy) measure node purity.
Pruning (max_depth, min_samples_leaf) is essential for regulatory acceptance.
Random Forests combine many trees via bootstrap sampling and random feature selection to reduce variance and improve accuracy.
Feature importance (Gini or permutation) provides insight into which drivers matter most – crucial for variable selection and model documentation.
In banking, use trees for exploratory analysis and rule discovery; use forests for predictive power; and always explain with SHAP/LIME if deploying forests in production.

[END OF LESSON 2 – MODULE 4]

Diploma in Financial Data Analytics LESSON 2: DECISION TREES AND RANDOM FORESTS – INTERPRETABLE ENSEMBLE METHODS FOR FINANCE

Diploma in Financial Data Analytics