Lesson 4.7: Model Interpretability and Explainability – SHAP, LIME, and Beyond

1. Learning Objectives

By the end of this lesson, you will be able to:

Understand the regulatory and practical requirements for model interpretability in finance.
Implement SHAP (SHapley Additive exPlanations) for feature attribution in any financial model.
Implement LIME (Local Interpretable Model-Agnostic Explanations) for local explanations.
Apply Integrated Gradients and DeepLIFT for deep learning models.
Build and interpret partial dependence plots (PDP) and individual conditional expectation (ICE) plots.
Apply model-agnostic methods for global and local interpretability.
Create comprehensive model documentation for Model Risk Management (MRM) audits.
Understand the trade-off between model complexity and interpretability.

2. Why Interpretability Matters in Finance

2.1 Regulatory Requirements

SR 11-7 (OCC): Models must be transparent, validated, and governed.
GDPR: Right to explanation for automated decisions.
EU AI Act: High-risk AI systems (credit scoring, insurance) require transparency.
Basel III/IV: Internal models must be explainable to regulators.

2.2 Business Requirements

Stakeholders need to trust the model.
Compliance teams need to audit decisions.
Risk managers need to understand model limitations.
Traders need to know why a signal was generated.

2.3 The Interpretability Spectrum

Model Type	Interpretability	Typical Use
Linear Regression	High	Risk factors, CAPM
Logistic Regression	High	Credit scoring
Decision Trees	Medium	Simple credit decisions
Random Forest	Low	Complex predictions
XGBoost	Low	Alpha generation
Deep Learning	Very Low	High-frequency trading

3. SHAP – SHapley Additive exPlanations

3.1 Mathematical Foundation – Shapley Values

From cooperative game theory. For a model f with features {x_1, ..., x_M}, the Shapley value for feature i is:

φ_i = Σ_{S ⊆ F \ {i}} ( |S|! (M - |S| - 1)! / M! ) * [ f(S ∪ {i}) - f(S) ]

Where:

S is a subset of features (coalition).
f(S) is the model’s prediction using only the features in S.
F is the set of all features.

Properties:

Efficiency: Σ_i φ_i = f(x) - E[f(x)]
Symmetry: Equal contributions → equal Shapley values.
Dummy: Zero contribution → zero Shapley value.
Additivity: Ensemble SHAP = sum of individual SHAP values.

3.2 Implementation

import shap
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

class SHAPExplainer:
    def __init__(self, model, feature_names, model_type='tree'):
        """
        Initialize SHAP explainer.
        model_type: 'linear', 'tree', 'deep', 'kernel'
        """
        self.model = model
        self.feature_names = feature_names
        self.model_type = model_type
        self.explainer = None
        self.expected_value = None

    def fit_explainer(self, X_background, n_samples=100):
        """
        Fit the SHAP explainer on background data.
        """
        if self.model_type == 'linear':
            self.explainer = shap.LinearExplainer(self.model, X_background)
        elif self.model_type == 'tree':
            self.explainer = shap.TreeExplainer(self.model, X_background)
        elif self.model_type == 'deep':
            self.explainer = shap.DeepExplainer(self.model, X_background[:n_samples])
        elif self.model_type == 'kernel':
            self.explainer = shap.KernelExplainer(
                self.model.predict_proba, X_background[:n_samples]
            )
        else:
            raise ValueError(f"Unknown model type: {self.model_type}")

        self.expected_value = self.explainer.expected_value
        return self.explainer

    def explain_instance(self, X_instance):
        """
        Explain a single instance.
        """
        shap_values = self.explainer.shap_values(X_instance)
        return shap_values

    def plot_summary(self, shap_values, X_data):
        """
        Create a summary plot of SHAP values.
        """
        shap.summary_plot(shap_values, X_data, feature_names=self.feature_names)
        plt.tight_layout()
        plt.show()

    def plot_force(self, shap_values, X_instance):
        """
        Create a force plot for an individual instance.
        """
        if isinstance(shap_values, list):
            # For multi-class
            shap.force_plot(
                self.expected_value[1],
                shap_values[1],
                X_instance,
                feature_names=self.feature_names,
                matplotlib=True,
                show=True
            )
        else:
            shap.force_plot(
                self.expected_value,
                shap_values,
                X_instance,
                feature_names=self.feature_names,
                matplotlib=True,
                show=True
            )

    def plot_dependence(self, shap_values, X_data, feature_idx):
        """
        Create a dependence plot for a specific feature.
        """
        shap.dependence_plot(
            feature_idx,
            shap_values,
            X_data,
            feature_names=self.feature_names,
            show=True
        )

    def get_feature_importance(self, shap_values):
        """
        Compute global feature importance from SHAP values.
        """
        if isinstance(shap_values, list):
            importance = np.mean([np.abs(sv).mean(axis=0) for sv in shap_values], axis=0)
        else:
            importance = np.abs(shap_values).mean(axis=0)

        return pd.DataFrame({
            'feature': self.feature_names,
            'importance': importance
        }).sort_values('importance', ascending=False)

3.3 Financial Application – Credit Scoring

def shap_credit_scoring(model, X_train, X_test, feature_names):
    """
    SHAP explainability for credit scoring.
    """
    explainer = SHAPExplainer(model, feature_names, model_type='tree')
    explainer.fit_explainer(X_train[:100])

    # Explain test set
    shap_values = explainer.explain_instance(X_test[:100])

    # Global importance
    importance = explainer.get_feature_importance(shap_values)
    print("Global Feature Importance:")
    print(importance)

    # Individual explanations
    print("\nExplaining first test instance:")
    explainer.plot_force(shap_values, X_test.iloc[0:1])

    # Dependence plots for top features
    top_features = importance.head(3)['feature'].values
    for feat in top_features:
        feat_idx = list(feature_names).index(feat)
        explainer.plot_dependence(shap_values, X_test[:100], feat_idx)

    return shap_values, importance

4. LIME – Local Interpretable Model-Agnostic Explanations

4.1 Mathematical Foundation

LIME approximates the black-box model locally with an interpretable model (e.g., linear regression).

ξ(x) = argmin_{g ∈ G} L(f, g, π_x) + Ω(g)

Where:

f is the original model.
g is the interpretable model.
π_x is the locality kernel (weights instances based on proximity to x).
Ω(g) penalises model complexity.

4.2 Implementation

import lime
import lime.lime_tabular

class LIMExplainer:
    def __init__(self, model, training_data, feature_names, class_names=None, mode='classification'):
        """
        Initialize LIME explainer.
        mode: 'classification' or 'regression'
        """
        self.model = model
        self.training_data = training_data
        self.feature_names = feature_names
        self.class_names = class_names
        self.mode = mode

        self.explainer = lime.lime_tabular.LimeTabularExplainer(
            training_data.values,
            feature_names=feature_names,
            class_names=class_names,
            mode=mode,
            discretize_continuous=True,
            discretize_cont_quantile=True
        )

    def explain_instance(self, X_instance, num_features=10, top_labels=2):
        """
        Explain a single instance.
        """
        if self.mode == 'classification':
            explanation = self.explainer.explain_instance(
                X_instance.values[0],
                self.model.predict_proba,
                num_features=num_features,
                top_labels=top_labels
            )
        else:
            explanation = self.explainer.explain_instance(
                X_instance.values[0],
                self.model.predict,
                num_features=num_features
            )

        return explanation

    def plot_explanation(self, explanation, show=True):
        """
        Visualise the LIME explanation.
        """
        if show:
            explanation.show_in_notebook(show_table=True, show_all=False)
        return explanation.as_list()

    def get_feature_weights(self, explanation):
        """
        Extract feature weights from LIME explanation.
        """
        return explanation.as_list()

4.3 LIME vs SHAP – When to Use

Feature	SHAP	LIME
Global Explanations	Yes (summary plots)	No (local only)
Local Explanations	Yes	Yes
Theoretical Foundation	Shapley values (axiomatic)	Local approximation
Speed	Slower for large datasets	Faster
Consistency	Consistent across instances	Can be unstable
Recommended Use	Global understanding, regulatory	Quick local explanations

5. Integrated Gradients – For Deep Learning Models

5.1 Mathematical Formulation

Integrated Gradients is a gradient-based attribution method.

IG_i(x) = (x_i - x'_i) * ∫_{α=0}^{1} ∂F(x' + α(x - x')) / ∂x_i dα

Where:

x is the input.
x' is a baseline (e.g., zeros).
∂F/∂x_i is the gradient of the model’s output with respect to feature i.

Properties:

Sensitivity: If the model depends on a feature, IG assigns it non-zero attribution.
Implementation Invariance: IG is invariant to model implementation (only depends on input-output mapping).

5.2 Implementation

import torch
from torch.autograd import grad

class IntegratedGradients:
    def __init__(self, model, device='cpu'):
        self.model = model
        self.device = device
        self.model.to(device)
        self.model.eval()

    def attribute(self, input_tensor, baseline=None, steps=50, target=None):
        """
        Compute Integrated Gradients for a given input.
        """
        input_tensor = input_tensor.to(self.device)

        if baseline is None:
            baseline = torch.zeros_like(input_tensor)

        input_tensor.requires_grad_(True)

        # Generate scaled inputs
        scaled_inputs = []
        for alpha in torch.linspace(0, 1, steps):
            scaled_input = baseline + alpha * (input_tensor - baseline)
            scaled_inputs.append(scaled_input)

        # Compute gradients at each step
        gradients = []
        for scaled_input in scaled_inputs:
            scaled_input.requires_grad_(True)
            output = self.model(scaled_input)

            if target is None:
                target_idx = torch.argmax(output)
            else:
                target_idx = target

            out_grad = torch.zeros_like(output)
            out_grad[0, target_idx] = 1.0

            grad_output = grad(
                outputs=output,
                inputs=scaled_input,
                grad_outputs=out_grad,
                create_graph=False,
                retain_graph=True
            )[0]

            gradients.append(grad_output.detach())

        # Average gradients and multiply by (input - baseline)
        avg_grad = torch.stack(gradients).mean(dim=0)
        ig = (input_tensor - baseline) * avg_grad

        return ig.detach().cpu().numpy()

6. Partial Dependence and ICE Plots

6.1 Partial Dependence Plots (PDP)
PDP shows the marginal effect of one or two features on the predicted outcome.

PDP_j(x_j) = E_{X_{-j}}[ f(x_j, X_{-j}) ]

from sklearn.inspection import partial_dependence
from sklearn.inspection import PartialDependenceDisplay

def plot_partial_dependence(model, X, features, feature_names, grid_resolution=50):
    """
    Create partial dependence plots for specified features.
    """
    fig, ax = plt.subplots(figsize=(12, 8))

    pdp = PartialDependenceDisplay.from_estimator(
        model,
        X,
        features=features,
        feature_names=feature_names,
        grid_resolution=grid_resolution,
        ax=ax
    )

    plt.tight_layout()
    plt.show()
    return pdp

6.2 Individual Conditional Expectation (ICE)
ICE plots show the relationship between a feature and the prediction for each individual observation.

ICE_j^i(x_j) = f(x_j, X_{-j}^i)

def plot_ice_curves(model, X, feature_idx, feature_name, n_samples=50):
    """
    Create ICE curves for a specific feature.
    """
    sample_indices = np.random.choice(len(X), n_samples, replace=False)
    X_sample = X.iloc[sample_indices].copy()

    x_min = X.iloc[:, feature_idx].min()
    x_max = X.iloc[:, feature_idx].max()
    grid = np.linspace(x_min, x_max, 50)

    predictions = []
    for val in grid:
        X_temp = X_sample.copy()
        X_temp.iloc[:, feature_idx] = val
        pred = model.predict_proba(X_temp)[:, 1]
        predictions.append(pred)

    predictions = np.array(predictions).T

    plt.figure(figsize=(10, 6))
    for i in range(min(20, n_samples)):
        plt.plot(grid, predictions[i, :], alpha=0.3, color='blue')

    plt.plot(grid, predictions.mean(axis=0), color='red', linewidth=2, label='PDP')

    plt.xlabel(feature_name)
    plt.ylabel('Predicted Probability')
    plt.title(f'ICE Curves for {feature_name}')
    plt.legend()
    plt.grid(True, alpha=0.3)
    plt.show()

7. Model Documentation for MRM Compliance

class ModelDocumentation:
    def __init__(self, model_name, model_version, model_type):
        self.model_name = model_name
        self.model_version = model_version
        self.model_type = model_type
        self.documentation = {}

    def add_section(self, section_name, content):
        self.documentation[section_name] = content

    def generate_report(self):
        """
        Generate a comprehensive model documentation report.
        """
        report = f"""
        ========================================
        MODEL DOCUMENTATION REPORT
        ========================================

        1. MODEL OVERVIEW
        ------------------
        Model Name: {self.model_name}
        Version: {self.model_version}
        Type: {self.model_type}
        Purpose: {self.documentation.get('purpose', 'Not specified')}
        Deployment Date: {self.documentation.get('deployment_date', 'Not specified')}

        2. CONCEPTUAL SOUNDNESS
        -----------------------
        Theoretical Foundation: {self.documentation.get('theory', 'Not specified')}
        Assumptions: {self.documentation.get('assumptions', 'Not specified')}
        Limitations: {self.documentation.get('limitations', 'Not specified')}

        3. DATA AND FEATURES
        --------------------
        Data Sources: {self.documentation.get('data_sources', 'Not specified')}
        Date Range: {self.documentation.get('date_range', 'Not specified')}
        Feature List:
        {self.documentation.get('features', 'Not specified')}
        Target Variable: {self.documentation.get('target', 'Not specified')}

        4. MODEL DEVELOPMENT
        --------------------
        Training Algorithm: {self.documentation.get('algorithm', 'Not specified')}
        Hyperparameters:
        {self.documentation.get('hyperparameters', 'Not specified')}
        Training Sample Size: {self.documentation.get('train_size', 'Not specified')}
        Validation Sample Size: {self.documentation.get('val_size', 'Not specified')}

        5. PERFORMANCE METRICS
        ----------------------
        In-Sample Performance:
        {self.documentation.get('in_sample_metrics', 'Not specified')}
        Out-of-Sample Performance:
        {self.documentation.get('out_of_sample_metrics', 'Not specified')}
        Backtest Results:
        {self.documentation.get('backtest_results', 'Not specified')}

        6. VALIDATION
        -------------
        Statistical Tests:
        {self.documentation.get('statistical_tests', 'Not specified')}
        Stress Tests:
        {self.documentation.get('stress_tests', 'Not specified')}
        Sensitivity Analysis:
        {self.documentation.get('sensitivity_analysis', 'Not specified')}

        7. EXPLAINABILITY
        -----------------
        Feature Importance (SHAP):
        {self.documentation.get('shap_analysis', 'Not specified')}
        Local Explanations (LIME):
        {self.documentation.get('lime_analysis', 'Not specified')}

        8. GOVERNANCE
        -------------
        Model Owner: {self.documentation.get('model_owner', 'Not specified')}
        Validation Team: {self.documentation.get('validation_team', 'Not specified')}
        Approval Date: {self.documentation.get('approval_date', 'Not specified')}
        Review Frequency: {self.documentation.get('review_frequency', 'Not specified')}
        """

        return report

    def save_report(self, filepath='model_documentation.txt'):
        with open(filepath, 'w') as f:
            f.write(self.generate_report())
        print(f"Report saved to {filepath}")

8. Summary for the AI Practitioner

Interpretability is mandatory for financial AI due to regulatory requirements (SR 11-7, GDPR, EU AI Act).
SHAP is the gold standard. It is model-agnostic, satisfies desirable mathematical properties, and provides both global and local explanations.
LIME is faster but less stable. Use it for quick local explanations.
Integrated Gradients is for deep learning models. It satisfies sensitivity and implementation invariance.
PDP and ICE plots show the marginal effect of features. Essential for understanding non-linear relationships.
Model documentation must cover conceptual soundness, data, development, validation, explainability, and governance.
Trade-off: Simpler models (linear, tree-based) are more interpretable. Use them when explainability is critical.

Diploma in Artificial Intelligence in Finance Lesson 4.7: Model Interpretability and Explainability – SHAP, LIME, and Beyond

Diploma in Artificial Intelligence in Finance

1. Learning Objectives

2. Why Interpretability Matters in Finance

3. SHAP – SHapley Additive exPlanations

4. LIME – Local Interpretable Model-Agnostic Explanations

5. Integrated Gradients – For Deep Learning Models

6. Partial Dependence and ICE Plots

7. Model Documentation for MRM Compliance

8. Summary for the AI Practitioner