Lesson 3.7: Model Interpretability and Explainability for Financial AI

1. Learning Objectives

By the end of this lesson, you will be able to:

Understand the regulatory and practical requirements for model interpretability in finance.
Implement SHAP (SHapley Additive exPlanations) for feature attribution in financial models.
Implement LIME (Local Interpretable Model-Agnostic Explanations) for local explanations.
Apply Integrated Gradients for deep learning model explanations.
Build and interpret partial dependence plots (PDP) and individual conditional expectation (ICE) plots.
Implement counterfactual explanations for regulatory compliance.
Understand the trade-off between model complexity and interpretability.
Create comprehensive model documentation for Model Risk Management (MRM) audits.

2. The Regulatory Landscape – Why Explainability Matters

Financial AI models are subject to regulatory scrutiny under SR 11-7 (OCC), GDPR (Right to Explanation), and the EU AI Act.

2.1 SR 11-7 – Model Risk Management Principles
SR 11-7 requires that models be:

Conceptually Sound: The model’s design and methodology must be grounded in economic theory and empirical evidence.
Validated: Models must undergo independent validation before deployment and periodically thereafter.
Transparent: Model inputs, outputs, and decision-making processes must be documented and explainable to stakeholders.
Governed: There must be clear accountability and oversight.

2.2 GDPR – Right to Explanation
Under GDPR Article 22, individuals have the right to not be subject to decisions based solely on automated processing. They also have the right to obtain an explanation of the decision.

2.3 The EU AI Act
The EU AI Act classifies AI systems by risk level. High-risk applications (including credit scoring and insurance) require:

Technical documentation.
Transparency and explainability.
Human oversight.
Robustness and accuracy.

3. Feature Attribution Methods – Understanding What Drives Predictions

3.1 SHAP (SHapley Additive exPlanations)

SHAP values are based on cooperative game theory. The Shapley value of a feature is its average marginal contribution across all possible feature coalitions.

Mathematical Formulation:
For a model f with features {x_1, ..., x_M}, the Shapley value for feature i is:
φ_i = Σ_{S ⊆ F \ {i}} ( |S|! (M - |S| - 1)! / M! ) * [ f(S ∪ {i}) - f(S) ]
where:

S is a subset of features (coalition).
f(S) is the model’s prediction using only the features in S.
F is the set of all features.

Interpretation: φ_i is the average contribution of feature i to the prediction, averaged over all possible feature subsets.

SHAP Properties:

Efficiency: Σ_i φ_i = f(x) - E[f(x)] (predictions sum to the difference from the baseline).
Symmetry: If two features contribute equally, they have the same Shapley value.
Dummy: If a feature adds no value, its Shapley value is zero.
Additivity: For an ensemble model, SHAP values of the ensemble are the sum of SHAP values of individual models.

SHAP Implementation:

import shap
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

class SHAPExplainer:
    def __init__(self, model, feature_names, model_type='linear'):
        """
        Initialize SHAP explainer.
        model_type: 'linear', 'tree', 'deep', 'kernel'
        """
        self.model = model
        self.feature_names = feature_names
        self.model_type = model_type
        self.explainer = None

    def fit_explainer(self, X_background, n_samples=100):
        """
        Fit the SHAP explainer on background data.
        """
        if self.model_type == 'linear':
            self.explainer = shap.LinearExplainer(self.model, X_background)
        elif self.model_type == 'tree':
            self.explainer = shap.TreeExplainer(self.model, X_background)
        elif self.model_type == 'deep':
            self.explainer = shap.DeepExplainer(self.model, X_background[:n_samples])
        elif self.model_type == 'kernel':
            self.explainer = shap.KernelExplainer(
                self.model.predict, X_background[:n_samples]
            )
        else:
            raise ValueError(f"Unknown model type: {self.model_type}")

        return self.explainer

    def explain_instance(self, X_instance):
        """
        Explain a single instance.
        """
        shap_values = self.explainer.shap_values(X_instance)
        return shap_values

    def plot_summary(self, shap_values, X_data):
        """
        Create a summary plot of SHAP values.
        """
        shap.summary_plot(shap_values, X_data, feature_names=self.feature_names)
        plt.tight_layout()
        plt.show()

    def plot_force(self, shap_values, X_instance, matplotlib=True):
        """
        Create a force plot for an individual instance.
        """
        if matplotlib:
            shap.force_plot(
                self.explainer.expected_value,
                shap_values,
                X_instance,
                feature_names=self.feature_names,
                matplotlib=True,
                show=True
            )
        else:
            shap.force_plot(
                self.explainer.expected_value,
                shap_values,
                X_instance,
                feature_names=self.feature_names
            )

    def plot_dependence(self, shap_values, X_data, feature_idx):
        """
        Create a dependence plot for a specific feature.
        """
        shap.dependence_plot(
            feature_idx,
            shap_values,
            X_data,
            feature_names=self.feature_names,
            show=True
        )

    def get_feature_importance(self, shap_values):
        """
        Compute global feature importance from SHAP values.
        """
        if isinstance(shap_values, list):
            # For multi-class models
            importance = np.mean([np.abs(sv).mean(axis=0) for sv in shap_values], axis=0)
        else:
            importance = np.abs(shap_values).mean(axis=0)

        return pd.DataFrame({
            'feature': self.feature_names,
            'importance': importance
        }).sort_values('importance', ascending=False)

SHAP for Financial Risk Modeling Example:

# Load data and train model
import yfinance as yf
from sklearn.ensemble import RandomForestClassifier

# Download data
data = yf.download('AAPL', start='2020-01-01', end='2024-01-01')

# Create features
data['Return'] = data['Adj Close'].pct_change()
data['Volatility'] = data['Return'].rolling(20).std()
data['SMA_20'] = data['Adj Close'].rolling(20).mean()
data['SMA_50'] = data['Adj Close'].rolling(50).mean()
data['RSI'] = compute_rsi(data['Adj Close'])  # From earlier lesson

# Target: next day return > 0
data['Target'] = (data['Return'].shift(-1) > 0).astype(int)
data = data.dropna()

# Train model
features = ['Return', 'Volatility', 'SMA_20', 'SMA_50', 'RSI']
X = data[features]
y = data['Target']

model = RandomForestClassifier(n_estimators=100, max_depth=5, random_state=42)
model.fit(X, y)

# SHAP explanations
explainer = SHAPExplainer(model, features, model_type='tree')
explainer.fit_explainer(X[:100])

# Explain a single instance
shap_values = explainer.explain_instance(X.iloc[0:1])

# Visualise
explainer.plot_force(shap_values, X.iloc[0:1])
explainer.plot_summary(shap_values, X[:100])

# Feature importance
importance = explainer.get_feature_importance(shap_values)
print(importance)

3.2 LIME (Local Interpretable Model-Agnostic Explanations)

LIME explains individual predictions by approximating the model locally with an interpretable model (e.g., linear regression).

Mathematical Formulation:
ξ(x) = argmin_{g ∈ G} L(f, g, π_x) + Ω(g)
where:

f is the original model.
g is the interpretable model (e.g., linear model).
π_x is the locality kernel (weights instances based on proximity to x).
Ω(g) penalises model complexity.

LIME Implementation:

import lime
import lime.lime_tabular

class LIMExplainer:
    def __init__(self, model, training_data, feature_names, class_names=None, mode='classification'):
        """
        Initialize LIME explainer.
        mode: 'classification' or 'regression'
        """
        self.model = model
        self.training_data = training_data
        self.feature_names = feature_names
        self.class_names = class_names
        self.mode = mode

        self.explainer = lime.lime_tabular.LimeTabularExplainer(
            training_data.values,
            feature_names=feature_names,
            class_names=class_names,
            mode=mode,
            discretize_continuous=True,
            discretize_cont_quantile=True
        )

    def explain_instance(self, X_instance, num_features=10, top_labels=2):
        """
        Explain a single instance.
        """
        if self.mode == 'classification':
            explanation = self.explainer.explain_instance(
                X_instance.values[0],
                self.model.predict_proba,
                num_features=num_features,
                top_labels=top_labels
            )
        else:
            explanation = self.explainer.explain_instance(
                X_instance.values[0],
                self.model.predict,
                num_features=num_features
            )

        return explanation

    def plot_explanation(self, explanation, show=True):
        """
        Visualise the LIME explanation.
        """
        if show:
            explanation.show_in_notebook(show_table=True, show_all=False)
        return explanation.as_list()

    def get_feature_weights(self, explanation):
        """
        Extract feature weights from LIME explanation.
        """
        return explanation.as_list()

3.3 Integrated Gradients (IG) – For Deep Learning Models

Integrated Gradients is a gradient-based attribution method designed specifically for deep neural networks. It satisfies sensitivity and implementation invariance axioms.

Mathematical Formulation:
IG_i(x) = (x_i - x'_i) * ∫_{α=0}^{1} ∂F(x' + α(x - x')) / ∂x_i dα
where:

x is the input.
x' is a baseline input (e.g., zeros, or mean of training data).
∂F/∂x_i is the gradient of the model’s output with respect to feature i.

Interpretation: IG attributes the prediction difference between the input and baseline to each feature. The integral is approximated numerically.

Integrated Gradients Implementation:

import torch
from torch.autograd import grad

class IntegratedGradients:
    def __init__(self, model, device='cpu'):
        self.model = model
        self.device = device
        self.model.to(device)
        self.model.eval()

    def attribute(self, input_tensor, baseline=None, steps=50, target=None):
        """
        Compute Integrated Gradients for a given input.

        Args:
            input_tensor: (batch_size, features) or (batch_size, seq_len, features)
            baseline: Reference input (default: zeros)
            steps: Number of steps for integral approximation
            target: Target class for classification (default: argmax)
        """
        input_tensor = input_tensor.to(self.device)

        if baseline is None:
            baseline = torch.zeros_like(input_tensor)

        # Scale input and baseline
        input_tensor.requires_grad_(True)

        # Generate scaled inputs
        scaled_inputs = []
        for alpha in torch.linspace(0, 1, steps):
            scaled_input = baseline + alpha * (input_tensor - baseline)
            scaled_inputs.append(scaled_input)

        # Compute gradients at each step
        gradients = []
        for scaled_input in scaled_inputs:
            scaled_input.requires_grad_(True)
            output = self.model(scaled_input)

            if target is None:
                target_idx = torch.argmax(output)
            else:
                target_idx = target

            # Compute gradient of output w.r.t input
            out_grad = torch.zeros_like(output)
            out_grad[0, target_idx] = 1.0

            grad_output = grad(
                outputs=output,
                inputs=scaled_input,
                grad_outputs=out_grad,
                create_graph=False,
                retain_graph=True
            )[0]

            gradients.append(grad_output.detach())

        # Average gradients and multiply by (input - baseline)
        avg_grad = torch.stack(gradients).mean(dim=0)
        ig = (input_tensor - baseline) * avg_grad

        return ig.detach().cpu().numpy()

    def attribute_sequence(self, input_tensor, baseline=None, steps=50):
        """
        Attribute for sequence models (LSTM/Transformer).
        """
        # For sequence models, we attribute each time step
        ig_values = []

        # For each time step, compute IG
        for t in range(input_tensor.shape[1]):
            # Zero out other time steps for baseline
            seq_baseline = torch.zeros_like(input_tensor)
            seq_baseline[:, t, :] = input_tensor[:, t, :]

            # This is a simplification; in practice, use a more sophisticated baseline
            ig_t = self.attribute(input_tensor, baseline=seq_baseline, steps=steps)
            ig_values.append(ig_t)

        return np.stack(ig_values, axis=1)

4. Partial Dependence Plots (PDP) and Individual Conditional Expectation (ICE)

4.1 Partial Dependence Plots
PDP shows the marginal effect of one or two features on the predicted outcome.

PDP_j(x_j) = E_{X_{-j}}[ f(x_j, X_{-j}) ] = ∫ f(x_j, x_{-j}) dP(x_{-j})

Implementation:

from sklearn.inspection import partial_dependence
from sklearn.inspection import PartialDependenceDisplay

def plot_partial_dependence(model, X, features, feature_names, grid_resolution=50):
    """
    Create partial dependence plots for specified features.
    """
    fig, ax = plt.subplots(figsize=(12, 8))

    pdp = PartialDependenceDisplay.from_estimator(
        model,
        X,
        features=features,
        feature_names=feature_names,
        grid_resolution=grid_resolution,
        ax=ax
    )

    plt.tight_layout()
    plt.show()
    return pdp

# Example: PDP for volatility and RSI
features_to_plot = [1, 4]  # Indices of Volatility and RSI
pdp = plot_partial_dependence(model, X, features_to_plot, features)

4.2 Individual Conditional Expectation (ICE)
ICE plots show the relationship between a feature and the prediction for each individual observation.

ICE_j^i(x_j) = f(x_j, X_{-j}^i)

def plot_ice_curves(model, X, feature_idx, feature_name, n_samples=50):
    """
    Create ICE curves for a specific feature.
    """
    import matplotlib.pyplot as plt

    # Select random samples
    sample_indices = np.random.choice(len(X), n_samples, replace=False)
    X_sample = X.iloc[sample_indices].copy()

    # Create grid of feature values
    x_min = X.iloc[:, feature_idx].min()
    x_max = X.iloc[:, feature_idx].max()
    grid = np.linspace(x_min, x_max, 50)

    # Compute predictions for each sample at each grid point
    predictions = []
    for val in grid:
        X_temp = X_sample.copy()
        X_temp.iloc[:, feature_idx] = val
        pred = model.predict_proba(X_temp)[:, 1]
        predictions.append(pred)

    predictions = np.array(predictions).T

    # Plot
    plt.figure(figsize=(10, 6))
    for i in range(min(20, n_samples)):
        plt.plot(grid, predictions[i, :], alpha=0.3, color='blue')

    # Add average line (PDP)
    plt.plot(grid, predictions.mean(axis=0), color='red', linewidth=2, label='PDP (average)')

    plt.xlabel(feature_name)
    plt.ylabel('Predicted Probability')
    plt.title(f'ICE Curves for {feature_name}')
    plt.legend()
    plt.grid(True, alpha=0.3)
    plt.show()

5. Counterfactual Explanations – What-If Analysis

Counterfactual explanations answer: “What changes to the input would change the model’s decision?”

Mathematical Formulation:
x' = argmin_{x'} [ |f(x') - y_target| + λ * distance(x, x') ]
where:

y_target is the desired outcome.
distance is a metric (e.g., L1, L2) measuring the similarity between x and x'.
λ balances the two objectives.

Counterfactual Implementation:

class CounterfactualExplainer:
    def __init__(self, model, feature_names, feature_bounds=None):
        self.model = model
        self.feature_names = feature_names
        self.feature_bounds = feature_bounds

    def generate_counterfactual(self, X_instance, target_class=1, n_steps=100, lr=0.01):
        """
        Generate a counterfactual explanation.
        """
        # Convert to tensor
        x = torch.FloatTensor(X_instance.values[0]).clone().detach().requires_grad_(True)

        # Define loss
        optimizer = torch.optim.Adam([x], lr=lr)

        for step in range(n_steps):
            optimizer.zero_grad()

            # Forward pass
            pred = self.model(x.unsqueeze(0))
            if hasattr(pred, 'logits'):
                pred = pred.logits

            # Loss: cross-entropy to target class + distance penalty
            target = torch.tensor([target_class])
            ce_loss = torch.nn.CrossEntropyLoss()(pred, target)

            # Distance penalty (L1)
            original = torch.FloatTensor(X_instance.values[0])
            distance = torch.abs(x - original).mean()

            # Total loss
            loss = ce_loss + 0.01 * distance

            loss.backward()
            optimizer.step()

            # Clip to feature bounds
            if self.feature_bounds is not None:
                with torch.no_grad():
                    for i, (low, high) in enumerate(self.feature_bounds):
                        x[i] = torch.clamp(x[i], low, high)

        # Check if counterfactual is valid
        final_pred = self.model(x.unsqueeze(0))
        if hasattr(final_pred, 'logits'):
            final_pred = final_pred.logits
        new_class = torch.argmax(final_pred).item()

        return {
            'counterfactual': x.detach().numpy(),
            'original': X_instance.values[0],
            'original_class': torch.argmax(self.model(torch.FloatTensor(X_instance.values[0]).unsqueeze(0))).item(),
            'new_class': new_class,
            'success': new_class == target_class
        }

6. Model Documentation for MRM Compliance

6.1 Model Documentation Template

class ModelDocumentation:
    def __init__(self, model_name, model_version, model_type):
        self.model_name = model_name
        self.model_version = model_version
        self.model_type = model_type
        self.documentation = {}

    def add_section(self, section_name, content):
        self.documentation[section_name] = content

    def generate_report(self):
        """
        Generate a comprehensive model documentation report.
        """
        report = f"""
        ========================================
        MODEL DOCUMENTATION REPORT
        ========================================

        1. MODEL OVERVIEW
        ------------------
        Model Name: {self.model_name}
        Version: {self.model_version}
        Type: {self.model_type}
        Purpose: {self.documentation.get('purpose', 'Not specified')}
        Deployment Date: {self.documentation.get('deployment_date', 'Not specified')}

        2. CONCEPTUAL SOUNDNESS
        -----------------------
        Theoretical Foundation: {self.documentation.get('theory', 'Not specified')}
        Assumptions: {self.documentation.get('assumptions', 'Not specified')}
        Limitations: {self.documentation.get('limitations', 'Not specified')}

        3. DATA AND FEATURES
        --------------------
        Data Sources: {self.documentation.get('data_sources', 'Not specified')}
        Date Range: {self.documentation.get('date_range', 'Not specified')}
        Feature List:
        {self.documentation.get('features', 'Not specified')}
        Target Variable: {self.documentation.get('target', 'Not specified')}

        4. MODEL DEVELOPMENT
        --------------------
        Training Algorithm: {self.documentation.get('algorithm', 'Not specified')}
        Hyperparameters:
        {self.documentation.get('hyperparameters', 'Not specified')}
        Training Sample Size: {self.documentation.get('train_size', 'Not specified')}
        Validation Sample Size: {self.documentation.get('val_size', 'Not specified')}

        5. PERFORMANCE METRICS
        ----------------------
        In-Sample Performance:
        {self.documentation.get('in_sample_metrics', 'Not specified')}
        Out-of-Sample Performance:
        {self.documentation.get('out_of_sample_metrics', 'Not specified')}
        Backtest Results:
        {self.documentation.get('backtest_results', 'Not specified')}

        6. VALIDATION
        -------------
        Statistical Tests:
        {self.documentation.get('statistical_tests', 'Not specified')}
        Stress Tests:
        {self.documentation.get('stress_tests', 'Not specified')}
        Sensitivity Analysis:
        {self.documentation.get('sensitivity_analysis', 'Not specified')}

        7. EXPLAINABILITY
        -----------------
        Feature Importance:
        {self.documentation.get('feature_importance', 'Not specified')}
        SHAP/LIME Analysis:
        {self.documentation.get('shap_analysis', 'Not specified')}

        8. GOVERNANCE
        -------------
        Model Owner: {self.documentation.get('model_owner', 'Not specified')}
        Validation Team: {self.documentation.get('validation_team', 'Not specified')}
        Approval Date: {self.documentation.get('approval_date', 'Not specified')}
        Review Frequency: {self.documentation.get('review_frequency', 'Not specified')}

        9. MONITORING
        -------------
        Performance Metrics Tracked:
        {self.documentation.get('monitoring_metrics', 'Not specified')}
        Drift Detection Methods:
        {self.documentation.get('drift_detection', 'Not specified')}
        Retraining Triggers:
        {self.documentation.get('retraining_triggers', 'Not specified')}

        10. APPENDIX
        -------------
        Code Repository: {self.documentation.get('code_repo', 'Not specified')}
        Data Dictionary: {self.documentation.get('data_dict', 'Not specified')}
        """

        return report

    def save_report(self, filepath='model_documentation.txt'):
        with open(filepath, 'w') as f:
            f.write(self.generate_report())
        print(f"Report saved to {filepath}")

7. Summary for the AI Practitioner

Regulatory compliance (SR 11-7, GDPR, EU AI Act) requires model explainability. This is non-negotiable in finance.
SHAP is the gold standard for feature attribution. It is model-agnostic and satisfies desirable mathematical properties.
LIME provides local explanations by fitting interpretable models locally. It is fast but less stable than SHAP.
Integrated Gradients is for deep learning models. It satisfies sensitivity and implementation invariance axioms.
PDP and ICE plots show the marginal effect of features. They are essential for understanding non-linear relationships.
Counterfactual explanations answer what-if questions. They are valuable for regulatory approval and stakeholder communication.
Model documentation is mandatory. It should cover conceptual soundness, data, development, validation, explainability, governance, and monitoring.
Trade-off: More complex models (deep learning) are less interpretable. Use simpler models (linear, tree-based) where explainability is critical.

Diploma in Artificial Intelligence in Finance Lesson 3.7: Model Interpretability and Explainability for Financial AI