LESSON 1: LOGISTIC REGRESSION – THE CORNERSTONE OF CREDIT SCORING

SECTION 1: LEARNING OBJECTIVES

By the end of this lesson, you will be able to:

Distinguish logistic regression from linear regression and explain why it is the industry standard for binary classification (default / no‑default, fraud / legitimate).
Derive the logistic (sigmoid) function and understand its role in mapping linear combinations to probabilities.
Formulate the log‑odds (logit) and interpret coefficients in terms of odds ratios – a critical skill for regulatory explainability.
Estimate model parameters via Maximum Likelihood Estimation (MLE) – extending the MLE concepts from Module 3.
Evaluate model performance using the Confusion Matrix, Accuracy, Precision, Recall, F1‑Score, and the ROC‑AUC curve.
Apply the Kolmogorov‑Smirnov (KS) statistic – the regulatory benchmark for credit scoring model separation power.
Interpret coefficients to satisfy fair lending and model validation requirements.
Implement logistic regression in Python using statsmodels (for inference) and scikit‑learn (for prediction), and handle class imbalance.

SECTION 2: WHY LOGISTIC REGRESSION IN BANKING?

In finance, many critical outcomes are binary:

Outcome	Target = 1	Target = 0
Loan default	Borrower defaults	Borrower repays
Fraud	Transaction is fraudulent	Transaction is legitimate
Churn	Customer closes account	Customer stays
Response to marketing	Customer accepts offer	Customer declines
Credit card delinquency	90+ days past due	Current / <90 days

Linear regression fails here because:

It can predict probabilities < 0 or > 1.
The relationship between predictors and probability is almost always S‑shaped (sigmoidal), not linear.
The error terms are heteroscedastic and non‑normal.

Logistic regression solves all these problems by modelling the log‑odds as a linear function, then transforming back to probabilities. It is:

Interpretable: Coeffcients translate directly to odds ratios.
Regulator‑friendly: Clear functional form; easy to validate and document.
Efficient: Converges quickly even with large banking datasets.
Extensible: Forms the foundation for more complex models (e.g., neural networks with sigmoid output layers).

SECTION 3: THE LOGISTIC MODEL – MATHEMATICAL FOUNDATION

3.1 The Odds and Log‑Odds

Let $p = P (Y = 1 ∣ X_{1}, X_{2}, \dots, X_{k})$ be the probability of default (or fraud, etc.).

Odds are defined as:

$Odds = 1 - p p$

Odds range from 0 (when p=0) to ∞ (when p=1). Odds of 2 mean the event is twice as likely to occur as not.

The log‑odds (or logit) is the natural logarithm of the odds:

$logit (p) = ln (1 - p p)$

The logit ranges from −∞ to +∞, which makes it suitable as the dependent variable in a linear model.

3.2 The Logistic Regression Equation

We model the log‑odds as a linear combination of predictors:

$ln (1 - p p) = β_{0} + β_{1} X_{1} + β_{2} X_{2} + \dots + β_{k} X_{k}$

Solving for $p$ gives the sigmoid (logistic) function:

$p = 1 + e ^{- (β 0 + β 1 X 1 + \dots + β k X k)} 1 = 1 + e ^{- X β} 1$

The sigmoid maps any real value to the interval (0,1), making it a proper probability function.

3.3 Interpreting Coefficients – The Odds Ratio

This is the most critical business interpretation.

Take the exponential of both sides of the log‑odds equation:

$1 - p p = e^{β 0 + β 1 X 1 + \dots + β k X k}$

Consider a predictor $X_{j}$ . If $X_{j}$ increases by 1 unit, holding all else constant, the odds multiply by $e^{β j}$ .

If $β_{j} > 0$ : $e^{β j} > 1$ → odds increase → higher probability of default (risk factor).
If $β_{j} < 0$ : $e^{β j} < 1$ → odds decrease → lower probability of default (protective factor).

Example: In a credit model, the coefficient for Debt‑to‑Income (DTI) is $β = 0.05$ . Then $e^{0.05} = 1.051$ . A 1‑percentage‑point increase in DTI increases the odds of default by 5.1%. This is a tangible, business‑friendly statement that regulators expect.

SECTION 4: PARAMETER ESTIMATION – MAXIMUM LIKELIHOOD (MLE)

Unlike linear regression (which uses Ordinary Least Squares), logistic regression has no closed‑form solution. We use Maximum Likelihood Estimation – a direct extension of Module 3, Lesson 8.

For each observation $i$ , with true label $y_{i} \in {0, 1}$ and predicted probability $p_{i}$ , the likelihood of observing that single outcome is:

$L_{i} = p_{i y i} (1 - p_{i})^{1 - y i}$

The total likelihood across $n$ independent observations is:

$L (β) = i = 1 \prod n p_{i y i} (1 - p_{i})^{1 - y i}$

The log‑likelihood is:

$ℓ (β) = i = 1 \sum n [y_{i} ln (p_{i}) + (1 - y_{i}) ln (1 - p_{i})]$

Substituting $p_{i} = 1 + e ^{- x i β} 1$ , we get:

$ℓ (β) = i = 1 \sum n [y_{i} x_{i} β - ln (1 + e^{x i β})]$

MLE finds the $β$ that maximises $ℓ (β)$ . There is no analytical solution; we use iterative numerical optimisation (Newton‑Raphson, Gradient Descent). The negative of this log‑likelihood is the log‑loss (cross‑entropy), which we minimise.

SECTION 5: EVALUATION METRICS – BEYOND ACCURACY

In banking, accuracy is often misleading because datasets are imbalanced (e.g., 98% good loans, 2% defaults). Predicting “no default” for everyone gives 98% accuracy but is worthless.

5.1 Confusion Matrix

	Predicted Positive (Default)	Predicted Negative (No Default)
Actual Positive	True Positive (TP)	False Negative (FN) – Type II error
Actual Negative	False Positive (FP) – Type I error	True Negative (TN)

5.2 Key Metrics

Precision (Positive Predictive Value): $TP + FP TP$ . Of all loans flagged as default, what fraction actually default? Critical for minimising false alarms.
Recall (Sensitivity / True Positive Rate): $TP + FN TP$ . Of all actual defaults, what fraction did we catch? Critical for minimising losses.
F1‑Score: Harmonic mean of Precision and Recall: $2 \cdot Precision + Recall Precision \cdot Recall$ .
Specificity (True Negative Rate): $TN + FP TN$ . Of all good loans, what fraction did we correctly identify?

5.3 ROC‑AUC (Receiver Operating Characteristic – Area Under Curve)

The ROC curve plots True Positive Rate (Recall) against False Positive Rate (1‑Specificity) at all classification thresholds.
AUC (Area Under the Curve) is a single number between 0.5 (random) and 1.0 (perfect).
Regulatory benchmark: In credit scoring, AUC > 0.75 is considered acceptable; > 0.80 is good; > 0.85 is excellent for consumer lending.

5.4 Kolmogorov‑Smirnov (KS) Statistic

The KS statistic measures the maximum vertical distance between the cumulative distribution of predicted probabilities for defaults and non‑defaults.
Formula: KS = \max_i \left( \text{% of good loans with score ≤ i} – \text{% of bad loans with score ≤ i} \right).
Regulatory importance: Under Basel and SR 11‑7, KS is the primary metric to rank‑order risk – a high KS (> 0.30) indicates the model effectively separates good from bad borrowers.

SECTION 6: IMPLEMENTATION IN PYTHON

Below is a comprehensive implementation covering data preparation, model fitting, coefficient interpretation, and performance evaluation.

# ===================================================================
# MODULE 4, LESSON 1: LOGISTIC REGRESSION FOR CREDIT SCORING
# ===================================================================

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (confusion_matrix, classification_report, 
                             roc_curve, roc_auc_score, precision_recall_curve)
import statsmodels.api as sm
from scipy import stats

# Set style and reproducibility
np.random.seed(42)
sns.set_style("whitegrid")

print("="*70)
print("LOGISTIC REGRESSION – CREDIT DEFAULT PREDICTION")
print("="*70)

# ----------------------------------------------------------------
# PART A: GENERATE SYNTHETIC BANKING DATA
# ----------------------------------------------------------------

n_customers = 10000

# Generate features (realistic distributions)
income = np.random.gamma(5, 15, n_customers) + 20          # in $000s (20-120k)
age = np.random.normal(45, 12, n_customers).clip(18, 80)
debt_to_income = np.random.beta(2, 5, n_customers) * 60    # 0-60%
credit_score = np.random.normal(700, 50, n_customers).clip(550, 850)
loan_amount = np.random.gamma(4, 50, n_customers) + 50     # in $000s

# Generate default probability (based on a true underlying logic)
# Higher DTI and lower credit score increase default risk
log_odds = -4.5 + 0.04 * debt_to_income - 0.005 * credit_score + 0.01 * (loan_amount/1000)
prob_default = 1 / (1 + np.exp(-log_odds))
default = np.random.binomial(1, prob_default, n_customers)

df = pd.DataFrame({
    'income': income,
    'age': age,
    'dti': debt_to_income,
    'credit_score': credit_score,
    'loan_amount': loan_amount,
    'default': default
})

print(f"\nDataset shape: {df.shape}")
print(f"Default rate: {df['default'].mean():.4f} ({df['default'].mean()*100:.2f}%)")
print("\nFeature statistics:\n", df.describe().round(2))

# ----------------------------------------------------------------
# PART B: TRAIN / TEST SPLIT & SCALING
# ----------------------------------------------------------------

X = df[['income', 'age', 'dti', 'credit_score', 'loan_amount']]
y = df['default']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Standardise features (important for convergence and coefficient interpretation)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Convert scaled arrays back to DataFrames for statsmodels
X_train_scaled_df = pd.DataFrame(X_train_scaled, columns=X.columns)
X_test_scaled_df = pd.DataFrame(X_test_scaled, columns=X.columns)

# Add intercept for statsmodels
X_train_sm = sm.add_constant(X_train_scaled_df)
X_test_sm = sm.add_constant(X_test_scaled_df)

print(f"\nTraining set: {len(X_train)} samples")
print(f"Test set:     {len(X_test)} samples")

# ----------------------------------------------------------------
# PART C: STATSMODELS – INFERENTIAL LOGISTIC REGRESSION
# ----------------------------------------------------------------

print("\n" + "="*70)
print("PART C: STATSMODELS – COEFFICIENT ESTIMATION & INFERENCE")
print("="*70)

# Fit using statsmodels (gives p-values, confidence intervals)
logit_model = sm.Logit(y_train, X_train_sm)
result = logit_model.fit(method='bfgs', maxiter=1000, disp=0)

print("\nModel Summary:")
print(result.summary())

# Extract coefficients and odds ratios
coefficients = result.params
odds_ratios = np.exp(coefficients)
p_values = result.pvalues

print("\n--- COEFFICIENT INTERPRETATION (Odds Ratios) ---")
for var in coefficients.index:
    if var == 'const':
        continue
    coef = coefficients[var]
    or_val = odds_ratios[var]
    p_val = p_values[var]
    sig = "***" if p_val < 0.001 else "**" if p_val < 0.01 else "*" if p_val < 0.05 else "ns"
    direction = "Risk factor" if or_val > 1 else "Protective factor"
    print(f"{var:>15}: β={coef:.4f}, OR={or_val:.4f}, p={p_val:.4f} {sig} → {direction}")

# For the intercept (not scaled), interpret carefully
print(f"\nIntercept (const): β={coefficients['const']:.4f} (baseline log-odds at mean of all features)")

# ----------------------------------------------------------------
# PART D: SCIKIT-LEARN – PREDICTION & PERFORMANCE
# ----------------------------------------------------------------

print("\n" + "="*70)
print("PART D: PREDICTIVE PERFORMANCE (SCIKIT-LEARN)")
print("="*70)

# Fit logistic regression with L2 penalty (default)
clf = LogisticRegression(C=1.0, solver='lbfgs', max_iter=1000, random_state=42)
clf.fit(X_train_scaled, y_train)

# Predict probabilities
y_pred_prob = clf.predict_proba(X_test_scaled)[:, 1]
y_pred_class = clf.predict(X_test_scaled)

# Confusion Matrix
cm = confusion_matrix(y_test, y_pred_class)
print("\nConfusion Matrix:")
print(pd.DataFrame(cm, columns=['Pred 0', 'Pred 1'], index=['Actual 0', 'Actual 1']))

# Classification Report
print("\nClassification Report:")
print(classification_report(y_test, y_pred_class, target_names=['No Default', 'Default']))

# ROC-AUC
auc = roc_auc_score(y_test, y_pred_prob)
fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)

print(f"\nROC-AUC Score: {auc:.4f}")

# KS Statistic
# Compute cumulative distributions of predicted probabilities for default vs non-default
default_scores = y_pred_prob[y_test == 1]
non_default_scores = y_pred_prob[y_test == 0]

# Calculate KS
ks_stat = np.max(np.abs(np.percentile(default_scores, np.linspace(0, 100, 1000)) - 
                        np.percentile(non_default_scores, np.linspace(0, 100, 1000))))
# More accurate: use empirical CDF
from statsmodels.distributions.empirical_distribution import ECDF
ecdf_def = ECDF(default_scores)
ecdf_nondef = ECDF(non_default_scores)
x_vals = np.linspace(0, 1, 1000)
ks_stat = np.max(np.abs(ecdf_def(x_vals) - ecdf_nondef(x_vals)))
print(f"KS Statistic: {ks_stat:.4f}")
print("  (KS > 0.30 indicates strong discrimination; >0.40 is excellent)")

# ----------------------------------------------------------------
# PART E: VISUALISATIONS – ROC CURVE & KS
# ----------------------------------------------------------------

fig, axes = plt.subplots(1, 3, figsize=(18, 5))

# ROC Curve
ax = axes[0]
ax.plot(fpr, tpr, 'b-', linewidth=2, label=f'AUC = {auc:.3f}')
ax.plot([0, 1], [0, 1], 'r--', linewidth=1, label='Random')
ax.set_xlabel('False Positive Rate (1 - Specificity)')
ax.set_ylabel('True Positive Rate (Recall)')
ax.set_title('ROC Curve', fontsize=12)
ax.legend()
ax.grid(True, alpha=0.3)

# KS Plot (Cumulative distributions)
ax = axes[1]
x_plot = np.linspace(0, 1, 500)
cdf_def = ecdf_def(x_plot)
cdf_nondef = ecdf_nondef(x_plot)
diff = cdf_def - cdf_nondef
max_idx = np.argmax(np.abs(diff))
ax.plot(x_plot, cdf_def, 'r-', label='Defaults', linewidth=2)
ax.plot(x_plot, cdf_nondef, 'g-', label='Non-Defaults', linewidth=2)
ax.vlines(x_plot[max_idx], cdf_nondef[max_idx], cdf_def[max_idx], 
          colors='blue', linestyle='--', linewidth=3, 
          label=f'KS = {ks_stat:.3f}')
ax.set_xlabel('Predicted Probability')
ax.set_ylabel('Cumulative Proportion')
ax.set_title('KS Statistic', fontsize=12)
ax.legend()
ax.grid(True, alpha=0.3)

# Precision-Recall Curve
ax = axes[2]
precision, recall, _ = precision_recall_curve(y_test, y_pred_prob)
ax.plot(recall, precision, 'purple', linewidth=2)
ax.set_xlabel('Recall')
ax.set_ylabel('Precision')
ax.set_title('Precision-Recall Curve', fontsize=12)
ax.grid(True, alpha=0.3)

plt.tight_layout()
plt.savefig('logistic_regression_metrics.png', dpi=300)
plt.show()

# ----------------------------------------------------------------
# PART F: CALIBRATION – ARE PREDICTED PROBABILITIES RELIABLE?
# ----------------------------------------------------------------

print("\n" + "="*70)
print("PART F: MODEL CALIBRATION")
print("="*70)

# Calibration plot: group predicted probabilities into bins and compare to observed default rates
df_test = pd.DataFrame({'y_true': y_test, 'y_prob': y_pred_prob})
df_test['bin'] = pd.cut(df_test['y_prob'], bins=10, labels=False, include_lowest=True)

calibration_data = df_test.groupby('bin').agg(
    observed_rate=('y_true', 'mean'),
    predicted_rate=('y_prob', 'mean'),
    count=('y_true', 'count')
).reset_index()

print("\nCalibration by probability decile:")
print(calibration_data)

# Visual calibration
fig, ax = plt.subplots(figsize=(8, 6))
ax.plot(calibration_data['predicted_rate'], calibration_data['observed_rate'], 
        'bo-', linewidth=2, label='Model')
ax.plot([0, 1], [0, 1], 'r--', linewidth=1, label='Perfect Calibration')
ax.set_xlabel('Mean Predicted Probability')
ax.set_ylabel('Observed Default Rate')
ax.set_title('Calibration Plot', fontsize=12)
ax.legend()
ax.grid(True, alpha=0.3)
plt.savefig('calibration_plot.png', dpi=300)
plt.show()

print("\n Business Interpretation:")
print("  - If points lie below the diagonal, the model overestimates risk (conservative).")
print("  - If above the diagonal, it underestimates risk (aggressive).")
print("  - For regulatory models, calibration must be validated across all score bands.")

SECTION 7: REGULATORY AND BUSINESS INTERPRETATION

SR 11‑7 (Supervisory Guidance on Model Risk Management) requires:

Clear documentation of variables, coefficients, and their economic rationale.
Out‑of‑sample testing (we performed train/test split).
Ongoing monitoring of KS and AUC over time (model drift).
Explainability: Each coefficient must be justifiable – for example, a negative coefficient on credit score is expected (higher score → lower default). An unexpected sign would trigger a regulatory review.

Fair Lending (ECOA, Fair Housing Act): We must test for disparate impact – i.e., whether the model inadvertently discriminates against protected groups (race, gender, age). We do this by:

Removing prohibited variables (or proving they are not proxies).
Running a disparate impact analysis (the 4/5 rule) using predicted probabilities across groups.
Logistic regression excels here because it provides transparent, auditable feature contributions.

SECTION 8: SUMMARY FOR THE DATA PRACTITIONER

Logistic regression models the log‑odds of a binary outcome as a linear function of predictors.
MLE is used to estimate coefficients; interpretation is via odds ratios $e^{β}$ .
Evaluation relies on AUC, KS, and calibrated probability plots – not just accuracy.
In Python, use statsmodels for statistical inference (p‑values, confidence intervals) and scikit‑learn for scalable prediction and cross‑validation.
Regulatory compliance demands transparency, variable justification, and continuous performance monitoring.

[END OF LESSON 1 – MODULE 4]

Diploma in Financial Data Analytics LESSON 1: LOGISTIC REGRESSION – THE CORNERSTONE OF CREDIT SCORING

Diploma in Financial Data Analytics