LESSON 6: FOUNDATIONS OF EDA FOR FINANCIAL DATA

SECTION 1: LEARNING OBJECTIVES

By the end of this lesson, you will be able to:

Understand the EDA process and why it is critical before building any financial model.
Load and inspect financial datasets using Python’s pandas library.
Calculate and interpret summary statistics for financial variables (mean, median, standard deviation, skewness, kurtosis).
Create visualizations to understand data distributions, relationships, and patterns.
Identify data quality issues including missing values, outliers, and inconsistencies.
Understand the concept of data profiling and how it applies to banking data.
Document EDA findings effectively for stakeholders and regulators.
Recognize common patterns in financial data (heavy tails, volatility clustering, seasonality).

SECTION 2: INTRODUCTION TO EXPLORATORY DATA ANALYSIS

2.1 What Is EDA and Why Does It Matter in Banking?

Exploratory Data Analysis (EDA) is the process of analyzing datasets to summarize their main characteristics, often using visual methods. In banking, EDA is the critical first step before any modeling or reporting.

Why EDA Is Essential in Banking:

Reason	Example
Understand Data Quality	Find missing credit scores before building a risk model
Identify Patterns	Detect seasonality in transaction volumes
Detect Anomalies	Find suspicious transactions that might indicate fraud
Validate Assumptions	Check if loan amounts follow expected distributions
Guide Feature Engineering	Discover that log transformation improves model performance
Communicate Insights	Present data characteristics to stakeholders

2.2 The EDA Process

EDA WORKFLOW:

Step 1: Data Collection & Loading
    ↓
Step 2: Data Profiling (Understanding Structure)
    ↓
Step 3: Summary Statistics & Distributions
    ↓
Step 4: Visual Exploration
    ↓
Step 5: Missing Data Analysis
    ↓
Step 6: Outlier Detection
    ↓
Step 7: Correlation Analysis
    ↓
Step 8: Insights & Documentation

2.3 Setting Up Your EDA Environment

# Import required libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from datetime import datetime, timedelta
import warnings
warnings.filterwarnings('ignore')

# Set visualization style
plt.style.use('seaborn-v0_8-darkgrid')
sns.set_palette("Set2")
plt.rcParams['figure.figsize'] = (12, 8)

# Create sample banking dataset for EDA
def generate_banking_data(n_customers=1000, n_transactions=5000):
    """Generate realistic banking data for EDA."""
    np.random.seed(42)
    
    # Customers
    customers = pd.DataFrame({
        'customer_id': range(1, n_customers + 1),
        'age': np.random.normal(45, 15, n_customers).astype(int).clip(18, 80),
        'annual_income': np.random.lognormal(10.5, 0.6, n_customers).astype(int).clip(20000, 250000),
        'credit_score': np.random.normal(700, 50, n_customers).astype(int).clip(500, 850),
        'state': np.random.choice(['CA', 'NY', 'TX', 'FL', 'IL', 'PA', 'OH', 'GA', 'NC', 'MI'], n_customers),
        'segment': np.random.choice(['Premium', 'Standard', 'Basic'], n_customers, p=[0.15, 0.55, 0.30]),
        'tenure_months': np.random.exponential(36, n_customers).astype(int).clip(1, 240)
    })
    
    # Introduce some missing values
    customers.loc[np.random.choice(customers.index, size=50, replace=False), 'credit_score'] = np.nan
    customers.loc[np.random.choice(customers.index, size=30, replace=False), 'annual_income'] = np.nan
    
    # Transactions (last 12 months)
    transactions = []
    for i in range(n_transactions):
        cust_id = np.random.randint(1, n_customers + 1)
        days_ago = np.random.randint(0, 365)
        amount = np.random.lognormal(4, 1.2)
        trans_type = np.random.choice(['Deposit', 'Withdrawal', 'Purchase', 'Transfer'], 
                                     p=[0.25, 0.30, 0.35, 0.10])
        status = np.random.choice(['Completed', 'Pending', 'Failed'], p=[0.85, 0.08, 0.07])
        
        transactions.append({
            'transaction_id': i + 1,
            'customer_id': cust_id,
            'date': datetime.now() - timedelta(days=days_ago),
            'amount': round(amount, 2),
            'transaction_type': trans_type,
            'status': status,
            'merchant': np.random.choice(['Amazon', 'Walmart', 'Target', 'Starbucks', 'Whole Foods', 
                                         'Shell Gas', 'CVS', 'Home Depot', 'Apple Store', 'Netflix'], size=1)[0],
            'category': np.random.choice(['Shopping', 'Groceries', 'Dining', 'Transportation', 
                                          'Entertainment', 'Health', 'Home Improvement'], size=1)[0]
        })
    
    transactions = pd.DataFrame(transactions)
    
    # Convert date
    transactions['date'] = pd.to_datetime(transactions['date'])
    transactions['year'] = transactions['date'].dt.year
    transactions['month'] = transactions['date'].dt.month
    transactions['day'] = transactions['date'].dt.day
    transactions['day_of_week'] = transactions['date'].dt.day_name()
    
    return customers, transactions

# Generate data
customers, transactions = generate_banking_data()
print(f"Customers: {len(customers)}")
print(f"Transactions: {len(transactions)}")
print("\nCustomers Sample:")
print(customers.head())
print("\nTransactions Sample:")
print(transactions.head())

SECTION 3: DATA PROFILING – UNDERSTANDING YOUR DATA

3.1 Basic Data Inspection

# ============= DATA PROFILING =============

# 1. Data Overview
print("="*60)
print("DATA PROFILING REPORT")
print("="*60)

print(f"\n📊 Data Shape:")
print(f"  Customers: {customers.shape[0]} rows, {customers.shape[1]} columns")
print(f"  Transactions: {transactions.shape[0]} rows, {transactions.shape[1]} columns")

# 2. Data Types
print("\n📋 Data Types:")
print(f"Customers:\n{customers.dtypes}")
print(f"\nTransactions:\n{transactions.dtypes}")

# 3. Memory Usage
print("\n💾 Memory Usage:")
print(f"  Customers: {customers.memory_usage(deep=True).sum() / 1024**2:.2f} MB")
print(f"  Transactions: {transactions.memory_usage(deep=True).sum() / 1024**2:.2f} MB")

# 4. Column Info
print("\n📝 Column Info:")
print("Customers:")
print(customers.info())
print("\nTransactions:")
print(transactions.info())

# 5. Unique Values
print("\n🔑 Unique Values:")
print(f"  Customer states: {customers['state'].unique()}")
print(f"  Customer segments: {customers['segment'].unique()}")
print(f"  Transaction types: {transactions['transaction_type'].unique()}")
print(f"  Transaction statuses: {transactions['status'].unique()}")

3.2 Summary Statistics

# ============= SUMMARY STATISTICS =============

# Numerical columns
print("\n" + "="*60)
print("SUMMARY STATISTICS - NUMERICAL COLUMNS")
print("="*60)

print("\nCustomers - Numerical:")
print(customers.describe())

print("\nTransactions - Numerical:")
print(transactions[['amount']].describe())

# Categorical columns
print("\n" + "="*60)
print("SUMMARY STATISTICS - CATEGORICAL COLUMNS")
print("="*60)

print("\nCustomers - Categorical:")
for col in ['state', 'segment']:
    print(f"\n{col.upper()}:")
    print(customers[col].value_counts())
    print(f"  Unique values: {customers[col].nunique()}")
    print(f"  Most common: {customers[col].mode().values[0]} ({customers[col].value_counts().iloc[0]} records)")

print("\nTransactions - Categorical:")
for col in ['transaction_type', 'status', 'category', 'merchant']:
    print(f"\n{col.upper()}:")
    print(transactions[col].value_counts().head(5))
    print(f"  Unique values: {transactions[col].nunique()}")

3.3 Understanding Distribution Statistics

# ============= DISTRIBUTION ANALYSIS =============

def calculate_distribution_stats(data):
    """Calculate detailed distribution statistics."""
    stats = {
        'mean': data.mean(),
        'median': data.median(),
        'mode': data.mode().iloc[0] if not data.mode().empty else np.nan,
        'std': data.std(),
        'variance': data.var(),
        'skewness': data.skew(),
        'kurtosis': data.kurtosis(),
        'range': data.max() - data.min(),
        'iqr': data.quantile(0.75) - data.quantile(0.25),
        'q1': data.quantile(0.25),
        'q3': data.quantile(0.75),
        'min': data.min(),
        'max': data.max()
    }
    return stats

print("\n" + "="*60)
print("DISTRIBUTION STATISTICS")
print("="*60)

# Customer age distribution
print("\n📊 Customer Age Distribution:")
age_stats = calculate_distribution_stats(customers['age'].dropna())
for key, value in age_stats.items():
    print(f"  {key}: {value:.2f}")

# Customer income distribution
print("\n📊 Customer Income Distribution:")
income_stats = calculate_distribution_stats(customers['annual_income'].dropna())
for key, value in income_stats.items():
    print(f"  {key}: ${value:,.2f}" if key in ['mean', 'median'] else f"  {key}: {value:.2f}")

# Credit score distribution
print("\n📊 Credit Score Distribution:")
credit_stats = calculate_distribution_stats(customers['credit_score'].dropna())
for key, value in credit_stats.items():
    print(f"  {key}: {value:.2f}")

# Transaction amount distribution
print("\n📊 Transaction Amount Distribution:")
txn_stats = calculate_distribution_stats(transactions['amount'].dropna())
for key, value in txn_stats.items():
    print(f"  {key}: ${value:,.2f}" if key in ['mean', 'median'] else f"  {key}: {value:.2f}")

SECTION 4: VISUAL EXPLORATION

4.1 Visualizing Distributions

# ============= VISUAL EXPLORATION =============

# 1. Histogram: Customer Age Distribution
fig, axes = plt.subplots(2, 3, figsize=(18, 10))

# Age distribution
customers['age'].hist(bins=30, ax=axes[0, 0], edgecolor='black', alpha=0.7)
axes[0, 0].set_title('Customer Age Distribution', fontsize=14)
axes[0, 0].set_xlabel('Age')
axes[0, 0].set_ylabel('Count')
axes[0, 0].axvline(customers['age'].mean(), color='red', linestyle='--', label=f"Mean: {customers['age'].mean():.1f}")
axes[0, 0].axvline(customers['age'].median(), color='blue', linestyle='--', label=f"Median: {customers['age'].median():.1f}")
axes[0, 0].legend()

# Income distribution (with log transform)
customers['annual_income'].hist(bins=50, ax=axes[0, 1], edgecolor='black', alpha=0.7)
axes[0, 1].set_title('Annual Income Distribution', fontsize=14)
axes[0, 1].set_xlabel('Annual Income ($)')
axes[0, 1].set_ylabel('Count')
axes[0, 1].axvline(customers['annual_income'].mean(), color='red', linestyle='--', label=f"Mean: ${customers['annual_income'].mean():,.0f}")
axes[0, 1].axvline(customers['annual_income'].median(), color='blue', linestyle='--', label=f"Median: ${customers['annual_income'].median():,.0f}")
axes[0, 1].legend()

# Log income distribution (better visualization)
np.log(customers['annual_income'].dropna()).hist(bins=30, ax=axes[0, 2], edgecolor='black', alpha=0.7)
axes[0, 2].set_title('Log Annual Income Distribution', fontsize=14)
axes[0, 2].set_xlabel('Log(Annual Income)')
axes[0, 2].set_ylabel('Count')

# Credit score distribution
customers['credit_score'].dropna().hist(bins=30, ax=axes[1, 0], edgecolor='black', alpha=0.7)
axes[1, 0].set_title('Credit Score Distribution', fontsize=14)
axes[1, 0].set_xlabel('Credit Score')
axes[1, 0].set_ylabel('Count')
axes[1, 0].axvline(customers['credit_score'].mean(), color='red', linestyle='--', label=f"Mean: {customers['credit_score'].mean():.1f}")
axes[1, 0].axvline(customers['credit_score'].median(), color='blue', linestyle='--', label=f"Median: {customers['credit_score'].median():.1f}")
axes[1, 0].legend()

# Tenure distribution
customers['tenure_months'].hist(bins=30, ax=axes[1, 1], edgecolor='black', alpha=0.7)
axes[1, 1].set_title('Customer Tenure Distribution', fontsize=14)
axes[1, 1].set_xlabel('Tenure (Months)')
axes[1, 1].set_ylabel('Count')

# Transaction amount distribution
transactions['amount'].hist(bins=50, ax=axes[1, 2], edgecolor='black', alpha=0.7)
axes[1, 2].set_title('Transaction Amount Distribution', fontsize=14)
axes[1, 2].set_xlabel('Amount ($)')
axes[1, 2].set_ylabel('Count')
axes[1, 2].axvline(transactions['amount'].mean(), color='red', linestyle='--', label=f"Mean: ${transactions['amount'].mean():.2f}")
axes[1, 2].axvline(transactions['amount'].median(), color='blue', linestyle='--', label=f"Median: ${transactions['amount'].median():.2f}")
axes[1, 2].legend()

plt.tight_layout()
plt.savefig('distribution_plots.png', dpi=300)
plt.show()

4.2 Box Plots for Outlier Detection

# ============= BOX PLOTS =============

fig, axes = plt.subplots(2, 2, figsize=(14, 10))

# Box plot by segment
customers.boxplot(column='annual_income', by='segment', ax=axes[0, 0])
axes[0, 0].set_title('Income by Customer Segment', fontsize=14)
axes[0, 0].set_xlabel('Segment')
axes[0, 0].set_ylabel('Annual Income ($)')

# Box plot by state
customers.boxplot(column='credit_score', by='state', ax=axes[0, 1])
axes[0, 1].set_title('Credit Score by State', fontsize=14)
axes[0, 1].set_xlabel('State')
axes[0, 1].set_ylabel('Credit Score')
axes[0, 1].tick_params(axis='x', rotation=45)

# Box plot for transaction amounts by type
transactions.boxplot(column='amount', by='transaction_type', ax=axes[1, 0])
axes[1, 0].set_title('Transaction Amount by Type', fontsize=14)
axes[1, 0].set_xlabel('Transaction Type')
axes[1, 0].set_ylabel('Amount ($)')

# Box plot for transaction amounts by status
transactions.boxplot(column='amount', by='status', ax=axes[1, 1])
axes[1, 1].set_title('Transaction Amount by Status', fontsize=14)
axes[1, 1].set_xlabel('Status')
axes[1, 1].set_ylabel('Amount ($)')

plt.tight_layout()
plt.savefig('box_plots.png', dpi=300)
plt.show()

4.3 Categorical Data Visualization

# ============= CATEGORICAL VISUALIZATION =============

fig, axes = plt.subplots(2, 2, figsize=(15, 10))

# Customer segment distribution
segment_counts = customers['segment'].value_counts()
segment_counts.plot(kind='bar', ax=axes[0, 0], color=['#2ecc71', '#f1c40f', '#e74c3c'])
axes[0, 0].set_title('Customer Segment Distribution', fontsize=14)
axes[0, 0].set_xlabel('Segment')
axes[0, 0].set_ylabel('Count')
axes[0, 0].set_ylim(0, max(segment_counts) * 1.1)
for i, v in enumerate(segment_counts.values):
    axes[0, 0].text(i, v + 10, str(v), ha='center', va='bottom')

# Transaction type distribution
txn_type_counts = transactions['transaction_type'].value_counts()
txn_type_counts.plot(kind='bar', ax=axes[0, 1], color=['#3498db', '#e67e22', '#2ecc71', '#9b59b6'])
axes[0, 1].set_title('Transaction Type Distribution', fontsize=14)
axes[0, 1].set_xlabel('Transaction Type')
axes[0, 1].set_ylabel('Count')

# Transaction status distribution
txn_status_counts = transactions['status'].value_counts()
txn_status_counts.plot(kind='bar', ax=axes[1, 0], color=['#2ecc71', '#f1c40f', '#e74c3c'])
axes[1, 0].set_title('Transaction Status Distribution', fontsize=14)
axes[1, 0].set_xlabel('Status')
axes[1, 0].set_ylabel('Count')

# Top merchants
top_merchants = transactions['merchant'].value_counts().head(10)
top_merchants.plot(kind='bar', ax=axes[1, 1], color='#3498db')
axes[1, 1].set_title('Top 10 Merchants by Transaction Volume', fontsize=14)
axes[1, 1].set_xlabel('Merchant')
axes[1, 1].set_ylabel('Count')
axes[1, 1].tick_params(axis='x', rotation=45)

plt.tight_layout()
plt.savefig('categorical_plots.png', dpi=300)
plt.show()

4.4 Time Series Visualization

# ============= TIME SERIES VISUALIZATION =============

# Aggregate transactions by date
daily_volume = transactions.groupby('date')['amount'].agg(['sum', 'count'])
daily_volume.columns = ['total_amount', 'transaction_count']

# Monthly aggregation
monthly_volume = transactions.groupby(['year', 'month'])['amount'].agg(['sum', 'count'])
monthly_volume.columns = ['total_amount', 'transaction_count']
monthly_volume['date'] = pd.to_datetime(monthly_volume.index.map(lambda x: f"{x[0]}-{x[1]:02d}-01"))

fig, axes = plt.subplots(2, 2, figsize=(15, 10))

# Daily transaction volume
daily_volume['total_amount'].plot(ax=axes[0, 0], color='#2c3e50', alpha=0.8)
axes[0, 0].set_title('Daily Transaction Volume (Last 365 Days)', fontsize=14)
axes[0, 0].set_xlabel('Date')
axes[0, 0].set_ylabel('Total Amount ($)')
axes[0, 0].axhline(daily_volume['total_amount'].mean(), color='red', linestyle='--', label=f"Mean: ${daily_volume['total_amount'].mean():,.0f}")
axes[0, 0].legend()

# Daily transaction count
daily_volume['transaction_count'].plot(ax=axes[0, 1], color='#e67e22', alpha=0.8)
axes[0, 1].set_title('Daily Transaction Count (Last 365 Days)', fontsize=14)
axes[0, 1].set_xlabel('Date')
axes[0, 1].set_ylabel('Transaction Count')
axes[0, 1].axhline(daily_volume['transaction_count'].mean(), color='red', linestyle='--', label=f"Mean: {daily_volume['transaction_count'].mean():.0f}")
axes[0, 1].legend()

# Monthly total amount
monthly_volume.sort_values('date').plot(x='date', y='total_amount', ax=axes[1, 0], kind='bar', color='#2c3e50')
axes[1, 0].set_title('Monthly Transaction Volume', fontsize=14)
axes[1, 0].set_xlabel('Month')
axes[1, 0].set_ylabel('Total Amount ($)')
axes[1, 0].tick_params(axis='x', rotation=45)

# Monthly transaction count
monthly_volume.sort_values('date').plot(x='date', y='transaction_count', ax=axes[1, 1], kind='bar', color='#e67e22')
axes[1, 1].set_title('Monthly Transaction Count', fontsize=14)
axes[1, 1].set_xlabel('Month')
axes[1, 1].set_ylabel('Transaction Count')
axes[1, 1].tick_params(axis='x', rotation=45)

plt.tight_layout()
plt.savefig('time_series_plots.png', dpi=300)
plt.show()

4.5 Correlation Analysis

# ============= CORRELATION ANALYSIS =============

# Select numerical columns for correlation
numerical_cols = ['age', 'annual_income', 'credit_score', 'tenure_months']
correlation_matrix = customers[numerical_cols].corr()

# Heatmap
fig, axes = plt.subplots(1, 2, figsize=(15, 6))

# Correlation heatmap
sns.heatmap(correlation_matrix, annot=True, cmap='RdBu_r', center=0, ax=axes[0], 
            fmt='.2f', square=True, cbar_kws={'shrink': 0.8})
axes[0].set_title('Correlation Matrix - Customer Features', fontsize=14)

# Scatter plot with regression
sns.regplot(x='age', y='credit_score', data=customers, ax=axes[1], 
            scatter_kws={'alpha': 0.5}, line_kws={'color': 'red'})
axes[1].set_title('Age vs Credit Score', fontsize=14)
axes[1].set_xlabel('Age')
axes[1].set_ylabel('Credit Score')

plt.tight_layout()
plt.savefig('correlation_plots.png', dpi=300)
plt.show()

# Print correlation insights
print("\n" + "="*60)
print("CORRELATION INSIGHTS")
print("="*60)
print("\nCorrelation Matrix:")
print(correlation_matrix.round(3))
print("\n🔍 Key Observations:")
print(f"  • Age vs Income: {correlation_matrix.loc['age', 'annual_income']:.3f} (weak relationship)")
print(f"  • Age vs Credit Score: {correlation_matrix.loc['age', 'credit_score']:.3f} (weak relationship)")
print(f"  • Income vs Credit Score: {correlation_matrix.loc['annual_income', 'credit_score']:.3f} (moderate relationship)")
print(f"  • Tenure vs Income: {correlation_matrix.loc['tenure_months', 'annual_income']:.3f} (weak relationship)")

SECTION 5: MISSING DATA ANALYSIS

5.1 Identifying Missing Data

# ============= MISSING DATA ANALYSIS =============

def analyze_missing_data(df, dataset_name):
    """Analyze and report missing data."""
    missing_data = df.isnull().sum()
    missing_percent = (missing_data / len(df)) * 100
    
    missing_df = pd.DataFrame({
        'Column': missing_data.index,
        'Missing Count': missing_data.values,
        'Missing %': missing_percent.values
    }).sort_values('Missing %', ascending=False)
    
    missing_df = missing_df[missing_df['Missing Count'] > 0]
    
    print(f"\n{'='*60}")
    print(f"MISSING DATA ANALYSIS: {dataset_name}")
    print(f"{'='*60}")
    print(f"Total records: {len(df)}")
    
    if missing_df.empty:
        print("✅ No missing data found!")
    else:
        print(f"\n📊 Columns with Missing Data:")
        print(missing_df.to_string(index=False))
        print(f"\nTotal missing values: {missing_df['Missing Count'].sum()}")
        print(f"Overall completeness: {(1 - missing_df['Missing Count'].sum() / (len(df) * len(df.columns))) * 100:.1f}%")
    
    return missing_df

# Analyze missing data
missing_customers = analyze_missing_data(customers, "Customers")
missing_transactions = analyze_missing_data(transactions, "Transactions")

# Visualize missing data
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Customers missing data
missing_customers_vis = customers.isnull().sum()
missing_customers_vis = missing_customers_vis[missing_customers_vis > 0]
if not missing_customers_vis.empty:
    missing_customers_vis.sort_values().plot(kind='barh', ax=axes[0], color='#e74c3c')
    axes[0].set_title('Missing Data - Customers', fontsize=14)
    axes[0].set_xlabel('Missing Count')
    axes[0].set_ylabel('Column')

# Transactions missing data
missing_transactions_vis = transactions.isnull().sum()
missing_transactions_vis = missing_transactions_vis[missing_transactions_vis > 0]
if not missing_transactions_vis.empty:
    missing_transactions_vis.sort_values().plot(kind='barh', ax=axes[1], color='#e74c3c')
    axes[1].set_title('Missing Data - Transactions', fontsize=14)
    axes[1].set_xlabel('Missing Count')
    axes[1].set_ylabel('Column')

plt.tight_layout()
plt.savefig('missing_data_plots.png', dpi=300)
plt.show()

5.2 Patterns in Missing Data

# ============= PATTERNS IN MISSING DATA =============

def analyze_missing_patterns(df):
    """Analyze patterns in missing data."""
    # Check if missing data is related to other columns
    missing_cols = df.columns[df.isnull().any()].tolist()
    
    patterns = {}
    for col in missing_cols:
        # Check if missingness is related to another column
        pattern = []
        for other_col in df.columns:
            if other_col != col:
                # Compare distributions when missing vs not missing
                missing_grp = df[df[col].isnull()][other_col]
                not_missing_grp = df[df[col].notnull()][other_col]
                
                if len(missing_grp) > 0 and len(not_missing_grp) > 0:
                    if pd.api.types.is_numeric_dtype(missing_grp):
                        # Compare means
                        diff = missing_grp.mean() - not_missing_grp.mean()
                        if abs(diff) > 0.1 * not_missing_grp.std():
                            pattern.append((other_col, 'numeric', diff))
                    else:
                        # Compare value counts
                        if len(missing_grp.unique()) > 0:
                            pattern.append((other_col, 'categorical', len(missing_grp.unique())))
        
        if pattern:
            patterns[col] = pattern
    
    return patterns

# Analyze patterns
patterns = analyze_missing_patterns(customers)
if patterns:
    print("\n" + "="*60)
    print("PATTERNS IN MISSING DATA")
    print("="*60)
    for col, pattern in patterns.items():
        print(f"\n📊 {col}:")
        for other_col, type_info, value in pattern:
            if type_info == 'numeric':
                print(f"  • Missingness related to {other_col} (difference: {value:.2f})")
            else:
                print(f"  • Missingness related to {other_col}")

SECTION 6: OUTLIER DETECTION

6.1 Statistical Methods for Outlier Detection

# ============= OUTLIER DETECTION =============

def detect_outliers_iqr(data, column, multiplier=1.5):
    """Detect outliers using IQR method."""
    q1 = data[column].quantile(0.25)
    q3 = data[column].quantile(0.75)
    iqr = q3 - q1
    
    lower_bound = q1 - multiplier * iqr
    upper_bound = q3 + multiplier * iqr
    
    outliers = data[(data[column] < lower_bound) | (data[column] > upper_bound)]
    
    return outliers, lower_bound, upper_bound

def detect_outliers_zscore(data, column, threshold=3):
    """Detect outliers using Z-score method."""
    mean = data[column].mean()
    std = data[column].std()
    
    z_scores = (data[column] - mean) / std
    outliers = data[abs(z_scores) > threshold]
    
    return outliers

# Analyze outliers for key columns
outlier_columns = ['age', 'annual_income', 'credit_score', 'tenure_months']

print("\n" + "="*60)
print("OUTLIER DETECTION - IQR METHOD")
print("="*60)

for col in outlier_columns:
    outliers, lb, ub = detect_outliers_iqr(customers, col)
    print(f"\n📊 {col}:")
    print(f"  Normal range: [{lb:.2f}, {ub:.2f}]")
    print(f"  Outliers found: {len(outliers)} ({len(outliers)/len(customers)*100:.1f}%)")
    if len(outliers) > 0:
        print(f"  Outlier values: {outliers[col].min():.2f} to {outliers[col].max():.2f}")

# Visualize outliers
fig, axes = plt.subplots(2, 2, figsize=(14, 10))

for idx, col in enumerate(outlier_columns):
    row = idx // 2
    col_idx = idx % 2
    
    # IQR method
    outliers, lb, ub = detect_outliers_iqr(customers, col)
    
    # Histogram with outlier boundaries
    customers[col].hist(bins=30, ax=axes[row, col_idx], edgecolor='black', alpha=0.7)
    axes[row, col_idx].axvline(lb, color='red', linestyle='--', label=f'Lower Bound: {lb:.2f}')
    axes[row, col_idx].axvline(ub, color='red', linestyle='--', label=f'Upper Bound: {ub:.2f}')
    axes[row, col_idx].set_title(f'{col} - Outlier Detection', fontsize=12)
    axes[row, col_idx].set_xlabel(col)
    axes[row, col_idx].set_ylabel('Count')
    axes[row, col_idx].legend()

plt.tight_layout()
plt.savefig('outlier_plots.png', dpi=300)
plt.show()

6.2 Handling Outliers

# ============= HANDLING OUTLIERS =============

def handle_outliers(data, column, method='winsorize', threshold=1.5):
    """Handle outliers using different methods."""
    
    if method == 'winsorize':
        # Winsorization: cap extreme values
        q1 = data[column].quantile(0.25)
        q3 = data[column].quantile(0.75)
        iqr = q3 - q1
        
        lower_bound = q1 - threshold * iqr
        upper_bound = q3 + threshold * iqr
        
        data_processed = data[column].copy()
        data_processed = data_processed.clip(lower=lower_bound, upper=upper_bound)
        
        return data_processed, f"Winsorized at {lower_bound:.2f} and {upper_bound:.2f}"
    
    elif method == 'remove':
        # Remove outliers
        q1 = data[column].quantile(0.25)
        q3 = data[column].quantile(0.75)
        iqr = q3 - q1
        
        lower_bound = q1 - threshold * iqr
        upper_bound = q3 + threshold * iqr
        
        mask = (data[column] >= lower_bound) & (data[column] <= upper_bound)
        data_processed = data.loc[mask, column]
        
        return data_processed, f"Removed {len(data) - len(data_processed)} records"
    
    elif method == 'log_transform':
        # Log transform (works well for right-skewed data)
        data_processed = np.log(data[column] + 1)
        
        return data_processed, "Applied log transformation"
    
    else:
        return data[column], "No transformation applied"

# Test outlier handling methods
test_col = 'annual_income'
print("\n" + "="*60)
print(f"OUTLIER HANDLING - {test_col}")
print("="*60)

# Original
print(f"\nOriginal Data:")
print(f"  Mean: ${customers[test_col].mean():,.2f}")
print(f"  Median: ${customers[test_col].median():,.2f}")
print(f"  Min: ${customers[test_col].min():,.2f}")
print(f"  Max: ${customers[test_col].max():,.2f}")
print(f"  Std Dev: ${customers[test_col].std():,.2f}")

# Winsorize
winsorized, info = handle_outliers(customers, test_col, method='winsorize')
print(f"\nWinsorized ({info}):")
print(f"  Mean: ${winsorized.mean():,.2f}")
print(f"  Median: ${winsorized.median():,.2f}")
print(f"  Min: ${winsorized.min():,.2f}")
print(f"  Max: ${winsorized.max():,.2f}")
print(f"  Std Dev: ${winsorized.std():,.2f}")

# Log transform
log_transformed, info = handle_outliers(customers, test_col, method='log_transform')
print(f"\nLog Transformed ({info}):")
print(f"  Mean: {log_transformed.mean():,.2f}")
print(f"  Median: {log_transformed.median():,.2f}")
print(f"  Std Dev: {log_transformed.std():,.2f}")

# Visualize methods
fig, axes = plt.subplots(1, 3, figsize=(15, 5))

# Original
customers[test_col].hist(bins=50, ax=axes[0], edgecolor='black', alpha=0.7)
axes[0].set_title('Original Distribution', fontsize=14)
axes[0].set_xlabel('Annual Income ($)')
axes[0].set_ylabel('Count')

# Winsorized
winsorized.hist(bins=50, ax=axes[1], edgecolor='black', alpha=0.7)
axes[1].set_title('Winsorized Distribution', fontsize=14)
axes[1].set_xlabel('Annual Income ($)')
axes[1].set_ylabel('Count')

# Log transformed
log_transformed.hist(bins=50, ax=axes[2], edgecolor='black', alpha=0.7)
axes[2].set_title('Log Transformed Distribution', fontsize=14)
axes[2].set_xlabel('Log(Annual Income)')
axes[2].set_ylabel('Count')

plt.tight_layout()
plt.savefig('outlier_handling.png', dpi=300)
plt.show()

SECTION 7: BUSINESS RISK & FINANCIAL IMPACT

7.1 Why EDA Matters for Risk Management

Risk	How EDA Helps	Example
Credit Risk	Identifies patterns in default behavior	Finding that customers with DTI > 40% have 5x default rate
Market Risk	Detects volatility patterns	Identifying volatility clustering in returns
Operational Risk	Finds unusual transaction patterns	Detecting fraud rings through transaction networks
Compliance Risk	Ensures data quality for reporting	Finding missing data in regulatory reports

7.2 Regulatory Requirements

Regulation	EDA Requirement
SR 11-7	Must understand data distribution before modeling
BASEL III	Must validate data quality and completeness
CCAR	Must identify outliers and anomalies in stress scenarios
GDPR	Must understand what data is collected and stored

7.3 EDA Best Practices for Banking

Start with Business Questions: Know what you’re looking for before you start.
Document Everything: Regulators will want to see your EDA process.
Visualize Before Modeling: Charts reveal patterns that numbers hide.
Check Data Quality Early: Bad data leads to bad models.
Understand Distributions: Financial data is rarely normal.
Look for Seasonality: Many banking metrics have monthly/quarterly patterns.
Identify Outliers Carefully: Some outliers are fraud, others are just unusual.
Correlation ≠ Causation: Just because two things move together doesn’t mean one causes the other.

SECTION 8: SUMMARY FOR THE DATA PRACTITIONER

8.1 The 1-Minute Elevator Pitch

“Exploratory Data Analysis is the critical first step in any banking analytics project. We inspect data structure, calculate summary statistics, visualize distributions, identify missing values and outliers, and understand relationships between variables. EDA ensures we understand our data before building models or making decisions. In banking, EDA helps us detect fraud, manage risk, and ensure regulatory compliance. A thorough EDA saves time, prevents costly mistakes, and builds confidence in our analytics.”

8.2 Key Takeaways

EDA is the first step in any analytics project—understand before you model.
Data Profiling reveals the structure, types, and quality of your data.
Summary Statistics (mean, median, std, skewness, kurtosis) describe distributions.
Visualizations reveal patterns that numbers alone can’t show.
Missing Data must be analyzed and handled appropriately.
Outliers can be detected using IQR or Z-score methods.
Correlation Analysis reveals relationships between variables.
Time Series Plots show trends, seasonality, and volatility.
Documentation is essential for regulatory compliance.
EDA saves time by identifying issues before modeling.

8.3 Recommended Next Steps

Practice EDA on real banking data (try Kaggle datasets)
Build an EDA template for your work
Learn advanced visualization libraries (Plotly, Bokeh)
Document your EDA process for regulatory purposes

[END OF LESSON 6]

Diploma in Financial Data Analytics LESSON 6: FOUNDATIONS OF EDA FOR FINANCIAL DATA

Diploma in Financial Data Analytics