┌─────────────────────────────────────────────────────────────────────────────┐
│                    AML TRANSACTION MONITORING                             │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌──────────────────────────────────────────────────────────────────────┐   │
│  │                    DATA COLLECTION                                  │   │
│  │  (Transaction data, customer data, external data)                  │   │
│  └──────────────────────────────────────────────────────────────────────┘   │
│                                    │                                        │
│                                    v                                        │
│  ┌──────────────────────────────────────────────────────────────────────┐   │
│  │                    TRANSFORMATION                                   │   │
│  │  (Feature engineering, data enrichment)                            │   │
│  └──────────────────────────────────────────────────────────────────────┘   │
│                                    │                                        │
│                                    v                                        │
│  ┌──────────────────────────────────────────────────────────────────────┐   │
│  │                    SCORING                                          │   │
│  │  (Risk scoring, anomaly detection)                                 │   │
│  └──────────────────────────────────────────────────────────────────────┘   │
│                                    │                                        │
│                                    v                                        │
│  ┌──────────────────────────────────────────────────────────────────────┐   │
│  │                    ALERT GENERATION                                 │   │
│  │  (Alerts for suspicious transactions)                              │   │
│  └──────────────────────────────────────────────────────────────────────┘   │
│                                    │                                        │
│                                    v                                        │
│  ┌──────────────────────────────────────────────────────────────────────┐   │
│  │                    INVESTIGATION                                   │   │
│  │  (Manual review, case management)                                 │   │
│  └──────────────────────────────────────────────────────────────────────┘   │
│                                    │                                        │
│                                    v                                        │
│  ┌──────────────────────────────────────────────────────────────────────┐   │
│  │                    REPORTING                                        │   │
│  │  (SAR filing, regulatory reporting)                                │   │
│  └──────────────────────────────────────────────────────────────────────┘   │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

# ===================================================================
# MODULE 4, LESSON 5: FRAUD DETECTION AND AML WITH AI
# ===================================================================

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import IsolationForest, RandomForestClassifier
from xgboost import XGBClassifier
from sklearn.metrics import (roc_auc_score, classification_report, confusion_matrix,
                             precision_recall_curve, f1_score)
from datetime import datetime, timedelta
import warnings
warnings.filterwarnings('ignore')

print("="*70)
print("FRAUD DETECTION AND AML WITH AI")
print("="*70)

# ----------------------------------------------------------------
# PART A: GENERATE TRANSACTION DATA
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART A: Generating Transaction Data")
print("-"*60)

np.random.seed(42)
n_transactions = 10000
n_customers = 500

# Generate customer profiles
customers = pd.DataFrame({
    'customer_id': range(1, n_customers + 1),
    'age': np.random.normal(45, 15, n_customers).clip(18, 80).astype(int),
    'income': np.random.gamma(5, 20, n_customers) + 20,
    'avg_transaction': np.random.lognormal(3, 0.5, n_customers).clip(10, 500),
    'risk_score': np.random.uniform(0, 100, n_customers)
})

# Generate transactions
customer_ids = np.random.choice(range(1, n_customers + 1), n_transactions)
transactions = pd.DataFrame({
    'transaction_id': range(1, n_transactions + 1),
    'customer_id': customer_ids,
    'amount': np.random.lognormal(3, 1, n_transactions).clip(1, 10000),
    'time_of_day': np.random.randint(0, 24, n_transactions),
    'day_of_week': np.random.randint(0, 7, n_transactions),
    'location': np.random.choice(['US', 'UK', 'EU', 'Asia', 'Other'], n_transactions, 
                                 p=[0.6, 0.15, 0.1, 0.1, 0.05]),
    'merchant_category': np.random.choice(['Retail', 'E-commerce', 'Food', 'Travel', 
                                           'Utilities', 'Entertainment'], n_transactions)
})

# Add customer features to transactions
transactions = transactions.merge(customers, on='customer_id')

# Generate fraud labels (2% fraud rate)
fraud_prob = 0.02
fraud_indices = np.random.choice(n_transactions, int(n_transactions * fraud_prob), replace=False)
transactions['is_fraud'] = 0
transactions.loc[fraud_indices, 'is_fraud'] = 1

# Make fraudulent transactions look different
for idx in fraud_indices:
    transactions.loc[idx, 'amount'] = transactions.loc[idx, 'amount'] * np.random.uniform(2, 5)
    transactions.loc[idx, 'location'] = np.random.choice(['Other', 'Asia'])
    transactions.loc[idx, 'time_of_day'] = np.random.randint(0, 6)

print(f"Generated {len(transactions)} transactions")
print(f"Fraud rate: {transactions['is_fraud'].mean():.2%}")

# ----------------------------------------------------------------
# PART B: EXPLORATORY ANALYSIS
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART B: Exploratory Analysis")
print("-"*60)

# Summary statistics by fraud status
fraud_summary = transactions.groupby('is_fraud').agg({
    'amount': ['mean', 'std', 'count'],
    'time_of_day': 'mean',
    'location': lambda x: x.value_counts().index[0]
}).round(2)

print("Fraud Summary:")
print(fraud_summary)

# Visualise
fig, axes = plt.subplots(2, 2, figsize=(14, 10))

# Amount distribution
ax = axes[0, 0]
transactions[transactions['is_fraud'] == 0]['amount'].hist(bins=50, alpha=0.5, label='Normal', color='blue')
transactions[transactions['is_fraud'] == 1]['amount'].hist(bins=50, alpha=0.5, label='Fraud', color='red')
ax.set_xlabel('Amount')
ax.set_ylabel('Frequency')
ax.set_title('Transaction Amount Distribution')
ax.legend()
ax.grid(True, alpha=0.3)

# Time of day
ax = axes[0, 1]
transactions.boxplot(column='time_of_day', by='is_fraud', ax=ax)
ax.set_title('Time of Day by Fraud Status')
ax.set_xlabel('Fraud Status')
ax.set_ylabel('Time of Day')
ax.grid(True, alpha=0.3)

# Location
ax = axes[1, 0]
location_fraud = transactions.groupby(['location', 'is_fraud']).size().unstack(fill_value=0)
location_fraud_pct = location_fraud.div(location_fraud.sum(axis=1), axis=0) * 100
location_fraud_pct.plot(kind='bar', stacked=True, ax=ax, color=['blue', 'red'])
ax.set_xlabel('Location')
ax.set_ylabel('Percentage')
ax.set_title('Fraud by Location')
ax.legend(['Normal', 'Fraud'])
ax.grid(True, alpha=0.3)

# Amount by merchant category
ax = axes[1, 1]
transactions.boxplot(column='amount', by=['merchant_category', 'is_fraud'], ax=ax)
ax.set_title('Amount by Merchant Category and Fraud')
ax.set_xlabel('Merchant Category / Fraud')
ax.set_ylabel('Amount')
ax.tick_params(axis='x', rotation=45)
ax.grid(True, alpha=0.3)

plt.tight_layout()
plt.savefig('fraud_eda.png', dpi=300, bbox_inches='tight')
plt.show()
print("Fraud EDA visualisation saved as 'fraud_eda.png'")

# ----------------------------------------------------------------
# PART C: FEATURE ENGINEERING
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART C: Feature Engineering")
print("-"*60)

# Customer-level features
customer_features = transactions.groupby('customer_id').agg({
    'amount': ['mean', 'std', 'max', 'count'],
    'transaction_id': 'count'
}).reset_index()
customer_features.columns = ['customer_id', 'avg_amount', 'std_amount', 'max_amount', 'tx_count']
customer_features['tx_count'] = customer_features['tx_count']  # Already count

# Merge back
transactions = transactions.merge(customer_features, on='customer_id')

# Time-based features
transactions['is_weekend'] = transactions['day_of_week'].isin([5, 6]).astype(int)
transactions['is_night'] = (transactions['time_of_day'] < 6).astype(int)

# Amount relative to average
transactions['amount_relative'] = transactions['amount'] / transactions['avg_amount']

# Features for modelling
features = ['amount', 'time_of_day', 'day_of_week', 'is_weekend', 'is_night',
            'avg_amount', 'std_amount', 'max_amount', 'tx_count', 'amount_relative']

# Encode categorical
transactions = pd.get_dummies(transactions, columns=['location', 'merchant_category'])

# Final feature list
feature_cols = features + ['location_Asia', 'location_EU', 'location_Other', 
                           'location_UK', 'location_US',
                           'merchant_category_E-commerce', 'merchant_category_Entertainment',
                           'merchant_category_Food', 'merchant_category_Retail',
                           'merchant_category_Travel', 'merchant_category_Utilities']

X = transactions[feature_cols]
y = transactions['is_fraud']

# ----------------------------------------------------------------
# PART D: MODEL TRAINING AND EVALUATION
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART D: Model Training and Evaluation")
print("-"*60)

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# 1. Isolation Forest (Unsupervised)
iso_forest = IsolationForest(contamination=0.02, random_state=42)
iso_forest.fit(X_train_scaled)
y_pred_iso = iso_forest.predict(X_test_scaled)
y_pred_iso_binary = (y_pred_iso == -1).astype(int)

# 2. Random Forest (Supervised)
rf_model = RandomForestClassifier(n_estimators=100, max_depth=10, class_weight='balanced', random_state=42)
rf_model.fit(X_train_scaled, y_train)
y_pred_rf_proba = rf_model.predict_proba(X_test_scaled)[:, 1]
y_pred_rf = (y_pred_rf_proba >= 0.5).astype(int)

# 3. XGBoost (Supervised)
xgb_model = XGBClassifier(n_estimators=100, max_depth=6, learning_rate=0.1,
                          scale_pos_weight=len(y_train[y_train==0])/len(y_train[y_train==1]),
                          random_state=42, use_label_encoder=False, eval_metric='logloss')
xgb_model.fit(X_train_scaled, y_train)
y_pred_xgb_proba = xgb_model.predict_proba(X_test_scaled)[:, 1]
y_pred_xgb = (y_pred_xgb_proba >= 0.5).astype(int)

# Evaluate
models = {
    'Isolation Forest': (y_pred_iso_binary, None, 'Unsupervised'),
    'Random Forest': (y_pred_rf, y_pred_rf_proba, 'Supervised'),
    'XGBoost': (y_pred_xgb, y_pred_xgb_proba, 'Supervised')
}

print("Model Performance:")
for name, (pred, proba, type) in models.items():
    auc = roc_auc_score(y_test, proba) if proba is not None else None
    f1 = f1_score(y_test, pred)
    print(f"\n{name} ({type}):")
    print(f"  F1-Score: {f1:.4f}")
    if auc:
        print(f"  AUC: {auc:.4f}")
    print(classification_report(y_test, pred, target_names=['Normal', 'Fraud']))

# ----------------------------------------------------------------
# PART E: CONFUSION MATRIX AND THRESHOLD TUNING
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART E: Confusion Matrix and Threshold Tuning")
print("-"*60)

# Best model (XGBoost)
y_pred_proba = xgb_model.predict_proba(X_test_scaled)[:, 1]

# Find optimal threshold based on precision-recall
precision, recall, thresholds = precision_recall_curve(y_test, y_pred_proba)
f1_scores = 2 * (precision * recall) / (precision + recall + 1e-10)
optimal_idx = np.argmax(f1_scores[:-1])
optimal_threshold = thresholds[optimal_idx] if len(thresholds) > 0 else 0.5

print(f"Optimal Threshold: {optimal_threshold:.4f}")

# Apply optimal threshold
y_pred_optimal = (y_pred_proba >= optimal_threshold).astype(int)

cm = confusion_matrix(y_test, y_pred_optimal)
print("\nConfusion Matrix (XGBoost with Optimal Threshold):")
print(pd.DataFrame(cm, columns=['Pred Normal', 'Pred Fraud'], 
                   index=['Actual Normal', 'Actual Fraud']))

# Feature importance
importance_xgb = pd.DataFrame({
    'Feature': feature_cols,
    'Importance': xgb_model.feature_importances_
}).sort_values('Importance', ascending=False)

print("\nTop 10 Fraud Predictors:")
print(importance_xgb.head(10).to_string(index=False))

# ----------------------------------------------------------------
# PART F: REAL-TIME FRAUD DETECTION SIMULATION
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART F: Real-Time Fraud Detection Simulation")
print("-"*60)

class FraudDetector:
    """Simulate a real-time fraud detection system."""
    
    def __init__(self, model, scaler, threshold=0.5):
        self.model = model
        self.scaler = scaler
        self.threshold = threshold
        self.alerts = []
        self.transaction_count = 0
    
    def check_transaction(self, transaction_data):
        """Check a transaction for fraud."""
        self.transaction_count += 1
        
        # Prepare features
        X_input = np.array([[
            transaction_data.get('amount', 0),
            transaction_data.get('time_of_day', 12),
            transaction_data.get('day_of_week', 3),
            transaction_data.get('is_weekend', 0),
            transaction_data.get('is_night', 0),
            transaction_data.get('avg_amount', 100),
            transaction_data.get('std_amount', 50),
            transaction_data.get('max_amount', 200),
            transaction_data.get('tx_count', 10),
            transaction_data.get('amount_relative', 1.0),
            transaction_data.get('location_Asia', 0),
            transaction_data.get('location_EU', 0),
            transaction_data.get('location_Other', 0),
            transaction_data.get('location_UK', 0),
            transaction_data.get('location_US', 1),
            transaction_data.get('merchant_category_E-commerce', 0),
            transaction_data.get('merchant_category_Entertainment', 0),
            transaction_data.get('merchant_category_Food', 0),
            transaction_data.get('merchant_category_Retail', 1),
            transaction_data.get('merchant_category_Travel', 0),
            transaction_data.get('merchant_category_Utilities', 0)
        ]])
        
        X_scaled = self.scaler.transform(X_input)
        prob = self.model.predict_proba(X_scaled)[0, 1]
        is_fraud = prob >= self.threshold
        
        if is_fraud:
            self.alerts.append({
                'transaction_id': self.transaction_count,
                'probability': prob,
                'transaction_data': transaction_data
            })
        
        return {
            'transaction_id': self.transaction_count,
            'is_fraud': is_fraud,
            'probability': prob
        }

# Create detector
detector = FraudDetector(xgb_model, scaler, optimal_threshold)

# Simulate transactions
sample_transactions = transactions.sample(50, random_state=42)

print("Simulating real-time fraud detection...")
for idx, tx in sample_transactions.iterrows():
    # Prepare transaction data
    tx_data = {col: tx[col] for col in feature_cols}
    result = detector.check_transaction(tx_data)
    if result['is_fraud']:
        print(f"⚠ FRAUD ALERT! Transaction #{result['transaction_id']}: Probability={result['probability']:.3f}")

print(f"\nDetected {len(detector.alerts)} fraud alerts out of {detector.transaction_count} transactions")
print(f"Alert rate: {len(detector.alerts)/detector.transaction_count*100:.2f}%")

# ----------------------------------------------------------------
# PART G: AML REGULATORY FRAMEWORK
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART G: AML Regulatory Framework")
print("-"*60)

print("""
Key AML Regulations:

1. FATF (Financial Action Task Force):
   - Global standards for AML/CFT.
   - 40 recommendations.

2. Bank Secrecy Act (US):
   - Record keeping and reporting.
   - SAR filing requirements.

3. EU AML Directives (5AMLD, 6AMLD):
   - Harmonised AML rules.
   - Beneficial ownership registers.

4. UK AML Regulations:
   - PEP screening.
   - Risk-based approach.

5. SAR (Suspicious Activity Report):
   - Filing requirements.
   - Confidential reporting.

Compliance Requirements:
  - Customer Due Diligence (CDD)
  - Enhanced Due Diligence (EDD)
  - Transaction Monitoring
  - Record Keeping
  - Training Programs
  - Independent Audits
""")

# ----------------------------------------------------------------
# PART H: SUMMARY AND RECOMMENDATIONS
# ----------------------------------------------------------------

print("\n" + "="*70)
print("PART H: Summary and Recommendations")
print("="*70)

print("""
Fraud Detection and AML – Key Takeaways:

1. Financial crime includes fraud, money laundering, and sanctions evasion.
2. AI/ML approaches: anomaly detection, supervised learning, graph-based, deep learning.
3. Key models: Isolation Forest, Random Forest, XGBoost, Neural Networks.
4. AML transaction monitoring: data collection → transformation → scoring → alerts → investigation → reporting.
5. Key metrics: precision, recall, F1-score, AUC, alert rate.
6. Regulatory framework: FATF, AML/KYC, SAR filing.
7. Best practices: real-time monitoring, threshold tuning, feature engineering, explainability.

Recommendations:
  - Implement real-time fraud detection.
  - Use ensemble models for better performance.
  - Tune thresholds based on business requirements.
  - Implement explainability for regulatory compliance.
  - Integrate with case management for investigation.
  - Regularly update models with new fraud patterns.
""")

print("="*70)
print("END OF LESSON 5 – MODULE 4")
print("="*70)