┌─────────────────────────────────────────────────────────────────────────────┐
│                           MLOPS LIFECYCLE                                  │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌─────────┐    ┌─────────┐    ┌─────────┐    ┌─────────┐    ┌─────────┐  │
│  │  Data   │    │  Model  │    │  Model  │    │  Model  │    │  Model  │  │
│  │  Ingestion│ →│  Training│ →│  Validation│ →│  Deployment│ →│  Monitoring│  │
│  └─────────┘    └─────────┘    └─────────┘    └─────────┘    └─────────┘  │
│       │              │              │              │              │         │
│       v              v              v              v              v         │
│  ┌─────────┐    ┌─────────┐    ┌─────────┐    ┌─────────┐    ┌─────────┐  │
│  │  Data   │    │  Model  │    │  Model  │    │  Model  │    │  Model  │  │
│  │  Versioning│ │  Registry│ │  Testing│ │  Serving│ │  Drift  │  │
│  │         │    │         │    │         │    │         │    │  Detection│  │
│  └─────────┘    └─────────┘    └─────────┘    └─────────┘    └─────────┘  │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────────┐│
│  │                     CI/CD PIPELINE                                      ││
│  │  (Automated testing, validation, deployment, rollback)                  ││
│  └─────────────────────────────────────────────────────────────────────────┘│
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

import mlflow
import mlflow.sklearn

# Start an experiment
mlflow.set_experiment("credit_score_model")

# Log parameters, metrics, and models
with mlflow.start_run():
    mlflow.log_param("max_depth", 8)
    mlflow.log_param("n_estimators", 100)
    mlflow.log_metric("auc", 0.85)
    mlflow.log_metric("ks", 0.42)
    mlflow.sklearn.log_model(model, "model")
    mlflow.log_artifact("feature_importance.png")

# ===================================================================
# MODULE 8, LESSON 3: MODEL DEPLOYMENT AND MLOPS
# ===================================================================

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score, roc_curve
import joblib
import pickle
import json
import requests
from datetime import datetime, timedelta
import warnings
warnings.filterwarnings('ignore')

# Set style
sns.set_style("whitegrid")
np.random.seed(42)

print("="*70)
print("MLOps IMPLEMENTATION IN FINANCIAL SERVICES")
print("="*70)

# ----------------------------------------------------------------
# PART A: SIMULATED MODEL TRAINING PIPELINE
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART A: Model Training Pipeline")
print("-"*60)

# Generate credit data
def generate_credit_data(n=5000):
    np.random.seed(42)
    df = pd.DataFrame({
        'income': np.random.gamma(5, 15, n) + 20,
        'dti': np.random.beta(2, 5, n) * 60,
        'credit_score': np.random.normal(700, 50, n).clip(550, 850).astype(int),
        'loan_amount': np.random.gamma(4, 50, n) + 30,
        'employment_years': np.random.gamma(3, 5, n).clip(0, 30).astype(int),
        'age': np.random.normal(45, 12, n).clip(22, 75).astype(int),
    })
    log_odds = -4.5 + 0.04*df['dti'] - 0.005*df['credit_score'] + 0.01*(df['loan_amount']/1000)
    prob = 1/(1+np.exp(-log_odds))
    df['default'] = np.random.binomial(1, prob)
    return df

df = generate_credit_data(5000)
features = ['income', 'dti', 'credit_score', 'loan_amount', 'employment_years', 'age']
X = df[features]
y = df['default']

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train model
model = RandomForestClassifier(n_estimators=100, max_depth=8, random_state=42)
model.fit(X_train, y_train)

# Evaluate
y_pred_proba = model.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, y_pred_proba)
print(f"Model AUC: {auc:.4f}")

# Log parameters (simulated)
training_metadata = {
    'model_type': 'RandomForestClassifier',
    'n_estimators': 100,
    'max_depth': 8,
    'random_state': 42,
    'features': features,
    'train_size': len(X_train),
    'test_size': len(X_test),
    'auc': auc,
    'timestamp': datetime.now().isoformat()
}

# Save model and metadata
joblib.dump(model, 'credit_model.pkl')
with open('training_metadata.json', 'w') as f:
    json.dump(training_metadata, f, indent=2)

print("Model and metadata saved.")
print("Training metadata:")
print(json.dumps(training_metadata, indent=2))

# ----------------------------------------------------------------
# PART B: MODEL VALIDATION AND TESTING
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART B: Model Validation and Testing")
print("-"*60)

def validate_model(model, X_test, y_test):
    """Run a suite of validation tests."""
    y_pred_proba = model.predict_proba(X_test)[:, 1]
    y_pred = (y_pred_proba >= 0.5).astype(int)
    
    # Performance metrics
    auc = roc_auc_score(y_test, y_pred_proba)
    accuracy = (y_pred == y_test).mean()
    
    # KS Statistic
    from scipy.stats import ks_2samp
    scores_good = y_pred_proba[y_test == 0]
    scores_bad = y_pred_proba[y_test == 1]
    ks_stat, _ = ks_2samp(scores_good, scores_bad)
    
    # Calibration (Hosmer-Lemeshow)
    def hl_test(y_true, y_pred, n_groups=10):
        df_test = pd.DataFrame({'y_true': y_true, 'y_pred': y_pred})
        df_test['decile'] = pd.qcut(df_test['y_pred'], q=n_groups, labels=False, duplicates='drop')
        observed = df_test.groupby('decile')['y_true'].sum().values
        expected = df_test.groupby('decile')['y_pred'].sum().values
        n_obs = df_test.groupby('decile').size().values
        hl_stat = np.sum((observed - expected)**2 / (expected * (1 - expected/n_obs)))
        return hl_stat
    
    hl_stat = hl_test(y_test, y_pred_proba)
    
    results = {
        'auc': auc,
        'accuracy': accuracy,
        'ks_stat': ks_stat,
        'hl_stat': hl_stat,
        'pass': auc > 0.75 and ks_stat > 0.30
    }
    return results

validation_results = validate_model(model, X_test, y_test)
print("Validation Results:")
for key, value in validation_results.items():
    print(f"  {key}: {value:.4f}" if isinstance(value, float) else f"  {key}: {value}")

# ----------------------------------------------------------------
# PART C: DEPLOYMENT SIMULATION (API)
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART C: Deployment Simulation (API)")
print("-"*60)

# Simple API simulation
class ModelAPI:
    """Simulated model serving API."""
    
    def __init__(self, model_path, threshold=0.5):
        self.model = joblib.load(model_path)
        self.threshold = threshold
        self.features = ['income', 'dti', 'credit_score', 'loan_amount', 'employment_years', 'age']
        self.request_count = 0
        self.prediction_log = []
    
    def predict(self, data):
        """Make a prediction."""
        self.request_count += 1
        # Validate input
        if not all(f in data for f in self.features):
            raise ValueError(f"Missing features. Required: {self.features}")
        
        # Create feature vector
        X = np.array([[data[f] for f in self.features]])
        
        # Predict
        prob = self.model.predict_proba(X)[0, 1]
        pred = 1 if prob >= self.threshold else 0
        
        # Log
        self.prediction_log.append({
            'timestamp': datetime.now().isoformat(),
            'input': data,
            'probability': prob,
            'prediction': pred
        })
        
        return {'probability': prob, 'prediction': pred, 'request_id': self.request_count}

# Create API instance
api = ModelAPI('credit_model.pkl', threshold=0.5)

# Test a sample prediction
test_application = {
    'income': 65.4,
    'dti': 28.5,
    'credit_score': 710,
    'loan_amount': 145,
    'employment_years': 8,
    'age': 42
}

result = api.predict(test_application)
print("Sample Prediction:")
print(f"  Probability: {result['probability']:.3f}")
print(f"  Prediction: {'Default' if result['prediction'] else 'No Default'}")
print(f"  Request ID: {result['request_id']}")

# ----------------------------------------------------------------
# PART D: MODEL MONITORING DASHBOARD
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART D: Model Monitoring Simulation")
print("-"*60)

def generate_monitoring_data(n_days=30):
    """Simulate monitoring data over time."""
    dates = [datetime(2024, 1, 1) + timedelta(days=i) for i in range(n_days)]
    data = []
    for i, date in enumerate(dates):
        # Simulate performance drift
        drift = np.sin(i / 5) * 0.05
        auc = 0.85 + drift + np.random.normal(0, 0.01)
        ks = 0.40 + drift * 0.5 + np.random.normal(0, 0.01)
        psi = np.random.gamma(0.1, 0.05) + max(0, drift * 2)
        requests = np.random.poisson(500 + i * 5)
        latency = np.random.normal(50 + drift * 20, 10).clip(20, 150)
        
        data.append({
            'date': date,
            'auc': auc.clip(0, 1),
            'ks': ks.clip(0, 1),
            'psi': psi,
            'requests': requests,
            'latency': latency
        })
    return pd.DataFrame(data)

monitoring_df = generate_monitoring_data(30)
print("Monitoring data generated:")
print(monitoring_df.head().round(3))

# Visualise monitoring dashboard
fig, axes = plt.subplots(2, 3, figsize=(15, 10))

# AUC over time
ax = axes[0, 0]
ax.plot(monitoring_df['date'], monitoring_df['auc'], 'b-', linewidth=2)
ax.axhline(y=0.80, color='green', linestyle='--', label='Acceptable (0.80)')
ax.axhline(y=0.75, color='orange', linestyle='--', label='Warning (0.75)')
ax.axhline(y=0.70, color='red', linestyle='--', label='Critical (0.70)')
ax.set_xlabel('Date')
ax.set_ylabel('AUC')
ax.set_title('Model Performance (AUC)')
ax.legend(fontsize=8)
ax.grid(True, alpha=0.3)

# KS over time
ax = axes[0, 1]
ax.plot(monitoring_df['date'], monitoring_df['ks'], 'g-', linewidth=2)
ax.axhline(y=0.35, color='green', linestyle='--', label='Acceptable (0.35)')
ax.axhline(y=0.30, color='orange', linestyle='--', label='Warning (0.30)')
ax.axhline(y=0.25, color='red', linestyle='--', label='Critical (0.25)')
ax.set_xlabel('Date')
ax.set_ylabel('KS')
ax.set_title('Model Performance (KS)')
ax.legend(fontsize=8)
ax.grid(True, alpha=0.3)

# PSI (Data Drift)
ax = axes[0, 2]
ax.plot(monitoring_df['date'], monitoring_df['psi'], 'r-', linewidth=2)
ax.axhline(y=0.10, color='green', linestyle='--', label='Stable (<0.10)')
ax.axhline(y=0.25, color='orange', linestyle='--', label='Warning (0.10-0.25)')
ax.axhline(y=0.25, color='red', linestyle='--', label='Critical (>0.25)')
ax.set_xlabel('Date')
ax.set_ylabel('PSI')
ax.set_title('Data Drift (PSI)')
ax.legend(fontsize=8)
ax.grid(True, alpha=0.3)

# Request volume
ax = axes[1, 0]
ax.bar(monitoring_df['date'], monitoring_df['requests'], color='blue', alpha=0.7)
ax.set_xlabel('Date')
ax.set_ylabel('Requests')
ax.set_title('API Request Volume')
ax.grid(True, alpha=0.3)

# Latency
ax = axes[1, 1]
ax.plot(monitoring_df['date'], monitoring_df['latency'], 'purple', linewidth=2)
ax.axhline(y=100, color='red', linestyle='--', label='SLA (100ms)')
ax.set_xlabel('Date')
ax.set_ylabel('Latency (ms)')
ax.set_title('API Latency')
ax.legend(fontsize=8)
ax.grid(True, alpha=0.3)

# Summary alert (bottom right)
ax = axes[1, 2]
ax.axis('off')
last = monitoring_df.iloc[-1]
alerts = []
if last['auc'] < 0.75:
    alerts.append(f"⚠ AUC Alert: {last['auc']:.3f} < 0.75")
if last['ks'] < 0.30:
    alerts.append(f"⚠ KS Alert: {last['ks']:.3f} < 0.30")
if last['psi'] > 0.25:
    alerts.append(f"⚠ PSI Alert: {last['psi']:.3f} > 0.25")
if last['latency'] > 100:
    alerts.append(f"⚠ Latency Alert: {last['latency']:.1f}ms > 100ms")

alert_text = "ALERTS:\n" + "\n".join(alerts) if alerts else "✅ All systems nominal"
ax.text(0.5, 0.5, alert_text, ha='center', va='center', fontsize=12,
        bbox=dict(boxstyle='round', facecolor='lightyellow', edgecolor='red' if alerts else 'green'),
        transform=ax.transAxes)

plt.tight_layout()
plt.savefig('model_monitoring_dashboard.png', dpi=300, bbox_inches='tight')
plt.show()
print("Monitoring dashboard saved as 'model_monitoring_dashboard.png'")

# ----------------------------------------------------------------
# PART E: MLOPs PIPELINE (CONCEPTUAL)
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART E: MLOps Pipeline Overview")
print("-"*60)

print("""
MLOps Pipeline Implementation:

1. Data Ingestion:
   - Scheduled ETL from core banking system.
   - Real-time streaming via Apache Kafka.

2. Data Validation:
   - Schema validation.
   - Data quality checks (nulls, outliers, distributions).

3. Feature Engineering:
   - Feature extraction from raw data.
   - Transformation (scaling, encoding).
   - Feature validation.

4. Model Training:
   - Automated training pipeline (Airflow/Dagster).
   - Hyperparameter tuning (Optuna, Hyperopt).
   - Model selection (cross-validation).

5. Model Validation:
   - Performance tests (AUC, KS, calibration).
   - Fairness tests (disparate impact).
   - Regulatory compliance tests.

6. Model Registry:
   - Versioned model storage (MLflow).
   - Stage transitions (Staging → Production → Archived).
   - Approval workflow.

7. Model Deployment:
   - Containerisation (Docker).
   - CI/CD pipeline (Jenkins/GitHub Actions).
   - Canary/Shadow deployment.

8. Model Monitoring:
   - Performance monitoring (AUC, KS).
   - Drift detection (PSI, CSI).
   - Alerting and escalation.
   - Automated retraining triggers.

9. Governance:
   - Audit trails.
   - Model inventory.
   - Documentation.
""")

# ----------------------------------------------------------------
# PART F: AUTOMATED RETRAINING STRATEGY
# ----------------------------------------------------------------

print("\n" + "-"*60)
print("PART F: Automated Retraining Strategy")
print("-"*60)

print("""
Automated Retraining Triggers:

1. Time-Based Retraining:
   - Daily (high-frequency models).
   - Weekly (medium-frequency).
   - Monthly (low-frequency).
   - Quarterly (regulatory models).

2. Performance-Based Retraining:
   - AUC drops below threshold (e.g., <0.75).
   - KS drops below threshold (e.g., <0.30).
   - PSI exceeds threshold (e.g., >0.25).
   - Calibration fails (HL p < 0.05).

3. Data-Based Retraining:
   - New data accumulated (e.g., >10,000 new samples).
   - Significant data distribution change detected.

4. Event-Based Retraining:
   - Major market events (e.g., recession).
   - Regulatory changes.
   - New feature availability.

Retraining Pipeline:
  ┌─────────────────────────────────────────────────────────────┐
  │                    RETRAINING PIPELINE                     │
  ├─────────────────────────────────────────────────────────────┤
  │  Detect Trigger → Extract New Data → Validate Data →       │
  │  Train Model → Validate Model → Compare with Champion →    │
  │  Deploy if Better → Update Monitoring → Audit Log         │
  └─────────────────────────────────────────────────────────────┘
""")

# ----------------------------------------------------------------
# PART G: SUMMARY AND RECOMMENDATIONS
# ----------------------------------------------------------------

print("\n" + "="*70)
print("PART G: Summary and Recommendations")
print("="*70)

print("""
MLOps in Financial Services – Key Takeaways:

1. MLOps extends DevOps to ML, enabling reliable and scalable ML deployment.
2. Model Versioning (MLflow) ensures reproducibility and auditability.
3. Deployment Strategies: Batch, Real-time, Streaming, Edge.
4. Model Monitoring: Performance, Data Drift, Concept Drift.
5. Regulatory Requirements: SR 11-7, BCBS 239, GDPR, Fair Lending.
6. Automated Retraining ensures models stay current.
7. CI/CD pipelines automate testing, validation, and deployment.
8. Governance is critical: audit trails, model inventory, documentation.

Recommendations:
  - Start with a small, non-critical model to build MLOps capabilities.
  - Implement MLflow for experiment tracking and model registry.
  - Use containerisation (Docker) for consistent deployment.
  - Set up monitoring dashboards for performance and drift.
  - Automate retraining based on performance triggers.
  - Document all processes for regulatory compliance.
  - Build a cross-functional MLOps team (DS, DE, Ops, Security).
""")

print("="*70)
print("END OF LESSON 3 – MODULE 8")
print("="*70)