1. Learning Objectives

By the end of this lesson, you will be able to:

  • Design a robust infrastructure architecture for AI systems in finance, addressing performance, reliability, and security.

  • Integrate AI models with existing financial systems (core banking, trading platforms, risk systems).

  • Design and implement data pipelines for AI: ingestion, transformation, feature engineering, and serving.

  • Manage infrastructure costs and optimize resource utilization.

  • Address disaster recovery, business continuity, and high availability requirements.

  • Implement security controls for AI systems: authentication, authorization, encryption, and audit logging.

2. Infrastructure Architecture

2.1 Reference Architecture
text
┌─────────────────────────────────────────────────────────────────────────────┐
│                         AI INFRASTRUCTURE ARCHITECTURE                      │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                        DATA SOURCES                                 │    │
│  │  (Core banking, Market data feeds, News APIs, Internal databases)    │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                    │                                        │
│                                    ▼                                        │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                      DATA PIPELINE                                  │    │
│  │  (Ingestion, Validation, Transformation, Feature Engineering)        │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                    │                                        │
│                                    ▼                                        │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                      FEATURE STORE                                  │    │
│  │  (Low-latency feature serving, Feature versioning)                  │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                    │                                        │
│                                    ▼                                        │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                      MODEL SERVER                                   │    │
│  │  (Model serving, Inference, Batching, Caching)                      │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                    │                                        │
│                                    ▼                                        │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                      APPLICATION                                    │    │
│  │  (Trading system, Risk system, Reporting, Compliance)                │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                    │                                        │
│                                    ▼                                        │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                      MONITORING                                     │    │
│  │  (Performance, Drift, System health, Alerts)                       │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
2.2 Key Components
 
 
Component Description Technologies
Data Pipeline Ingests, validates, and transforms data. Apache Kafka, Apache Flink, dbt, Airflow.
Feature Store Serves features for inference, versions features. Feast, Tecton, Redis.
Model Server Hosts models and serves predictions. TensorFlow Serving, TorchServe, MLflow, Seldon.
API Gateway Routes requests and enforces security. Kong, AWS API Gateway, Envoy.
Monitoring Monitors model performance and system health. Prometheus, Grafana, ELK Stack.
Storage Stores data and model artifacts. S3, MinIO, PostgreSQL, Cassandra.
2.3 Deployment Patterns
 
 
Pattern Description Use Case
Containers Model runs in a container (Docker). Standardized, portable deployments.
Serverless Model runs on-demand (AWS Lambda, Google Cloud Functions). Low-volume, infrequent inference.
Kubernetes Container orchestration. Scalable, complex deployments.
Bare metal Model runs on dedicated hardware. High-performance, low-latency.
2.4 Scaling and Performance

Horizontal scaling: Add more instances of the model server to handle more requests.

Vertical scaling: Increase the resources (CPU, memory, GPU) of the model server.

Load balancing: Distribute requests across multiple instances.

Auto-scaling: Automatically scale based on load.

Mathematical model for scaling: The required number of instances N is:

N = ceil(λ * L / T)

where:

  • λ is the request rate.

  • L is the average inference latency.

  • T is the target response time.

3. Integration with Existing Systems

3.1 Integration Patterns
 
 
Pattern Description Example
API integration Model is exposed as a REST/gRPC API. Trading system calls model API.
Message-based Model consumes messages from a queue. Risk system sends requests to a queue.
Database integration Model reads/writes directly to the database. Model stores predictions in the database.
Batch integration Model is called as a batch job. End-of-day risk calculations.
3.2 Legacy System Challenges

Financial institutions often have legacy systems that are difficult to integrate with.

Challenges:

  • Proprietary protocols: Legacy systems may use non-standard protocols.

  • Data formats: Legacy systems may use outdated data formats (e.g., COBOL copybooks).

  • Security: Legacy systems may have weak security.

  • Latency: Legacy systems may be slow.

Mitigations:

  • Adapter pattern: Build a translation layer between the legacy system and the AI system.

  • Message-based integration: Use message queues to decouple systems.

  • Data replication: Replicate data from legacy systems to a modern data store.

3.3 Real-Time vs. Batch Integration
 
 
Aspect Real-Time Batch
Latency Milliseconds Minutes to hours
Data freshness Current Stale
Use cases Trading, fraud detection Risk reporting, stress testing
Integration API, streaming Scheduled jobs

4. Feature Engineering Pipelines

4.1 Online vs. Offline Features
 
 
Aspect Online Features Offline Features
Purpose Real-time inference Model training, batch inference
Latency Low (<100ms) High (seconds to minutes)
Compute High (must be fast) Lower (can be slower)
Implementation In-memory, Redis, Feature Store Data warehouse, Data lake
4.2 Feature Engineering Pipeline
text
┌─────────────────────────────────────────────────────────────────────────────┐
│                      FEATURE ENGINEERING PIPELINE                           │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐    ┌──────────┐ │
│  │   INGESTION  │───▶│  CLEANING    │───▶│  FEATURE    │───▶│  FEATURE │ │
│  │              │    │  & VALIDATION│    │  ENGINEERING│    │  STORE   │ │
│  └──────────────┘    └──────────────┘    └──────────────┘    └──────────┘ │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

1. Ingestion: Collect data from multiple sources (databases, APIs, message queues).

2. Cleaning and validation:

  • Handle missing values.

  • Remove duplicates.

  • Validate data types and ranges.

3. Feature engineering:

  • Compute derived features (e.g., moving averages, ratios).

  • Encode categorical features.

  • Normalize/standardize numerical features.

4. Feature store:

  • Store features for both training and inference.

  • Version features for reproducibility.

4.3 Feature Store Implementation

Feast (Feature Store for ML):

python
from feast import FeatureStore

store = FeatureStore(repo_path=".")

# Retrieve features for inference
features = store.get_online_features(
    features=["transaction:amount", "transaction:time"],
    entity_rows=[{"entity_id": "123"}]
).to_dict()

# Register a new feature view
from feast import FeatureView, Entity

entity = Entity(name="transaction_id", description="Transaction ID")
feature_view = FeatureView(
    name="transaction_features",
    entities=[entity],
    ttl=timedelta(days=7),
    schema=Field(
        name="amount",
        dtype=ValueType.FLOAT,
    )
)

5. Data Pipelines for AI

5.1 ETL vs. ELT vs. ETEL
 
 
Pattern Description Use Case
ETL Extract, Transform, Load. Batch processing, data warehousing.
ELT Extract, Load, Transform. Cloud data warehouses (Snowflake, BigQuery).
ETEL Extract, Transform, Embed, Load. Embedding generation for RAG.
5.2 Data Pipeline Tools
 
 
Category Tools
Orchestration Apache Airflow, Prefect, Dagster, AWS Step Functions
Streaming Apache Kafka, Apache Pulsar, AWS Kinesis
Batch Processing Apache Spark, Apache Flink, AWS Glue
Data Storage Snowflake, BigQuery, Redshift, S3, Delta Lake
5.3 Data Quality Testing

Great Expectations is a popular tool for data quality testing.

python
import great_expectations as ge

# Load data
df = ge.dataset.PandasDataset(data)

# Define expectations
expectations = [
    ("expect_column_to_exist", "amount"),
    ("expect_column_values_to_not_be_null", "amount"),
    ("expect_column_values_to_be_between", "amount", 0, 1000000),
    ("expect_column_values_to_be_in_set", "currency", ["USD", "EUR", "GBP"])
]

# Run validation
for expectation in expectations:
    method = getattr(df, expectation[0])
    method(*expectation[1:])

6. Cost Management

6.1 Cost Drivers
 
 
Cost Driver Description
Compute Model training and inference costs.
Storage Data and model storage costs.
Network Data transfer costs.
Development Engineering and data science costs.
6.2 Cost Optimization Strategies
 
 
Strategy Description
Right-sizing Choose the correct instance size for workloads.
Spot instances Use spot/preemptible instances for batch jobs.
Auto-scaling Scale down during low-demand periods.
Caching Cache frequent inference results.
Model compression Use smaller models (distillation, pruning, quantization).
Batching Batch inference requests to improve utilization.
6.3 Cost Monitoring

  • Tagging: Tag resources with cost center and project.

  • Budget alerts: Set budget alerts to notify of overspend.

  • Cost dashboards: Visualize costs by team, project, or resource.

7. Disaster Recovery and Business Continuity

7.1 High Availability (HA) Requirements

Financial systems require high availability:

 
 
Availability Downtime/year Downtime/month Downtime/week
99.9% 8.76 hours 43.2 minutes 10.1 minutes
99.99% 52.6 minutes 4.3 minutes 1.0 minute
99.999% 5.3 minutes 25.9 seconds 6.0 seconds
7.2 DR Strategies
 
 
Strategy Description RTO RPO
Backup and restore Backup data, restore on failure. Hours Hours
Active-passive Standby environment ready to take over. Minutes Minutes
Active-active Multiple active environments. Seconds Seconds
7.3 BCDR Planning for AI Systems

  1. Identify critical components: Model server, data pipeline, feature store.

  2. Define RTO (Recovery Time Objective): How long can the system be down?

  3. Define RPO (Recovery Point Objective): How much data can be lost?

  4. Design recovery procedures:

    • Data recovery.

    • Model recovery (reload from model registry).

    • Application recovery.

  5. Test recovery procedures regularly.

8. Security

8.1 Security Controls
 
 
Control Description Technology
Authentication Verify identity. OAuth2, OIDC, LDAP.
Authorization Control access. RBAC, ABAC.
Encryption (at rest) Encrypt stored data. AES-256.
Encryption (in transit) Encrypt data in transit. TLS 1.2+.
Audit logging Log all actions. CloudTrail, Azure Monitor.
Network security Control network access. VPC, Firewalls, VPN.
8.2 Securing the ML Pipeline
 
 
Stage Security Concern Mitigation
Data ingestion Data poisoning. Data validation, provenance tracking.
Model training Model theft. Access controls, encryption.
Model serving Inference attacks. Rate limiting, input validation.
Model storage Model tampering. Digital signatures, access controls.
8.3 Adversarial AI Threats
 
 
Threat Description Mitigation
Data poisoning Adversary injects malicious data. Data validation, robust training.
Model extraction Adversary steals the model. Access controls, differential privacy.
Adversarial examples Small input changes cause misclassification. Adversarial training, input sanitization.
Model inversion Adversary reconstructs training data. Differential privacy.

9. Summary for the AI Practitioner

  • AI infrastructure requires a robust architecture: data pipeline, feature store, model server, and monitoring.

  • Integration with legacy systems uses adapters and message-based patterns.

  • Feature engineering pipelines must support both online and offline features.

  • Cost management requires right-sizing, auto-scaling, and caching.

  • Disaster recovery requires high availability, defined RTO/RPO, and regular testing.

  • Security requires authentication, authorization, encryption, and adversarial AI defenses.

 
PreviousNext