1. LEARNING OBJECTIVES
By the end of this lesson, you will be able to:
-
Understand REST API architecture and HTTP methods (GET, POST, PUT, DELETE).
-
Authenticate with financial APIs using API keys, OAuth, and JWT.
-
Make API requests using the requests library.
-
Parse JSON responses from financial data APIs.
-
Implement webhooks to receive real-time financial data.
-
Build a simple API client for a financial data provider.
-
Apply asynchronous programming with asyncio and aiohttp.
-
Handle rate limiting and retry logic.
-
Build a production-grade financial data ingestion pipeline.
2. REST API ARCHITECTURE
2.1 What is REST?
REST (Representational State Transfer) is an architectural style for designing networked applications.
Key Principles:
-
Stateless: Each request contains all necessary information.
-
Client-Server: Separation of concerns.
-
Cacheable: Responses can be cached.
-
Uniform Interface: Consistent API design.
2.2 HTTP Methods
| Method | Purpose | Financial Application |
|---|---|---|
| GET | Retrieve data | Get stock prices, account balance |
| POST | Create new resource | Place an order, create a subscription |
| PUT | Update a resource | Update account settings |
| DELETE | Delete a resource | Cancel an order, delete an API key |
| PATCH | Partial update | Update a specific field |
2.3 HTTP Status Codes
| Code Range | Meaning | Example |
|---|---|---|
| 2xx | Success | 200 OK, 201 Created |
| 3xx | Redirection | 301 Moved Permanently |
| 4xx | Client Error | 400 Bad Request, 401 Unauthorized, 404 Not Found |
| 5xx | Server Error | 500 Internal Server Error, 503 Service Unavailable |
3. MAKING API REQUESTS WITH REQUESTS
3.1 Installation and Import
pip install requests import requests import json
3.2 Basic GET Request
# Simple GET request url = "https://api.example.com/v1/stock/AAPL" response = requests.get(url) # Check status if response.status_code == 200: data = response.json() # Parse JSON print(data) else: print(f"Error: {response.status_code}") print(response.text)
3.3 GET Request with Parameters
# Query parameters params = { 'symbol': 'AAPL', 'start_date': '2024-01-01', 'end_date': '2024-12-31', 'interval': '1d' } response = requests.get( "https://api.example.com/v1/historical_prices", params=params ) # URL becomes: https://api.example.com/v1/historical_prices?symbol=AAPL&start_date=2024-01-01&...
3.4 POST Request (Creating Resources)
# POST request with JSON body order_data = { 'symbol': 'AAPL', 'side': 'BUY', 'quantity': 100, 'order_type': 'LIMIT', 'limit_price': 150.50 } headers = { 'Content-Type': 'application/json', 'Authorization': 'Bearer YOUR_API_KEY' } response = requests.post( "https://api.example.com/v1/orders", json=order_data, headers=headers ) if response.status_code == 201: order_id = response.json()['order_id'] print(f"Order placed: {order_id}")
3.5 PUT and DELETE Requests
# PUT (Update) update_data = {'limit_price': 151.00} response = requests.put( f"https://api.example.com/v1/orders/{order_id}", json=update_data, headers=headers ) # DELETE (Cancel) response = requests.delete( f"https://api.example.com/v1/orders/{order_id}", headers=headers )
4. API AUTHENTICATION
4.1 API Key Authentication
# API key in headers headers = { 'X-API-Key': 'YOUR_API_KEY', 'Content-Type': 'application/json' } response = requests.get(url, headers=headers) # API key in query string params = {'apikey': 'YOUR_API_KEY'} response = requests.get(url, params=params)
4.2 OAuth 2.0 Authentication
# Get access token auth_data = { 'grant_type': 'client_credentials', 'client_id': 'CLIENT_ID', 'client_secret': 'CLIENT_SECRET' } response = requests.post( "https://auth.example.com/oauth/token", data=auth_data ) access_token = response.json()['access_token'] # Use token in subsequent requests headers = { 'Authorization': f'Bearer {access_token}', 'Content-Type': 'application/json' } response = requests.get( "https://api.example.com/v1/data", headers=headers )
4.3 JWT (JSON Web Token)
import jwt import time # Generate JWT payload = { 'sub': 'user_id_123', 'iat': int(time.time()), 'exp': int(time.time()) + 3600 # 1 hour expiry } token = jwt.encode(payload, 'SECRET_KEY', algorithm='HS256') # Decode JWT decoded = jwt.decode(token, 'SECRET_KEY', algorithms=['HS256'])
5. BUILDING A FINANCIAL API CLIENT
5.1 Complete Client Class
import requests import time from typing import Dict, List, Optional class FinancialAPIClient: def __init__(self, api_key: str, base_url: str, timeout: int = 30): self.api_key = api_key self.base_url = base_url self.timeout = timeout self.session = requests.Session() self.session.headers.update({ 'X-API-Key': self.api_key, 'Content-Type': 'application/json', 'Accept': 'application/json' }) def _request(self, method: str, endpoint: str, params: Optional[Dict] = None, data: Optional[Dict] = None) -> Dict: """Generic request handler with error handling.""" url = f"{self.base_url}{endpoint}" try: response = self.session.request( method=method, url=url, params=params, json=data, timeout=self.timeout ) response.raise_for_status() return response.json() except requests.exceptions.Timeout: raise Exception(f"Request to {url} timed out") except requests.exceptions.HTTPError as e: raise Exception(f"HTTP Error: {e.response.status_code} - {e.response.text}") except requests.exceptions.RequestException as e: raise Exception(f"Request failed: {str(e)}") def get_stock_price(self, symbol: str) -> Dict: """Get current stock price.""" return self._request('GET', f'/stock/{symbol}/price') def get_historical_prices(self, symbol: str, start: str, end: str) -> Dict: """Get historical prices.""" params = { 'symbol': symbol, 'start_date': start, 'end_date': end } return self._request('GET', '/historical_prices', params=params) def place_order(self, symbol: str, side: str, quantity: int, order_type: str, limit_price: Optional[float] = None) -> Dict: """Place an order.""" data = { 'symbol': symbol, 'side': side, 'quantity': quantity, 'order_type': order_type } if limit_price: data['limit_price'] = limit_price return self._request('POST', '/orders', data=data) def cancel_order(self, order_id: str) -> Dict: """Cancel an existing order.""" return self._request('DELETE', f'/orders/{order_id}') def get_portfolio(self) -> Dict: """Get portfolio holdings.""" return self._request('GET', '/portfolio')
5.2 Usage Example
# Initialize client client = FinancialAPIClient( api_key="YOUR_API_KEY", base_url="https://api.example.com/v1" ) # Get stock price price = client.get_stock_price("AAPL") print(f"AAPL: ${price['price']:.2f}") # Get historical data historical = client.get_historical_prices( symbol="AAPL", start="2024-01-01", end="2024-12-31" ) # Place an order order = client.place_order( symbol="AAPL", side="BUY", quantity=100, order_type="LIMIT", limit_price=150.50 ) # Get portfolio portfolio = client.get_portfolio()
6. RATE LIMITING AND RETRY LOGIC
6.1 Rate Limiting Strategies
import time from functools import wraps def rate_limiter(calls_per_second=10): """Decorator to limit API call rate.""" min_interval = 1.0 / calls_per_second last_called = [0.0] def decorator(func): @wraps(func) def wrapper(*args, **kwargs): elapsed = time.time() - last_called[0] if elapsed < min_interval: time.sleep(min_interval - elapsed) last_called[0] = time.time() return func(*args, **kwargs) return wrapper return decorator # Usage @rate_limiter(calls_per_second=5) def make_api_request(): response = requests.get("https://api.example.com/v1/data") return response.json()
6.2 Exponential Backoff Retry
def retry_with_backoff(max_retries=5, backoff_factor=2): """Decorator for retrying requests with exponential backoff.""" def decorator(func): @wraps(func) def wrapper(*args, **kwargs): retries = 0 delay = 1 while retries < max_retries: try: return func(*args, **kwargs) except Exception as e: retries += 1 if retries >= max_retries: raise print(f"Retry {retries}/{max_retries} after {delay}s delay") time.sleep(delay) delay *= backoff_factor return None return wrapper return decorator # Usage @retry_with_backoff(max_retries=5, backoff_factor=2) def fetch_data(): response = requests.get("https://api.example.com/v1/data") response.raise_for_status() return response.json()
6.3 Handling Rate Limit Headers
def rate_limited_request(url, headers=None): """Handle rate limit headers from response.""" response = requests.get(url, headers=headers) # Check rate limit headers if 'X-RateLimit-Remaining' in response.headers: remaining = int(response.headers['X-RateLimit-Remaining']) reset = int(response.headers['X-RateLimit-Reset']) if remaining == 0: wait_time = reset - time.time() if wait_time > 0: time.sleep(wait_time) return response
7. WEBHOOKS
7.1 What are Webhooks?
Webhooks are HTTP callbacks that send real-time notifications when an event occurs. Instead of polling, the server pushes data to the client.
7.2 Building a Webhook Listener
from flask import Flask, request, jsonify import hashlib import hmac app = Flask(__name__) WEBHOOK_SECRET = "YOUR_WEBHOOK_SECRET" @app.route('/webhook', methods=['POST']) def handle_webhook(): """Handle incoming webhook events.""" try: # Verify signature signature = request.headers.get('X-Signature') payload = request.get_data() expected = hmac.new( WEBHOOK_SECRET.encode(), payload, hashlib.sha256 ).hexdigest() if signature != expected: return jsonify({'error': 'Invalid signature'}), 401 # Parse event event = request.get_json() event_type = event.get('event_type') # Process different event types if event_type == 'ORDER_FILLED': process_order_filled(event['data']) elif event_type == 'PRICE_UPDATE': process_price_update(event['data']) elif event_type == 'ACCOUNT_BALANCE': process_balance_update(event['data']) else: print(f"Unknown event type: {event_type}") return jsonify({'status': 'success'}), 200 except Exception as e: print(f"Webhook error: {str(e)}") return jsonify({'error': str(e)}), 500 def process_order_filled(data): print(f"Order filled: {data['order_id']} at ${data['price']}") # Update database, send notification, etc. def process_price_update(data): print(f"Price update: {data['symbol']} ${data['price']}") # Store in database, trigger alerts, etc. def process_balance_update(data): print(f"Balance updated: ${data['balance']}") # Update portfolio, check margin, etc. if __name__ == '__main__': app.run(host='0.0.0.0', port=5000)
7.3 Webhook Security
def verify_webhook(payload, signature, secret): """Verify HMAC signature.""" expected = hmac.new( secret.encode(), payload, hashlib.sha256 ).hexdigest() return hmac.compare_digest(signature, expected) def verify_timestamp(timestamp, max_age=300): """Prevent replay attacks.""" return abs(time.time() - int(timestamp)) <= max_age
8. ASYNCHRONOUS PROGRAMMING
8.1 Why Async?
-
I/O Bound Tasks: API calls, database queries, file operations.
-
Concurrency: Handle multiple requests simultaneously.
-
Efficiency: No need for threading (reduces overhead).
8.2 Basic Asynchronous Code
import asyncio import aiohttp import time async def fetch_price(session, symbol): """Asynchronously fetch stock price.""" url = f"https://api.example.com/v1/stock/{symbol}" async with session.get(url) as response: data = await response.json() return symbol, data['price'] async def fetch_prices(symbols): """Fetch multiple prices concurrently.""" async with aiohttp.ClientSession() as session: tasks = [fetch_price(session, symbol) for symbol in symbols] results = await asyncio.gather(*tasks) return dict(results) # Usage symbols = ['AAPL', 'GOOGL', 'MSFT', 'AMZN', 'META'] start = time.time() prices = asyncio.run(fetch_prices(symbols)) print(f"Time: {time.time() - start:.2f}s")
8.3 Asynchronous API Client
import aiohttp import asyncio from typing import Dict, List class AsyncFinancialAPIClient: def __init__(self, api_key: str, base_url: str): self.api_key = api_key self.base_url = base_url self.session = None async def __aenter__(self): self.session = aiohttp.ClientSession( headers={ 'X-API-Key': self.api_key, 'Content-Type': 'application/json' } ) return self async def __aexit__(self, exc_type, exc_val, exc_tb): await self.session.close() async def get_stock_price(self, symbol: str) -> float: url = f"{self.base_url}/stock/{symbol}" async with self.session.get(url) as response: data = await response.json() return data['price'] async def get_multiple_prices(self, symbols: List[str]) -> Dict[str, float]: tasks = [self.get_stock_price(sym) for sym in symbols] results = await asyncio.gather(*tasks) return dict(zip(symbols, results)) async def place_order(self, symbol: str, side: str, quantity: int, order_type: str, limit_price: float = None) -> Dict: data = { 'symbol': symbol, 'side': side, 'quantity': quantity, 'order_type': order_type } if limit_price: data['limit_price'] = limit_price url = f"{self.base_url}/orders" async with self.session.post(url, json=data) as response: return await response.json() # Usage async def main(): async with AsyncFinancialAPIClient( api_key="YOUR_API_KEY", base_url="https://api.example.com/v1" ) as client: prices = await client.get_multiple_prices(['AAPL', 'GOOGL', 'MSFT']) print(prices) asyncio.run(main())
8.4 Semaphore (Rate Limiting Async)
class AsyncRateLimiter: def __init__(self, calls_per_second=10): self.semaphore = asyncio.Semaphore(calls_per_second) self.min_interval = 1.0 / calls_per_second self._last_call = 0 async def __aenter__(self): await self.semaphore.acquire() # Enforce minimum interval now = time.time() elapsed = now - self._last_call if elapsed < self.min_interval: await asyncio.sleep(self.min_interval - elapsed) self._last_call = time.time() return self async def __aexit__(self, *args): self.semaphore.release() # Usage async def rate_limited_request(session, url): async with AsyncRateLimiter(calls_per_second=5): async with session.get(url) as response: return await response.json()
9. BUILDING A DATA INGESTION PIPELINE
9.1 Complete Data Pipeline
import asyncio import aiohttp import pandas as pd from typing import List, Dict import logging logging.basicConfig(level=logging.INFO) class DataIngestionPipeline: def __init__(self, api_key: str, base_url: str): self.api_key = api_key self.base_url = base_url self.logger = logging.getLogger(__name__) async def fetch_historical_data(self, symbol: str, start: str, end: str) -> pd.DataFrame: """Fetch historical data for a single symbol.""" url = f"{self.base_url}/historical/{symbol}" params = {'start': start, 'end': end} async with aiohttp.ClientSession() as session: async with session.get(url, params=params, headers={'X-API-Key': self.api_key}) as response: data = await response.json() df = pd.DataFrame(data['prices']) df['Date'] = pd.to_datetime(df['Date']) df.set_index('Date', inplace=True) df['Symbol'] = symbol self.logger.info(f"Fetched {len(df)} rows for {symbol}") return df async def fetch_multiple(self, symbols: List[str], start: str, end: str) -> pd.DataFrame: """Fetch historical data for multiple symbols concurrently.""" tasks = [self.fetch_historical_data(sym, start, end) for sym in symbols] results = await asyncio.gather(*tasks) return pd.concat(results, axis=0) def process_data(self, df: pd.DataFrame) -> pd.DataFrame: """Clean and process the data.""" # Add returns df['Return'] = df.groupby('Symbol')['Close'].pct_change() # Add rolling statistics df['MA_50'] = df.groupby('Symbol')['Close'].rolling(50).mean().reset_index(0, drop=True) df['MA_200'] = df.groupby('Symbol')['Close'].rolling(200).mean().reset_index(0, drop=True) # Add volatility df['Volatility'] = df.groupby('Symbol')['Return'].rolling(20).std().reset_index(0, drop=True) # Drop NaN values df = df.dropna() return df async def run_pipeline(self, symbols: List[str], start: str, end: str) -> pd.DataFrame: """Run the complete pipeline.""" self.logger.info(f"Starting data ingestion for {len(symbols)} symbols") # Fetch data raw_data = await self.fetch_multiple(symbols, start, end) self.logger.info(f"Fetched {len(raw_data)} rows") # Process data processed_data = self.process_data(raw_data) self.logger.info(f"Processed {len(processed_data)} rows") return processed_data # Usage async def main(): pipeline = DataIngestionPipeline( api_key="YOUR_API_KEY", base_url="https://api.example.com/v1" ) symbols = ['AAPL', 'GOOGL', 'MSFT', 'AMZN', 'META'] data = await pipeline.run_pipeline( symbols=symbols, start='2024-01-01', end='2024-12-31' ) print(data.head()) data.to_csv('historical_prices.csv') asyncio.run(main())
10. SUMMARY FOR THE FINANCE PRACTITIONER
-
REST APIs are the standard for financial data access. Use GET for data retrieval, POST for order placement.
-
Authentication is critical. Use API keys, OAuth, or JWT depending on the provider.
-
Rate Limiting must be respected. Implement exponential backoff and retry logic.
-
Webhooks provide real-time data delivery. Build secure endpoints with signature verification.
-
Asynchronous Programming improves performance for I/O-bound tasks. Use asyncio and aiohttp.
-
Data Pipelines should be modular, error-resistant, and well-documented.