1. Learning Objectives

By the end of this lesson, you will be able to:

  • Implement a robust data ingestion pipeline that correctly handles multiple exchanges, time zones, and corporate actions.

  • Diagnose and treat missing data using probabilistic imputation, interpolation, and forward-fill with decay factors.

  • Detect and treat outliers using statistical (Z-score, IQR) and robust (MCD, Winsorisation) methods tailored for financial jumps.

  • Engineer a comprehensive feature set including price-derived, volume-derived, and microstructure features with exact mathematical formulations.

  • Construct point-in-time feature matrices that completely eliminate look-ahead and survivorship biases.

  • Apply dimensionality reduction (PCA, Sparse Autoencoders) to high-dimensional financial datasets with mathematical rigor.

2. The Data Ingestion Pipeline – The First Line of Defence

Financial data arrives from multiple vendors (Bloomberg, Reuters, Quandl) in different formats, frequencies, and time zones.

2.1 Time Zone Alignment
All timestamps must be converted to a single canonical time zone (usually UTC or Exchange Local Time with a flag). For AI, we use Unix timestamps (seconds since 1970-01-01) as the absolute index.
t_UTC = t_local - offset

2.2 Corporate Actions Adjustment
Raw prices are contaminated by stock splits, dividends, and rights issues. We must use Adjustment Factors to create a continuous, tradable price series.

  • Cumulative Adjustment Factor C_t: At each corporate action, compute C_t = C_{t-1} * (P_{before} / P_{after}).

  • Adjusted Close Price: P_{adj,t} = P_{raw,t} * C_t.
    Crucial Rule: Always use Adj Close from Yahoo Finance or PX_LAST adjusted from Bloomberg. If you adjust yourself, ensure you apply the factor backwards so that historical prices are scaled to current capitalisation.

2.3 Resampling and Frequency Alignment
Markets have different trading calendars (NYSE vs. LSE). For a multi-asset AI model, we must align all series to a common frequency.

  • Downsampling (Tick → Minute): Use OHLCV (Open, High, Low, Close, Volume). The Close is the last traded price.

  • Upsampling (Daily → Hourly): Requires interpolation. Never use simple linear interpolation for prices (creates false trends). Use Previous Value Carried Forward (forward-fill) and treat the gaps as missing for model training.

  • Aggregation Rule for Returns: If you have daily returns r_d and want weekly returns r_w:
    r_w = Π_{d=1}^{5} (1 + r_d) - 1. In log-space: r_w = Σ ln(1 + r_d).

3. Missing Data – The Silent Killer of AI Models

Financial data is never complete. Holidays, trading halts, and data vendor errors create gaps.

3.1 Missing Data Mechanisms

  • MCAR (Missing Completely at Random): Rare in finance. E.g., a random power outage.

  • MAR (Missing at Random): Missingness depends on observed data. E.g., small-cap stocks have more missing days than large-caps.

  • MNAR (Missing Not at Random): Missingness depends on the missing value itself. E.g., a stock halts trading because it dropped 50%. This is critical – if you forward-fill the last price, you will completely miss the drop.

3.2 Imputation Strategies – From Naive to Bayesian

  • Last Observation Carried Forward (LOCF): P_t = P_{t-1} if P_t = NaN. Use for weekends/holidays only. Danger: Creates flat periods that distort volatility estimates.

  • Linear Interpolation: P_t = P_{t-k} + (P_{t+m} - P_{t-k}) * (k / (k+m)). Use only for high-frequency sparse data (e.g., illiquid FX crosses). Danger: Introduces look-ahead bias if used before the prediction point.

  • Multiple Imputation by Chained Equations (MICE): For cross-sectional features (e.g., fundamentals), train a small Random Forest on observed features to predict missing ones. This is AI-safe as it uses only past information.

  • Missing Indicator: Create a binary column IsMissing_t. Often, the fact that data is missing is itself a powerful signal (e.g., a company delaying its earnings report).

AI Rule: For time-series AI, never impute using future data. Use only expanding window imputation: at time t, your imputation model is trained only on data up to t-1.

4. Outlier Detection and Treatment – Separating Jumps from Noise

Financial returns have fat tails. A 5-sigma move is not an error; it is a crash. We must distinguish between data errors (fat-finger trades) and real extreme events.

4.1 Statistical Methods

  • Z-Score Method: z_t = (r_t - \bar{r}) / s. If |z_t| > 3.5, flag as outlier. Problem: Mean and variance are themselves skewed by outliers (masking effect).

  • Median Absolute Deviation (MAD) – Robust: MAD = median( |r_t - median(r)| ). Modified Z-score: M_z = 0.6745 * (r_t - median(r)) / MAD. If |M_z| > 3.5, flag. This is robust to up to 50% outliers.

  • IQR Method: Q1 = 25th percentile, Q3 = 75th percentile. IQR = Q3 - Q1. Lower fence = Q1 - 1.5*IQR, Upper fence = Q3 + 1.5*IQR.

4.2 Winsorisation – The Preferred Finance Treatment
Instead of deleting outliers (which removes valuable crash information), we clip them to a specified percentile.
r_t^* = min( max(r_t, P_{0.01}), P_{0.99} )
This caps extremes at the 1st and 99th percentiles, preserving the distribution’s shape while preventing a single trade from dominating the gradient descent in your neural network.

4.3 Time-Series Contextual Outliers
A 5% move is normal for a small-cap but abnormal for a Treasury bond. Use rolling Z-score with a lookback of L days:
z_roll_t = (r_t - mean_{t-L, t-1}) / std_{t-L, t-1}.
If |z_roll_t| > 4, consider capping, but only if you also verify with volume spikes (a huge move on low volume is likely a data error).

5. Normalisation and Standardisation – Preparing Features for Neural Networks

AI architectures (especially deep learning with sigmoid/tanh activations) require features to be on a similar scale to avoid gradient saturation.

5.1 Z-Score Standardisation (Mean-Variance Scaling)
x'_t = (x_t - μ) / σ

  • Expanding Window: μ_t = mean_{s < t} x_s, σ_t = std_{s < t} x_s. Use this for live deployment to avoid look-ahead.

  • Rolling Window: μ_t = mean_{s=t-L}^{t-1} x_s. Better for non-stationary data.
    AI Warning: Standardising prices directly is useless because prices are non-stationary; the rolling mean will lag far behind. Always standardise returns or log-returns.

5.2 Min-Max Scaling (Range Scaling)
x'_t = (x_t - min) / (max - min) → maps to [0,1].
Danger: Extremely sensitive to outliers. A single crash will compress all historical data into a tiny band. Use only for bounded indicators (e.g., RSI which is already in [0,100]).

5.3 Robust Scaling
Uses median and IQR instead of mean and std:
x'_t = (x_t - median) / IQR.
This is the safest default for financial features, as it is immune to fat tails.

6. Feature Engineering – Price-Derived Indicators (The Technical Toolbox)

AI models can learn non-linear relationships, but providing high-quality, domain-specific features reduces the burden on the network’s capacity.

6.1 Momentum and Trend Features

  • Simple Moving Average (SMA): SMA_t(L) = (1/L) Σ_{i=0}^{L-1} P_{t-i}.

  • Exponential Moving Average (EMA): EMA_t = P_t * α + EMA_{t-1} * (1-α), where α = 2/(L+1). EMA gives higher weight to recent prices.

  • Rate of Change (ROC): ROC_t(L) = (P_t / P_{t-L}) - 1. Measures raw momentum.

  • MACD (Moving Average Convergence Divergence):
    MACD_line = EMA_t(12) - EMA_t(26).
    Signal_line = EMA_t(9, MACD_line).
    MACD_hist = MACD_line - Signal_line.
    The histogram crossing zero indicates momentum shifts.

6.2 Volatility and Mean-Reversion Features

  • Bollinger Bands:
    Middle = SMA_t(20).
    Upper = Middle + 2 * σ_{20}.
    Lower = Middle - 2 * σ_{20}.
    %B = (P_t - Lower) / (Upper - Lower) (fractional position inside bands). %B > 1 = overbought, < 0 = oversold.

  • Average True Range (ATR): Measures market volatility for risk-sizing.
    TR_t = max( High_t - Low_t, |High_t - Close_{t-1}|, |Low_t - Close_{t-1}| ).
    ATR_t = SMA(TR, 14).

6.3 Volume and Flow Features

  • Volume-Weighted Average Price (VWAP): VWAP_t = Σ (P_i * Vol_i) / Σ Vol_i over a day. Used as the “fair” price.

  • On-Balance Volume (OBV): OBV_t = OBV_{t-1} + Volume_t * sign(Close_t - Close_{t-1}). Divergence between OBV and price signals accumulation/distribution.

  • Money Flow Index (MFI): A volume-weighted RSI.
    TP_t = (High + Low + Close) / 3.
    MF = TP * Volume.
    If TP_t > TP_{t-1}, add to Positive Money Flow, else to Negative.
    MFI = 100 - 100 / (1 + (Positive MF / Negative MF)).

7. Target Engineering – Defining What the AI Learns

The choice of target variable Y is more important than the choice of AI architecture.

7.1 Forward Returns (Prediction Horizon H)
Y_t = R_{t, t+H} = P_{t+H} / P_t - 1 (Simple) or r_{t, t+H} = ln(P_{t+H} / P_t) (Log).

  • Classification: Convert to binary classes: Y_t = 1 if R_{t,t+H} > 0, else 0. Or use triple-barrier method: Y_t = +1 if price hits upper threshold before lower threshold.

  • Regression: Predict the exact forward return. Use Huber loss to mitigate outlier effects.

7.2 Risk-Adjusted Targets
Instead of raw returns, predict the Information Ratio or Sharpe Ratio of a strategy over the next period. This is complex but aligns the AI directly with the investor’s utility function.

7.3 Triple-Barrier Labelling (Marcos Lopez de Prado)
Define an upper barrier U_t = P_t * (1 + tol) and lower barrier L_t = P_t * (1 - tol), and a vertical barrier T (max holding period).

  • Y_t = 1 (Win) if U_t touched first.

  • Y_t = -1 (Loss) if L_t touched first.

  • Y_t = 0 if time barrier T expires first.
    This creates a meta-label that is robust to noise and exactly simulates a stop-loss/take-profit strategy.

8. Point-in-Time Feature Construction – The Cardinal Rule of Finance

This is where most AI models fail in production. A “Point-in-Time” (PIT) feature matrix ensures that for a timestamp t, all features are constructed only from data available strictly before t.

8.1 The Flattened Tensor
For a model with N assets, F features, and L lags, the input at time t is a tensor of shape (N, F, L). Each feature x_{n, f, τ} where τ ∈ {t-L, ..., t-1} must only use data from the past.

8.2 Avoiding Look-Ahead Bias – Concrete Examples

  • Using Close to predict Close: If you are predicting the next day’s close, your features can include today’s open, high, and low, but never today’s close. If your feature includes Close_t and your target is Close_{t+1}, you have leaked 100% of the information.

  • Earnings Data: Financial statements are released with a delay. If Q1 earnings are reported on April 15th, you cannot use them in a feature for a model predicting on April 1st. You must use a lagged fundamental database (e.g., Compustat Point-in-Time).

8.3 Implementation Strategy – The Expanding Window

plaintext
For each time t from T_min to T_max:
    Train_window = data[0 : t-1]  # Strictly past
    Model.fit(Train_window)
    Features_for_prediction = data[t-1 : t]  # Only past info
    Prediction = Model.predict(Features_for_prediction)

Never use TimeSeriesSplit with shuffle=True. Always use TimeSeriesSplit with a fixed gap (e.g., gap=1) to prevent any temporal leakage.

9. Dimensionality Reduction – Handling the Curse of Dimensionality

When dealing with 500 stocks and 50 features, you have 25,000 input nodes. This is too many for a small dataset.

9.1 Principal Component Analysis (PCA) – The Classic Factor Model
Given an (T x N) return matrix R, we compute the covariance matrix Σ = Cov(R). PCA finds eigenvectors W (loadings) and eigenvalues λ (variance explained).
F = R * W, where F are the principal component scores (factors).
Math: Σ = W Λ W^T. We keep the top K components that explain 95% of variance.
AI Application: Instead of feeding 500 stock prices to an LSTM, feed the first 10 PCA factors. This reduces noise and forces the model to learn macro-level patterns.

9.2 Sparse Autoencoders – Non-Linear Factor Extraction
A neural network with a narrow bottleneck layer. Training:
Loss = MSE( X_input, X_reconstructed ) + λ * |latent_weights|_1.
The L1 penalty forces the latent layer to learn sparse, interpretable factors. The output of the bottleneck z is your new low-dimensional feature vector.

9.3 t-SNE / UMAP – For Visualisation Only
These are non-linear manifold learners. They are excellent for clustering regimes (e.g., visualising bull vs. bear clusters) but cannot be used for live prediction because they require the entire dataset to compute pairwise similarities (they are transductive, not inductive). Never feed t-SNE output into a trading model.

10. Summary for the AI Practitioner

  1. Ingestion: Always adjust for corporate actions and align to UTC. Raw prices are invalid.

  2. Missing Data: Use LOCF for holidays, but flag MNAR gaps. Impute using only expanding windows.

  3. Outliers: Use MAD or rolling Z-score for detection; use Winsorisation (clipping) rather than deletion.

  4. Scaling: Use Robust Scaler (Median/IQR) or Z-Score on returns, not prices.

  5. Target Engineering: Triple-Barrier labelling is superior to simple binary classification for noisy data.

  6. Cardinal Rule: Point-in-Time construction is mandatory. Use gap in cross-validation. Implement a strict chronological pipeline.

  7. Dimensionality: Use PCA for linear factor reduction; use Sparse Autoencoders for non-linear reduction. Avoid t-SNE for live models.

 
PreviousNext