1. Learning Objectives

By the end of this lesson, you will be able to:

  • Understand the critical importance of out-of-sample validation in financial ML.

  • Implement walk-forward validation and expanding window backtesting.

  • Compute and interpret key performance metrics: Sharpe Ratio, Sortino Ratio, Calmar Ratio, Maximum Drawdown.

  • Understand and apply the Deflated Sharpe Ratio for statistical significance testing.

  • Implement backtesting engines for trading strategies with transaction costs and slippage.

  • Understand the pitfalls of backtesting (survivorship bias, look-ahead bias, data snooping).

  • Apply the False Discovery Rate (FDR) correction for multiple testing in strategy development.

  • Benchmark strategies against buy-and-hold and factor models.

2. The Validation Framework – Why Out-of-Sample Matters

In financial ML, in-sample performance is almost always positive due to overfitting. Out-of-sample performance is the only reliable metric.

2.1 Types of Validation

 
 
Method Description Use Case
Hold-Out Split data into train/validation/test. Large datasets, quick evaluation.
Walk-Forward Train on expanding window, test on fixed window. Time series, realistic deployment simulation.
Rolling Window Train on fixed window, test on fixed window. Captures regime changes.
Time Series CV Multiple train/test splits preserving chronological order. Standard for financial ML.

2.2 Walk-Forward Validation (The Gold Standard)

text
def walk_forward_validation(data, model_class, train_size, test_size, step_size=None):
    """
    Walk-forward validation for financial time series.
    """
    if step_size is None:
        step_size = test_size

    results = []

    for start in range(0, len(data) - train_size - test_size, step_size):
        train_end = start + train_size
        test_end = train_end + test_size

        # Split data
        train_data = data.iloc[start:train_end]
        test_data = data.iloc[train_end:test_end]

        # Train model
        X_train = train_data.drop('target', axis=1)
        y_train = train_data['target']
        model = model_class()
        model.fit(X_train, y_train)

        # Test
        X_test = test_data.drop('target', axis=1)
        y_test = test_data['target']
        y_pred = model.predict(X_test)

        # Evaluate
        results.append({
            'train_start': start,
            'train_end': train_end,
            'test_end': test_end,
            'sharpe': compute_sharpe_ratio(y_pred, y_test),
            'accuracy': np.mean(y_pred == y_test)
        })

    return pd.DataFrame(results)

3. Performance Metrics for Trading Strategies

3.1 Returns and Risk Measures

Simple Return:
R_t = (P_t - P_{t-1}) / P_{t-1}

Log Return:
r_t = ln(P_t / P_{t-1})

Cumulative Return:
R_cum = Π_{t=1}^{T} (1 + R_t) - 1

Excess Return:
R_excess = R_p - R_f where R_f is the risk-free rate.

3.2 Risk Metrics

 
 
Metric Formula Interpretation
Volatility σ = std(R_p) * sqrt(252) Annualised standard deviation of returns.
Downside Deviation σ_down = std(min(R_p - MAR, 0)) * sqrt(252) Only negative returns below a Minimum Acceptable Return (MAR).
Maximum Drawdown (MDD) MDD = max_{t} (Peak_t - Trough_t) / Peak_t Largest peak-to-trough decline.
Average Drawdown AvgDD = average of drawdowns Average decline from peaks.

3.3 Risk-Adjusted Performance Metrics

Sharpe Ratio:
Sharpe = (R_p - R_f) / σ_p

Sortino Ratio (Downside Risk):
Sortino = (R_p - R_f) / σ_down

Calmar Ratio (Drawdown Risk):
Calmar = (R_p - R_f) / MDD

Omega Ratio (All Moments):
Omega = (∫_{MAR}^{∞} (1 - F(x)) dx) / (∫_{-∞}^{MAR} F(x) dx)
where F(x) is the cumulative distribution of returns.

3.4 Implementation

text
def compute_performance_metrics(returns, risk_free=0.02/252):
    """
    Compute comprehensive performance metrics.
    """
    # Annualisation factor
    ann_factor = 252

    # Returns
    mean_return = returns.mean() * ann_factor
    std_return = returns.std() * np.sqrt(ann_factor)

    # Downside deviation
    downside = returns[returns < 0].std() * np.sqrt(ann_factor)

    # Sharpe Ratio
    sharpe = (mean_return - risk_free * ann_factor) / std_return if std_return > 0 else 0

    # Sortino Ratio
    sortino = (mean_return - risk_free * ann_factor) / downside if downside > 0 else 0

    # Maximum Drawdown
    cumulative = (1 + returns).cumprod()
    running_max = cumulative.expanding().max()
    drawdown = (cumulative - running_max) / running_max
    max_drawdown = drawdown.min()

    # Calmar Ratio
    calmar = (mean_return - risk_free * ann_factor) / abs(max_drawdown) if max_drawdown < 0 else 0

    # Average Drawdown
    avg_drawdown = drawdown[drawdown < 0].mean()

    # Win Rate
    win_rate = (returns > 0).mean()

    # Average Winner / Average Loser
    avg_winner = returns[returns > 0].mean()
    avg_loser = returns[returns < 0].mean()
    profit_factor = abs(avg_winner / avg_loser) if avg_loser != 0 else np.inf

    # Information Ratio (relative to benchmark)
    # (Requires benchmark returns)

    # Skewness and Kurtosis
    skewness = returns.skew()
    kurtosis = returns.kurtosis()

    return {
        'mean_return': mean_return,
        'std_return': std_return,
        'sharpe_ratio': sharpe,
        'sortino_ratio': sortino,
        'calmar_ratio': calmar,
        'max_drawdown': max_drawdown,
        'avg_drawdown': avg_drawdown,
        'win_rate': win_rate,
        'profit_factor': profit_factor,
        'skewness': skewness,
        'kurtosis': kurtosis
    }

4. The Deflated Sharpe Ratio – Statistical Significance

The Deflated Sharpe Ratio (DSR) adjusts the Sharpe Ratio for the number of strategies tested (data snooping bias).

4.1 Mathematical Formulation

The DSR is based on the non-central t-distribution:
DSR = SR * sqrt(T) / (1 + SR * (T - 1) * γ_3 / (2 * T))
where γ_3 is the skewness of the returns.

The Deflated Formula:
DSR = SR / sqrt( Var(SR) ) with:
Var(SR) = (1 + (1/2) SR²) / T - SR² / (4T) + ...

The DSR p-value:
p = 1 - Φ( DSR / sqrt(Var(DSR)) )

Interpretation:

  • DSR > 2.0 is considered statistically significant at the 95% level.

  • If you have tested N strategies, the DSR must exceed z_{1-α/N} (Bonferroni correction).

4.2 Implementation

text
def compute_deflated_sharpe(returns, n_strategies=1, confidence=0.95):
    """
    Compute the Deflated Sharpe Ratio.
    """
    T = len(returns)
    sr = compute_sharpe_ratio(returns)

    # Variance of Sharpe Ratio (standard formula)
    skewness = returns.skew()
    kurtosis = returns.kurtosis()

    # Non-central t-distribution approximation
    dsr = sr * np.sqrt(T) / (1 + sr * (T - 1) * skewness / (2 * T))

    # Adjust for number of strategies tested (Bonferroni)
    from scipy.stats import norm
    z_score = norm.ppf(confidence)
    dsr_adjusted = dsr - z_score * np.sqrt(n_strategies)

    # P-value
    p_value = 1 - norm.cdf(dsr_adjusted)

    return {
        'raw_sharpe': sr,
        'dsr': dsr,
        'dsr_adjusted': dsr_adjusted,
        'p_value': p_value,
        'significant': p_value < (1 - confidence) / n_strategies
    }

5. Backtesting Engines – Building a Realistic Simulation

5.1 Full Backtesting Engine

text
class BacktestEngine:
    def __init__(self, initial_capital=100000, transaction_cost=0.001, slippage=0.0005):
        self.initial_capital = initial_capital
        self.transaction_cost = transaction_cost
        self.slippage = slippage
        self.capital = initial_capital
        self.positions = []
        self.portfolio_values = []
        self.trades = []

    def reset(self):
        self.capital = self.initial_capital
        self.positions = []
        self.portfolio_values = []
        self.trades = []

    def execute_trade(self, symbol, quantity, price, timestamp):
        """
        Execute a trade with transaction costs and slippage.
        """
        # Slippage: price moves against you
        if quantity > 0:  # Buy
            exec_price = price * (1 + self.slippage)
        else:  # Sell
            exec_price = price * (1 - self.slippage)

        # Transaction cost
        trade_value = abs(quantity) * exec_price
        cost = trade_value * self.transaction_cost

        # Update capital
        self.capital -= (quantity * exec_price + cost)

        # Record trade
        self.trades.append({
            'timestamp': timestamp,
            'symbol': symbol,
            'quantity': quantity,
            'price': exec_price,
            'cost': cost
        })

        return exec_price

    def run_backtest(self, data, signals, price_col='Close'):
        """
        Run a backtest given price data and trading signals.
        """
        self.reset()

        # Ensure aligned indices
        aligned_data = data.align(signals, join='inner')[0]

        # Initialise position
        current_position = 0
        entry_price = None

        for i in range(1, len(aligned_data)):
            current_time = aligned_data.index[i]
            current_price = aligned_data.iloc[i][price_col]

            # Get signal
            signal = signals.iloc[i]

            if signal == 1 and current_position == 0:  # Buy signal
                # Buy with all capital
                quantity = self.capital // current_price
                if quantity > 0:
                    self.execute_trade('asset', quantity, current_price, current_time)
                    current_position = quantity
                    entry_price = current_price

            elif signal == -1 and current_position > 0:  # Sell signal
                # Sell all position
                self.execute_trade('asset', -current_position, current_price, current_time)
                current_position = 0
                entry_price = None

            # Record portfolio value
            portfolio_value = self.capital + current_position * current_price
            self.portfolio_values.append({
                'timestamp': current_time,
                'portfolio_value': portfolio_value,
                'position': current_position,
                'price': current_price
            })

        # Close any open position at the end
        if current_position > 0:
            final_price = aligned_data.iloc[-1][price_col]
            self.execute_trade('asset', -current_position, final_price, aligned_data.index[-1])

        return pd.DataFrame(self.portfolio_values).set_index('timestamp')

    def compute_returns(self, portfolio_df):
        """
        Compute returns from portfolio values.
        """
        portfolio_df['return'] = portfolio_df['portfolio_value'].pct_change()
        return portfolio_df.dropna()

5.2 Transaction Cost Modelling

text
def compute_transaction_costs(trades, volatility, spread):
    """
    Advanced transaction cost model (Almgren-Chriss style).
    """
    total_cost = 0

    for trade in trades:
        quantity = abs(trade['quantity'])
        price = trade['price']

        # Fixed cost (spread)
        spread_cost = quantity * price * spread / 2

        # Market impact (power law)
        market_impact = quantity * price * 0.01 * (quantity / 100000)**0.6 * volatility

        total_cost += spread_cost + market_impact

    return total_cost

6. Pitfalls in Backtesting – The “Gotchas”

6.1 Survivorship Bias
Historical datasets often only include assets that are still trading. Failed companies are excluded, making backtests overly optimistic.

Solution: Use point-in-time databases or include delisted assets.

6.2 Look-Ahead Bias
Using future information to make decisions. Examples:

  • Using earnings data that wasn’t available at the time.

  • Using adjusted prices without proper lag.

  • Using hindsight to select parameters.

Solution: Strict chronological ordering. Use TimeSeriesSplit for CV.

6.3 Data Snooping
Testing many strategies on the same data and reporting only the best results.

Solution: Use the Deflated Sharpe Ratio. Perform out-of-sample testing on unseen data.

6.4 Overfitting
Complex models fit the noise in the training data.

Solution: Simpler models with strong regularisation. Cross-validation.

6.5 The False Discovery Rate (FDR)
When testing N independent strategies, the expected number of false positives is N * α.

Bonferroni Correction:
α_corrected = α / N

Benjamini-Hochberg Procedure:

text
def benjamini_hochberg(p_values, q=0.05):
    """
    Benjamini-Hochberg procedure for FDR control.
    """
    p_values = np.array(p_values)
    sorted_idx = np.argsort(p_values)
    sorted_p = p_values[sorted_idx]

    m = len(p_values)
    threshold = (np.arange(1, m+1) / m) * q

    significant = sorted_p <= threshold
    if np.any(significant):
        k = np.where(significant)[0][-1]
        return sorted_idx[:k+1]
    else:
        return []

7. Benchmarking – Comparing Against Alternatives

7.1 Common Benchmarks

 
 
Benchmark Description
Buy-and-Hold Buy and hold the market portfolio.
S&P 500 Market-cap weighted index.
60/40 60% equities, 40% bonds.
Equal Weight Equal weights across all assets.
Minimum Variance Minimum variance portfolio.

7.2 Implementation

text
def benchmark_strategy(returns, benchmark_returns):
    """
    Compare strategy performance against benchmark.
    """
    # Correlation
    correlation = returns.corr(benchmark_returns)

    # Beta
    beta = returns.cov(benchmark_returns) / benchmark_returns.var()

    # Alpha (Jensen's alpha)
    alpha = returns.mean() - beta * benchmark_returns.mean()

    # Information Ratio
    tracking_error = (returns - beta * benchmark_returns).std()
    info_ratio = (returns.mean() - benchmark_returns.mean()) / tracking_error if tracking_error > 0 else 0

    # Up-capture and Down-capture
    benchmark_up = benchmark_returns[benchmark_returns > 0]
    benchmark_down = benchmark_returns[benchmark_returns < 0]
    up_capture = returns[benchmark_returns > 0].mean() / benchmark_up.mean() if len(benchmark_up) > 0 else 0
    down_capture = returns[benchmark_returns < 0].mean() / benchmark_down.mean() if len(benchmark_down) > 0 else 0

    return {
        'correlation': correlation,
        'beta': beta,
        'alpha': alpha,
        'information_ratio': info_ratio,
        'up_capture': up_capture,
        'down_capture': down_capture
    }

8. Summary for the AI Practitioner

  1. Walk-forward validation is the gold standard for financial ML. Never use random shuffle CV.

  2. Sharpe Ratio is the most common metric. Sortino and Calmar are better for downside risk.

  3. Maximum Drawdown is critical for risk management. A high Sharpe with a high drawdown is not acceptable.

  4. The Deflated Sharpe Ratio adjusts for data snooping. Always use it when evaluating multiple strategies.

  5. Transaction costs (spread, market impact) must be included in backtests. They can eliminate apparent alpha.

  6. Survivorship bias and look-ahead bias are the most common backtesting errors. Always use point-in-time data.

  7. Benchmarking is mandatory. A strategy must beat a simple buy-and-hold to be valuable.

  8. FDR correction (Bonferroni or Benjamini-Hochberg) is essential when testing multiple strategies.

 
PreviousNext