1. Learning Objectives
By the end of this lesson, you will be able to:
-
Understand the principles of hyperparameter tuning for financial models.
-
Implement Grid Search and Random Search for hyperparameter optimisation.
-
Apply Bayesian optimisation using Gaussian Processes for efficient tuning.
-
Implement cross-validation strategies specific to financial time series.
-
Understand the tradeoff between model complexity, training time, and performance.
-
Apply automated machine learning (AutoML) tools for financial modelling.
-
Implement ensemble selection and model stacking for robust predictions.
-
Understand the importance of stability and robustness over peak performance.
2. Hyperparameters vs. Parameters
Parameters:Â Learned from data (e.g., regression coefficients, neural network weights).
Hyperparameters:Â Set before training (e.g., learning rate, number of trees, depth of trees, regularisation strength).
Financial Implication:Â Hyperparameters must be tuned on validation data (not test data). Over-tuning on the validation set can lead to overfitting.
3. Cross-Validation Strategies for Time Series
3.1 TimeSeriesSplit (Expanding Window)
from sklearn.model_selection import TimeSeriesSplit
def time_series_cv_evaluation(model, X, y, n_splits=5, test_size=60):
"""
Evaluate model using TimeSeriesSplit.
"""
tscv = TimeSeriesSplit(n_splits=n_splits, test_size=test_size, gap=1)
scores = []
for train_idx, val_idx in tscv.split(X):
X_train, X_val = X.iloc[train_idx], X.iloc[val_idx]
y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]
model.fit(X_train, y_train)
y_pred = model.predict(X_val)
score = np.mean(y_pred == y_val) # Classification accuracy
scores.append(score)
return np.mean(scores), np.std(scores)
3.2 Purged Walk-Forward (Lopez de Prado)
The purged walk-forward removes overlapping training and validation periods to prevent data leakage.
def purged_walk_forward(X, y, train_size, test_size, purge_gap=0):
"""
Purged walk-forward cross-validation.
"""
for start in range(0, len(X) - train_size - test_size, test_size):
train_end = start + train_size
test_start = train_end + purge_gap
train_idx = range(start, train_end)
val_idx = range(test_start, test_start + test_size)
yield train_idx, val_idx
4. Grid Search – Exhaustive Search
Grid Search tries all combinations of hyperparameters in a specified grid.
from sklearn.model_selection import GridSearchCV
def grid_search_tuning(model, param_grid, X_train, y_train, cv_strategy):
"""
Perform grid search with time series cross-validation.
"""
gs = GridSearchCV(
estimator=model,
param_grid=param_grid,
cv=cv_strategy,
scoring='roc_auc',
n_jobs=-1,
verbose=1
)
gs.fit(X_train, y_train)
print(f"Best parameters: {gs.best_params_}")
print(f"Best score: {gs.best_score_:.4f}")
return gs.best_estimator_, gs.best_params_, gs.cv_results_
4.1 Example – XGBoost Grid Search
def xgboost_grid_search(X_train, y_train):
"""
Grid search for XGBoost hyperparameters.
"""
param_grid = {
'n_estimators': [50, 100, 200],
'max_depth': [3, 5, 7, 9],
'learning_rate': [0.01, 0.05, 0.1, 0.2],
'subsample': [0.7, 0.8, 0.9],
'colsample_bytree': [0.7, 0.8, 0.9],
'reg_alpha': [0, 0.1, 1.0],
'reg_lambda': [0, 0.1, 1.0]
}
model = xgb.XGBClassifier(objective='binary:logistic', random_state=42)
gs = GridSearchCV(
model,
param_grid,
cv=TimeSeriesSplit(n_splits=3, test_size=60),
scoring='roc_auc',
n_jobs=-1,
verbose=1
)
gs.fit(X_train, y_train)
return gs
5. Random Search – Efficient Exploration
Random Search samples hyperparameters randomly from distributions. It is more efficient than Grid Search for high-dimensional spaces.
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import uniform, randint
def random_search_tuning(model, param_distributions, X_train, y_train, n_iter=100):
"""
Perform random search with time series cross-validation.
"""
rs = RandomizedSearchCV(
estimator=model,
param_distributions=param_distributions,
n_iter=n_iter,
cv=TimeSeriesSplit(n_splits=3, test_size=60),
scoring='roc_auc',
n_jobs=-1,
random_state=42,
verbose=1
)
rs.fit(X_train, y_train)
print(f"Best parameters: {rs.best_params_}")
print(f"Best score: {rs.best_score_:.4f}")
return rs.best_estimator_, rs.best_params_
5.1 Example – Random Search Parameter Distributions
param_dist = {
'n_estimators': randint(50, 500),
'max_depth': randint(3, 15),
'learning_rate': uniform(0.01, 0.3),
'subsample': uniform(0.5, 0.5),
'colsample_bytree': uniform(0.5, 0.5),
'reg_alpha': uniform(0, 2),
'reg_lambda': uniform(0, 2)
}
6. Bayesian Optimisation – Informed Search
Bayesian Optimisation builds a probabilistic model of the objective function and uses it to select the most promising hyperparameters to evaluate.
6.1 Mathematical Formulation
-
Surrogate Model:Â Gaussian Process (GP) approximating the objective function.
-
Acquisition Function:Â Determines the next point to evaluate.
Common Acquisition Functions:
-
Expected Improvement (EI):Â
EI(x) = E[max(f(x) - f(x^+), 0)] -
Upper Confidence Bound (UCB):Â
UCB(x) = μ(x) + κ σ(x) -
Probability of Improvement (PI):Â
PI(x) = P(f(x) > f(x^+))
6.2 Implementation with Optuna
import optuna
def bayesian_optimisation_xgboost(X_train, y_train, n_trials=100):
"""
Bayesian optimisation for XGBoost using Optuna.
"""
def objective(trial):
params = {
'n_estimators': trial.suggest_int('n_estimators', 50, 500),
'max_depth': trial.suggest_int('max_depth', 3, 15),
'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3, log=True),
'subsample': trial.suggest_float('subsample', 0.5, 1.0),
'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
'reg_alpha': trial.suggest_float('reg_alpha', 0, 2),
'reg_lambda': trial.suggest_float('reg_lambda', 0, 2),
'min_child_weight': trial.suggest_int('min_child_weight', 1, 10)
}
model = xgb.XGBClassifier(**params, objective='binary:logistic', random_state=42)
# Time series cross-validation
tscv = TimeSeriesSplit(n_splits=3, test_size=60)
scores = []
for train_idx, val_idx in tscv.split(X_train):
X_tr, X_val = X_train.iloc[train_idx], X_train.iloc[val_idx]
y_tr, y_val = y_train.iloc[train_idx], y_train.iloc[val_idx]
model.fit(X_tr, y_tr)
y_pred = model.predict_proba(X_val)[:, 1]
score = roc_auc_score(y_val, y_pred)
scores.append(score)
return np.mean(scores)
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=n_trials, n_jobs=-1)
print(f"Best trial: {study.best_trial}")
print(f"Best parameters: {study.best_params}")
print(f"Best AUC: {study.best_value:.4f}")
return study
6.3 Financial Application – Optimising for Sharpe Ratio
def objective_sharpe(trial, data, model_type='xgboost'):
"""
Objective function optimising for Sharpe Ratio.
"""
# Define hyperparameters
if model_type == 'xgboost':
params = {
'n_estimators': trial.suggest_int('n_estimators', 50, 500),
'max_depth': trial.suggest_int('max_depth', 3, 15),
'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3, log=True)
}
model = xgb.XGBRegressor(**params, random_state=42)
else:
# LSTM hyperparameters
params = {
'hidden_dim': trial.suggest_int('hidden_dim', 32, 256),
'num_layers': trial.suggest_int('num_layers', 1, 4),
'learning_rate': trial.suggest_float('learning_rate', 1e-5, 1e-2, log=True)
}
model = LSTMTimeSeries(**params)
# Walk-forward validation with Sharpe ratio as objective
returns = []
for train_idx, val_idx in purged_walk_forward(data, train_size=1000, test_size=60):
X_train, X_val = data.iloc[train_idx], data.iloc[val_idx]
model.fit(X_train.drop('target', axis=1), X_train['target'])
y_pred = model.predict(X_val.drop('target', axis=1))
returns.append(y_pred * X_val['target']) # Position * return
sharpe = compute_sharpe_ratio(pd.Series(returns))
return sharpe
7. AutoML – Automated Machine Learning
AutoML tools automate the entire ML pipeline: feature engineering, model selection, hyperparameter tuning.
7.1 TPOT (Tree-based Pipeline Optimisation Tool)
from tpot import TPOTClassifier
def automl_tpot(X_train, y_train, generations=50, population_size=50):
"""
Automated ML with TPOT.
"""
tpot = TPOTClassifier(
generations=generations,
population_size=population_size,
cv=TimeSeriesSplit(n_splits=3, test_size=60),
scoring='roc_auc',
random_state=42,
n_jobs=-1,
verbosity=2
)
tpot.fit(X_train, y_train)
# Export the best pipeline
tpot.export('best_pipeline.py')
return tpot
7.2 AutoML with H2O
import h2o
from h2o.automl import H2OAutoML
def automl_h2o(X, y, max_models=50, max_runtime_secs=3600):
"""
Automated ML with H2O.
"""
h2o.init()
# Convert to H2O frames
hf = h2o.H2OFrame(pd.concat([X, y], axis=1))
x = X.columns.tolist()
y = y.name
# AutoML
aml = H2OAutoML(
max_models=max_models,
max_runtime_secs=max_runtime_secs,
seed=42,
nfolds=5
)
aml.train(x=x, y=y, training_frame=hf)
# Get leaderboard
leaderboard = aml.leaderboard
print(leaderboard)
return aml.leader, aml
8. Model Selection – Choosing the Best Model
8.1 Ensemble Selection
Instead of choosing one model, combine multiple models.
def ensemble_selection(models, X_val, y_val, top_k=3):
"""
Select the top k models based on validation performance.
"""
scores = []
for name, model in models.items():
y_pred = model.predict_proba(X_val)[:, 1]
auc = roc_auc_score(y_val, y_pred)
scores.append((name, model, auc))
# Sort by performance
scores.sort(key=lambda x: x[2], reverse=True)
# Select top k
selected = scores[:top_k]
print("Selected models:")
for name, model, auc in selected:
print(f" {name}: AUC = {auc:.4f}")
return selected
8.2 Stacking with Selected Models
def stack_selected_models(selected_models, X_train, y_train, X_val, y_val):
"""
Stack selected models with a meta-learner.
"""
from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
# Base models
base_models = [(name, model) for name, model, _ in selected_models]
# Meta-learner
meta_model = LogisticRegression()
stacking = StackingClassifier(
estimators=base_models,
final_estimator=meta_model,
cv=TimeSeriesSplit(n_splits=3, test_size=60)
)
stacking.fit(X_train, y_train)
return stacking
9. Stability and Robustness – Beyond Peak Performance
In finance, stability is often more important than peak performance.
9.1 Coefficient of Variation of Performance
def performance_stability(returns, window=252):
"""
Compute rolling Sharpe Ratio and its coefficient of variation.
"""
rolling_sharpe = returns.rolling(window).apply(
lambda x: compute_sharpe_ratio(x)
)
stability = rolling_sharpe.std() / (rolling_sharpe.mean() + 1e-8)
return stability
9.2 Regime-Specific Performance
def regime_performance(returns, regime_labels):
"""
Evaluate performance in different market regimes.
"""
results = {}
for regime in regime_labels.unique():
regime_returns = returns[regime_labels == regime]
results[regime] = compute_performance_metrics(regime_returns)
return pd.DataFrame(results).T
9.3 Stress Testing
def stress_test_strategy(returns, stress_scenarios):
"""
Stress test the strategy under extreme scenarios.
"""
results = {}
for scenario_name, scenario_multiplier in stress_scenarios.items():
stressed_returns = returns * scenario_multiplier
results[scenario_name] = compute_performance_metrics(stressed_returns)
return pd.DataFrame(results).T
10. Summary for the AI Practitioner
-
Hyperparameter tuning is essential but must be done on validation data. Never tune on test data.
-
TimeSeriesSplit is mandatory for financial data. Never useÂ
KFold (random shuffle) as it leaks information. -
Grid Search is exhaustive but slow. Use it for small parameter spaces.
-
Random Search is more efficient than Grid Search for high-dimensional spaces.
-
Bayesian Optimisation (Optuna) is the state-of-the-art. It uses fewer evaluations and finds better parameters.
-
AutoMLÂ tools (TPOT, H2O) automate the entire pipeline. Useful for rapid prototyping.
-
Ensemble selection and stacking combine multiple models for superior robustness.
-
Stability is often more important than peak performance in finance. Evaluate across regimes and stress scenarios.
-
Validation performance should guide model selection, not training performance.
-
Regularisation is critical. A slightly worse validation model with strong regularisation is better than a slightly better model with none.