Learning Objectives

By the end of this lesson, learners should be able to:

1. Explain the purpose of predictive models in business analytics.
2. Distinguish between model development, validation and deployment.
3. Explain the difference between classification and regression models.
4. Identify appropriate evaluation metrics for different predictive tasks.
5. Explain accuracy, precision, recall and F1-score.
6. Interpret confusion matrices in business contexts.
7. Explain ROC curves and AUC.
8. Explain regression evaluation measures.
9. Distinguish model fit from predictive performance.
10. Explain cross-validation and holdout evaluation.
11. Identify overfitting, underfitting and data leakage.
12. Evaluate predictive models from both technical and business perspectives.

1. Introduction to Predictive Models

A predictive model is a mathematical or statistical representation used to estimate an outcome from available information.

In business analytics, predictive models can support decisions such as:

• Which customers are likely to leave?
• Which transactions are likely to be fraudulent?
• What will next month’s sales be?
• Which applicants are likely to default?
• Which products are likely to experience high demand?

The quality of a predictive model cannot be judged solely by how well it fits historical data.

The more important question is:

How well does the model perform on data it has not previously seen?

2. Classification and Regression Models

Predictive models commonly address two broad types of outcomes.

Classification

The outcome belongs to a category.

Examples:

• Fraud / legitimate.
• Churn / retain.
• Default / non-default.

Regression

The outcome is numerical.

Examples:

• Revenue.
• Sales volume.
• Customer spending.
• Delivery time.

The evaluation criteria should reflect the type of prediction being made.

3. Training, Validation and Testing

A predictive dataset can be divided into different subsets.

Training Set

Used to develop the model.

Validation Set

Used to compare or tune alternative models and settings.

Test Set

Used to provide an approximately unbiased evaluation of the final model on unseen data.

The exact approach depends on the modelling method and available data.

4. Why Data Splitting Matters

Suppose a model is evaluated using exactly the same observations used to train it.

Its performance may appear unusually strong because the model has already been exposed to those observations.

This can lead to an overly optimistic assessment of real-world performance.

Evaluation on unseen data provides stronger evidence of generalization.

5. Confusion Matrix

A confusion matrix summarizes classification predictions.

For a binary classification problem, it contains:

	Actual Positive	Actual Negative
Predicted Positive	True Positive	False Positive
Predicted Negative	False Negative	True Negative

Each category has a different business interpretation.

6. True Positive

A true positive occurs when:

The model predicts the positive class and the actual outcome is positive.

Example:

A fraud model flags a transaction as fraudulent and investigation confirms that it was fraudulent.

7. True Negative

A true negative occurs when:

The model predicts the negative class and the actual outcome is negative.

Example:

A transaction is classified as legitimate and is genuinely legitimate.

8. False Positive

A false positive occurs when:

The model predicts positive but the actual outcome is negative.

Example:

A legitimate transaction is incorrectly flagged as fraudulent.

9. False Negative

A false negative occurs when:

The model predicts negative but the actual outcome is positive.

Example:

A fraudulent transaction is incorrectly classified as legitimate.

10. Accuracy

Accuracy is the proportion of predictions that are correct.

Conceptually:

Accuracy = Correct Predictions / Total Predictions

Accuracy can be useful when classes are reasonably balanced and the costs of errors are similar.

However, accuracy can be misleading for highly imbalanced datasets.

11. Accuracy and Imbalanced Data

Suppose a fraud dataset contains:

• 99,000 legitimate transactions.
• 1,000 fraudulent transactions.

A model that classifies every transaction as legitimate achieves:

99% accuracy.

Yet it detects:

0% of fraud cases.

Therefore, the model may have high accuracy while being practically useless for fraud detection.

12. Precision

Precision measures the proportion of predicted positive cases that are actually positive.

Conceptually:

Precision = True Positives / (True Positives + False Positives)

Precision is particularly important when false positives are costly.

13. Example of Precision

Suppose a fraud model flags:

1,000 transactions.

Investigation shows:

700 are actually fraudulent.

Precision is:

700 / 1,000 = 70%.

This means 70% of transactions flagged by the model were actually fraudulent.

14. Recall

Recall measures the proportion of actual positive cases that the model successfully identifies.

Conceptually:

Recall = True Positives / (True Positives + False Negatives)

Recall becomes particularly important when failing to identify a positive case has significant consequences.

15. Example of Recall

Suppose:

Actual fraudulent transactions = 1,000.

Model detects = 700.

Recall:

700 / 1,000 = 70%.

The model therefore identifies 70% of the actual fraud cases.

16. Precision Versus Recall

A model can sometimes improve one at the expense of the other.

For example:

A highly cautious fraud model may flag only transactions with extremely high predicted fraud probability.

This may produce:

• High precision.
• Lower recall.

A more aggressive model may flag many transactions.

This may produce:

• Higher recall.
• Lower precision.

The appropriate balance depends on business consequences.

17. F1-Score

The F1-score combines precision and recall using their harmonic mean.

It is useful when both false positives and false negatives matter and a single summary measure is desired.

A model with:

• High precision.
• High recall.

will generally have a high F1-score.

18. ROC Curve

A Receiver Operating Characteristic (ROC) curve examines the trade-off between:

• True positive rate.
• False positive rate.

across different classification thresholds.

It helps analysts understand how model discrimination changes as the decision threshold changes.

19. AUC

Area Under the ROC Curve (AUC) summarizes the model’s ability to distinguish between positive and negative cases across thresholds.

A higher AUC generally indicates stronger ranking/discrimination performance.

However, AUC should not be interpreted as a direct statement that a model is “X% accurate.”

20. Choosing Classification Metrics

Different business problems require different priorities.

Fraud Detection

Recall may be especially important if missed fraud is expensive.

Customer Service Alerts

Precision may matter if unnecessary interventions are costly.

Medical Risk Screening

Both false negatives and false positives may have significant consequences.

Marketing

Precision, recall, lift and business conversion may all be relevant.

The metric should therefore reflect the decision problem.

21. Regression Model Evaluation

Regression models predict numerical values.

Common evaluation measures include:

• MAE.
• MSE.
• RMSE.
• R².

These measures answer different questions.

22. MAE

Mean Absolute Error measures the average absolute difference between actual and predicted values.

It is relatively easy to interpret because it uses the same units as the target.

Example:

MAE = KSh 200,000.

This indicates that the model’s predictions differ from actual values by an average absolute amount of approximately KSh 200,000 over the evaluation sample.

23. MSE

Mean Squared Error squares errors before averaging them.

This means larger errors receive disproportionately greater weight.

MSE can therefore be useful when large prediction mistakes are particularly undesirable.

24. RMSE

Root Mean Squared Error is the square root of MSE.

It is expressed in the same units as the target variable.

For example:

If predicting sales in KSh:

RMSE = KSh 350,000.

25. R² in Model Evaluation

R² can provide information about how much variation in the observed outcome is accounted for by the fitted regression model.

However, a high R² does not necessarily imply:

• Strong future predictions.
• Causation.
• Commercial usefulness.
• Good performance on unseen data.

It should therefore be interpreted alongside other evaluation evidence.

26. Cross-Validation

Cross-validation repeatedly divides available data into training and validation portions to evaluate model performance.

One common approach is k-fold cross-validation.

The data is divided into k groups.

The model is trained on:

k − 1 groups

and evaluated on:

the remaining group.

The process is repeated so that each group serves as validation data.

27. Why Cross-Validation Is Useful

Cross-validation can provide a more robust estimate of model performance than relying on a single arbitrary train-validation split.

It can be particularly useful when:

• The dataset is relatively limited.
• Multiple models need comparison.
• Hyperparameters need tuning.

However, the exact validation strategy should respect the structure of the data.

28. Time-Series Validation

Ordinary random cross-validation can be inappropriate for time-series data because it may allow future information to influence evaluation of earlier periods.

For time-dependent data, validation should preserve temporal order.

For example:

Train: January–September
Validate: October

Then:

Train: January–October
Validate: November

This better reflects how the model would operate in practice.

29. Overfitting

Overfitting occurs when a model captures training-specific patterns that do not generalize.

Typical pattern:

Very strong training performance.

but:

Significantly weaker unseen-data performance.

Overfitting is particularly concerning when the model is complex relative to the available information.

30. Underfitting

Underfitting occurs when a model is too simple to capture important patterns.

Typical pattern:

Poor performance on both training and unseen data.

The solution is not necessarily to make the model dramatically more complex.

The analyst should investigate:

• Features.
• Model assumptions.
• Data quality.
• Relationships.
• Appropriate model class.

31. Data Leakage

Data leakage occurs when information that would not legitimately be available at prediction time enters model development.

This can make evaluation appear exceptionally strong.

For example:

A model predicting loan default uses a variable created by the collections department after default-related action has already begun.

The model would be unrealistic in deployment.

32. Hyperparameter Tuning

Some models have settings that are not learned directly from the training observations.

These are called hyperparameters.

Examples include:

• Tree depth.
• Regularization strength.
• Number of neighbors.
• Learning rate.

These settings should be selected using appropriate validation procedures rather than optimizing directly against the final test set.

33. Model Selection

Suppose three models produce:

Model	Precision	Recall	Business Cost
A	90%	55%	High
B	80%	82%	Medium
C	68%	94%	Very High

The “best” model cannot be selected simply by looking at one metric.

Management must consider:

• Error costs.
• Operational capacity.
• Customer impact.
• Regulatory considerations.
• Financial consequences.

34. Technical Performance Versus Business Performance

A model can perform well statistically while delivering little business value.

For example:

A churn model may have excellent classification metrics but generate too many customer alerts for the retention team to handle.

The model is technically useful but operationally poorly integrated.

Business analytics therefore requires both:

Model performance

and

Decision usefulness.

35. Model Calibration

A classification model may produce probabilities rather than only categories.

For example:

Customer A: 80% churn probability.

A well-calibrated model should produce probabilities that correspond reasonably to observed frequencies.

If among customers assigned approximately 80% risk, only 30% actually churn, the probabilities may be poorly calibrated.

36. Threshold Selection

Many classification models produce a probability and then convert it into a class using a threshold.

For example:

Probability ≥ 0.50 → High risk.

Changing the threshold changes:

• Precision.
• Recall.
• False positives.
• False negatives.

The threshold should therefore reflect business objectives rather than automatically defaulting to 50%.

37. Model Explainability

Organizations may need to understand why a model generates particular predictions.

This can be important for:

• Management.
• Customers.
• Auditors.
• Regulators.
• Risk committees.

The appropriate level of explanation depends on the model, application and governance requirements.

38. Model Robustness

A robust model should perform reasonably well when conditions vary within the range expected in practice.

Analysts should consider:

• Different customer segments.
• Different time periods.
• Different geographic markets.
• Changing business conditions.

A model that works only in one narrow dataset may not be sufficiently robust.

39. Model Monitoring After Deployment

Evaluation does not end when a model is deployed.

Organizations should monitor:

• Prediction performance.
• Data quality.
• Distribution changes.
• Error patterns.
• Business outcomes.
• Model drift.

A model that was effective last year may not remain effective indefinitely.

40. Best Practices

Business analysts should:

1. Select metrics based on the business problem.
2. Evaluate models on appropriate unseen data.
3. Establish meaningful baseline models.
4. Avoid data leakage.
5. Use appropriate validation procedures.
6. Consider class imbalance.
7. Examine both technical and business performance.
8. Monitor deployed models.
9. Document modelling assumptions.
10. Communicate limitations and uncertainty.

Lesson Summary

Predictive model evaluation determines whether a model performs adequately on unseen information and whether its predictions are useful for the intended business decision.

For classification, important measures include:

• Accuracy.
• Precision.
• Recall.
• F1-score.
• ROC/AUC.

For regression, common measures include:

• MAE.
• MSE.
• RMSE.
• R².

Other important concepts include:

• Training, validation and testing.
• Cross-validation.
• Overfitting.
• Underfitting.
• Data leakage.
• Hyperparameter tuning.
• Calibration.
• Threshold selection.
• Model monitoring.

A technically strong model is not automatically the best business model. The appropriate model is the one whose performance, reliability, operational requirements and business consequences align with the decision it is intended to support