Model Evaluation and Metrics

Why Metrics Matter

You optimize what you measure. A poor metric choice yields models that look good in a notebook but fail user or business needs. Always tie the primary metric to the decision or experience you want to improve.

Core Families

Task	Common Metrics	When Accuracy Misleads
Classification	Accuracy, Precision, Recall, F1, ROC-AUC, PR-AUC	Class imbalance, asymmetric costs
Regression	MAE, MSE, RMSE, R²	Skewed errors, outlier sensitivity
Ranking / RecSys	MAP, NDCG, Hit Rate, MRR	Only top-K matters
Prob. Calibration	Brier Score, calibration curve	Threshold decisions rely on probability quality

Classification Metrics (Intuition)

Precision: Of predicted positives, how many were correct? (False positives expensive)
Recall: Of actual positives, how many did we catch? (False negatives expensive)
F1: Harmonic mean balancing precision & recall.
ROC-AUC: Probability a random positive outranks a random negative. Stable under imbalance but can mask poor minority handling.
PR-AUC: Focuses on positive class quality; better in heavy imbalance.

Scenario	Prefer Metric
Disease screening	Recall (with precision floor)
Spam detection	Precision (with recall floor)
Rare fraud detection	PR-AUC + Recall at fixed precision

Imbalanced Data Notes

Accuracy can stay >95% while the model predicts the majority class always. Always view a confusion matrix or classification report. Consider class weights (class_weight='balanced'), resampling, or threshold tuning.

Regression Metrics

Metric	Strength	Watch Out
MAE	Interpretable (same units)	Under-penalizes rare large errors
MSE	Penalizes large errors strongly	Units squared (hard to explain)
RMSE	Same units + punishes large errors	Still sensitive to outliers
R²	Variance explained vs baseline	Negative possible; not an error scale

Pick MAE to express typical error. Use RMSE if large deviations are especially costly (e.g. power demand spikes). Always include at least one error metric even if reporting R².

Cross-Validation (CV)

Variant	Use Case
k-fold (k=5/10)	Standard tabular data
Stratified k-fold	Classification with imbalance
TimeSeriesSplit	Temporal order matters
Group/Leave-One-Group-Out	User or entity grouped samples

Look at the mean and the spread (std) of CV scores. A slightly lower mean with much lower variance can be preferable for stability.

Data Leakage Examples

Leakage Pattern	Description	Mitigation
Scaling on full dataset	Test info in scaler stats	Fit on train only; reuse transformer
Encoded target ratio feature	Target info baked in	Compute inside CV folds or drop
Future timestamp used	Seeing the future	Use time-respecting splits
Duplicate entities across splits	Same user leaks behavior	Group-based split

Try a random target sanity test: shuffle target labels and retrain. If performance still looks “good,” leakage or label bleed exists.

Probability Calibration

If you act on thresholds (e.g., flag if p>0.7), calibration matters. Plot predicted probability bins vs actual frequency. Techniques: Platt scaling (logistic on logits) or Isotonic (non-parametric) — fit on validation set after final model training.

See also: foundational stats concepts in Intro Statistics for ML and baseline error reasoning in Regression Basics.

Evaluation Workflow

Define primary metric + guardrail (e.g., F1 primary; precision >= 0.90).
Compute simple baseline (dummy classifier; mean predictor).
Perform train/validation or CV; tune only on validation folds.
Analyze errors: confusion matrix, most common mislabels, residual plots.
Check for leakage (scalers, engineered features, duplicates).
Calibrate probabilities if downstream threshold decisions exist.
Estimate uncertainty (bootstrap or CV std).
Document evaluation (data slice, date, metric definitions, limitations).

Code Snippets

from sklearn.metrics import classification_report, roc_auc_score, average_precision_score
print(classification_report(y_true, y_pred_labels))
print('ROC-AUC:', roc_auc_score(y_true, y_pred_prob))
print('PR-AUC:', average_precision_score(y_true, y_pred_prob))

import numpy as np
from sklearn.metrics import mean_absolute_error

def bootstrap_metric(y_true, y_pred, metric=mean_absolute_error, n=200, seed=0):
    """Return (mean_metric, (low_ci, high_ci)) via bootstrap."""
    rng = np.random.default_rng(seed)
    vals = []
    y_true = np.array(y_true); y_pred = np.array(y_pred)
    for _ in range(n):
        idx = rng.integers(0, len(y_true), len(y_true))
        vals.append(metric(y_true[idx], y_pred[idx]))
    return float(np.mean(vals)), (float(np.percentile(vals,2.5)), float(np.percentile(vals,97.5)))

Checklist

Baseline vs model improvement measured
Primary + guardrail metrics defined
Validation / CV strategy documented
Confusion matrix or residuals inspected
Leakage sanity check done
Probability calibration (if thresholded decisions)
Uncertainty/variance reported
Evaluation note written

Resources

scikit-learn metrics docs
"Rules of Machine Learning" – error analysis
Google ML Crash Course – ROC/PR curves