Intro Statistics for ML

Goal: give just enough statistical intuition to make sensible modeling and evaluation decisions—without diving into heavy math.

Core Quantities

Term	Intuition	Tip
Mean	Average value	Sensitive to outliers
Median	Middle value	Robust central tendency
Variance / Std	Spread of values	Large spread may require scaling
Quantiles	Value at given cumulative percent	Useful for detecting skew/outliers
IQR	Middle 50% range	Basis for simple outlier rules

Distributions & Skew

Highly skewed variables (e.g., income) can mislead models. Consider log1p transform or binning if tail dominates.

Correlation ≠ Causation

Correlation measures association, not cause. Two features highly correlated with the target may both be proxies for a hidden factor.

Type	Use
Pearson	Linear relationship
Spearman	Monotonic (rank-based)

Sampling & Leakage

Always compute dataset-level statistics (mean, std, encodings) using only the training split. Computing from full data introduces subtle leakage.

Confidence Intuition (Plain Language)

A 95% interval around a metric says: if we re-sampled data many times, the true value would fall in this band most of the time. It does not guarantee correctness for a single model run.

Hypothesis Testing (Light Touch)

You rarely need full tests early. Focus on practical significance: "Does Model B reduce MAE enough to matter for users?" Bootstrapping residuals can approximate intervals if needed.

Feature Scaling Rationale

Standardization (zero mean, unit variance) helps models that rely on gradient descent or regularization treat features comparably. Tree models usually ignore scaling.

Variance vs Bias (Mental Model)

Aspect	High Bias	High Variance
Behavior	Underfits	Overfits
Training Error	High	Low
Validation Error	High	Higher than train
Fix	Add complexity/features	Simplify / regularize / more data

Practical Sanity Checks

Compare model to naïve baseline; large gap? Good. Tiny gap? Maybe problem framed poorly.
Re-train with a different random seed—performance stable? Variance low.
Randomize target (for classification) — does model still get high accuracy? If yes, pipeline leakage.

Mini Exercise

Take a numeric feature set; compute mean, median, std.
Log-transform a skewed column; compare histogram shapes.
Compute Pearson and Spearman between two features; interpret difference.

Checklist

Central tendency & spread reviewed
Skewed features identified (and strategy noted)
Correlation matrix inspected
Train-only statistic fitting respected
Baseline vs model gap assessed

Resources

OpenIntro Statistics (first chapters)
"Rules of Machine Learning" (data focus)
scikit-learn: preprocessing & model selection modules