Regression Basics

When to Use

Use regression when the target is continuous (price, temperature, time-on-site). Start with the simplest model that can beat a naïve baseline.

Intuition

Linear regression finds coefficients (weights) so the line / hyperplane minimizes the sum of squared differences between predictions and actual values.

Core Metrics

Metric	Interprets	Notes
MAE	Average absolute error	Robust to outliers (linear penalty)
MSE	Average squared error	Larger errors penalized more
RMSE	sqrt(MSE)	Same units as target; sensitive to outliers
R²	Variance explained	Can be negative if model is worse than baseline

Pick MAE for interpretability, RMSE when large errors are costly, R² to communicate variance explained (but never alone).

Baselines

Always compare against: - Mean predictor (y_mean) - Median predictor (robust) - Domain simple heuristic (e.g., last known value for time series)

Quick Example

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = LinearRegression().fit(X_train, y_train)
pred = model.predict(X_test)
print('MAE:', round(mean_absolute_error(y_test, pred), 3))
print('R²:', round(r2_score(y_test, pred), 3))

Regularization

Prevents overfitting by shrinking weights: - L2 (Ridge): pushes weights toward zero (smooth shrinkage) - L1 (Lasso): can zero out some weights (feature selection) - Elastic Net: combination of both

Assumptions (Practical View)

Concept	Why You Care
Linearity	Severe non-linearity -> poor fit (check residual plots)
Independence	Correlated errors may hide true performance
Homoscedasticity	Unequal variance = biased uncertainty estimates
Multicollinearity	Inflated variance of coefficients

You do not need perfection—just be aware when violations are extreme.

Residual Checks

import matplotlib.pyplot as plt
resid = y_test - pred
plt.scatter(pred, resid, alpha=.5)
plt.axhline(0, color='red')
plt.xlabel('Predicted')
plt.ylabel('Residual')
plt.title('Residual Plot')
plt.show()

Look for structure (curves, funnels) — indicates missed non-linearity or heteroscedasticity.

Common Pitfalls

Pitfall	Fix
Reporting only R²	Include an error metric (MAE/RMSE)
Fitting on unscaled data with regularization	Apply scaling first (standardize)
Ignoring outliers that dominate MSE	Inspect residual distribution
Data leakage (future info)	Ensure train/test split before feature creation
Overfitting with polynomial terms	Use cross-validation + regularization

Next Steps

Bias vs Variance Mini-View: If both training and validation errors are high, the model is underfitting (increase model flexibility or add features). If training error low but validation error high, regularize or gather more data.
Feature Scaling Note: Algorithms relying on gradient descent or distance (linear regression with regularization, k-NN, neural nets) benefit from standardized features (zero mean, unit variance) to stabilize optimization.

flowchart LR
    A[Collect Data] --> B[Train/Test Split]
    B --> C[Baseline (Mean)]
    C --> D[Linear Regression]
    D --> E[Residual Analysis]
    E --> F{Adequate?}
    F -- No --> G[Add Features / Regularize]
    F -- Yes --> H[Report & Deploy]

- Try tree-based models for non-linear patterns - Add interaction or polynomial terms carefully - Explore time-aware splits if data is temporal

Checklist

Resources

scikit-learn linear models guide
"An Introduction to Statistical Learning"