ML Project Lifecycle

Building an ML system is not just “train a model.” It’s a loop of defining, building, evaluating, deploying, and monitoring—with careful documentation at each step to enable reproducibility and safe iteration.

1. High-Level Flow

flowchart TD
    A[Problem & Metric] --> B[Data Collection & Labeling]
    B --> C[Exploration & Validation]
    C --> D[Feature Engineering]
    D --> E[Model Training & Tuning]
    E --> F[Evaluation & Error Analysis]
    F --> G[Deployment]
    G --> H[Monitoring & Drift Detection]
    H --> B

This is iterative: monitoring uncovers drift → triggers new data labeling → retraining.

2. Stage Details

Stage	Goal	Typical Artifacts	Common Risks
Problem & Metric	Define objective & success criteria	Problem brief, metric spec	Vague target → misaligned optimization
Data Collection & Labeling	Acquire representative labeled data	Raw dataset snapshot, label guidelines	Sampling bias, noisy labels
Exploration (EDA)	Understand distributions & issues	EDA notebook, data quality report	Hidden leakage, unbalanced classes
Feature Engineering	Transform raw → model-ready	Feature scripts, schema docs	Undocumented transformations
Training & Tuning	Learn parameters; choose model	Training logs, model weights, config	Overfitting, non-reproducible runs
Evaluation & Error Analysis	Assess & understand failures	Metrics table, confusion matrices, error slices	Using wrong / single metric
Deployment	Serve model predictions	Packaged model + preprocessing bundle	Env mismatch, missing dependency
Monitoring & Drift	Detect performance decay, drift	Dashboards, drift reports, alerts	Silent degradation, stale thresholds

3. Problem Framing & Metric Selection

Write a one-paragraph problem statement: inputs, desired output, users impacted, decision using the output. Define primary metric + guardrail. Example (spam filter): Primary = Recall@0.95 Precision; Guardrail = False positive rate under 2%.

Avoid metric mismatch: optimizing accuracy on 2% positive class is misleading; use PR-AUC or recall at fixed precision.

4. Data Collection & Labeling

Consideration	Guiding Question
Coverage	Do we include all user segments/time periods?
Label Quality	Clear instructions? Inter-annotator agreement measured?
Refresh Cadence	How often does data drift?
Privacy	Are we storing only necessary attributes?

Track dataset version: create a hash or manifest of file list + counts.

5. Train / Validation / Test Split

Split	Purpose	Notes
Train	Fit model parameters	Often 60–80%
Validation	Tune hyperparameters & early stopping	Keep untouched until tuning
Test	Final unbiased evaluation	Use only once per cycle

Special cases: Time series → split chronologically; user-centric tasks → group by user to prevent leakage.

6. Feature Engineering

Transform raw data into informative signals. Keep code deterministic & versioned. Examples: * Aggregations: past 7-day event counts * Normalization / scaling * Text vectorization (TF-IDF or embeddings) * Missing value indicators

Document a feature schema: name, type, description, transformation source.

7. Model Training & Hyperparameter Tuning

Aspect	Best Practice
Random Seeds	Set seeds for reproducibility (model + data split)
Search Strategy	Start with coarse grid/random; refine
Early Stopping	Prevent overfitting when validation loss plateaus
Regularization	Simplify model and improve generalization

Keep a config file (YAML/JSON) capturing: model type, hyperparameters, feature version, data split seed.

8. Evaluation & Error Analysis

Move beyond a single aggregate score. | Technique | Purpose | |-----------|---------| | Confusion matrix | See error types (FP vs FN) | | Slice analysis | Performance across user segments | | Calibration plot | Probability reliability | | Residual plots (regression) | Detect non-linearity / heteroscedasticity |

Perform counterfactual checks: modify one feature to see if prediction changes sensibly.

9. Model Documentation (Model Card Starter)

## Model Card: Spam Filter v1.2
Purpose: Classify emails as spam/ham.
Data: 120k labeled emails (Jan–Mar 2025), stratified by source.
Primary Metric: Recall@0.95 Precision (achieved 0.91 at 0.95 precision).
Guardrails: FP rate <2%; latency <50ms P95.
Limitations: Under-represents non-English emails (<3%).
Ethical Considerations: False positives hide legitimate emails; user appeal process required.
Versioning: Model hash abc123; feature pipeline commit 4f9e2; dataset manifest dataset_v5.json.

10. Deployment Patterns

Pattern	Use Case	Notes
Batch Scoring	Nightly recommendations	Low latency not required
Online API	Real-time user decisions	Need low latency & autoscaling
Edge / On-device	Privacy or offline	Model size constraints
Streaming	Continuous risk scoring	Integrate with event bus

Package preprocessing + model together to avoid training/serving skew.

11. Monitoring & Drift

Drift Type	Signal	Example Metric
Data Drift	Input distribution shifts	Population Stability Index (PSI)
Concept Drift	Relationship input→target changes	Decline in primary metric
Prediction Drift	Output distribution shifts	Mean probability vs historical

Set alert thresholds in coordination with business impact (avoid noisy alerts). For classification: track recall/precision on a labeled rolling sample if feasible.

12. Automated Retraining Triggers

Trigger	Threshold Example
Metric decay	F1 drops >5% absolute from baseline
Data drift	PSI > 0.2 on core feature
Volume shift	Input volume ±30% sustained 24h

Ensure a human review step before promotion unless risk is minimal.

13. Governance & Reproducibility Artifacts

Artifact	Content
Dataset Manifest	File list, counts, hash, schema
Feature Schema	Names, types, derivations
Training Config	Hyperparams, seed, feature version
Model Weights	Serialized model + hashing
Evaluation Report	Metrics, slice analysis, date
Model Card	Human-readable summary

Store artifacts in a structured directory (e.g., artifacts/run_2025_09_28/).

14. Risks & Mitigations

Risk	Impact	Mitigation
Hidden bias in data	Unfair decisions	Diverse sampling; bias metrics
Data leakage	Inflated metrics	Strict split before transformations
Skew train vs serve	Poor live performance	Serialize preprocessing pipeline
Silent model decay	Business impact	Continuous monitoring & alerts
Irreproducible experiment	Cannot debug	Version configs & data hashes

15. Quick Start Example (Mini Pipeline Pseudocode)

def load_data(): ...
def split(df): ... # returns train, val, test
def build_features(df): ...
def train(X_train, y_train, cfg): ...
def evaluate(model, X_val, y_val): ...
def package(model, feature_pipeline): ...

raw = load_data()
train_df, val_df, test_df = split(raw)
X_train, y_train = build_features(train_df)
X_val, y_val = build_features(val_df)
model = train(X_train, y_train, cfg)
metrics = evaluate(model, X_val, y_val)
if metrics['f1'] >= cfg['min_f1']:
        package(model, 'feature_pipeline.pkl')

16. Checklist

17. Further Resources

Google ML Crash Course
Model Cards (Mitchell et al.)
"Rules of Machine Learning" (Google)
MLOps community guides (mlops.community)
Practical Monitoring (for alert design)