MLOps Basics

MLOps brings software engineering discipline to machine learning systems: automation, versioning, reproducibility, testing, continuous delivery, and monitoring. The goal is to shorten iteration time while maintaining reliability and traceability.

1. Why MLOps?

Pain Without MLOps	Consequence	MLOps Countermeasure
"Which code produced this model?"	Lost reproducibility	Versioned configs + model registry
Manual training steps	Human error, slow iteration	Automated pipelines / CI triggers
Drift undetected	Silent accuracy decay	Monitoring dashboards + alerts
Notebook-only experiments	Hard to productionize	Modular, testable feature & train code

2. Core Lifecycle with MLOps Lens

flowchart LR
    A[Data Ingest] --> B[Feature Pipeline]
    B --> C[Train & Tune]
    C --> D[Evaluate]
    D --> E[Register Model]
    E --> F[Stage/Canary Deploy]
    F --> G[Monitor & Drift]
    G --> H{Retrain Trigger}
    H --> B

3. Key Components

Component	Purpose	Tooling Examples
Version Control	Track code + configs	Git, GitHub/GitLab
Data Versioning	Immutable dataset lineage	DVC, Delta Lake, LakeFS
Feature Store	Reuse & consistent features (online/offline)	Feast, Tecton
Experiment Tracking	Log params, metrics, artifacts	MLflow, Weights & Biases
Orchestration	Define & schedule pipelines	Airflow, Prefect, Dagster
Model Registry	Manage promote/archived states	MLflow Registry, SageMaker Registry
CI/CD	Automate tests/build/deploy	GitHub Actions, GitLab CI
Monitoring	Detect drift & performance drops	Prometheus + custom, Evidently, WhyLabs

4. Reproducibility & Experiment Tracking

Aspect	Practice	Tooling
Randomness	Set + log seeds	`numpy.random.seed(42)`
Environment	Lock dependencies	`requirements.txt`, `poetry.lock`
Data Snapshot	Hash or manifest + storage path	DVC, object store versioning
Code Version	Git commit hash embedded	Auto-injected at run start
Config	Structured YAML/JSON	Hydra, OmegaConf
Artifacts	Persist model + preprocessing	MLflow artifacts, S3 bucket
Metrics	Auto-log scalar & curves	Tracking UI dashboards

Minimal reproducibility recipe: 1. Never overwrite raw data.
2. Log training config (hyperparameters + feature list).
3. Store model + preprocessing pipeline atomically.
4. Record git commit + dataset hash inside the model metadata.

5. Pipeline Design Principles

Principle	Description	Example
Determinism	Same inputs → same outputs	Fixed seeds; pure functions
Idempotency	Re-running doesn't corrupt state	Write to versioned output dirs
Modularity	Swap steps independently	Separate feature build vs train step
Observability	Each step logs metrics & duration	Step-level timing metrics
Failure Isolation	One failing step doesn't corrupt others	Retry policies per step

6. Model Registry Workflow

State	Meaning	Allowed Transitions
None (new)	Not yet tracked	→ Staging
Staging	Candidate under evaluation	→ Production / Archived
Production	Actively serving	→ Archived / Staging (rollback)
Archived	Deprecated; kept for audit	→ (usually end)

Promotion requires: evaluation report attached, bias & slice checks reviewed, approval recorded (PR or ticket).

7. Integrating with CI/CD

Pipeline Stage	Automation Example
Lint & Unit Tests	`pytest`, static type checks on feature code
Data Validation	Schema / drift check step (Great Expectations)
Train Job Trigger	On merged PR or scheduled cron
Evaluation Gate	Compare metrics vs baseline; fail if worse
Packaging	Build Docker image embedding model & pipeline
Deployment	Canary release via infrastructure as code

8. Testing Layers (Beyond Accuracy)

Test Type	Scope	Example
Unit	Individual transforms	Scaling function returns expected shape
Data Quality	Input schema/values	No new nulls in key features
Integration	End-to-end pipeline	Train script produces model + metrics JSON
Regression (Model)	Compare performance	New model F1 not < baseline - 2%
Fairness / Bias	Slice metrics	FPR parity across groups within threshold

9. Monitoring & Drift Detection

Category	Metric Examples	Tooling
Data Drift	PSI, KL divergence on features	Evidently, WhyLabs
Prediction Drift	Output distribution shift	Custom metrics store
Performance	Primary metric on labeled sample	Shadow labeling pipeline
Operational	Latency, error rate	Prometheus, Grafana

Alert on rate of change (burn of error budget) rather than static thresholds only.

10. Automated Retraining Strategy

Pick triggers conservatively to avoid thrash. Example policy:

If (PSI(feature_x) > 0.2 OR F1_drop > 5% absolute) for 3 consecutive days
THEN queue retraining job using latest approved data snapshot.

Always produce a comparative evaluation (new vs production) before promotion.

11. Security & Compliance Touchpoints

Area	Consideration	Action
Data Access	Principle of least privilege	Scoped IAM roles
Sensitive Features	PII handling	Mask/encrypt before logging
Artifact Integrity	Tampering risk	Sign model artifacts (hash)
Audit Trail	Who promoted model?	PR + registry event log

12. Example Lightweight Makefile Targets

make data      # download / stage raw data
make features  # generate feature parquet
make train     # train model, log to MLflow
make eval      # compare against baseline
make package   # bundle model + preprocessing
make deploy    # push image & update service

13. Example Directory Layout

ml_project/
    data/
        raw/
        processed/
    features/
        build_features.py
    models/
        registry_metadata.json
    pipelines/
        train_pipeline.py
    configs/
        train.yaml
    notebooks/
        exploration.ipynb
    scripts/
        evaluate.py
    Makefile

14. Checklist

15. Further Resources

dvc.org (Data versioning)
mlflow.org (Tracking & registry)
weightsandbiases.com (Experiment tracking)
Feast (Feature store)
"Building Machine Learning Pipelines" (Hannes Hapke & Catherine Nelson)
MLOps community (https://mlops.community)