Serving and Inference

Serving and Inference

Training produces a model artifact; serving makes it available to applications at the required latency, throughput, reliability, and cost. A good serving design prevents training/serving skew, protects infrastructure, and enables monitoring & iteration.

---

1. Serving Modalities

Modality	Description	Use When	Latency	Example
Batch Scoring	Score large dataset offline	Nightly recommendations	Minutes–Hours	Generate product feed
Online Synchronous	Request/response API	Personalized UI decisions	< 200 ms	Spam filter on send
Streaming / Event	Consume events, emit predictions	Real-time fraud detection	< 500 ms end-to-end	Kafka consumer -> scorer
On-Device / Edge	Model runs locally	Offline mode / privacy	< 50 ms	Mobile image classification

Start with batch if latency isn't critical—simplest to operationalize.

---

2. CPU vs GPU (and When)

Factor	CPU	GPU
Startup cost	Low	Higher
Parallelism	Limited vectorization	Massive parallel (matrix ops)
Best for	Small / tabular models, light tree ensembles	Deep nets (vision, NLP)
Cost per hour	Lower	Higher
Elastic scaling	Easier	Harder (scarcity)

Optimization path: quantization (INT8), pruning, distillation for deep models to reduce GPU need.

---

3. Architecture Overview (Online API)

sequenceDiagram
  participant C as Client
  participant GW as API Gateway
  participant S as Model Service
  participant FS as Feature Store
  participant REG as Model Registry

  C->>GW: POST /predict
  GW->>S: Forward validated request
  S->>FS: Fetch latest features
  FS-->>S: Feature vector
  S->>REG: (On startup) Load model artifact
  S-->>GW: JSON prediction + metadata
  GW-->>C: Response

Key components: feature retrieval, model artifact loading, input validation, observability.

---

4. Avoiding Training/Serving Skew

Risk	Cause	Mitigation
Different preprocessing code	Reimplemented logic	Serialize feature pipeline (e.g. sklearn ColumnTransformer)
Time leakage	Using features not available at prediction time	Audit feature generation timestamps
Drifted feature distributions	Upstream changes	Monitor feature stats vs training baseline

---

5. Example (FastAPI Real-Time Endpoint)

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import joblib, time

class PredictRequest(BaseModel):
    feature_a: float
    feature_b: float

class PredictResponse(BaseModel):
    prediction: float
    model_version: str
    latency_ms: float

app = FastAPI()
model = joblib.load('artifacts/model.pkl')  # includes preprocessing pipeline
MODEL_VERSION = '1.0.3'

@app.post('/predict', response_model=PredictResponse)
def predict(req: PredictRequest):
    start = time.time()
    try:
        X = [[req.feature_a, req.feature_b]]
        pred = model.predict(X)[0]
    except Exception as e:
        raise HTTPException(status_code=400, detail=f'Bad input: {e}')
    latency = (time.time() - start) * 1000
    return PredictResponse(prediction=float(pred), model_version=MODEL_VERSION, latency_ms=latency)

Add: request size limits, timeout middleware, logging (trace_id), metrics (Prometheus histogram), and input schema version.

---

6. Caching & Throughput

Pattern	Use Case	Notes
In-process LRU cache	Small repeated feature lookups	Speeds hot keys
CDN / Edge caching	Public model responses (rare)	Must be cache-safe
Embedding cache	Expensive embedding generation	TTL + memory management

Batch similar requests (e.g., multiple inputs in one call) to increase GPU utilization.

---

7. Model Versioning & Canary

Strategy	Benefit
Shadow deployment	Compare new model silently
Canary (small %)	Gradual risk exposure
A/B experiment	Statistical performance comparison

Return model version in every response; log it for attribution.

---

8. Monitoring Metrics

Category	Metric Examples
System	Latency (P50/P95/P99), throughput req/s, error rate
Model	Prediction distribution, confidence score histogram
Data	Feature means/std vs baseline, missing rate
Business	Conversion lift, fraud capture rate

Set alert thresholds for change, not just absolute values.

---

9. Security & Safety

Concern	Mitigation
Prompt/Model abuse (gen models)	Content filtering, rate limiting
Input injection (text models)	Sanitize inputs, guardrails
DOS via large payload	Enforce size & timeout
Sensitive data logging	Redaction / hash selective fields

---

10. Cost Optimization

Lever	Impact
Autoscaling min/max	Avoid idle GPU burn
Model compression	Lower latency & cost
Spot/preemptible instances (batch)	Cheaper offline scoring
Request batching	Higher hardware utilization

---

11. Failure Handling

Failure	Desired Behavior
Feature store timeout	Fallback to cached features or default values
Model load failure on deploy	Fail health check → rollback
Single replica crash	Orchestrator restarts (K8s readiness/liveness probes)
Latency spike	Trigger autoscale; shed low-priority traffic

---

12. Checklist

• Preprocessing pipeline bundled with model
• Health + readiness endpoints
• Latency, throughput, error, and feature drift metrics exported
• Model version in responses & logs
• Input validation + size/time limits
• Canary or shadow strategy documented
• Secure dependency versions and container image scan
• Logging excludes PII / secrets
• Autoscaling rules tuned (scale on CPU/GPU or latency)
• Retraining trigger path defined

---

13. Further Resources

• FastAPI docs
• BentoML / Seldon Core (model serving frameworks)
• MLflow Model Registry
• Nvidia Triton (GPU optimized serving)
• Alibi Detect (drift detection)