Skip to content

Logging, Monitoring, and Observability

Logging, Monitoring, and Observability

Observability answers: "Why is it slow / failing / behaving oddly right now?" Logging alone tells stories after the fact; monitoring warns you; true observability lets you ask new, ad‑hoc questions without adding more code first.

1. The Three Pillars (and Their Roles)

Pillar Data Shape Typical Storage Primary Questions
Logs Event lines (text / JSON) Log index (e.g. Loki, Elasticsearch) What happened? Context + errors
Metrics Numeric time series TSDB (Prometheus) Is behavior within normal bounds?
Traces Spans linked by IDs Trace store (Jaeger/Tempo) Where is the latency/error occurring across services?

Modern practice: Use structured JSON logs, Prometheus metrics, and OpenTelemetry traces emitted from the same code paths sharing correlation IDs.

2. Request Correlation

Generate a unique trace_id (or reuse from reverse proxy) at request entry, then propagate via headers (e.g., traceparent / x-request-id). Include that ID in every log line and span.

sequenceDiagram
  participant C as Client
  participant GW as API Gateway
  participant S1 as Service A
  participant S2 as Service B

  C->>GW: HTTP Request
  GW->>S1: Adds trace_id
  S1->>S2: Propagates trace_id
  S2-->>S1: Response (trace annotated)
  S1-->>GW: Aggregated response
  GW-->>C: Returns trace_id header

With a trace ID the frontend or support team can paste it into dashboards to retrieve logs, metrics overlays, and the distributed trace visualization quickly.

3. Logging Guidelines

Aspect Practice Example
Format JSON, one object per line { "level":"info", "msg":"user login", "user_id":42 }
Levels Use debug < info < warn < error Limit error to actionable failures
Context Key/value pairs over string concat {"order_id":123,"duration_ms":87}
PII Avoid raw emails, tokens Hash or truncate if needed
Volume Rate-limit noisy debug Sample high-frequency events

Bad: print("User 42 failed login again!!!")
Better: 

{"ts":"2025-01-12T15:04:05Z","level":"warn","event":"auth.login.failed","user_id":42,"ip":"203.0.113.5"}

Python Example (Structure)

import json, sys, time, uuid

def log(level, **fields):
    record = {
        "ts": time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
        "level": level,
        **fields
    }
    sys.stdout.write(json.dumps(record) + "\n")

trace_id = str(uuid.uuid4())
log("info", event="order.create.start", trace_id=trace_id, order_id=101)
try:
    # ... business logic ...
    log("info", event="order.create.ok", trace_id=trace_id, order_id=101, duration_ms=53)
except Exception as e:
    log("error", event="order.create.error", trace_id=trace_id, order_id=101, error=str(e))

4. Metrics Fundamentals

Type Use Example
Counter Ever-increasing count http_requests_total
Gauge Arbitrary up/down value queue_depth
Histogram Distribution (latency, sizes) request_latency_seconds_bucket
Summary Client-side quantiles (avoid on Prometheus server) Not widely recommended for backend latency

Expose metrics endpoint (e.g., /metrics) and scrape with Prometheus.

Example (Python + Prometheus client)

from prometheus_client import Counter, Histogram, start_http_server
import time, random

REQS = Counter('api_requests_total', 'Total API requests', ['route'])
LAT = Histogram('api_request_latency_seconds', 'Latency', ['route'])

def handle(route):
    with LAT.labels(route).time():
        time.sleep(random.random()/20)
    REQS.labels(route).inc()

if __name__ == '__main__':
    start_http_server(9100)
    while True:
        handle('/projects')

5. Alerting Philosophy

Alerts should be: Actionable, Urgent, Owned. Avoid noisy alerts = alert fatigue.

Anti-Pattern Fix
Alert on every 500 Alert on error rate > X% for Y mins
CPU > 80% single sample Use sustained window + correlate with latency
Paging on low-priority batch failure Triage via daily report

Define SLO (Service Level Objective): e.g., 99.5% of requests under 300ms over 30 days. Alert when error budget burn rate spikes (fast consumption of the allowed failure budget).

6. Tracing Deep Dive

Trace = tree of spans. Each span: operation name, start time, duration, attributes, status.

Common span attributes: http.method, http.route, db.statement (sanitized), error flag. Propagate traceparent header (W3C standard) across service boundaries.

Benefits: * Pinpoint slow service or DB call. * Visualize fan-out patterns causing latency. * Combine with logs via trace_id for rich debugging.

7. Combining Pillars (Example Flow)

  1. User action triggers request (trace started).
  2. Middleware starts span, adds trace_id to logging MDC (mapped diagnostic context).
  3. Each downstream call creates child span (DB, cache, external API).
  4. Logs automatically include trace_id.
  5. Metrics histogram records latency buckets.
  6. On error, span status = error + log entry at error level triggers alert if rate threshold exceeded.

8. Dashboards & Visual Overlays

Essential starter panels: * Request rate (per route) + error rate overlay. * P95 and P99 latency with deploy markers. * Saturation: CPU, memory, queue depth. * Top error codes & top slow queries.

Overlay deployments (Git commit hash) so regressions correlate with release events.

9. Maturity Ladder

Level Characteristics Next Step
0: Blind Logs only on local dev Add centralized logging
1: Basic Centralized logs + basic metrics Add tracing & structured logs
2: Correlated Trace IDs unify logs/metrics/traces Define SLOs & alerting burn rates
3: Proactive SLO-based alerts, chaos drills Add anomaly detection, auto-rollbacks

Aim for Level 2 early in a project: biggest insight ROI.

10. Costs & Sampling

High cardinality (e.g., user_id labels) explodes storage cost. Strategies: * Sample traces (keep all errors + small % of healthy).
Aggregate metrics labels sparingly (avoid free-form labels).
 Log at info only essential business events; push verbose details to debug and sample them.

11. Local Development Tips

  • Run a lightweight stack: docker-compose with Prometheus + Grafana + Tempo (or Jaeger) + Loki.
  • Use OpenTelemetry SDK auto-instrumentation to bootstrap quickly.
  • Practice reading a trace for a known bug to build team muscle memory.

12. Checklist

  • Structured JSON logs with trace_id
  • Central log aggregation configured
  • Key service & route latency histograms
  • Error rate + latency SLO defined
  • Alert rules based on burn/error thresholds
  • Distributed tracing enabled & propagated headers
  • Dashboard: rate, errors, latency, saturation
  • Sampling rules (errors full; success partial)
  • Deployment markers on charts
  • Run book for top 3 alerts

13. Quick Reference

Need Look At
Which service is slow? Trace waterfall
Sudden 500 spike Error rate panel + recent deploys
Intermittent latency P95/P99 vs CPU/memory overlay
User reported issue Search logs by trace_id or user id (hashed)
Capacity planning Saturation panels + growth trend

14. Resources

  • https://opentelemetry.io/
  • Prometheus + Grafana docs
  • "Distributed Systems Observability" (Cindy Sridharan)
  • Loki / Tempo (Grafana OSS stack)
  • Honeycomb (high-cardinality observability SaaS)