Resilience and Reliability Patterns

Building reliable services means planning for things to break: networks partition, dependencies slow down, caches evict, threads deadlock. Resilience patterns give you graceful degradation instead of hard failure.

1. Core Concepts

Term	Meaning
Fault	A defect or failure of a component
Failure	Service behavior no longer meets expectation
Resilience	Ability to recover / continue under faults
Graceful Degradation	Partial functionality instead of total outage
Backpressure	Signaling upstream to slow down

Mental model: Think in terms of "bulkheads" on a ship—compartments isolate flooding.

2. Typical Failure Modes

Slow dependency (adds latency)
Dependency returns errors (burst or sustained)
Network partition / DNS failure
Resource exhaustion (threads, file handles, memory)
Thundering herd after outage (all clients retry simultaneously)

Document discovered modes in postmortems to build playbooks.

3. Patterns Overview

Pattern	Problem Addressed	Key Idea
Timeout	Infinite waiting	Fail fast after N ms
Retry w/ Jitter	Transient errors	Try again w/ backoff randomness
Circuit Breaker	Cascading failure	Stop calling when failure rate high
Bulkhead	Resource isolation	Limit concurrency per dependency
Fallback / Cache	Partial unavailability	Serve stale / reduced data
Rate Limiting	Overload	Shed excess load early
Load Shifting	Hot spots	Distribute traffic to healthy nodes
Hedging (duplicate)	Tail latency	Send second request after delay

4. Timeouts

Always set explicit timeouts for outbound calls (HTTP, DB queries). A few fast failures are better than threads piling up. Tune timeouts slightly above typical P99 latency of dependency.

5. Retries (with Backoff & Jitter)

Good for transient network glitches or 503 responses. Bad for: non-idempotent operations (double-charging a card) unless you use an idempotency key.

Pseudo-code (exponential backoff + full jitter):

import random, time

def retry(op, retries=3, base=0.1):
    for attempt in range(retries):
        try:
            return op()
        except TransientError as e:
            sleep = random.uniform(0, base * (2 ** attempt))
            time.sleep(sleep)
    raise

6. Circuit Breaker

States: CLOSED (normal) → OPEN (failing fast) → HALF-OPEN (probe). Reduces pressure on a failing dependency so it can recover.

graph LR
  A[CLOSED] -- failure threshold reached --> B[OPEN]
  B -- after cool-down --> C[HALF-OPEN]
  C -- probe success --> A
  C -- probe failure --> B

Track: failure rate, open counts, mean open duration.

7. Bulkheads

Limit concurrency per dependency: e.g., allocate a separate thread/connection pool. A slow analytics call should not starve login API threads.

8. Fallbacks & Graceful Degradation

Scenario	Fallback
Recommendation service down	Serve popular items list
User profile picture CDN failing	Show default avatar
Feature flag service slow	Use last known snapshot

Log when fallback used (warn level) to measure impact.

9. Rate Limiting & Load Shedding

If you process beyond safe capacity, everyone suffers. Use a token bucket or leaky bucket algorithm. When near saturation: shed lower-priority requests with 429 before the system collapses.

10. Tail Latency Mitigation

Hedging: after waiting P95 latency, send a second request to another replica; use first successful response. Beware increased load; apply only to idempotent GETs.

11. Observability Hooks

Expose metrics: * dependency_latency_seconds (histogram)
dependency_failures_total (counter)
circuit_state (gauge)
* fallback_invocations_total

Dashboards should highlight: error rate % vs SLO, open circuit count, retry attempts per minute.

12. Chaos & Game Days

Inject failure intentionally (chaos monkey, fault injection proxies) to validate that patterns behave as expected. Start in staging. Document surprising outcomes.

13. Checklist

14. Further Resources

"Release It!" (Michael Nygard) – foundational patterns
Polly (.NET), Resilience4j (JVM), Envoy (circuit breaking)
Google SRE Workbook – error budget concepts