Designing for Failure

Everything up to this point has assumed the happy path. The actual measure of a system design — the thing interviewers probe hardest and the thing that determines whether a 3am page happens — is what it does when a component fails, because in a large enough system, something is always failing.

Redundancy: eliminate single points of failure

Every component in the diagrams throughout this guide — the load balancer, the database primary, the cache — is a single point of failure if there's only one of it. The fix is the same pattern every time: run more than one, with a way to detect failure and fail over.

Single point of failure:        Redundant:
                                 ┌─────────┐
  ┌─────────┐                   │Primary  │
  │ Server  │  ← if this dies,  └────┬────┘
  └─────────┘    everything          │ failover
                  is down       ┌────┴────┐
                                │ Standby │
                                └─────────┘

The circuit breaker pattern

When service A calls service B, and B starts failing or timing out, the naive behavior — A keeps calling B, waiting for each timeout — makes things worse: A's own threads/connections pile up waiting on a dead service, and A goes down too. A circuit breaker wraps the call to B and tracks its failure rate; once failures cross a threshold, the breaker "trips" and A fails fast (returns an error immediately, or a fallback) without even attempting the call, for a cooldown period, then allows a trial request through to check if B has recovered.

CLOSED (normal) ──[failure rate > threshold]──→ OPEN (fail fast)
    ↑                                                  │
    └────[trial request succeeds]── HALF-OPEN ←────[cooldown elapses]

This single pattern is what stops one struggling service from taking down every service that depends on it — a cascading failure, historically one of the most common causes of major outages.

Retries need backoff (and jitter)

A naive retry ("try again immediately") under a real outage makes it worse — thousands of clients retrying simultaneously creates a synchronized traffic spike (a "thundering herd") right as the struggling service is trying to recover. The fix is exponential backoff with jitter: wait progressively longer between retries, with a random component so clients don't retry in lockstep.

import random
import time

def retry_with_backoff(fn, max_attempts=5, base_delay=0.5):
    for attempt in range(max_attempts):
        try:
            return fn()
        except TransientError:
            if attempt == max_attempts - 1:
                raise
            delay = base_delay * (2 ** attempt) + random.uniform(0, 0.5)
            time.sleep(delay)

Health checks

A load balancer or orchestrator can only route around a dead node if it actually knows the node is dead. A health check is a lightweight endpoint (commonly /healthz) that reports whether an instance is ready to serve traffic. Two checks are worth distinguishing:

• Liveness — "is the process still running?" A failed liveness check usually means "restart me."
• Readiness — "can I currently handle traffic?" A service can be alive but not ready (e.g. still warming up a cache, or its database connection pool is exhausted) — readiness checks keep traffic away from it without killing the process.

Graceful degradation

When a dependency fails, the options aren't just "fully working" or "fully down." A product page whose recommendation service is unavailable can still render the product itself, just without the "related items" section. Designing which features are load-bearing versus optional ahead of time — and having an explicit fallback for the optional ones — turns a dependency outage into a degraded experience instead of a full outage.

The bulkhead pattern

Named after ship design — a bulkhead is a wall that stops a hull breach in one compartment from flooding the entire ship. Applied to systems: isolate resources (thread pools, connection pools) per dependency, so a slow or failing downstream service exhausts only the resources allocated to it, not every resource in the calling service.

Without bulkheads:              With bulkheads:
  ┌─────────────────┐             ┌──────┬──────┬──────┐
  │  shared pool of  │             │ pool │ pool │ pool │
  │  100 connections │             │  A   │  B   │  C   │
  │  (any dependency │             │ (30) │ (30) │ (40) │
  │  can exhaust it) │             └──────┴──────┴──────┘
  └─────────────────┘        one slow dependency can only
                              ever consume its own slice

Timeouts

A call with no timeout is a call that can hang forever, quietly consuming a thread or connection while it waits. Every network call — every single one — needs an explicit timeout. This sounds obvious and is still one of the most common real-world causes of cascading failure, because the default in most HTTP clients and database drivers is either no timeout or one set far too high to be useful.

Chaos engineering

The natural extension of "design for failure": if failure is inevitable, test your system's failure handling deliberately, in a controlled way, instead of discovering it for the first time during a real incident. Chaos engineering means intentionally injecting failure — killing a random instance, adding artificial network latency, cutting off a dependency — in a non-production (or carefully scoped production) environment, and verifying the system degrades the way you designed it to. Netflix's Chaos Monkey, which randomly terminates production instances, is the best-known example of this practice taken seriously.

The single most useful mental shift in this entire guide: don't ask "how do I build this so it never fails" — nothing built by people, running on machines, connected by networks, ever achieves that. Ask "how does this behave, and recover, when the failure that's guaranteed to eventually happen, happens." That question is what separates a system design from a happy-path diagram.