Consistency Models and the CAP Theorem

This is the theory that explains why distributed databases force trade-offs instead of just being "fast and correct and always available" — and once you understand it, most database marketing claims become easy to see through.

The CAP theorem

In a distributed system, when a network partition happens (some nodes can't talk to others — and on a large enough network, this is a when, not an if), you must choose between:

• Consistency (C) — every read gets the most recent write, or an error.
• Availability (A) — every request gets a response, even if it might not be the latest data.

You cannot have both during a partition. This gives systems a rough classification:

CP systems  (consistent, sacrifice availability during a partition)
  → e.g. traditional relational databases in a strict-consistency config,
    ZooKeeper, etcd — used where being wrong is worse than being down
    (leader election, financial ledgers, configuration state)

AP systems  (available, sacrifice strict consistency during a partition)
  → e.g. Cassandra, DynamoDB (default mode) — used where being briefly
    stale is fine, but being down is not (shopping carts, social feeds,
    view/like counters)

Note what CAP is actually about: behavior during a network partition, not general system design. When the network is healthy (the common case), most systems are both available and consistent — CAP only forces a choice in the failure case, which is exactly why it's so often misunderstood.

Consistency isn't binary

In practice, "consistency" is a spectrum, and most real systems pick a specific point on it deliberately:

• Strong consistency — every read reflects the most recent write. Expensive: usually requires coordination between nodes on every operation.
• Eventual consistency — replicas converge to the same value eventually, given no new writes. Cheap and highly available, but a read right after a write might return stale data.
• Read-your-writes consistency — a practical middle ground: a user always sees their own writes immediately (routed to the primary or a replica guaranteed to be caught up), even if other users see them with a short delay.

The URL shortener from Chapter 1 is a textbook AP + eventually-consistent system: a few seconds of delay before a new link resolves on every replica is invisible to users, and the availability of the redirect path matters far more than perfect consistency.