CAP and PACELC

Every system-design interview eventually asks about CAP. The honest answer — the one I wish more candidates would give — is that the famous theorem is technically correct, often misquoted, and not nearly as useful for building real systems as the follow-up theorem (PACELC) that almost nobody talks about. Both deserve their own paragraphs, because the gap between them is where most real architectural decisions actually live.

Eric Brewer published CAP as a conjecture in 2000 and it was formalized as a theorem two years later. It says, simplified to death: when a distributed system suffers a network partition (the P), it must choose between consistency (the C) and availability (the A). You cannot have both. The math is rigorous; the practical implications get scrambled the moment you try to explain it quickly.

The misquote I see most often is "CAP says pick two of three." It doesn't. Partitions are not optional in distributed systems — networks fail, packets drop, the cloud provider has a bad day. You don't pick the P. The P happens. CAP is really saying: "when (not if) the P happens, are you willing to trade C or A?" The choice is forced, and your job as an architect is to know which side of it your system falls on, on purpose, with eyes open.

The two real corners

When you accept that partitions will happen, only two corners of the triangle are actually inhabitable.

CP systems preserve consistency at the cost of availability. If a partition splits the cluster, the side that can't reach a quorum refuses to serve writes (and often refuses to serve reads too, to avoid returning stale data). The system effectively goes into a "no, sorry, come back later" state on the affected nodes until the partition heals. etcd, Zookeeper, Spanner, and almost every consensus-based store live here. The user-facing implication is that the system sometimes says "no" — slowly, with a timeout — and that's by design.

AP systems preserve availability at the cost of consistency. When a partition splits the cluster, every node continues to accept reads and writes locally. The system stays up; the trade-off is that writes made on one side of the partition don't reach the other side until it heals, and now you have divergent data that somebody has to reconcile. DynamoDB, Cassandra, Riak, and most geographically-distributed caches live here. The user-facing implication is that reads might show stale data, writes might collide, and the application has to handle conflict resolution.

The third corner — CA — is sometimes drawn on slides for completeness. It isn't a real option in a distributed system, because there's no way to opt out of partitions. Single-node databases are technically "CA" only because they have no other nodes for a partition to separate them from. The moment you add a replica, you're back to choosing between CP and AP.

PACELC: the question CAP doesn't answer

The thing CAP leaves out is what your system does the 99% of the time when nothing is partitioned. PACELC fills that gap with a slightly clunky but powerful framing: if there's a partition, you choose between A and C; else (the normal case), you choose between L (low latency) and C.

The "else" trade-off is the one you live with every day. A read across regions, with the goal of strongest possible consistency, costs a cross-region round-trip — 50-200 ms. A read from the closest replica with whatever it has costs maybe 2 ms. Both are "correct" answers to "what's this value"; they just describe different versions of correct.

The pattern: CP systems are also EC (else-consistent), and AP systems are also EL (else-latency). That's not coincidence — the same engineering trade-offs that make a system safe under partition also make it slow when there's no partition, and vice versa. Reading either column tells you most of what you need to know about what a database feels like in production.

The bumper sticker is "CAP says pick two." The real question is: "When the network partitions, will my users see stale data, or will they see an error?" Both are acceptable answers — neither is free.

What the "else" actually costs

The "else" column isn't abstract — it shows up as latency on every request, all day, partition or no partition. Run the numbers before you reach for a strongly-consistent default.

Say a single API request fans out to 6 sequential reads against your store (a profile, its settings, two permission checks, a feature flag, a counter — utterly ordinary). Compare a close-replica read against a strong cross-region read with the regions ~80 ms apart:

That's the same data, the same product, the same database — and a 40× difference in tail latency on a single page, driven entirely by the else-pick. The trap is paying that 480 ms on reads that never needed to be strong (a display counter, a recommendation). The skill PACELC trains is spending your consistency budget only where a stale read actually costs something.

The corner is not the whole answer

A senior engineer's instinct, after a few years, is to stop classifying systems by corner and start asking which trade-off each feature makes. Most production systems mix both. A bank might use Spanner (CP) for the ledger and DynamoDB (AP) for the session store. A SaaS app might use CockroachDB (CP) for billing and Cassandra (AP) for activity feeds. The label on the bottle matters less than what you're using it for.

The question to ask of each operation, not each system, is: if the network partitions right now, what do I prefer this request to do? The three reasonable answers are:

• Succeed with a stale value (AP). Right for product price display, recommendation widgets, social-graph counts, anything where "approximately right" is better than "nothing".
• Succeed with the latest value, even if it costs blocking for 30 seconds (CP). Right for payments, inventory commitments, anything where wrong-but-fast is worse than slow-but-right.
• Fail loudly so the user knows to retry (CP, in availability terms). Right for write operations that can't be safely retried later — better an honest 503 than a successful response that hides a divergence.

The same database can serve all three depending on how you configure each query. DynamoDB does eventually-consistent and strongly-consistent reads in the same table. Cassandra's R+W>N is the per-query knob. MongoDB has readConcern and writeConcern. The system gives you the levers; the architectural work is knowing which feature pulls which lever.

A war story about the wrong default

A team I worked with built a multi-region inventory service on Cassandra, configured for AP "because that's what Cassandra is". Their happy path worked great. Then their first cross-region partition happened and they discovered, simultaneously, that:

1. Both regions had been accepting reservations against the same physical inventory, because each region thought it was the authoritative copy during the partition.
2. When the partition healed, they had two reservation records for several items where they only had one item in stock.
3. The last-write-wins conflict resolution Cassandra defaulted to was technically deterministic, in the sense that both regions eventually agreed on which reservation to keep — but the user whose reservation got overwritten only found out when their order shipped a different item, or didn't ship at all.

The eventual fix had three parts. They added a CP-flavoured secondary store for the actual inventory count, so reservations could check a globally-coordinated value before committing. Reservation history stayed in Cassandra (eventually consistent is fine for an audit trail). And the inventory-count writes used W = quorum so a partition that left a region in the minority would refuse to commit reservations until the partition healed — making the user see a 503 instead of an order that mysteriously disappeared.

The lesson is the one most teams take a year too long to learn: the CAP corner of your database doesn't pick the CAP corner of your feature. You have to design at the feature level, even when the storage is uniform.

When you don't need to think about CAP

If you're running a single-region service with a single database (possibly replicated for HA, possibly not), you can stop reading this lesson. CAP is genuinely a story about multi-region or multi-cluster systems where partitions are common enough to plan for. A normal small-to-mid-size production system that lives inside one AZ rarely experiences partitions and rarely needs to make these trade-offs explicit.

You'll know you're crossing into CAP territory when one of two things happens: latency requirements push you to multi-region deployment, or compliance requirements demand data residency in specific regions. At that point, you're picking between CP and AP whether you call it that or not.

[CONCEPT]consensus-raft is the formal machinery underneath most CP systems — it's how they decide whether a quorum exists. [CONCEPT]replication explains the durability story that underlies both corners. And [CONCEPT]distributed-transactions is what you reach for when you need stronger guarantees than CAP alone provides.