Consensus and Raft

There's a problem at the bottom of every distributed database, every service-discovery system, every Kubernetes control plane, that goes something like this: five computers, sitting in different racks, have to agree on a single ordered list of events. Not "mostly agree" — identically agree. A network blip can't make two of them think different things are committed. A slow GC pause can't either. A crashed and restarted node has to come back and pick up the same list everyone else has, with no gaps. The problem sounds simple. It is one of the deepest problems in distributed systems, and algorithms for solving it cleanly only emerged in the late 1980s (Paxos) and the mid 2010s (Raft).

The reason this matters for you, as someone building systems on top of databases like etcd, CockroachDB, or Spanner, is that everything those systems guarantee — leader election, strong consistency, no-data-loss failover — bottoms out in a consensus algorithm. When the marketing page says "we use Raft" or "we use Paxos" they mean: trust us, your data won't go to two different places at once. Understanding the algorithm a little bit lets you read those guarantees with the right eyes.

What Raft is actually doing

Strip Raft down to its essence and there are only two ideas: a single leader at any moment, and a majority quorum for every commit. Everything else is mechanics.

A Raft cluster picks one node to be the leader. Every write goes through the leader. The leader appends the write to its own log, then ships the entry to every follower via an AppendEntries RPC. Each follower writes the entry to its own log and acks. Once a majority of nodes — including the leader — have the entry in their logs, the leader declares it committed and replies to the client. The majority quorum is what makes it safe; nothing is committed until enough nodes agree to remember it that no minority can lose it.

In steady state, the leader sends a heartbeat (an empty AppendEntries) every 50-150 ms so followers know it's still alive. If a heartbeat is missed for longer than the election timeout (typically 150-300 ms, randomized to avoid split votes), followers assume the leader is dead and start an election.

Why "majority" and why odd numbers

The majority rule is what protects against split-brain. If a network partition splits a 5-node cluster into a 3-node side and a 2-node side, the 3-node side can still reach majority and keep serving writes; the 2-node side can't reach 3 acks and refuses to elect a new leader or commit anything. When the partition heals, the 2-node side's logs get overwritten by the leader's. There's exactly one history; minorities don't get to write it.

This is also why every Raft tutorial nags you about odd-sized clusters. The fault tolerance of an N-node cluster is floor((N-1)/2). A 3-node cluster tolerates 1 failure. A 4-node cluster also tolerates 1 failure, because it still needs 3 acks (floor(4/2)+1 = 3). You paid for an extra node and got no extra safety. Pick 3 for small clusters, 5 for production clusters, rarely 7 — past 5, the marginal availability gain is small and the leader's AppendEntries chatter goes up linearly.

Leader election in detail

When a follower's election timeout fires, it does three things in order: bumps its term number, votes for itself, and sends RequestVote to every other node. A node that receives a vote request grants its vote if and only if two conditions hold:

1. It hasn't already voted in this term. (Each node votes at most once per term — this prevents two candidates from both winning.)
2. The candidate's log is at least as up-to-date as the voter's own. (Up-to-date means: the candidate's last log entry has a higher term, or has the same term but a higher index.)

The second condition is the subtle one and it's the entire safety story. It guarantees that any node that wins an election has every entry that was committed under previous leaders. A node that's behind on the log can't gather a majority of votes, because the nodes ahead of it on the log will vote no. The new leader's log might contain uncommitted entries from the old leader that need to be cleaned up, but no committed entry can ever be lost.

Election timeouts are randomized (e.g., uniformly distributed between 150 and 300 ms) specifically to prevent split votes. If two followers had identical timeouts they'd both candidate at the same instant and split the votes; with randomization, one candidates ~10ms before the other and usually wins outright. If a split vote does happen, both candidates wait for their next randomized timeout and try again.

What can go wrong and how Raft handles it

The reason production systems trust Raft is that the algorithm has been pulled apart and put back together by enough people that every failure mode has a defined behavior. A few cases worth seeing:

The leader crashes after the client sends a write but before the leader replicates it. The write is lost. The client either times out and retries (idempotent operations only — see [CONCEPT]idempotency) or sees a server error. There's no inconsistency; the entry was never committed.

The leader crashes after replicating but before responding to the client. This is the nasty one. The entry might be committed (replicated to majority) but the client doesn't know. The client retries; if the operation is idempotent (write with the same key), no harm. If it's not idempotent (POST /orders without an idempotency key), the order is created twice. Raft alone doesn't solve this — the application has to.

A network partition isolates the leader. The leader is in the minority and can't reach majority for new writes. It stops committing. The majority side elects a new leader (after election timeout) and continues. When the partition heals, the old leader sees a higher term, steps down, and reconciles its log with the new leader.

The cluster splits exactly in half. Neither half has majority. Both halves refuse to commit. The cluster is unavailable until the partition heals. This is the right behavior — better unavailability than two leaders committing different things.

Where Raft actually runs in production

You almost never implement Raft yourself. You import a battle- tested library (etcd-io/raft is the canonical Go implementation; tikv/raft-rs for Rust; hashicorp/raft for Go) or you run a database that uses one. The most common deployments:

• etcd — the entire Kubernetes control plane writes through etcd, which is a 3 or 5-node Raft cluster. If etcd is sick, your Kubernetes cluster can't schedule pods.
• Consul — HashiCorp's service-discovery store. Raft cluster per datacenter.
• CockroachDB — every range (~64MB shard) is its own Raft group. A single database might have thousands of Raft groups running concurrently.
• TiKV / TiDB — same pattern as CockroachDB, smaller ranges.
• MongoDB (since 4.0) — replica-set elections use a Raft-like protocol.
• Patroni — distributes Postgres failover using etcd or Consul, both of which are Raft underneath.

You probably don't run a Raft cluster directly. You run a database that does. The day Raft becomes your problem is the day someone pulls a network cable in the wrong rack and your "etcd is sick" metric starts firing.

Paxos, ZAB, and why we ended up with Raft

Paxos came first (Lamport, late 1980s), is mathematically gorgeous, and is famously, genuinely difficult to implement correctly. Lamport's paper proved the algorithm; making it work for a real database — log compaction, dynamic membership, batched commits — required years of follow-up papers and a small number of brave teams. Google's Chubby and Spanner use Paxos variants; ZooKeeper uses ZAB, which is a Paxos cousin.

Raft was designed in 2014 specifically to be understandable. The paper title is "In Search of an Understandable Consensus Algorithm" and they meant it. The guarantees are identical to Paxos, but the protocol is presented as a state machine with three states (follower, candidate, leader) and a few RPCs. Almost everything new in the consensus space since 2015 uses Raft, not because it's faster (it isn't really) but because it's easier to implement correctly, debug when broken, and onboard new engineers to.

What Raft does NOT give you

Three common misconceptions worth dispelling:

Read scalability. In standard Raft, every read must go through the leader to guarantee strong consistency. Followers have the data, but they might be slightly behind, and reading from them gives you eventual consistency. Production systems opt into follower reads with explicit consistency trade-offs (SELECT AS OF SYSTEM TIME in CockroachDB, for example).

Write throughput per cluster. A single Raft cluster is bound by a single leader. Throughput per cluster maxes out around 10-50k writes/sec on commodity hardware (depending on entry size, network, etc.). Scale beyond that comes from running many independent Raft groups — one per data range (CockroachDB, TiKV) or one per partition (Kafka with KRaft). Each group runs its own protocol, and the system orchestrates them.

Geo-distribution without latency. A cross-region Raft cluster pays cross-region latency on every commit, because every commit requires a majority and the leader has to wait for majority acks. At 70 ms between US East and US West, that's 70 ms minimum added to every write. This is why globally-distributed databases (Spanner, CockroachDB) either accept the latency for strong consistency or provide locality-aware features so most writes stay local to one region.

Back-of-envelope: what a commit actually costs

The latency floor of a Raft write is one round trip from the leader to the nearest enough follower to form a majority — not to all of them. In a 5-node cluster the leader needs 2 follower acks (itself plus 2 = 3 = majority), so the commit lands as soon as the 2nd-fastest follower replies. The slowest 2 nodes don't gate the write. That's the quiet superpower of quorum: it's a tail-latency shock absorber.

Plug in real round-trip times and the topology choice stops being abstract:

Two lessons fall out of the table. First, spreading across AZs in one region is nearly free — a couple of milliseconds buys you single-AZ-loss survival. Second, spreading across regions is not — every commit eats the cross-region RTT to whichever follower completes the quorum, and there is no batching trick that makes the speed of light shorter. Pin the leader near the writers, or accept the tax.

A war story about a 3-node etcd

A team I worked with ran a 3-node etcd cluster for their Kubernetes control plane, with all three nodes in the same AZ "because cross-AZ latency hurt write performance". A datacenter power event took out the entire AZ. All three etcd nodes went down simultaneously. Kubernetes was unable to schedule, scale, or even report pod status for the four hours it took to bring the AZ back. Worker nodes kept running existing pods, so user- facing services were degraded but not dead; everything that needed a control-plane operation (deploys, rollbacks, scaling events) was frozen.

The fix was painful: rebuild etcd as a 5-node cluster spanning three AZs (3+1+1), accept the 2-3 ms cross-AZ latency on every write, and verify with chaos testing that single-AZ loss didn't take down the cluster. The latency tax was real — Kubernetes control-plane writes got slower — but the cluster now tolerated the failure mode that had cost them an outage. The lesson, as with most consensus deployments, is that the AZ topology of your Raft cluster is your real availability budget. Co-locating for performance is co-locating for blast radius.

[CONCEPT]cap-theorem for why these trade-offs are unavoidable. [CONCEPT]replication is the simpler version of the same problem (one writer instead of consensus); Raft is what you reach for when "one writer" isn't enough.