Replication

Replication is the easiest distributed-systems problem to start and one of the hardest to finish. The starting version takes an afternoon: spin up a second Postgres, point its primary_conninfo at the first, watch the WAL stream copy over. You've doubled your read capacity and you have a hot standby in case the primary dies. The finishing version takes years, because consistency is a contract you didn't realize you were signing, and every read you route to the replica is a small bet that the lag is short enough not to embarrass the application.

The mental shift that separates engineers who reason about replicated systems well from ones who don't is treating the primary's view of "now" and each replica's view of "now" as different variables that need to be tracked separately. They're never identical. They're not even guaranteed to converge under load. Most bugs in replicated systems trace to someone writing code that assumed they were.

Sync versus async — picking your poison

Every replication setup has to decide whether the primary waits for the replica before acknowledging a write. This is a real fork in the road; you can't have both speed and zero data loss, only one or the other (or a careful mix).

Synchronous replication blocks the commit until at least one replica has applied the change. The win is provable: if the primary dies right now, the synchronous replica has every committed write the primary did, so failover loses zero data — RPO of zero, in the operations vocabulary. The cost is that every write you do pays the round-trip latency to that replica. Cross-AZ in AWS is 1-3 ms, which sounds small until you remember it's added to every commit. A service that did 50 ms transactions now does 53 ms transactions, and your TPS ceiling moves with it.

Asynchronous replication acknowledges the write the moment the primary's local fsync is done, then streams the change to replicas in the background. Writes are fast — the replicas are out of the critical path entirely. But the primary's just-acknowledged writes might not have made it to any replica yet, and if the primary dies in the next 30 ms, those writes are gone. This is the RPO greater than zero world, and the number depends on how far the replicas had lagged.

Production setups usually mix both: one synchronous replica that exists purely to provide a safe failover target, plus several asynchronous replicas to scale read traffic. Postgres's synchronous_standby_names lets you name exactly one (or a quorum of) replicas as the durability anchor and treat the rest as read-only scale-out. You get the no-loss failover story and the fast-write story by paying for the replica that gives you both.

Replication lag, the silent killer

Lag is the time between "committed on primary" and "visible on replica". On a healthy day it's a handful of milliseconds. Under load it's seconds. Under a long-running DDL on the primary — say, ALTER TABLE on a 200 GB table — it can be minutes or hours, and the only visible symptom is reports that are mysteriously stale.

The thing about lag is that you don't notice it. The replica is serving queries, the queries succeed, the dashboards (reading from the replica) are green. The first sign you have a problem is usually a customer ticket: "I clicked save five minutes ago and the value hasn't updated." You investigate and find your read replica is 47 seconds behind because someone ran a backfill on the primary.

The lesson most teams learn the hard way is that any read from a replica needs a freshness contract. "It's OK to be one second stale" is a fine contract for a public profile page. It's a terrible contract for a checkout total. Different code paths need different contracts, and the application — not the database — is where those decisions live.

The leader-election trap

When the primary dies, one of the replicas becomes the new primary. The new one needs to be at least as up-to-date as the failed one, or the cluster has lost data. There are exactly two ways to guarantee this, and "the replica was probably caught up when the primary died" is not one of them.

The first way is synchronous replication to the candidate replica. If the writes that mattered to you were synced to that specific replica, promoting it is safe — by construction it has every acknowledged commit.

The second way is a consensus protocol that requires a quorum of replicas to acknowledge each write before commit. Raft and Paxos are the two practical algorithms; the guarantee is the same as synchronous replication except across more than one replica, so you can survive a minority of replicas being down at the moment of failover. This is what etcd, Consul, Spanner, CockroachDB, and Patroni's recommended setup all use under the hood. See [CONCEPT]consensus-raft for the protocol.

Anything else — async replication plus "we'll promote whichever replica looks most caught up" — is best-effort HA. You will lose data when a real failure happens. The only question is how much, and how often, and whether your customers notice. (They usually do.)

Read-only scale-out and its ceiling

The other reason to run replicas is to absorb read traffic. Postgres saturates a primary at some write rate plus some read rate; if you move reads to N replicas, the primary's read load drops to roughly zero, and reads scale with replica count. The math is simple, the operational story is simple, the gains are real, and you should do this before you ever consider sharding.

Run the back-of-envelope before you provision anything. Say one Postgres node tops out around 12,000 queries/sec, your workload is 90% reads / 10% writes, and every write must be shipped to every replica regardless of how reads are split:

The pattern the table makes visible: read capacity grows linearly with replicas, but the write cost on the primary grows too, because each replica is one more WAL stream to feed. The ceiling is less obvious than the linear part.

The primary still has to ship every WAL record to every replica. At 4-5 replicas the ship-out bandwidth starts to dominate the primary's CPU — it's literally spending more time sending changes than processing new transactions. Past that point you need to either reduce write traffic, cascade replication (replica-of-a-replica, which adds another lag layer), or accept that this is the read ceiling and the next move is sharding to split the write workload across multiple primaries.

Add a cache before you add the fifth replica. Add a read replica before you add a shard. Sharding is what you reach for when the cheaper moves have actually run out, not before.

A war story about silent divergence

A team I worked with ran two Postgres replicas behind PgBouncer, one sync, one async. They'd been operating happily for two years when the sync replica's network card started flapping — not failing hard, just dropping packets occasionally. The replication stream slowed down, the primary started waiting longer for the sync ack, write latency crept up.

The sysadmin "fixed" it by quietly demoting the bad replica to async. This was the right operational move for write latency. It was the wrong move for safety, because the team forgot to promote a different replica to sync. For three weeks both replicas were async. Then the primary's RAID card died at 4am and the on-call promoted the most caught-up replica. It was 2.3 seconds behind. About 800 commits were on the failed primary's WAL that hadn't replicated yet. Those 800 commits were customer-facing financial events — small ones, but enough that the team spent the next month reconciling.

The bug was not technical. The bug was that "we have a sync replica" had been a comforting fact in the team's head for years, and the moment that fact became false, nothing alerted anyone. The postmortem fix was a Prometheus alert: pg_synchronous_standby_count < 1 for more than 5 minutes pages. Two lines of YAML, fixed forever. The lesson is that every distributed-system invariant you care about needs to be a metric somewhere. If "we have a sync replica" matters, monitor it. If "the lag is under 1 second" matters, monitor that.

When to walk away

• You need strong consistency across the cluster — use a single primary with no replica reads, or move to a system that provides serializable reads on followers (Spanner, CockroachDB, FoundationDB). Garden-variety Postgres replicas won't give you this no matter what you configure.
• The workload is write-heavy — replicas don't add write capacity, only sharding does. See [CONCEPT]sharding-strategies.
• You only need a backup — a backup is not a replica. Backups protect against logical mistakes (drop table, bad migration), replicas don't. Run both.

[CONCEPT]cap-theorem explains the deeper trade-off you're buying into. [CONCEPT]consensus-raft is the protocol that turns "asynchronous replication with manual failover" into "automatic failover without data loss".