Load balancing

Most architectures are born the day a single API process can no longer keep up. You spin up a second copy on another machine, point DNS at both, and immediately discover the question nobody briefed you on: which copy does the next request go to? That's load balancing in one sentence, and almost every scaling decision you'll make for the next year sits downstream of how you answer it.

I always think of the load balancer as the cheapest move on the board. A single CPU saturates around 5,000 requests per second of real work. Two CPUs get you to 10,000 — if and only if both halves of your traffic find both halves of your fleet. That "if and only if" is the whole job. Once you have it, you can keep adding replicas almost mechanically; without it, the second replica sits idle while the first one melts.

You have two structural choices. You can teach every client about every server (client-side balancing — Eureka, Consul, gRPC's xDS), or you can put one box in the middle that owns the routing knowledge. Most production systems eventually pick the middle box. Not because it's faster — it isn't; the extra hop costs you a millisecond. They pick it because one box to operate and patch is cheaper than N clients you have to keep synchronized with the fleet topology.

How many replicas do you actually need?

Before you argue about algorithms, do the arithmetic, because the answer is usually smaller than the panic suggests. The whole point of the balancer is that capacity adds up linearly — if the traffic spreads. So you can size the fleet on the back of an envelope.

Say each replica is one CPU that tops out at 5,000 RPS of real work, and you want headroom: never run a box past 70% so a single failure doesn't cascade. Your peak traffic is 40,000 RPS.

The jump from 12 to 20 isn't waste — it's the cost of not paging someone when a box or a zone dies. The mistake teams make is sizing for the average (12) and discovering during the first incident that 70% utilization at peak means 100% the instant one replica drops, and now the survivors are the next dominoes.

The algorithms, and when each breaks

There are really only three balancing algorithms you'll meet in production. Their names are mnemonic for what they do; the interesting part is the failure mode each one hides.

Round-robin picks replica i % N. It's stateless, takes O(1) to dispatch, and works beautifully when every request costs about the same. It breaks the first time one of your endpoints does a heavier query — say, a report endpoint that hits a join across two million rows. Round-robin still dispatches one in N to the unlucky replica, the queue piles up behind that one slow request, and the "per-replica latency" graph in Grafana starts looking like a heart monitor.

Least-connections fixes that by sending the next request to whichever replica has the fewest in-flight calls right now. It adapts gracefully to heterogeneous workloads, which is what almost every real API turns out to have. The trade-off is that the balancer now has to maintain per-replica counters; in a multi-core balancer that costs an atomic increment per dispatch. Cheap in absolute terms, but you can measure it at sustained 100K RPS.

Weighted round-robin lets you assign each replica a number. If you mark your canary at weight 1 and prod at weight 9, exactly 10% of traffic hits the canary — which is the cleanest way to do a slow rollout. It's also how most teams handle replicas of different sizes ("we have one m5.xlarge and three m5.large; weight them 2-1-1-1").

A comparison you'll come back to

The first failure mode everybody hits

It's almost always this. You ship a load balancer, the health checks are green, and a customer emails to say their last forty requests returned 500. You log in to find one replica running but unable to reach the database. The health endpoint — GET /healthz — returns 200 because it's a stub that says "yes, the process is alive". The real request path needs the database, and the replica can't reach it. The balancer keeps shipping a third of traffic to the broken box.

Here's the shape of the placebo and the cure, side by side:

# Placebo: proves the process is scheduled, nothing more.
GET /healthz  ->  200 "ok"          # a string literal. always passes.

# Readiness: exercises the same dependencies a real request needs.
GET /readyz:
  ping database (1 cheap query, 200ms budget)   -> fail -> 503
  ping cache / downstream the request path uses  -> fail -> 503
  else                                           -> 200

# The balancer polls /readyz, not /healthz. A box that can't
# reach the DB returns 503 and gets evicted — instead of
# silently 500-ing one in three requests.

A health check that doesn't exercise the same code path as a real request is a placebo. It only proves the process didn't die.

The fix isn't to make the health check smarter overnight; it's to recognize that "alive" and "ready to serve" are different states. Kubernetes formalizes this with liveness and readiness probes. Liveness answers "is the process up?" — fail this and the orchestrator restarts the container. Readiness answers "should traffic come here right now?" — fail this and the balancer evicts the replica until it recovers, without killing it. Both probes are worth wiring; the second is the one that prevents most outages.

When the balancer isn't the right tool

There are workloads where putting a balancer in front does more harm than good. The two that catch most teams:

Long-lived connections — WebSockets, gRPC streams, anything that holds the socket open for minutes. A layer-4 balancer (HAProxy in TCP mode, NLB) handles these by pinning each connection to one backend and leaving it alone. A layer-7 balancer that re-evaluates per request will shred your streams the first time it tries to reload its config. Pick the layer that matches the shape of your traffic, not the layer that has the prettier dashboards.

Sticky sessions — if your backend keeps session state in local memory, you need affinity ("Alice always lands on replica 3") and you've just thrown away half of the reason you wanted multiple replicas in the first place. You can't drain replica 3 for a deploy without dropping every Alice. The right move is almost always to push session state into a shared store — Redis, the database, a signed cookie — and let the balancer be stateless again.

What comes next

Once load is balanced, the next bottleneck is almost always the database. Most teams reach for a cache before they add a third replica, because absorbing reads is cheaper than absorbing writes. That's [CONCEPT]caching-patterns — the next lesson — and once you've added a cache the question of which data lives where opens up [CONCEPT]sharding-strategies and [CONCEPT]replication.