System Design
Latency Numbers
How long things actually take. The reference card for "is this design even physically possible?"
Latency numbers every engineer should know
You will spend more time waiting on physics than on your code. A function call is a few nanoseconds. A trip across the Atlantic and back is eighty milliseconds — roughly twenty million function calls of wall-clock time, for a packet that did nothing but travel. Most "why is this slow?" investigations end at exactly this gap, and the engineers who can answer the question fastest are the ones who already carry the table in their head.
The reason to memorize these numbers isn't trivia. It's that the table lets you reject impossible designs before you write a line of code. "Can we do a synchronous fan-out to twelve services in our 100ms budget?" isn't a question you answer by building it and measuring. You answer it with arithmetic, in the design review, in thirty seconds. The numbers haven't moved much in fifteen years — CPUs got wider, networks got denser, but the speed of light is a hard constant and a disk seek is still a disk seek.
The hierarchy
Read this top to bottom once. The only thing you need to retain is the shape: each tier is roughly 100× slower than the one above it, and the network tier is where the wall-clock disappears.
| Operation | Latency | Relative to L1 |
|---|---|---|
| CPU register | 0.3 ns | ×0.3 |
| L1 cache | 1 ns | ×1 |
| Branch mispredict | 3 ns | ×3 |
| L2 cache | 4 ns | ×4 |
| Mutex lock/unlock | 17 ns | ×17 |
| Main memory access | 100 ns | ×100 |
| Send 2KB over 1 Gbps | 800 ns | ×800 |
| Read 1MB sequentially from RAM | 3 μs | ×3,000 |
| SSD random read | 16 μs | ×16,000 |
| Read 1MB sequentially from SSD | 49 μs | ×49,000 |
| Round-trip in same datacenter | 500 μs | ×500,000 |
| Read 1MB sequentially from disk | 825 μs | ×825,000 |
| Disk seek | 2 ms | ×2,000,000 |
| Cross-region (US east ↔ west) | 50 ms | ×50,000,000 |
| Cross-continent (US ↔ EU) | 80 ms | ×80,000,000 |
| US ↔ Asia | 150 ms | ×150,000,000 |
| Reboot a server | 5 min | — |
A datacenter round-trip is 500 microseconds. A cross-continent round-trip is 80 milliseconds — 160× more. The single biggest lever in most systems is not making the work faster; it's not crossing the ocean to do it.
The jump that matters most is the one from memory (100 ns) to a same-datacenter round-trip (500 μs). That's a 5,000× cliff, and it's the moment your work stops being a computation and starts being a conversation. Everything past that line — the network — is where real systems spend their time, and where almost all the latency you'll ever chase actually lives.
The seven-orders-of-magnitude rule
From a CPU register (0.3 ns) to a cross-Pacific round-trip (150 ms) is about nine orders of magnitude. Most of the design tricks you already know are just attempts to move work up the hierarchy by one or two orders:
- Caching moves a read from the disk tier to the RAM tier. That's a ~100× win, which is exactly why it's the second thing almost every system reaches for. See [CONCEPT]caching-patterns.
- CDNs move a read from cross-continent (80 ms) to a local edge (5–20 ms). Same trick, applied to the network tier.
- Batching turns N round-trips into one. If each round-trip is 500 μs and you have 100 of them, batching saves you ~50 ms of pure waiting — for free, with no faster hardware.
That last one is the lever people forget. You cannot make the speed of light faster. You can stop paying for it ten times in a row.
A war story: the chatty ORM that crossed the ocean
A team ran their primary database in us-east-1 and, for a compliance
reason that made sense at the time, served a slice of European users
from an app tier in eu-west-1. Latency budget for the page: 300 ms.
It tested fine. It shipped fine. Then European traffic ramped, and the
page started taking four to six seconds for those users — but only
in production, and only for them.
The app code looked innocent. It loaded an order, then looped over its line items and lazy-loaded each one. Classic N+1: one query for the order, then one query per line item. On a developer's laptop, with the database on localhost, each of those queries was ~0.5 ms and the whole thing finished in single-digit milliseconds. Nobody noticed.
In production, every one of those queries was a round-trip from
eu-west-1 to us-east-1 — about 80 ms each. An order with 60
line items did 61 sequential queries. Do the arithmetic:
| Quantity | Math | Result |
|---|---|---|
| Queries per page | 1 order + 60 items | 61 |
| RTT per query (cross-continent) | — | 80 ms |
| Total, sequential | 61 × 80 ms | 4,880 ms |
| Budget | — | 300 ms |
Nearly five seconds, and not one millisecond of it was the database "being slow." Postgres answered every query in under a millisecond. The time was entirely the ocean, paid 61 times. The fix wasn't a faster database, a bigger instance, or an index. It was a single eager-loading join that fetched the order and all its line items in one round-trip: 80 ms total instead of 4,880. A 60× win from deleting 60 trips.
N+1 latency is invisible on a laptop because localhost RTT is ~0.05 ms, so 61 queries cost 3 ms and feel instant. The bug only exists when the round-trip is real. This is why "it's fast in dev" tells you nothing about a system that crosses a network — and why latency numbers belong in code review, not just in production postmortems.
What fits in a user-felt budget
Users don't perceive milliseconds; they perceive feel. These thresholds are well-studied and stable: under ~100 ms feels instantaneous, ~1 s is the limit before attention wanders, ~10 s is where you've lost them entirely.
| Budget | Feel | What fits inside it |
|---|---|---|
| < 100 ms | Instant | A few same-DC hops + cache reads. No cross-region trips. |
| 100–300 ms | Snappy | 1 cross-region round-trip OR ~5 same-DC hops + a DB query |
| 300 ms – 1 s | Slow but tolerable | A couple of cross-region hops, or a moderate DB scan |
| > 1 s | Show a spinner | Batch work, cold caches, fan-out to many services |
The practical reading: a single cross-continent round-trip (80 ms) eats most of a "snappy" budget by itself. You get one. If your design does two sequential trips across an ocean, you are over budget before any code runs, and no amount of optimization downstream will save you. The fix is structural — move the data closer (replica, CDN, edge), or stop making the trip.
Things that are NOT free
The flip side of "the network dominates" is that engineers start treating everything above the network as free. It isn't. These are the costs that hide inside a single request handler and add up under load:
| "Free" thing | Actual cost | Why it bites |
|---|---|---|
| Hashmap insert | 100–500 ns | Allocator + hash; fine once, brutal in a tight loop |
| String concat in a loop | 100 ns each + GC | Builds garbage; the GC pause is the real cost |
| JSON parse 10KB | 50–200 μs | More than a same-DC round-trip — measure it |
| Regex compile | 1–100 μs per pattern | Compile once, not per request |
| Logger write to disk | 1–10 ms | Blocks the thread if synchronous |
new Date() | 50–200 ns | Cheap alone, thousands/sec is a syscall storm |
Notice the trap in that table: parsing a 10KB JSON body (50–200 μs) can cost more than the same-datacenter network round-trip that delivered it. The instinct "it's just CPU, it's free" is exactly backwards for the hot path. The rule that survives: profile before you optimize. Your intuition for which line is slow is wrong often enough that guessing is a waste of time — and the latency table is what makes the profiler's output legible when you read it.
A service logged every request to disk synchronously. Each log.info
blocked the request thread for 1–10 ms. Under low traffic, invisible.
Under load, those millisecond blocks serialized behind each other and
the p99 latency went vertical — the service was spending more time
writing logs than serving requests. Async logging (buffer + background
flush) made the problem vanish. The lesson: a 1–10 ms blocking call in
the hot path is a network round-trip you didn't know you were making.
Putting it together
The whole discipline reduces to three habits:
- Count the round-trips, not the operations. The network is seven orders of magnitude above the CPU. Ten queries to a distant database beat any micro-optimization you can name.
- Know where your data physically is. Same machine, same DC, same region, or across an ocean — that single fact sets the floor on what you can build. A cross-region call is not a local call with a bigger number; it's a different category.
- Profile the rest. Once round-trips are minimized, the "free" CPU work is where the remaining latency hides, and your intuition for it is unreliable. Measure.
[CONCEPT]back-of-envelope is the natural partner to this page: the latency table tells you whether a design is physically possible, and the sizing table tells you how big the machine behind it has to be. Run both before you build, not after the pager goes off.