Rate limiting

The first rate limiter I ever wrote was a single in-memory counter behind a single API process. It worked beautifully until the day we autoscaled the API from one instance to three, and the same user suddenly got three times the intended limit because each instance had its own counter and they had no way to coordinate. That's the first lesson rate limiting teaches: the algorithm is the easy part, the distributed version of the algorithm is the entire engineering job.

A rate limiter, stripped to essentials, is a small program that says "no" to too much traffic so the rest of the system doesn't fall over. It sits somewhere on the request path — the CDN, the API gateway, application middleware — counts requests against some key (user ID, IP, API token, tenant ID), and rejects new requests once that key has used its allowance. The interesting questions are how you count, where you count, and what you return when the answer is no.

Why you add one in the first place

Teams reach for a rate limiter for one of three reasons, and the reason matters because it shapes how you tune the limiter.

Protecting expensive resources. Every call to OpenAI, Stripe, or Twilio costs real money. A bug in your code that calls /v1/embeddings in a loop can burn through a month of budget in twenty minutes. A rate limiter scoped per workload (this batch job gets 100 RPM, that endpoint gets 10 RPM) is the only defense against your own infrastructure becoming your own denial-of-wallet attack.

Preventing abuse. Login endpoints get credential-stuffed within hours of going public; password-reset endpoints get used as email cannons; sign-up endpoints get hit by bots creating millions of throwaway accounts. A rate limit per IP (or per email, or per phone) makes the abuse economically unattractive without inconveniencing legitimate users.

Shaping multi-tenant load. In SaaS, one customer's runaway script shouldn't ruin every other customer's latency. Per-tenant limits ensure noisy neighbours stay in their own room. Without them, a single enterprise customer doing a one-time backfill at 10K RPS can saturate the database and make your other customers notice.

The reason matters because it determines the response to a limit hit. Abuse prevention should be silent or punitive ("blocked" with no Retry-After). Workload protection should be helpful (Retry-After: 30, X-RateLimit-Remaining: 0). Multi-tenant fairness should be transparent (the tenant should know they hit a limit and adjust). A single rate limiter implementation that doesn't distinguish these intents will get all three wrong.

The three algorithms, and why everyone picks token bucket

There are really three algorithms you'll meet in production. Each has fans and a small set of niches where it wins, but token bucket is the default for 90% of cases for good reasons.

Token bucket maintains, for each key, a bucket that holds up to N tokens and refills at R tokens per second. Every request consumes one token; if the bucket is empty, the request is rejected. The mathematical guarantee is "average rate R, with bursts up to N". This is what most production limiters use — Stripe's API, AWS API Gateway, Cloudflare's edge limiters, the X-RateLimit-* headers you see in every modern API. The reason it dominates is that it captures the most common real-world shape of traffic: users want to occasionally burst (a script syncing 50 records in a tight loop) but the long-run average should be bounded. Token bucket allows the burst while enforcing the average.

The whole algorithm is a handful of lines — you don't even store a running countdown, you store the last refill time and compute how many tokens have dripped in since:

allow(key, now):
  b = store.get(key)  or  {tokens: N, ts: now}   # full bucket on first sight
  refill   = (now - b.ts) * R                     # tokens accrued since last check
  b.tokens = min(N, b.tokens + refill)            # never overflow past N
  b.ts     = now
  if b.tokens >= 1:
    b.tokens -= 1
    store.set(key, b)
    return ALLOW
  store.set(key, b)
  return DENY   # bucket empty — this is your 429

That min(N, ...) is the entire burst policy: idle keys fill back up to N and no further, so a user who's been quiet for an hour gets exactly one bucket of burst, not an hour's worth of saved-up credit.

Leaky bucket flips the model: requests enter a queue at any rate, and the queue drains at a fixed rate R. Overflow drops. The result is perfectly smooth output — no bursts ever — at the cost of feeling punitive when a legitimate user occasionally needs to spike. Used in places where downstream really cannot handle bursts (legacy systems, hardware-constrained services), rare in modern web APIs.

Fixed window is the simplest: a counter resets every N seconds (often Math.floor(now / 60) for per-minute limits). Trivially cheap to implement (one increment, one comparison). The catastrophic edge case is the "double burst" at window boundaries: a client can fire bucket-size requests in the last second of window 1, then bucket-size more in the first second of window 2, doubling their effective rate briefly. Sliding- window-log fixes this by tracking individual request timestamps in a sorted set, at the cost of O(N) memory per key.

The distributed problem, which is the real problem

The single-process rate limiter is a fifty-line algorithm. Production rate limiters are not fifty lines because production APIs don't run in a single process. The moment you have N load-balanced backends, each running its own in-memory counter, the user effectively gets N times the intended limit before any one backend says no.

There are two practical solutions and one common bad one.

Centralized counter (the standard solution). Every backend checks and increments a shared counter, usually in Redis. Redis INCR is atomic and a single Redis instance can comfortably serve a million ops per second, which is enough for the entire edge of a large API. The cost is one network round-trip per rate-limited request — usually 1-2 ms, which is acceptable for edge enforcement but starts to matter if every internal call needs to be limited too. The latency tax is the price of correctness.

Probabilistic / approximate (the high-scale solution). At extreme scale (millions of RPS across hundreds of backends), even centralized Redis is a bottleneck. The trick is to let each backend approximate its share of the budget: backend i of N enforces bucket_size / N locally. It's not exact — a user might briefly exceed the global limit during backend skew — but it's cheap and adequately accurate. Used by Cloudflare, Akamai, AWS edge services.

Per-instance with no coordination (the bad solution). Every backend has its own in-memory counter, no Redis, no coordination. The "limit" is effectively N × the configured value. This is the version most teams accidentally ship first because the in-memory implementation is so much easier than the Redis one. The next limit-hit incident catches the bug.

A rate limiter has to be cheaper than the work it protects, or you've just moved the bottleneck. The Redis round-trip is almost always cheaper than the database query you're rate- limiting.

Does one Redis actually hold the edge? Run the numbers

Before you panic about Redis becoming the bottleneck, do the back-of-envelope. Say you're a large API doing 200K RPS at the edge, and every request costs one round-trip to a single Redis (a token-bucket check is typically a GET + a small Lua script — call it one op).

One Redis sits at a fifth of its ceiling and adds well under a millisecond to guard a request that would otherwise spend hundreds of milliseconds in your database. That's the trade that makes the centralized counter the default: you only graduate to the probabilistic approach when this table stops closing — when the edge climbs past ~700K–1M RPS and Redis runs out of headroom, not before.

What to return when you say no

The HTTP convention is 429 Too Many Requests, and the useful headers are well-established:

• Retry-After: 30 — tells the client to wait 30 seconds before retrying. Either a number of seconds or an HTTP date.
• X-RateLimit-Limit: 100 — the current period's limit.
• X-RateLimit-Remaining: 0 — how many requests left in this period.
• X-RateLimit-Reset: 1729123456 — Unix timestamp when the bucket refills.

Well-behaved clients (every Stripe SDK, the AWS CLI, anything written by an engineer who's been burned by rate limits before) respect these headers and back off. Misbehaving clients ignore them and hammer your API harder when they hit a 429, which is why the third axis of rate limiting is...

Edge or application — pick both

Edge rate limits (CDN, WAF, API gateway) catch the high-volume garbage — DDoS, scrapers, botnets — before traffic ever reaches your origin. They're cheap (sub-millisecond), they're dumb (one counter per IP, no business context), and they protect the origin from being saturated by clearly-illegitimate traffic.

Application rate limits (middleware inside your service) catch the plausibly-legitimate abuse — a paying customer who accidentally wrote an infinite loop, a script doing a backfill that should be batched, a tenant whose usage pattern changed overnight. The middleware has business context: "this user is on plan X, this endpoint costs Y tokens, this tenant gets priority over that one". It's slower and smarter; it cannot absorb the firehose without the edge in front of it.

Production systems run both, and the edge limits are an order of magnitude looser than the application limits. The edge says "no more than 10K RPS from any one IP" — that's clearly an attack. The application says "no more than 100 RPS per user per endpoint" — that's policy. Both layers are necessary; the edge protects the origin from existing, the application protects fairness across legitimate users.

A war story about the limiter that limited the wrong thing

A team I worked with shipped a per-IP rate limit on their login endpoint to defeat credential stuffing. The default was 10 attempts per minute per IP. It worked great against the simple bots. Then a corporate customer reported that hundreds of their employees were locked out for an hour every Monday morning — turns out they all shared a single office IP, and the Monday-morning login wave (everyone logging in within a five-minute window) hit the limit.

The fix was to switch the limit key from IP to (IP, username prefix) or to username directly, so legitimate users on shared IPs weren't penalized by the limit. The deeper lesson — and it applies to every rate limiter — is that the key is the policy. Pick the wrong key and you'll either let abuse through (key too narrow) or block legitimate users (key too broad). There's no algorithm that compensates for the wrong key, and the right key requires understanding who your users actually are.

When you don't need a rate limiter

Three cases where adding one is overkill:

• Single-user tools — desktop apps, internal scripts, CLI tools. The "user" is also the operator; if they call the API too often they only hurt themselves.
• Trusted internal traffic on a private network — service- to-service inside your VPC, especially when capacity is controlled by deployment (you know how many instances are running and how often they call). Rate limits add complexity without adding safety here.
• Operations with a natural cost gate — anything that requires an expensive resource to even attempt (a paid API key, a captcha, a multi-step workflow). The natural friction is its own rate limit.

For everything else — anything public, anything multi-tenant, anything backed by a shared resource — rate limits are infrastructure, not a feature. Ship them on day one, not after your first abuse incident.

[CONCEPT]circuit-breaker is the complementary pattern: rate limiter says no on the callee side, circuit breaker says no on the caller side. Together they prevent the two shapes of cascading failure. [CONCEPT]observability is what tells you whether your limits are set at the right value — if you never see 429s, the limit is too loose; if legitimate users complain regularly, it's too tight.