System Design
Circuit Breaker
Stop calling a dependency that's clearly broken. The pattern is small; the failure modes it prevents are not.
Circuit breaker
You don't really understand the circuit breaker pattern until the first time you've watched a service die from cascading dependency failure. There's a particular kind of pager-at-3am incident where your service is healthy in every measurable way except that it isn't serving requests. CPU is low. Memory is fine. The error log isn't even particularly noisy. But every endpoint is timing out because every endpoint is waiting on a downstream call that isn't coming back, and your thread pool is full of threads holding sockets open to a service that died eight minutes ago. The fix is two lines of config: when the downstream looks dead, stop calling it for a minute. That's the circuit breaker.
Named after the electrical device of the same name, the pattern borrows the metaphor directly: a normal current flows through the circuit, but if the load spikes dangerously, the breaker trips and cuts the connection until somebody (or something) verifies the problem has passed. In software, that "somebody" is the breaker itself, and the verification is a single probe request.
The three states everybody draws
Every breaker has the same state machine, which means once you've seen one diagram, you've seen them all. There are three states.
Closed is the normal state. Requests flow through the breaker to the dependency, and the breaker observes whether they succeed. It's not doing anything special; it's just counting.
Open is the tripped state. Every call short-circuits at the breaker — the dependency isn't called at all. The caller either gets an immediate failure or hits a fallback path. The point of this state is to give the broken dependency room to recover without the caller hammering it with traffic it can't handle.
Half-open is the probe state. After a cooldown period (usually 30-60 seconds), the breaker lets exactly one request through. If that probe succeeds, the breaker closes and traffic resumes. If it fails, the breaker opens again and the cooldown starts over. This is how the breaker discovers the dependency has recovered without flooding it the moment it comes back online.
The state transitions are the entire pattern. Implementations vary in how they count failures and what they consider a probe, but every breaker — Hystrix, Resilience4j, Polly, Istio's outlier detection — has these three states and the same transitions between them.
Why "fail fast" is the whole point
The instinct, the first time you build a breaker, is to think about it as a way to prevent errors. It isn't. The dependency is going to keep being broken whether the breaker is there or not. What the breaker prevents is the caller wasting resources waiting on the broken thing.
A 30-second timeout against a dead service is 30 seconds of held threads, held sockets, held connection-pool slots, held attention from your alerting system. At 100 RPS against a dead dependency, that's 3,000 concurrent waiting threads after 30 seconds. Most thread pools are configured for 200. Your service runs out of threads, refuses to accept new HTTP connections, and the load balancer marks it unhealthy. Now it's not just the call to the dead dependency that's failing — it's everything, including the endpoints that don't even touch the broken thing. The breaker trades 30 seconds of pretending to work for 1 millisecond of failing fast, and that trade is the difference between a slow cascading outage and a fast contained one.
The breaker doesn't fix the broken dependency. It contains the blast radius so the broken dependency can't take down the caller while it's recovering.
The math behind thread exhaustion
The reason "we have a timeout" doesn't save you is arithmetic. A synchronous call holds a thread for the entire duration of the timeout. Little's Law gives the concurrency directly: concurrent threads held = arrival rate × time held per request. Plug in a few realistic numbers against a pool of 200 threads:
| Incoming RPS | Timeout (s) | Threads held = RPS × timeout | Pool of 200 |
|---|---|---|---|
| 100 | 0.5 (healthy p99) | 50 | Fine |
| 100 | 5 (dead, timing out) | 500 | Exhausted in ~2s |
| 500 | 5 (dead, Black Friday) | 2,500 | Exhausted instantly |
| 500 | 0.001 (breaker open) | 0.5 | Trivial |
The fourth row is the whole pitch. With the breaker open, each "call" returns in a millisecond, so 500 RPS holds half a thread on average instead of 2,500. The timeout never protected you — it just set the ceiling on how long each thread stayed stuck. The breaker is what removes the wait entirely.
Where the breaker lives, and how many you need
The breaker lives in the caller. The callee is already broken; adding code to a broken thing won't help. Practically this means the breaker is in the caller's HTTP client (or RPC client, or database client), and most language ecosystems have a battle- tested library — Resilience4j in the JVM world, Polly in .NET, go-resiliency or the patterns built into gRPC for Go, opossum.js in Node.
The other rule that catches people: you need one breaker per
dependency, not one per service. If your order-api calls
payment-svc, inventory-svc, and shipping-svc, that's
three separate breakers with three separate state machines and
three separate thresholds. The point is that payment-svc being
down shouldn't stop you from calling inventory-svc — and a
single breaker per service would do exactly that.
If you have lots of dependencies and want to keep configuration manageable, the standard pattern is to define a default breaker config (window size, threshold, cooldown) and only override per- dependency when one dependency has unusual characteristics. A payment gateway with a slow 99th-percentile shouldn't share its breaker tuning with an internal cache lookup.
Tuning the threshold without flapping
Most breakers use a sliding window: "open if X% of the last N requests failed". The numbers matter, and getting them wrong is why some teams' breakers do more harm than good.
| Setting | Too low | Too high |
|---|---|---|
| Failure threshold (%) | Flaps every time the network coughs | Stays closed while users hit errors |
| Window size (requests) | Noisy, false trips on small samples | Slow to react to a real outage |
| Cooldown (seconds) | Probes too aggressively, doesn't give recovery time | Slow to come back when dependency is fine |
Common defaults — window of 20 requests, threshold of 50%, cooldown of 30 seconds — are a reasonable starting point for most services. Adjust based on your dependency's characteristics: a flaky third- party API might need a longer window (so a 2-request blip doesn't trip), while a critical internal service might need a shorter cooldown (so recovery is detected quickly).
The metric that tells you whether your breaker is tuned right is the count of trips per day in production. Zero trips means it's not doing anything (or your dependencies are flawless — possible, unlikely). Many trips per hour means it's flapping or your dependencies are genuinely broken; either way, investigate. The goal is a breaker that opens during real outages and stays closed during normal noise.
A 500 is a failure. A timeout is a failure. A client-side validation error (4xx) is NOT a failure — the dependency answered correctly that you sent garbage. Counting 4xx as failures will trip the breaker every time a user mistypes their email, and that's not what you want. Be explicit in your config about which response codes count toward the threshold.
The fallback is half the design
When the breaker opens, the caller has to do something. The breaker doesn't make that decision for you; you do, and the quality of the fallback is what determines whether your users notice the outage.
The four fallback patterns, ordered by how customer-friendly they are:
- Return cached data, even if stale. "Here's the product price as of 10 minutes ago." Users would rather see slightly outdated information than an error page.
- Degrade gracefully, omitting the broken feature. The page loads without the recommendation widget, the search returns without the "similar items" sidebar. The user gets something useful.
- Queue the work for later, returning a 202 "we got your request, we'll process it." Works for fire-and-forget operations (send email, post analytics event); a saga or job queue handles the actual processing.
- Fail fast with 5xx if there's truly nothing useful to do. Better than hanging, but the worst customer experience.
A breaker without a fallback is just a faster way to fail. The combination — breaker + thoughtful fallback — is what turns a cascading outage into a partial degradation the user barely notices.
In code the whole thing is unglamorous — the value is in the fallback branch, not the breaker:
price = priceBreaker.call(
() => taxApi.getRate(cart), // the protected call
fallback = () => cache.get(cart) // stale-but-useful when open
)
When the breaker is closed, the first lambda runs. When it's open,
the call never touches taxApi — the second lambda runs in a
millisecond and the user sees a slightly stale rate instead of a
spinner. Replace the fallback with throw 503 and you have the
"fail fast" pattern; replace it with queue.enqueue(cart) and you
have async retry. The breaker is identical in all three — the
fallback is where the product decision lives.
A war story about the breaker that wasn't there
A team I worked with ran a checkout service that called a tax- calculation API hosted by a third party. The third party had a known 99.9% uptime, which translates to about 43 minutes of downtime per month — they hit it pretty regularly. The team had talked about adding a circuit breaker around the call but never prioritized it because "we have a 5-second timeout, what's the worst that could happen."
The worst that could happen, it turned out, was Black Friday. The tax API went down at 2pm Eastern, the checkout volume was 40× a normal day, and the team's 5-second-timeout pattern meant every checkout request held a thread for 5 seconds before failing. Within 4 minutes, the checkout service's thread pool was exhausted and the load balancer marked every instance unhealthy. Auto- scaling launched more instances; they joined the pool, accepted traffic, and exhausted their thread pools 4 minutes later. The service was effectively down for 40 minutes — until the team deployed an emergency PR that added a circuit breaker around the tax API call. The breaker opened immediately (the third party was still down), every request returned a 503 in 1ms instead of holding a thread for 5 seconds, and the service was healthy within 90 seconds.
The third party's outage lasted 90 minutes. The team's outage, without the breaker, would have lasted 90 minutes too. With the breaker (which they should have shipped a year earlier), they served degraded traffic in the last 50 minutes and only lost revenue during the time it took them to deploy the fix.
Autoscaling makes the cascade worse, not better, when the root cause is a stuck downstream. New instances don't add downstream capacity — they just bring fresh thread pools to exhaust against the same dead dependency, and you pay for the extra machines while the outage continues. Fix the wait (breaker), not the symptom (not enough threads).
When you don't need a breaker
Like every pattern, a circuit breaker is a tax. Don't pay it where you don't have to.
- Single-process applications with in-process function calls. There's no network to protect; if a function is broken, the program is broken.
- Operations that don't have a fallback. If the user can't proceed without the broken dependency anyway, failing fast doesn't help — you've replaced a 5-second wait with a 1ms failure, but the user sees an error either way. A retry-with- backoff is more useful here than a breaker.
- Truly idempotent operations to extremely reliable dependencies. If the dependency has six nines of uptime and the operation is free to retry, you're probably better served by a retry policy than by a breaker.
[CONCEPT]message-queues are the most common fallback target — turn a synchronous broken call into an async retried job. See also [CONCEPT]rate-limiting (what protects the callee from the same problem) and [CONCEPT]observability (so you can see the breaker trips before customers do).