Circuit Breaker Pattern

Overview

The Circuit Breaker is a resilience pattern that prevents cascading failures in distributed systems by cutting off requests to a failing downstream service. Instead of letting every request fail slowly (consuming threads, connections, and time), the circuit breaker fails fast and redirects traffic to a fallback path.

The pattern is fundamental to building fault-tolerant microservices and is implemented in many frameworks and platforms:

• Netflix Hystrix -- the library that popularized the pattern (now deprecated, but deeply influential)
• Resilience4j -- modern Java replacement for Hystrix
• Polly -- resilience library for .NET
• Istio / Envoy -- circuit breaking at the service mesh level
• Spring Cloud Circuit Breaker -- abstraction over multiple implementations
• Sentinel -- Alibaba's flow control and circuit breaking library

Architecture

Client Application

An external client that sends requests through the circuit breaker proxy. The client is unaware of the circuit breaker state and simply receives either a normal response or a degraded fallback response.

Circuit Breaker Proxy

The core component that wraps calls to the downstream service. It maintains a failure counter and a state machine. When the failure count exceeds the configured threshold (e.g., 3 failures), the circuit "opens" and all subsequent requests are short-circuited to the fallback without ever reaching the backend.

Fallback Service

Returns a degraded or cached response when the circuit is open. A degraded response (stale data, default values, reduced functionality) is almost always better than an error or a timeout. The fallback keeps the overall system usable even when a dependency is down.

Backend Service and Database

The downstream service protected by the circuit breaker. When this service is failing (overloaded, crashed, or its database is unreachable), the circuit breaker prevents it from receiving additional load, giving it time to recover.

Key Concepts

Three States

The circuit breaker operates as a state machine with three states:

• CLOSED: Normal operation. All requests pass through to the backend. Failures are counted. This is the default state.
• OPEN: Failure threshold exceeded. No requests reach the backend. All traffic is redirected to the fallback. A timeout timer starts (e.g., 30 seconds).
• HALF-OPEN: The timeout has expired. The circuit breaker allows exactly one probe request through to test if the backend has recovered. If the probe succeeds, the circuit closes. If it fails, the circuit opens again.

Failure Threshold

The circuit breaker tracks consecutive or windowed failures. When failures reach a configured threshold (e.g., 3 out of 5 requests, or 3 consecutive failures), the circuit transitions from CLOSED to OPEN. The threshold prevents a single transient error from tripping the circuit while still reacting quickly to sustained failures.

Timeout and Recovery

When the circuit opens, a timer starts. During this window, the backend receives zero traffic, giving it breathing room to recover. When the timer expires, the circuit enters HALF-OPEN and sends a single probe request. This probe-based recovery avoids flooding a barely-recovered service with full traffic.

Why Fallbacks Matter

Without a fallback, an open circuit simply returns errors faster. With a fallback, the system degrades gracefully: cached product listings instead of live data, default recommendations instead of personalized ones, or a "service temporarily limited" message instead of a 500 error. Degraded service is almost always preferable to no service.

Defense in Depth

The circuit breaker is most effective when combined with other resilience patterns:

• Retry with backoff handles transient failures before the circuit trips
• Bulkhead isolates failures to specific service pools, preventing one bad dependency from exhausting all threads
• Timeout ensures requests fail within a bounded time rather than hanging indefinitely
• Rate limiting protects the upstream from generating excessive load in the first place

Together, these patterns form a layered defense against the unpredictable failures inherent in distributed systems.

Scenarios

1. CLOSED (Normal) -- Client request flows through the circuit breaker to the backend and database, succeeds, and the failure counter stays at zero
2. CLOSED -> OPEN -- Three consecutive failures hit the threshold, the circuit opens, and subsequent requests are short-circuited to the fallback service returning cached/degraded responses
3. HALF-OPEN (Recovery) -- Timer expires, circuit enters HALF-OPEN, a single probe request succeeds, circuit closes, and normal traffic resumes
4. HALF-OPEN (Still Broken) -- Timer expires, probe request fails, circuit returns to OPEN state, timer restarts, and fallback continues serving degraded responses