Bulkhead Pattern

Overview

The Bulkhead pattern isolates resources into separate pools so that a failure in one component cannot cascade and exhaust resources shared by other components. The name comes from the watertight compartments (bulkheads) in a ship's hull -- if one compartment is breached and floods, the bulkhead walls prevent water from spreading to adjacent compartments, keeping the ship afloat.

In software systems, bulkheads partition thread pools, connection pools, or other finite resources on a per-service (or per-consumer) basis. Without this isolation, a single slow or failing downstream service can consume all available threads in a shared pool, causing every other service to starve and the entire system to collapse.

Real-world implementations:

• Netflix Hystrix -- thread pool isolation per dependency (now in maintenance mode)
• Resilience4j Bulkhead -- thread pool and semaphore isolation for JVM applications
• Envoy Proxy -- per-upstream connection pools and circuit breaking
• Istio -- connection pool settings in DestinationRule
• Polly (.NET) -- Bulkhead policy with configurable parallelism and queue depth
• Kubernetes -- resource requests/limits provide process-level isolation

Architecture

Client Layer

Three types of clients send requests to different backend services through a shared API Gateway: a Product Browser, an Order Client, and a Payment Client.

API Gateway (Router)

The gateway receives all incoming requests and routes them to the appropriate isolated thread pool based on the target service. It does not use a single shared thread pool -- each downstream service gets its own.

Isolated Thread Pools (Bulkheads)

Each backend service is fronted by a dedicated thread pool with its own capacity and overflow queue:

• Product Pool -- 10 threads, queue capacity 20
• Order Pool -- 10 threads, queue capacity 20
• Payment Pool -- 5 threads, queue capacity 10 (smaller pool for the critical payment path, where tighter control is preferred)

When all threads in a pool are busy, new requests overflow into the queue. When the queue is also full, requests are immediately rejected with HTTP 429.

Backend Services

Three independent services (Product, Order, Payment), each handling its own domain. A failure or slowdown in one service only affects its own thread pool.

Key Concepts

Resource Isolation Strategies

Thread Pool Isolation: Each dependency gets a dedicated thread pool. Requests execute on pool threads, not on the caller's thread. This provides the strongest isolation -- a hung request blocks a pool thread but cannot block the caller's I/O thread. The tradeoff is higher overhead from context switching and thread scheduling.

Semaphore Isolation: A lighter-weight approach that limits concurrency using a semaphore (counter) instead of a separate thread pool. Requests execute on the caller's thread but are bounded by the semaphore permit count. Lower overhead than thread pools, but a hung request blocks the caller's thread. Best suited for fast, high-volume calls where thread pool overhead is not justified.

Process Isolation: The strongest form of bulkheading. Each service runs in its own process (or container) with dedicated CPU and memory limits. Kubernetes resource requests and limits enforce this at the infrastructure level. A memory leak or CPU spike in one service cannot affect another.

Thread Pool Sizing

Choosing the right pool size is critical:

• Too small: Legitimate requests are rejected even under normal load (starvation). Users see unnecessary 429 errors.
• Too large: The pool can absorb so many requests that it provides no meaningful isolation. A slow service can still consume excessive resources before the pool fills.
• Right-sized: Pool size should reflect the downstream service's actual throughput capacity. A common formula: pool size = requests per second * p99 latency (in seconds) + small buffer.

Overflow and Rejection

When a thread pool is saturated:

1. New requests enter the overflow queue (bounded).
2. When the queue is full, requests are rejected immediately with HTTP 429 (Too Many Requests).
3. The client receives a fast failure instead of hanging indefinitely.
4. A Retry-After header signals when the client should retry.

This fail-fast behavior is preferable to letting clients hang, as it frees client-side resources and enables retries with backoff.

Relationship to Circuit Breaker

Bulkhead and Circuit Breaker are complementary patterns that address different aspects of fault tolerance:

• Bulkhead limits the number of concurrent requests to a service (controls parallelism). It does not track error rates.
• Circuit Breaker monitors error rates and response times, tripping open when failures exceed a threshold (controls failure propagation). It does not limit concurrency.

Used together: the bulkhead prevents resource exhaustion while the circuit breaker cuts off a failing service before the bulkhead even fills up. Resilience4j, for example, recommends wrapping calls with both: the bulkhead limits concurrency, and the circuit breaker trips if the service starts failing despite the concurrency limit.

Scenarios

1. Normal Operation -- All three bulkheads operate normally. Each service handles requests through its own isolated thread pool with capacity to spare.
2. Failure Isolation -- The Order Service becomes slow (30s responses). The Order Pool fills up, but the Product Pool and Payment Pool are completely unaffected. Without bulkheads, a shared pool would let the slow service starve all other services.
3. Pool Overflow (429 Rejection) -- The Order Pool is fully saturated (threads full, queue full). New order requests are immediately rejected with 429 Too Many Requests. Product and Payment services continue operating normally with full capacity.