Retry with Exponential Backoff

Overview

Retry with exponential backoff is a resilience pattern for handling transient failures in distributed systems. Instead of failing immediately or retrying in a tight loop, the caller waits progressively longer between each attempt, giving the failing service time to recover without being overwhelmed by retry traffic.

This pattern is a standard building block in virtually every distributed system and is built into many libraries and SDKs:

• AWS SDK -- all AWS SDKs implement exponential backoff with jitter by default
• gRPC -- built-in retry policy with exponential backoff
• Envoy Proxy -- configurable retry budgets with backoff
• Spring Retry -- annotation-driven retry with customizable backoff strategies
• Tenacity -- Python retry library with exponential and custom backoff
• Polly -- .NET resilience library with retry policies
• Axios Retry / Got -- HTTP client retry plugins for Node.js

Architecture

Client Application

The external caller that initiates requests. From the client's perspective, the retry logic is transparent -- it either receives a successful response (possibly after invisible retries) or an error after all attempts are exhausted.

API Gateway with Retry Policy

The gateway wraps outbound calls with a configurable retry policy: maximum retry count, initial delay, backoff multiplier, and maximum delay cap. The retry policy intercepts failures and decides whether to retry or give up based on these parameters.

Internal Service and External API

The internal service processes requests and depends on an external third-party API that is subject to intermittent failures: rate limiting (429), temporary overload (503), timeouts, and connection errors. These are the transient failures that retries are designed to handle.

Dead Letter Queue

When all retry attempts are exhausted, the failed request is persisted to a Dead Letter Queue rather than being silently dropped. This ensures no data is lost and allows operators or automated processes to review, diagnose, and replay failed requests later.

Key Concepts

Exponential Backoff

The delay between retries grows exponentially: delay = initialDelay * 2^retryNumber. With an initial delay of 1 second and a multiplier of 2, the sequence is 1s, 2s, 4s, 8s. This progressively longer wait gives the failing service more time to recover with each attempt, and reduces the load the retrying client places on the system. A maximum delay cap (e.g., 16s) prevents the wait from growing unbounded.

Jitter

Without jitter, thousands of clients that fail at the same moment will all retry at the same intervals (1s, 2s, 4s), creating synchronized traffic spikes known as the "thundering herd" problem. These synchronized waves can keep a recovering service down permanently. Jitter adds a random component to each delay, spreading retries across time. Common jitter strategies include:

• Full Jitter: random(0, baseDelay * 2^retry) -- maximum spread, recommended by AWS
• Equal Jitter: halfDelay + random(0, halfDelay) -- balanced between spread and minimum wait
• Decorrelated Jitter: min(maxDelay, random(baseDelay, prevDelay * 3)) -- adaptive, based on previous delay

Dead Letter Queue

When retries are exhausted, the request must go somewhere. Silently dropping it means data loss. Returning an error to the client is necessary, but the request payload and context should also be preserved. A Dead Letter Queue stores these failed requests with metadata (timestamps, error codes, attempt count) for later analysis. Operators can fix the root cause and replay the messages, or automated processes can retry them during off-peak hours.

Idempotency

Retries inherently mean a request may be executed more than once. If the operation is not idempotent (e.g., charging a credit card, sending an email), retries can cause duplicate side effects. Operations protected by retry policies must be designed to be safe for re-execution, typically through idempotency keys, deduplication checks, or conditional writes.

When NOT to Retry

Not all failures are transient. Retrying non-transient errors wastes time and resources:

• 400 Bad Request -- the request is malformed and will fail every time
• 401/403 Unauthorized -- authentication or authorization failure, retrying will not help
• 404 Not Found -- the resource does not exist
• 409 Conflict -- business logic conflict that requires client-side resolution
• 422 Unprocessable Entity -- validation failure

Retry policies should include an allowlist of retryable status codes (typically 429, 500, 502, 503, 504) and immediately propagate all other errors to the caller.

Retry Budget

In high-throughput systems, unlimited retries can amplify load on an already struggling service. A retry budget limits the total percentage of requests that can be retries (e.g., 20% of total traffic). Once the budget is exceeded, new retry attempts are suppressed, protecting the downstream service from retry storms.

Scenarios

1. Success on First Try -- Request succeeds immediately with no retries needed, demonstrating zero overhead from the retry policy when the service is healthy
2. Exponential Backoff (Success on 4th Try) -- Request fails three times (429, 503, 500) with exponential backoff delays of 1s, 2s, and 4s between attempts, then succeeds on the fourth try after 7 seconds of total backoff
3. Max Retries Exceeded -- All four attempts fail against a completely down service, the request is saved to the Dead Letter Queue for later processing, and the client receives a 502 error
4. Jitter (Anti-Thundering Herd) -- Multiple clients experience the same failure but retry at staggered times due to random jitter, preventing synchronized retry waves and allowing the service to recover gradually