retry-backoff.js·4 scenarios
Handle transient failures: retry with 1s -> 2s -> 4s backoff, jitter, dead letter queue on max retries
Retry with exponential backoff is a resilience pattern for handling transient failures in distributed systems. Instead of failing immediately or retrying in a tight loop, the caller waits progressively longer between each attempt, giving the failing service time to recover without being overwhelmed by retry traffic.
This pattern is a standard building block in virtually every distributed system and is built into many libraries and SDKs:
The external caller that initiates requests. From the client's perspective, the retry logic is transparent -- it either receives a successful response (possibly after invisible retries) or an error after all attempts are exhausted.
The gateway wraps outbound calls with a configurable retry policy: maximum retry count, initial delay, backoff multiplier, and maximum delay cap. The retry policy intercepts failures and decides whether to retry or give up based on these parameters.
The internal service processes requests and depends on an external third-party API that is subject to intermittent failures: rate limiting (429), temporary overload (503), timeouts, and connection errors. These are the transient failures that retries are designed to handle.
When all retry attempts are exhausted, the failed request is persisted to a Dead Letter Queue rather than being silently dropped. This ensures no data is lost and allows operators or automated processes to review, diagnose, and replay failed requests later.
The delay between retries grows exponentially: delay = initialDelay * 2^retryNumber. With an initial delay of 1 second and a multiplier of 2, the sequence is 1s, 2s, 4s, 8s. This progressively longer wait gives the failing service more time to recover with each attempt, and reduces the load the retrying client places on the system. A maximum delay cap (e.g., 16s) prevents the wait from growing unbounded.
Without jitter, thousands of clients that fail at the same moment will all retry at the same intervals (1s, 2s, 4s), creating synchronized traffic spikes known as the "thundering herd" problem. These synchronized waves can keep a recovering service down permanently. Jitter adds a random component to each delay, spreading retries across time. Common jitter strategies include:
random(0, baseDelay * 2^retry) -- maximum spread, recommended by AWShalfDelay + random(0, halfDelay) -- balanced between spread and minimum waitmin(maxDelay, random(baseDelay, prevDelay * 3)) -- adaptive, based on previous delayWhen retries are exhausted, the request must go somewhere. Silently dropping it means data loss. Returning an error to the client is necessary, but the request payload and context should also be preserved. A Dead Letter Queue stores these failed requests with metadata (timestamps, error codes, attempt count) for later analysis. Operators can fix the root cause and replay the messages, or automated processes can retry them during off-peak hours.
Retries inherently mean a request may be executed more than once. If the operation is not idempotent (e.g., charging a credit card, sending an email), retries can cause duplicate side effects. Operations protected by retry policies must be designed to be safe for re-execution, typically through idempotency keys, deduplication checks, or conditional writes.
Not all failures are transient. Retrying non-transient errors wastes time and resources:
Retry policies should include an allowlist of retryable status codes (typically 429, 500, 502, 503, 504) and immediately propagate all other errors to the caller.
In high-throughput systems, unlimited retries can amplify load on an already struggling service. A retry budget limits the total percentage of requests that can be retries (e.g., 20% of total traffic). Once the budget is exceeded, new retry attempts are suppressed, protecting the downstream service from retry storms.