Idempotency

The first production bug that taught me what idempotency really means involved a customer being charged for the same order eleven times in twenty minutes. The team had shipped a checkout flow where the mobile client would retry on any network hiccup — sensible enough on the face of it. What nobody had thought through was that the server's POST /charge endpoint had no way to tell two retries apart from two genuine clicks. The client's network was flaky, retries fired, and a customer who had clicked Pay exactly once watched their card statement fill up with identical charges. The team spent the next week refunding and writing apology emails.

The fix is conceptually trivial: tag each user intent with a unique key, and design the server so that two requests carrying the same key produce exactly one effect no matter how many times they arrive. That's idempotency — and it's one of those words that sounds intimidatingly mathematical and is actually just a contract between the client and server about what "safe to retry" means.

Why this matters at all

Distributed systems retry. Networks drop packets, gateways timeout, clients reconnect mid-flight. The only safe behavior in the face of an uncertain response is to retry — but the only safe retry is one that can't create a duplicate. Without idempotency, every layer of your stack that does anything sensible on a 502 is silently a source of duplicate writes. The retry logic isn't the bug; the absence of idempotency is.

A few concrete examples to anchor the idea:

• PUT /user/42 { email: x } is naturally idempotent: applying it twice produces the same final state. Safe to retry.
• DELETE /post/9 is idempotent: the second call finds it already gone. Safe to retry. (Though the response code matters — 200 vs 404 — and clients should treat both as success.)
• POST /orders { items: [...] } is not idempotent: two applications create two orders. Unsafe to retry without help.
• POST /charge { amount: 100 } is not idempotent: two applications charge twice. Catastrophically unsafe to retry without help.

The HTTP method matters less than people think; what really matters is whether the operation has a side effect that's observable and non-cumulative-by-design. Email sending is non-idempotent. Inventory decrement is non-idempotent. Anything that "creates" something is non-idempotent by default.

The Idempotency-Key pattern

Stripe popularized the convention that most modern APIs now follow: the client generates a unique key per intent and sends it as an HTTP header. Idempotency-Key: a8f73-... — usually a UUID, sometimes a content-derived hash for very pure cases.

The server does the following dance:

1. Receive the request with idempotency key K.
2. Attempt to atomically claim K (e.g., SETNX in Redis, or INSERT INTO idempotency_keys ... ON CONFLICT DO NOTHING in Postgres). The store should return whether this request was the first to claim K.
3. If K was unclaimed: this is the first attempt. Process the request. Store the response (status code, headers, body) under K. Return it.
4. If K was already claimed: a previous attempt already handled this. Look up the stored response. Return exactly that same response — same status code, same body — without re-processing.

In code, the whole pattern fits in a dozen lines. The ordering of these lines is the entire correctness story:

function handleCharge(key, body):
  # 1. RESERVE first — atomic, before any side effect
  claimed = db.insert("idempotency_keys", {key, status: "pending"})
                .onConflict().doNothing()
  if not claimed:
    record = db.get("idempotency_keys", key)
    if record.status == "pending":
      return 409  # in-flight; client should back off and retry
    return record.response  # replay the stored answer verbatim

  # 2. DO the side effect (this is the dangerous part)
  response = chargeCard(body.amount)

  # 3. STORE the result, flip pending -> done, atomically
  db.update("idempotency_keys", key, {status: "done", response})
  return response

The contract this gives the client is: "if you retry with the same key, you'll get the same answer." Whether the original attempt succeeded or failed, the retry gets the same outcome. The client doesn't have to know if their first attempt actually reached the server, because retrying is safe regardless.

Idempotency is what lets your retry logic be aggressive without being destructive. Without it, "retry on any 5xx" is a footgun pointing at your customers' wallets.

Where the key comes from

The key has to be generated by the client, at the moment of intent, not at the moment of HTTP send. This is subtle and critical. Consider the difference:

• Generated when the user clicks Pay (correct): a single intent has one key, every retry uses that key, the server deduplicates.
• Generated each time the HTTP client sends a request (wrong): every retry uses a new key, the server thinks they're separate intents, you get duplicate charges.

Mobile clients usually persist the key in local storage tied to the form submission, so even an app crash + restart preserves the key and a retry from the new app instance is correctly deduplicated. Web clients can use sessionStorage or hide the key in a form field.

If the client doesn't generate the key, the next-best option is the server generating it deterministically from request content — e.g., hash(user_id + intent_type + timestamp_rounded_to_5s). This is fragile (any difference in the input produces a different key, including timing jitter), and it's why Stripe's documentation pushes you so hard toward client-side keys.

What can go wrong

The three failure modes I've seen most often, in order of frequency:

Storing the result after the side effect. The server does the work, then stores the key + response. A crash between the two creates a duplicate when the client retries. The fix is to reserve the key first, do the work, then store the result. If the work fails, you have a choice: keep the key + remember the failure (so retries return the same error consistently), or release the key (so retries can re-attempt the operation). Stripe keeps it — they consider the request "done" from the API's perspective even if processing failed, because the client already saw the error and got their answer.

Wrong scope for the key. I've seen POST /payments keyed by user ID (so the same user could never pay twice in the TTL window) and keyed by request body hash (so a customer ordering two identical items on purpose got one of them deduplicated away). The right scope is one user intent = one key. The user clicked Pay; the key represents that click. If they click Pay again on the same screen the next day, that's a new intent and needs a new key.

TTL too short. If the idempotency record expires after 24 hours and the client retries on day 2, the server doesn't know the request is a retry and processes it again. The fix is a TTL longer than any plausible client retry window, plus a hard upper bound on how long the client itself retries. Stripe defaults to 24 hours; most other systems should pick the same.

How big does the key store get?

The fear that stops teams adopting idempotency keys is usually "won't that table grow forever?" Run the back-of-envelope before you worry. Say a payments API does 500 writes/second at peak and you store each key with its response — call it ~1 KB per record (UUID + status code + a small JSON body + timestamps):

The number that matters is the steady-state size, not the cumulative one — a TTL of 24 hours means records older than a day are swept, so the table converges to "one day's traffic" and stops growing. 86 GB of TTL'd key+response data is a rounding error next to the ledger it protects. If even that feels heavy, store only the key + status and re-fetch the response from the real record on replay; that drops you to ~64 bytes/record and the whole table fits in Redis.

Idempotency in queue-based systems

Synchronous APIs are the easy case for idempotency because the client controls the retry. Queues are the hard case because the broker controls the retry, and most brokers guarantee at-least-once delivery — meaning the consumer will sometimes see the same message twice and has to handle it.

The pattern in queues is the same idea applied at the consumer:

• Natural-key dedup: the message carries a unique ID, the consumer's first action is INSERT INTO processed_msgs (id) ON CONFLICT DO NOTHING and only proceeds if the insert affected a row. If the consumer crashes after processing but before committing the offset, the broker redelivers, the consumer sees the duplicate, the insert no-ops, and processing is skipped.
• Side-table dedup: similar to natural-key but in a separate table just for tracking. Useful when the natural primary key of the entity isn't a good idempotency token.
• Idempotent state transitions: design the operation so that re-applying it is a no-op. UPDATE orders SET status = 'paid' WHERE id = ? AND status = 'pending' is idempotent because the second application finds the row not in 'pending' and does nothing.

The third option is the most elegant when you can manage it, because it requires no dedup infrastructure — the database row's state IS the dedup mechanism. The first two are universal fallbacks for operations that don't have a natural state transition to exploit.

A war story about the dedup that wasn't

A team I worked with built a notification service that emitted push notifications via Firebase. The producer (their backend) wrote a "send notification" message to Kafka; the consumer (a worker) read the message and called Firebase. They added idempotency by using (user_id, notification_type, day) as a dedup key.

What they didn't anticipate was that for one specific notification type — a price-alert that should fire when a followed stock crosses a threshold — a user could legitimately want multiple notifications in a single day if the stock crossed the threshold multiple times. Their dedup key was too coarse. Users started complaining that they were missing the second and third alerts on volatile days. The team had to redesign the key per notification type — some used (user_id, type, day), others used (user_id, type, event_id) where event_id was the specific threshold-crossing event.

The lesson — and it's one every team learns the hard way — is that idempotency is per operation, not per type. The granularity at which you dedup has to match the granularity at which the user has distinct intents. Get it wrong, and idempotency that seemed correct in unit tests becomes silent dropped operations in production.

When you don't need it

Three cases where idempotency genuinely doesn't apply:

• Pure reads. GET handlers should be idempotent by HTTP semantics; if they have side effects, the side effect is the bug, not the lack of an idempotency key.
• Internal calls between trusted services where you control both sides. If you can guarantee no retries on the calling side (e.g., a single-shot batch job), the dedup overhead is pure cost.
• Operations that are naturally idempotent by virtue of their effect. SET key value is idempotent because the final state doesn't depend on how many times you set it. Don't add ceremony around something that's already safe.

For everything else — anything that creates, charges, sends, or otherwise alters state in an externally-observable way — the idempotency key is non-optional. Treat its absence as a bug report waiting to be written.

[CONCEPT]message-queues pairs with this directly — every consumer is responsible for dedupe, and at-least-once delivery means the dedup isn't optional. [CONCEPT]distributed-transactions relies on it too: every saga step must be idempotent or the retries will double-everything. [CONCEPT]circuit-breaker benefits from it indirectly: when the breaker opens and the fallback re-queues, the eventual reprocess is safe only if the original operation was idempotent.