Distributed transactions: 2PC and Saga

There's a moment in every system's life when the database transaction stops being enough. Yesterday a single BEGIN/COMMIT was sufficient because every write happened in one schema. Today the company split the monolith and now placing an order means charging the customer in the payment service, reserving inventory in the warehouse service, and scheduling shipping in the logistics service — three databases, three teams, three deployment cycles, and zero way to wrap them in one ACID boundary.

The honest framing is: you no longer have transactions. You have a workflow that has to either complete entirely or undo itself entirely, and the database isn't going to help you. The patterns in this lesson are how production systems make that workflow feel transactional from the outside without actually being one.

A distributed transaction is the system asking for a guarantee the network cannot give. Every pattern here is a different way of buying most of that guarantee back, and being honest about the part you didn't.

The textbook answer: Two-Phase Commit

Two-Phase Commit (2PC) is the protocol every distributed-systems course teaches first. A coordinator runs the show; the participants are the databases or services that need to commit together. The protocol has, as the name suggests, two phases.

In the prepare phase, the coordinator asks every participant "can you commit this transaction?" Each participant locks the rows involved, writes a "prepared" record to its WAL, and replies yes or no. If anyone says no, the coordinator tells everyone to roll back.

In the commit phase, if everyone said yes, the coordinator tells each participant to commit. The participants release their locks and the transaction is done. If anyone fails to receive the commit message, they consult the coordinator on recovery.

PREPARE phase:
  coordinator → payment:   "prepare?"  → payment   locks row, WAL "prepared", "yes"
  coordinator → inventory: "prepare?"  → inventory locks row, WAL "prepared", "yes"
  coordinator → shipping:  "prepare?"  → shipping  locks row, WAL "prepared", "yes"

COMMIT phase (all said yes):
  coordinator → all: "commit"  → each releases locks, done

  ── but if the coordinator crashes HERE, between phases ──
  payment, inventory, shipping all sit holding locks,
  "prepared", waiting. Nobody may commit or abort alone.

It works. It is provably correct. It is also so operationally miserable that the entire industry quietly stopped using it for anything customer-facing. Three reasons:

The blocking problem. Between prepare and commit, every participant is holding row locks. If the coordinator crashes after prepare and before commit, those locks are held until the coordinator recovers. In the meantime, every other transaction touching those rows blocks. A coordinator failure during peak traffic can lock thousands of rows for minutes.

The coordinator-is-a-single-point-of-failure problem. The protocol's correctness depends on the coordinator's log being durable and recoverable. The coordinator becomes a critical dependency for every cross-service write. You've replaced one shared database with one shared coordinator and gained nothing.

The chatty-network problem. Every transaction is two round-trips to every participant. Three participants means six round-trips. At cross-AZ or cross-region latency, this adds 50-200ms to every write. Compare this to a saga (below) which is one round-trip per step and can be designed to parallelize.

You will still see 2PC in two places: XA transactions in legacy enterprise systems (mostly bank cores), and inside distributed databases (Spanner, CockroachDB) where the participants are storage shards inside one logical system and the coordinator is part of the database. In those settings the engineering cost has been paid by the database vendor and you don't have to think about it. Outside of those, you almost never want 2PC.

How much does the blocking actually cost?

Numbers make the "locks for minutes" claim concrete. Say checkout runs 2PC across three services, and a coordinator restart takes 90 seconds (crash detection + process restart + WAL replay). Checkout peaks at 500 orders/sec, and each order's prepare phase touches one hot inventory row per SKU, with popular SKUs sharing rows.

The 300 locked rows aren't the scary number. The 45,000 blocked requests are — every customer trying to buy a popular item during that 90-second window hits a hung checkout, because their order needs a lock a dead coordinator is sitting on. That's the cost the industry decided it would rather not pay, and it's why sagas (which never hold a cross-service lock) won.

The production answer: Saga

A saga abandons the goal of atomic commit. Instead it accepts that the workflow will happen as a series of local transactions, one per service, and that if a later step fails the system has to undo the earlier ones with compensating actions. The result isn't ACID — there are intermediate states a reader might see — but the end-state is consistent and the system stays available even when participants are slow.

The classic example is the e-commerce order. The happy path is "charge payment → reserve inventory → schedule shipping → confirm order". The unhappy path, if inventory fails, is "refund payment → release whatever was reserved → tell the user". Each step is a normal database transaction inside one service; the choreography between them is the saga.

forward:   chargePayment → reserveInventory → scheduleShipping → confirm
                                    │ fails here
                                    ▼
compensate (reverse order): releaseInventory → refundPayment → markFailed

Sagas come in two flavors that look different in code but address the same problem:

Orchestration — a dedicated service (the orchestrator) calls each step in order and decides what to do on failure. The orchestrator's state machine is the source of truth: "we're at step 3, payment succeeded, inventory in progress." If something fails, the orchestrator runs the compensations for completed steps. Easy to reason about, easy to debug, but the orchestrator is a single service that knows about every workflow.

Choreography — there's no central conductor. Each service listens to events from the broker and acts on them: "order-created" triggers the payment service, which emits "payment-charged", which triggers the inventory service, and so on. Failures emit compensation events that other services react to. Decentralized, loose coupling, much harder to trace through when something goes wrong. Production teams reach for choreography when steps are naturally event-driven and orchestration when the workflow has a clear owner.

Compensating actions are the hard part

The phrase "just write a compensation" hides what is, in practice, the trickiest part of saga design. Some operations have no clean inverse. You can refund a payment, sure — but the customer's bank might charge a fee. You can release reserved inventory, but if the item went out of stock for the 30 seconds it was reserved, your real-time stock count was wrong during that window.

The discipline is that every action must either be truly reversible (you can refund the exact payment) or designed as semantically reversible (instead of "send email", queue "send email to address X with content Y" and have a compensation that sends "ignore the previous email" — ugly but honest). Some operations simply can't be undone (sent a physical letter, triggered a real-world side effect), and the saga has to be designed so those are either at the very end (after every other step has committed) or guarded by a pre-check that makes them extremely unlikely to need reversal.

The reason saga authors write playbooks for failure modes is that the system's correctness depends on those playbooks. Unlike a database transaction, the compensation isn't infrastructure — it's business logic.

Isolation: the saga's silent gotcha

The thing 2PC gave you that sagas don't is isolation. Inside a 2PC transaction, no other reader sees the partial state. Inside a saga, the gap between "payment charged" and "shipping scheduled" is visible — a customer-service rep refreshing the customer's view might see "order: pending, paid: yes, shipping: not yet" for the 800 ms it takes the workflow to complete.

This isn't always a problem. Often it's fine — the workflow completes, the intermediate state is temporary, nobody notices. Sometimes it's a real bug. Two anti-patterns to watch for:

Reading the partial state and acting on it. A reporting system that runs every minute might catch the order in the "paid but not shipped" state and surface it as a discrepancy. The fix is to filter by saga completion ("only count orders where workflow_state = done"), not to fight the visibility.

Allowing concurrent sagas on the same entity. Two sagas touching the same order can interleave in ways neither author expected. The standard fix is a saga-instance lock — a row or column on the entity that says "saga X is in progress, no other saga may touch this entity until X completes or aborts". You've just reinvented a coarse lock, but at the workflow level instead of the row level.

What breaks: saga failure modes

Sagas trade 2PC's blocking for a different bag of failure modes. These are the ones that page you.

The lost-compensation. A forward step succeeds, a later step fails, and the compensation for the early step never runs — the orchestrator crashed mid-rollback, or the compensation event got dropped. Result: payment charged, no order. The fix is that the saga log itself must be durable and the orchestrator must resume in-flight sagas on restart, re-running compensations from where it left off. The saga state machine is as critical as the data.

Compensation against a moved target. You charged $50, then went to refund $50, but in between the customer's balance changed or the payment partially settled. A blind "refund $50" can now be wrong. Compensations must verify current state, not replay the forward step's assumptions.

Non-idempotent retries. A chargePayment call times out, the orchestrator retries, and the customer is charged twice — the first call actually succeeded, the timeout lied. Every saga step needs an idempotency key so a retry of an already-applied step is a no-op, not a second side effect.

The success-that-reported-failure. The trap in the war story below: a step returns an error but the side effect actually happened. Compensating "undoes" something the real world still did.

A war story about the compensation that wasn't

A team I worked with built an order saga that did charge_payment → reserve_inventory → notify_warehouse. The compensation for inventory was release_reservation; the compensation for payment was refund. Everything worked in testing because the failures they tested were the ones they could think of: inventory says no, warehouse is down, etc.

What they didn't test was a partial network failure where notify_warehouse returned an error but actually succeeded — the warehouse received the message and started picking the items. The orchestrator dutifully ran the compensation: release inventory (now over-counted), refund payment. The customer got their money back, the warehouse shipped the items, and the company lost the goods.

It happened during a Black Friday spike when the network between the orchestrator and the warehouse was dropping ~2% of responses. At a few thousand orders an hour, "2% of responses lost on the final step" was dozens of shipped-and-refunded orders before anyone noticed the inventory ledger drifting. The loss was real product, out the door, paid back to the customer.

The fix was the same fix every saga eventually adopts: every side-effect step needs to be idempotent (so retries are safe) and every check needs to verify actual state, not just trust the return code. The team added a "confirm warehouse received it" step between notification and order-complete, and added a check before inventory release that the warehouse hadn't started picking. Six more lines of code; an entire class of bug gone.

# Before: trusts the return code
notifyWarehouse(order)            # error? assume it didn't happen
if failed: releaseInventory()     # WRONG — warehouse may be picking

# After: verifies actual state before compensating
notifyWarehouse(order)
status = warehouse.getOrderStatus(order.id)   # ask, don't assume
if status == "not_received":
    releaseInventory()            # safe — confirmed nothing happened
elif status == "picking":
    escalate(order)               # human-in-the-loop, do NOT auto-refund

When you don't need any of this

Distributed transactions are a tax. Pay it only when you have to. Three escape hatches that save you the trouble:

• Co-locate the data. If two services keep changing together, consider whether they should be one service. Splitting was a decision; un-splitting is also a decision.
• Use a transactional outbox. If one service writes and the other just needs to react, the writer's local transaction stores both its own state change and an outbox row. A separate process ships the outbox row to a broker. The reader is eventually consistent but the writer's commit is atomic. See [CONCEPT]outbox-pattern.
• Make the workflow async. Many "distributed transaction" needs go away when you stop demanding the user wait for the workflow to complete. Return "we got your order, we'll email you" instead of blocking. The user-facing latency drops, the saga runs in the background, and intermediate-state visibility stops mattering because nobody's looking.

[CONCEPT]idempotency is the prerequisite skill — every saga step must be idempotent or retries will double-charge, double-ship, double-everything. [CONCEPT]consensus-raft is what's running inside the databases that promise you 2PC for free.