Outbox pattern

Two storage systems, one logical action, no transaction spanning both. That's the entire problem. You save an order to Postgres and publish an OrderPlaced event to Kafka so the rest of the company can react — email, inventory, analytics. Those are two separate systems, and there is no COMMIT that covers both. Whichever you do second can fail on its own, and now your database and your event stream disagree about whether the order exists.

It looks like a problem you can dodge by being careful about ordering. You can't. Both orderings are broken:

BEGIN;
INSERT INTO orders ...;
COMMIT;                ← process dies HERE
kafka.publish(event);  ← never runs. Order exists, nobody told.

kafka.publish(event);  ← succeeds
BEGIN;
INSERT INTO orders ...;
COMMIT;                ← fails. Event fired for an order that doesn't exist.

Publish-first loses you data integrity; commit-first loses you the event. There is no third ordering. The flaw is structural: a crash in the gap between two systems leaves them inconsistent, and the gap can't be closed by reordering it.

You cannot make a write to a database and a publish to a broker atomic by doing them carefully in the right order. A crash in the gap breaks you either way. The only fix is to remove the second system from the critical write.

A war story: the night 14,000 emails didn't send

A team published UserSignedup straight after the commit — commit-first, the "safe" ordering. It worked for a year. Then a deploy rolled during a traffic spike, the service was killed mid-request thousands of times in the rollout window, and each kill landed in exactly that gap: row committed, event never published.

The users existed. They could log in. But the welcome email, the provisioning webhook, and the analytics signup event never fired for roughly 14,000 accounts. Nobody noticed for three days — there was no error, no exception, no dropped request. The signups simply had no downstream effects. They found it when a marketing dashboard showed a suspicious dip and someone cross-checked the users table against the email provider's send log.

There was no clean recovery. They wrote a one-off backfill that diffed the two systems and re-emitted events — and got to re-learn that their consumers weren't idempotent, so the backfill double-sent a few thousand emails on top. The whole incident is what the outbox pattern exists to prevent.

The fix: make the event part of the transaction

Add an outbox table to the same database as your business data. In the same transaction that writes the order, insert a row describing the event. Now the event and the data commit together — both or neither — because they're the same COMMIT.

BEGIN;
INSERT INTO orders (...) ...;
INSERT INTO outbox (event_type, payload, created_at) ...;
COMMIT;                ← atomic: the event can't be lost without the order

The broker is no longer on the critical write path. A separate relay reads unpublished outbox rows and ships them to Kafka, marking each sent:

SELECT * FROM outbox WHERE published_at IS NULL ORDER BY id LIMIT 100;
for each row:
  kafka.publish(row.event_type, row.payload);
  UPDATE outbox SET published_at = now() WHERE id = row.id;

If Kafka is down, rows pile up and the relay retries — nothing is lost, delivery is just delayed. If the relay crashes after publishing but before the UPDATE, it republishes on restart. Which leads to the one rule every outbox consumer must obey:

Naive alternatives, and why they lose

The outbox is the sweet spot: real atomicity at medium complexity. 2PC gets you atomicity too, but most brokers don't speak XA, and distributed two-phase commit is its own operational nightmare — see [CONCEPT]distributed-transactions.

Two ways to drain the outbox

Polling

SELECT ... WHERE published_at IS NULL ORDER BY id LIMIT 100 on a timer. Dead simple, deployable as a sidecar, runs anywhere your DB does. The costs: it adds query load, and end-to-end latency is roughly the poll interval.

Do the arithmetic before assuming it's free. Polling once per second is 86,400 queries/day per poller. One outbox is nothing. But teams often run an outbox per service, with redundant poller replicas:

A couple of million WHERE published_at IS NULL queries a day is fine if that predicate is indexed and the table stays small. It quietly becomes a problem when the outbox isn't compacted and the query starts scanning millions of already-published rows.

Change-data-capture

Point Debezium (or native logical replication) at the outbox table and stream its write-ahead log straight to Kafka. No polling, sub-second latency, no query load on the table. The price is operational: you're now running Debezium + Kafka Connect and the WAL-slot plumbing behind it. Heavyweight, and the right answer once polling load or latency actually hurts — not before.

What to monitor

An outbox fails silently — the whole point is that nothing errors — so you watch it directly:

• Outbox depth: COUNT(*) WHERE published_at IS NULL. A steadily climbing number means the relay is down or Kafka is rejecting; alert on it before customers notice missing downstream effects.
• Publish lag: now() - min(created_at) WHERE published_at IS NULL — the age of the oldest unsent event. This is your real end-to-end latency, and the first thing to page on.
• Relay dead-letter count: events that fail to publish repeatedly (bad payload, schema mismatch) shouldn't block the queue forever — route them aside and count them.

When NOT to use an outbox

• The event is genuinely fire-and-forget and losable — real-time presence, a "user is typing" ping. The atomicity machinery isn't worth it.
• One service owns both the DB and the only consumer — just call the consumer in-process, inside the same transaction's success path.
• You're already event-sourced ([CONCEPT]event-sourcing): the event store is your source of truth, so there's no second system to stay consistent with.

The outbox is the unglamorous plumbing that lets [CONCEPT]message-queues actually deliver every event your data implies — paired with [CONCEPT]idempotency on the consumer side, it gets you effectively-once delivery across two systems that share no transaction.