transactional-outbox.js·2 scenarios
Solve dual-write problem: DB + outbox in one transaction, relay publishes to Kafka, at-least-once delivery
The Transactional Outbox pattern is a reliability technique for microservice architectures that guarantees event publishing alongside database writes, without requiring distributed transactions. Instead of writing to a database and a message broker independently (which can fail partially), the service writes both the business data and the outgoing event into the same database transaction.
This pattern is widely adopted in event-driven systems:
The business service that processes commands. On each operation, it opens a single database transaction that writes both the business record (e.g., an order row) and an outbox event record. When the transaction commits, both writes succeed or both fail -- atomically.
The same database that holds business data also contains an outbox table. Each row represents a pending event with fields like aggregate_type, aggregate_id, event_type, payload, and a published flag. Because the outbox table lives in the same database, no distributed transaction is needed.
A separate process that periodically polls the outbox table for unpublished events (WHERE published = false). For each event found, it publishes the message to Kafka, waits for the broker acknowledgment, and then marks the outbox row as published. The relay tracks its position to resume correctly after restarts.
Receives events from the relay and stores them in topics partitioned by aggregate key. Downstream consumers subscribe to these topics and process events independently.
Downstream services that react to events. Because the outbox pattern provides at-least-once delivery (not exactly-once), consumers must handle duplicate messages through idempotent processing.
When a service needs to write to a database AND publish an event to a message broker, two independent systems must both succeed. If the database write succeeds but the Kafka publish fails (or vice versa), the system enters an inconsistent state: the data exists but the event was never sent, or the event was published but the data was never persisted. There is no way to make these two operations atomic without a coordination mechanism.
By co-locating the event record in the same database as the business data, the outbox pattern reduces the dual-write to a single-write problem. The database transaction guarantees atomicity: if the order row exists, the outbox event row exists too. The relay's job is purely delivery -- it reads committed events and forwards them to Kafka. The system no longer depends on two independent writes succeeding simultaneously.
The outbox pattern guarantees that every committed event will eventually be published to the broker. However, if the relay crashes after publishing to Kafka but before marking the outbox row as published, the event will be re-published on the next poll cycle. This means consumers may receive the same event more than once and must be designed for idempotency -- for example, by checking a unique event ID before processing.
The simplest relay implementation polls the outbox table on a timer (e.g., every 500ms). This works well but introduces polling latency and database load. A more efficient alternative is Change Data Capture (CDC), where a tool like Debezium tails the database's Write-Ahead Log and captures outbox inserts in near real-time. CDC eliminates polling overhead and reduces end-to-end latency to milliseconds, but adds operational complexity (WAL configuration, connector management).
Over time, the outbox table accumulates published rows. Production systems should periodically delete or archive rows where published = true to prevent unbounded table growth. Some implementations use a separate cleanup job or database partitioning to manage this.
Events for the same aggregate (e.g., the same order ID) are published in the order they were inserted into the outbox table, which matches the order of the original transactions. Cross-aggregate ordering is not guaranteed and typically not required. When publishing to Kafka, using the aggregate ID as the partition key ensures per-aggregate ordering within a topic partition.