I Built an Event-Driven Order System Twice: First Incorrectly, Then Reliably
Transactional outboxes, idempotent consumers, compensation, and why “exactly once” is the wrong promise
A new order looks like one operation to a customer.
Inside a distributed system, it is not.
The order must be recorded. Inventory must be reserved. Payment must be authorized. Fulfilment must be created. Each step owns a different database, communicates asynchronously, and can fail after doing useful work but before reporting success.
My first implementation ignored most of that complexity:
write order to PostgreSQL publish OrderCreated to Kafka return 202 Acceptedplain text
It worked perfectly—until I placed a failure between any two lines.
So I built the system twice. The first version was deliberately naive. I reproduced its inconsistencies, then added one reliability mechanism at a time: a transactional outbox, inbox deduplication, optimistic versions, bounded retries, dead-letter queues, compensation, replay controls, and failure injection.
The result is an event-driven order system written in Rust, using Tokio, Axum, SQLx, PostgreSQL, and Redpanda.
The important result is not the stack. It is the guarantee:
atomic local state + outbox + at-least-once publication + idempotent local and external effects + explicit compensation = a workflow that converges without pretending delivery happens exactly onceplain text
That statement is narrower than “exactly once.” It is also something the system can defend under failure.
Start with invariants, not components
Architecture diagrams make unreliable systems look calm. Invariants make failure visible.
For one order o, let:
R(o) = active inventory reservations A(o) = committed payment authorizations F(o) = created fulfilmentsplain text
The system permits duplicate deliveries and repeated handler attempts. It does not permit duplicate business effects:
R(o) ≤ 1 A(o) ≤ 1 F(o) ≤ 1plain text
Inventory has a conservation rule for every SKU:
available + reserved = initial - fulfilled available ≥ 0 reserved ≥ 0plain text
After a crash, a healthy result may therefore look like this:
event deliveries = 2 handler invocations = 2 inventory reservations = 1 payment authorizations = 1 fulfilments created = 1plain text
The duplicate is not eliminated. It is made harmless.
This distinction became the foundation of the project. Delivery count, execution count, and business-effect count are different things.
The first design had two ways to lie
The naive Orders API performed two independent commits:
Client Orders DB Orders API Redpanda | | | | | POST /orders | | | |---------------->| | | | |<-- INSERT -------| | | |--- COMMIT ------>| | | | |--- publish ----->| |<---------------------- 202 --------| |plain text
There is no transaction spanning PostgreSQL and Redpanda. That creates two failure windows.
Failure 1: the order commits, but the event disappears
PostgreSQL COMMIT ✓ | X process dies | Kafka publish never happensplain text
The database has an order. Inventory never hears about it. The order remains pending forever.
Retrying Kafka can help with a temporary broker error, but it cannot recover a process that lost the intention to publish.
Failure 2: the event publishes, but the response disappears
PostgreSQL COMMIT ✓ Kafka publish ✓ | X HTTP response is lost | client retries an ambiguous requestplain text
An HTTP idempotency key prevents a second order row. It does not prove whether the first event was published. If the naive path publishes again, consumers receive a duplicate.
Publishing before the database commit does not solve this. It merely creates an event for an order that might later roll back.
There is no safe ordering of two independent commits.
I kept this broken implementation runnable. make demo-naive-failure injects both failures and records the database and broker outcomes. Reliability mechanisms make more sense as responses to observed failures than as items on an architecture checklist.
The outbox moved the atomic boundary
The solution was not to make PostgreSQL and Kafka participate in one distributed transaction.
It was to store the intention to publish inside the same PostgreSQL transaction as the order:
BEGIN INSERT INTO orders ... INSERT INTO order_items ... INSERT INTO outbox_events ... COMMITplain text
Either the order and the event intention both commit, or neither does.
Client -> Orders transaction -> [orders + outbox_events] | v outbox publisher | v Redpandaplain text
A background publisher claims eligible rows using FOR UPDATE SKIP LOCKED. It attaches a short lease, commits the claim, and only then contacts Redpanda. Multiple publishers can share the backlog without holding database transactions open during network calls.
But the outbox does not make publication exactly once.
Consider this crash:
publish to Redpanda ✓ | X publisher dies before recording success | lease expires | another publisher sends the event againplain text
The publisher cannot know whether the first publication became durable. Retrying is the safe choice, so the transport remains at least once.
The outbox prevents lost publication intent. It deliberately leaves duplicate handling to consumers.
The inbox makes redelivery harmless
Each consumer keeps an inbox keyed by (consumer_name, event_id).
Handling an event is one local transaction:
BEGIN validate the envelope claim the inbox identity verify the payload hash check the semantic version apply the business mutation write resulting outbox events mark the inbox entry processed advance the local source offset COMMITplain text
If the same event arrives again, the inbox identity already exists. The consumer verifies that the payload hash matches and acknowledges the duplicate without repeating business work.
The hash check matters. The same event ID with different content is not a normal duplicate; it is an integrity failure.
This project uses rskafka, which does not provide Kafka consumer groups. Each service therefore stores its consumed offsets in PostgreSQL. The implementation differs from a standard group consumer, but the safety rule is the same:
Source progress advances only after the local effect—or a durable dead-letter handoff—has committed.
If the process crashes after committing the database transaction, it may read the event again. The inbox turns that redelivery into a no-op.
An inbox cannot protect an external payment provider
Now consider a different ambiguous boundary:
Payments service -> Provider: authorize $50 Provider commits authorization ✓ Provider -> Payments service: success X response is lost Payments service retriesplain text
The inbox cannot help. The consumer may still be processing its first event; the ambiguity occurred inside an external call.
The retry must reuse a stable provider operation key:
payment:{order_id}:authorize:v1plain text
The fake provider stores that key independently. A repeated request returns the result of the original logical operation instead of authorizing again.
The finished system therefore has several idempotency boundaries:
- The HTTP idempotency key protects order creation from client retries.
- The inbox protects local consumer state from broker redelivery.
- Unique domain constraints protect against distinct messages requesting the same logical effect.
- The provider operation key protects money from ambiguous network responses.
No single idempotency table solves all four problems.
Choreography made ownership explicit
The system contains four services:
- Orders owns lifecycle, transition history, and workflow decisions.
- Inventory owns stock and reservations.
- Payments owns authorizations, refunds, and the provider ledger.
- Fulfilment owns fulfilment creation.
They share a PostgreSQL server locally but use separate databases. No service reads another service’s tables.
The workflow separates commands from facts:
orders.order_created fact inventory.reserve_inventory command inventory.reservation_succeeded fact payments.authorize_payment command payments.payment_authorized fact fulfilment.create_fulfilment command fulfilment.fulfilment_created factplain text
Orders consumes outcomes, stores the relevant facts, and issues the next command. Inventory does not decide whether payment should start. Payments does not infer whether fulfilment is ready.
The happy path is simple:
PENDING -> INVENTORY_RESERVED -> PAYMENT_AUTHORIZED -> READY_FOR_FULFILMENT -> COMPLETEDplain text
The failure paths are more revealing.
Compensation is not rollback
Once several services have committed locally, there is no global rollback button.
If payment fails after inventory was reserved, Orders requests an inventory release.
If fulfilment fails after payment and reservation both succeeded, Orders requests both a refund and a release:
Fulfilment failed | +--> refund payment ------> PaymentRefunded ----+ | | +--> release inventory ---> InventoryReleased --+ | v CANCELLEDplain text
The order enters CANCELLING while those actions are in progress. It becomes CANCELLED only after every required confirmation arrives.
Marking it cancelled earlier would make the API lie. Money or stock could remain stranded behind a clean-looking terminal state.
Compensations are idempotent. Refunding an already-refunded payment and releasing an already-released reservation return logical success without repeating the effect.
If compensation repeatedly fails, the system does not pretend that the rollback succeeded. It enters MANUAL_REVIEW, emits an operator-visible failure, and preserves the audit trail.
That is not a weakness. It is an honest representation of unresolved state.
Kafka ordering is not business ordering
All workflow records use order_id as their Redpanda key. The project uses single-partition topics for deterministic learning.
That still does not create one universal version sequence.
Commands sent to different services are independent. The system uses a version counter per (order_id, target_service):
order 42 -> inventory v1: reserve order 42 -> payments v1: authorize order 42 -> inventory v2: release order 42 -> payments v2: refundplain text
Without per-target versions, Payments could receive version 2 simply because Inventory received version 1. It would report a gap for a message it was never supposed to see.
Consumers classify incoming versions:
incoming = last + 1 -> apply incoming <= last -> stale or duplicate incoming > last + 1 -> bounded gap recovery unresolved gap -> dead-letter queueplain text
Offsets advance only through a contiguous completed prefix. Finishing a later record must not cause an unresolved earlier record to be skipped after a crash.
Ordering is a semantic rule enforced by the application, not a blanket statement that “Kafka orders messages.”
Retry only failures that may change
Retries are useful when another attempt can produce a different result.
- Timeouts, connection failures, and temporary broker outages are transient.
- Deadlocks and resolvable version conflicts are contention.
- Rate limits should honor a delay.
- Insufficient stock and declined payments are valid business outcomes.
- Invalid schemas, corrupt envelopes, and identity/hash mismatches are poison messages.
Transient retries use exponential backoff with full jitter:
delay ∈ [0, min(cap, base × 2^attempt)]plain text
Attempts and total elapsed time are bounded. Concurrency is bounded too; otherwise a recovering dependency can be overwhelmed by a large population of well-jittered retries.
A declined payment produces a business event. It does not belong in a dead-letter queue.
An unsupported event schema does belong there. The DLQ record preserves the original location, envelope, error code, attempts, and replay metadata.
Replay is not diagnosis. An operator corrects the underlying problem first, then replays the original event identity so all existing idempotency protections remain active.
Correlation IDs are part of correctness
An asynchronous workflow cannot be debugged using one HTTP request log.
Every event carries its identity, type, producer, aggregate, semantic version, correlation ID, causation ID, timestamp, and trace context.
That creates a causal chain:
POST /orders └─ OrderCreated └─ ReserveInventory └─ ReservationSucceeded └─ AuthorizePayment └─ PaymentAuthorized └─ CreateFulfilment └─ FulfilmentCreatedplain text
Correlation reconstructs one order across four services. Metrics reveal whether the class of orders is healthy.
The useful metrics are bounded: outbox age, backlog, consumer lag, retry counts, duplicates, gaps, DLQ traffic, and compensation age.
Order IDs, event IDs, and SKUs do not become metric labels. That would create unbounded cardinality.
The most useful tests killed the process
The final suite covers more than successful requests:
- database commit followed by missing publication;
- publication followed by publisher death;
- consumer commit followed by redelivery;
- payment success followed by response loss;
- duplicate and reordered events;
- concurrent order requests sharing one idempotency key;
- concurrent reservations competing for the same stock;
- broker and database outages;
- poison records;
- compensation exhaustion;
- replay of ordinary and dead-lettered events.
Faults are named and deterministic. Property tests use recorded seeds, generated event histories, fake clocks, and bounded polling rather than unexplained sleeps.
One test found a particularly instructive bug: an outcome referencing an unknown order returned a fatal error. The consumer’s source position never advanced, so one poison record wedged the stream forever.
The fix was not another blind retry. It was to classify the record, publish a durable DLQ entry, advance safely, and allow unrelated work to continue.
The reliability test found a failure that the happy path could not.
What the system actually guarantees
After 202 Accepted, the order and its initial publication intentions are durable in one local commit.
Outbox events are attempted at least once. Publisher crashes may create duplicates.
Inbox identity, payload hashes, unique constraints, and versions make those duplicate deliveries harmless to local business state.
Stable provider keys make authorization and refund retries harmless inside the bundled fake provider.
Successful orders complete. Expected failures compensate and cancel. Exhausted compensation becomes visible manual work.
Poison records reach a DLQ rather than blocking the stream indefinitely.
What it does not guarantee
The system does not provide:
- exactly-once message delivery or handler execution;
- atomic commits across PostgreSQL and Redpanda;
- automatic idempotency for an arbitrary external provider;
- global ordering across topics or partitions;
- production consumer-group scaling;
- multi-region failover;
- PCI-compliant payment processing;
- automatic resolution of permanently failed compensation.
The single-partition design and PostgreSQL offset ledger are useful constraints for deterministic learning. They are not throughput claims.
A production evolution would add partitioned consumer coordination, schema governance, real provider reconciliation, retention policies, secured administrative endpoints, and disaster-recovery testing.
The reusable lesson
The most valuable part of the project was not a specific Rust crate or database query. It was the sequence:
state an invariant ↓ identify an ambiguous boundary ↓ reproduce the failure ↓ add the smallest mechanism that addresses it ↓ state what the mechanism still cannot guarantee ↓ test the complete workflow under failure againplain text
Every major mechanism entered because a test violated an invariant:
DB commit without event -> transactional outbox duplicate publication -> inbox + idempotent effects provider response loss -> provider idempotency key concurrent mutation -> versions + constraints out-of-order commands -> per-target sequences partial distributed success -> compensation permanent bad message -> DLQ dependency outage -> bounded retry + jitter invisible async failure -> correlation + telemetryplain text
The final system does not make failure disappear.
It makes failure durable, classifiable, observable, retryable where appropriate, and explicit when human intervention is required.
That is the reliability claim:
An accepted order and its publication intent commit together. Messages may arrive more than once. Each service makes repeated delivery harmless at its own boundary, external effects use stable idempotency keys, and incomplete workflows converge through compensation or visible manual review.
Less magical than “exactly once.” More useful in an incident.
The complete implementation, failure lab, architecture decisions, milestone evidence, chaos suite, runbook, and replay tool are available in the GitHub repository.
If you want to explore it locally:
make demo-naive-failure make test make chaos-smokebash