At-Most-Once, At-Least-Once, and Exactly-Once: What Message Delivery Guarantees Really Mean
Your broker's delivery count is the wrong metric. At-most-once, at-least-once, and exactly-once semantics, and how to guarantee one committed effect per event.

Distributed systems communicate through messages, but message transport and business correctness are different problems.
Consider this pipeline:
Producer → Broker → Consumer → Database → External systemplain text
A broker may deliver a message once while the consumer applies it twice. It may deliver a message twice while an idempotent consumer applies it once. It may provide exactly-once processing inside Kafka while an external PostgreSQL update still happens twice.
The useful question is therefore not:
How many times did the broker deliver the message?
It is:
How many times did the logical operation affect authoritative state?
TL;DR
| Semantic | Message loss | Duplicate delivery | Practical interpretation |
|---|---|---|---|
| At-most-once | Possible | Prevented | Execute zero or one time |
| At-least-once | Prevented under stated durability and liveness assumptions | Possible | Retry until acknowledged |
| Exactly-once | Prevented within a defined transactional boundary | Still occurs; suppressed before commit | One logical input produces one committed result |
For most correctness-sensitive systems, the practical design is:
At-least-once delivery + stable event IDs + idempotent consumers + atomic database transactions = effectively-once business effectsplain text
Effectively-once is the working term for that outcome. It concedes that the transport will redeliver and asserts only that the committed effect is applied once. It is a weaker claim than exactly-once and a far easier one to defend.
Exactly-once is not a broker-wide magic switch. It is a property of a specific processing boundary.
1. Formal model
Let m represent one logical message.
Define:
D(m): number of timesmis delivered to a consumer.H(m): number of times the handler executes.E(m): number of externally visible business effects produced bym.
These counts are not necessarily equal.
A broker may redeliver a message while an idempotent consumer produces only one effect:
D(m) = 2 H(m) = 2 E(m) = 1plain text
A buggy handler may produce two effects from one delivery:
D(m) = 1 H(m) = 1 E(m) = 2plain text
Transport-level exactly-once delivery would not fix the second case.
The application usually cares about:
E(m) = 1plain text
This distinction follows the end-to-end principle: lower system layers can help with reliability, but correctness often has to be enforced and verified at the application endpoints. Saltzer, Reed, and Clark, “End-to-End Arguments in System Design”
Every section that follows is tagged with the signature it produces, so the mechanism and its guarantee stay attached to each other.
2. Why the problem exists
Assume a consumer receives a trade event:
{ "eventId": "trade-123", "accountId": "alice", "market": "SOL-PERP", "signedQuantity": "10", "price": "100" }plain text
The consumer must:
- Update Alice’s position in PostgreSQL.
- Acknowledge the message to the broker.
These operations occur in independent systems.
A normal execution is:
Receive trade-123 Commit position update Acknowledge trade-123plain text
Now consider the failure windows.
Failure before database commit
Receive event Begin database transaction Worker crashesplain text
The database rolls back. The broker must redeliver the message or the event is lost.
Failure after commit but before acknowledgement
Receive event Commit database transaction Worker crashes Acknowledgement never reaches brokerplain text
The broker cannot determine whether the database committed.
The missing acknowledgement is consistent with several executions:
A. Database committed; worker crashed before ACK B. Database committed; ACK was lost C. Database did not commit; worker crashedplain text
If the broker redelivers, cases A and B can produce duplicates.
If it does not redeliver, case C loses the event.
Acknowledgement deadline expiry
The most frequent source of duplicates in production is not a crash at all.
Receive event Begin processing Visibility timeout or ack deadline expires Broker redelivers to a second worker First worker is still runningplain text
This produces two concurrent executions of the same event rather than two sequential ones. Any deduplication mechanism therefore has to be safe under concurrency, not merely under restart. Section 5 handles this case explicitly.
The underlying result
This ambiguity is fundamental to remote execution, and it is not an engineering deficiency that a better protocol removes. It is the Two Generals problem: over an unreliable channel, no finite exchange of messages lets both parties reach certainty about the other’s state. A sender can never distinguish a lost request from a lost response.
Early RPC systems addressed it with request identifiers and duplicate suppression rather than assuming that a lost response implied failed execution. Birrell and Nelson, “Implementing Remote Procedure Calls”
3. At-most-once delivery
At-most-once means:
D(m) ∈ {0, 1}plain text
The message is never delivered more than once, but it may not be delivered at all.
A simple implementation acknowledges before processing:
Receive event Acknowledge event Process eventplain text
If the worker crashes after acknowledgement but before processing, the event is permanently skipped.
Properties
At-most-once guarantees:
D(m) ≤ 1plain text
It does not guarantee:
D(m) ≥ 1plain text
Appropriate use cases
At-most-once is appropriate when losing an individual update is acceptable or recoverable:
- High-frequency telemetry.
- Presence information.
- Typing indicators.
- Cache hints.
- Periodic metrics.
- Replaceable sensor readings.
It is usually inappropriate for non-reconstructible financial transitions:
- Payments.
- Trades.
- Deposits.
- Withdrawals.
- Ledger entries.
- Inventory movements.
At-most-once delivery is not at-most-once execution
This section bounds a delivery count:
D(m) ≤ 1plain text
The RPC literature uses the same phrase to bound an effect count:
E(m) ≤ 1plain text
These are different properties. A system can hold either one without the other.
A sender may retry until acknowledged while attaching a stable request ID. The receiver keeps a duplicate table and applies each ID no more than once. The handler still runs on every retry; it detects the duplicate and skips the mutation.
D(m) ≥ 1 H(m) ≥ 1 E(m) ≤ 1plain text
That is at-least-once delivery combined with at-most-once execution. It is the effectively-once construction developed in section 5, not the loss-tolerant semantic described above.
Both meanings appear in vendor documentation and protocol specifications. The useful question is always which count is bounded.
At-most-once delivery accepts loss in order to prevent repetition.
At-most-once execution prevents repetition while retries supply the liveness that delivery gave up.
Duplicate suppression carries its own requirement: deduplication state must outlive the longest delayed retry. Section 5 develops that constraint.
Signature: D(m) ≤ 1, E(m) ≤ 1, no lower bound on either.
4. At-least-once delivery
At-least-once means that a broker retains and retries a message until it receives an acknowledgement.
Under durable storage, eventual consumer recovery, and eventual network reachability:
D(m) ≥ 1plain text
Duplicate delivery remains possible:
D(m) ∈ {1, 2, 3, ...}plain text
A consumer normally acknowledges only after processing:
Receive event Process event Acknowledge eventplain text
If it commits its database transaction and crashes before acknowledgement, the message is redelivered.
Trade-off
At-most-once prefers possible loss over duplication.
At-least-once prefers possible duplication over loss.
For business-critical events, duplication is usually easier to control because operations can be given stable identities. A lost event may be impossible to reconstruct.
Pat Helland argues that large-scale applications frequently operate without global distributed transactions and must tolerate retries, duplicate messages, and reordering at the application layer. “Life Beyond Distributed Transactions”
Signature: D(m) ≥ 1, E(m) unbounded without further work.
5. Idempotent consumers
A function f is idempotent when:
f(f(x)) = f(x)plain text
That is the pure-function definition. In a distributed system the property that matters is the effect on state, not the returned value. An API that applies a mutation on the first call and returns 409 Conflict on the second is idempotent in the sense that counts here: E(m) = 1.
Some operations are naturally idempotent:
Set account status to CLOSED Set target replica count to 5 Store object using content hash Hplain text
Other operations are not:
balance += 100 position += 10 sendEmail() chargeCard(50)plain text
Repeating an increment, charge, or send creates another effect.
A consumer can make a non-idempotent operation repeat-safe by assigning it a stable operation ID.
event_id = trade-123plain text
The consumer records which event IDs it has committed.
Inbox table
CREATE TABLE consumer_inbox ( consumer_name TEXT NOT NULL, event_id UUID NOT NULL, processed_at TIMESTAMPTZ NOT NULL DEFAULT now(), PRIMARY KEY (consumer_name, event_id) );sql
The inbox insertion and business mutation must execute in the same database transaction.
BEGIN; INSERT INTO consumer_inbox ( consumer_name, event_id ) VALUES ( 'position-projector', :event_id ) ON CONFLICT DO NOTHING RETURNING event_id;sql
If the insert succeeds, process the event:
SELECT * FROM positions WHERE account_id = :account_id AND market = :market FOR UPDATE; UPDATE positions SET signed_size = :new_size, entry_price = :new_entry_price, realized_pnl = :new_realized_pnl WHERE account_id = :account_id AND market = :market; COMMIT;sql
If the insert returns no row, the event has already been committed. The consumer must not repeat the position update.
Only acknowledge the broker after COMMIT.
The constraint this pattern imposes
The inbox row and the business mutation must live in the same transactional database.
This is the load-bearing assumption of the entire pattern, and it is the one most often violated in practice. If the inbox is in PostgreSQL and the business state is in Redis, a separate service, or a second database, there is no shared commit and the guarantee is gone. Two writes to two systems reintroduce the dual-write problem described in section 7, one layer down.
When the business state genuinely cannot share a transaction with the deduplication record, the consumer is in the same position as any caller of an external system, and section 10 applies.
Crash analysis
The inbox insertion and position update both roll back.
The broker redelivers the message, and the next worker processes it normally.
The inbox record and position update remain committed.
The broker redelivers the event. The inbox uniqueness constraint detects the duplicate, so no second position update occurs.
Two workers attempt to insert the same (consumer_name, event_id).
The unique index serializes the conflict. One transaction succeeds. The other blocks until the first resolves, then observes the conflict and skips the business mutation.
This is the case produced by acknowledgement deadline expiry in section 2, and it is the reason deduplication has to be enforced by a database constraint rather than by a read-then-write check in application code. A SELECT followed by an INSERT leaves a window in which both workers see no existing row.
The resulting semantics are:
D(m) ≥ 1 E(m) = 1plain text
for effects contained inside that database transaction.
Deduplication retention
Deduplication state must outlive the maximum possible replay window.
deduplication retention > maximum message replay ageplain text
If inbox records expire after seven days but events can be replayed after thirty days, an old event may be applied again.
If the replay horizon is unbounded, the system needs one of:
- Indefinite deduplication records.
- Monotonic entity sequence numbers.
- Compacted per-entity operation state.
- A permanently bounded replay policy.
Signature: D(m) ≥ 1, H(m) ≥ 1, E(m) = 1 inside one database.
6. Exactly-once semantics
Exactly-once is meaningful only after defining its boundary.
The phrase can refer to at least three different guarantees:
- One record appended to a broker log.
- One read-process-write result inside a streaming platform.
- One externally visible business effect across multiple systems.
These are not equivalent.
6.1 Exactly-once broker insertion
Kafka’s idempotent producer prevents producer retries from appending duplicate copies of a logical record to a partition.
Conceptually, the broker tracks:
producer identity producer epoch per-partition sequence numberplain text
A retried record with an already accepted sequence number is recognized as a duplicate.
The scope of this guarantee is narrower than it first appears, and the limits are where most misreadings originate:
- It covers retries performed by the producer client, within one producer session. It does not cover retries performed by your application code.
- If the application catches an error, constructs a new producer, and sends again, that producer receives a new identity. The broker has no basis for treating the second send as a duplicate of the first.
- Deduplication operates over a bounded sequence window, which is why in-flight requests per connection are capped when idempotence is enabled.
- A stable
transactional.idis what gives a producer identity that survives restart, along with epoch-based fencing of the previous instance.
This guarantees one append within Kafka’s documented producer scope. It does not guarantee one PostgreSQL update by a downstream consumer. Apache Kafka producer configuration
One logical producer send → one Kafka partition recordplain text
does not imply:
One Kafka record → one external database mutationplain text
Signature: E(m) = 1 where the effect is defined as an append to a Kafka partition, within one producer session.
6.2 Exactly-once Kafka read-process-write
Kafka can atomically combine:
- Output records written to Kafka.
- Consumer offsets describing which input records produced those outputs.
A transaction either:
- Publishes the output and commits the input offset.
- Or exposes neither to
read_committedconsumers.
The resulting boundary is:
Kafka input → processing → Kafka outputplain text
Two configuration details carry the guarantee, and it silently degrades without them. Downstream consumers must set isolation.level=read_committed, or they will read uncommitted and aborted records. Applications built on Kafka Streams must set the exactly-once processing guarantee explicitly; the default is at-least-once.
Kafka’s design documentation scopes this guarantee around partition ownership, transactional producers, and transactional offset commits. Apache Kafka transaction design
An unrelated PostgreSQL write is not automatically part of that transaction.
Signature: E(m) = 1 where both the input position and the output record are inside Kafka.
6.3 Exactly-once stateful stream processing
A stateful stream processor must recover source positions and operator state from the same logical point.
Suppose a checkpoint contains:
Input offset: 1,000 Position state: +10 SOLplain text
If recovery restores the position from offset 1,000 but resumes input at offset 900, events 901–1,000 are applied twice.
If it restores position state from offset 900 but resumes input at 1,000, those events are skipped.
A consistent checkpoint must bind:
source positions + operator state + pending transactional outputsplain text
Flink’s asynchronous barrier snapshot algorithm inserts barriers into the dataflow and captures consistent operator snapshots without globally pausing the pipeline. Recovery restores state and input positions from a common checkpoint. Carbone et al., “Lightweight Asynchronous Snapshots for Distributed Dataflows”
Google’s MillWheel combined persistent state, record identity, acknowledgements, and duplicate suppression to expose exactly-once processing within its dataflow model. Akidau et al., “MillWheel”
These systems provide exactly-once semantics inside their supported state and checkpoint boundary. Arbitrary external side effects remain outside that boundary unless the sink is transactional or idempotent.
Signature: E(m) = 1 where the effect is a mutation of managed operator state.
7. The dual-write problem
Suppose a matching engine must:
- Commit a trade to PostgreSQL.
- Publish
TradeExecutedto Kafka.
These are two independent writes.
Database first
Commit trade Publish eventplain text
Failure:
Trade commits Process crashes Event is never publishedplain text
The authoritative database contains the trade, but downstream consumers never observe it.
Broker first
Publish event Commit tradeplain text
Failure:
Event is published Database transaction failsplain text
Consumers observe a trade that the authoritative system rejected.
Retries do not make these writes atomic.
8. Transactional outbox
The outbox pattern converts the dual write into one local database transaction.
BEGIN; INSERT INTO trades ( trade_id, account_id, market, quantity, price ) VALUES (...); INSERT INTO outbox_events ( event_id, aggregate_id, event_type, payload, published_at ) VALUES ( :event_id, :account_id, 'TradeExecuted', :payload, NULL ); COMMIT;sql
The trade and the intent to publish now share one atomic fate.
A relay reads unpublished outbox records and publishes them to the broker.
Local database transaction ├── business row └── outbox row Outbox relay └── durable brokerplain text
The relay still has a failure window:
Publish event Crash before setting published_atplain text
It may publish the same event again.
That is safe because downstream consumers use stable event IDs and inbox deduplication.
Polling versus log tailing
The relay above polls the outbox table. The alternative is change data capture: tail the database’s replication log and publish rows as they commit.
Log tailing removes the polling interval from end-to-end latency and removes the read load from the primary. It does not remove the failure window. The position in the replication log is itself a checkpoint that can be committed before or after the publish, which reproduces the same ambiguity one layer down.
Either relay design is at-least-once. Neither one removes the consumer’s obligation to deduplicate.
The complete architecture
Producer: business mutation + outbox in one local transaction Transport: durable at-least-once broker Consumer: inbox + business mutation in one local transactionplain text
This provides effectively-once business state without requiring a global distributed transaction.
Signature: D(m) ≥ 1 end to end, E(m) = 1 at both the producer’s database and the consumer’s database.
9. Distributed transactions
A true atomic transaction across the broker and database requires every participant to join a common commit protocol.
Two-phase commit consists of:
- Prepare: each participant durably records whether it can commit.
- Decision: the coordinator tells every participant to commit or abort.
The major liveness problem is coordinator failure. A prepared participant may be unable to determine the final outcome safely and can remain blocked until the coordinator recovers. In the interim it holds locks it cannot release, which is why a stalled coordinator can degrade an entire system rather than one transaction.
Gray and Lamport formulate transaction commit as an agreement problem and describe Paxos Commit, which replicates the commit decision through consensus. This improves fault tolerance at the cost of more protocol machinery and messages. Gray and Lamport, “Consensus on Transaction Commit”
The practical obstacle is participation. PostgreSQL supports prepared transactions and can act as an XA participant. Kafka does not implement XA; its transactions are internal to Kafka and cannot enlist an external resource manager. A protocol that requires every participant to join cannot be assembled from components where one of them structurally cannot.
Distributed transactions may be appropriate inside controlled infrastructure. They are frequently unavailable across heterogeneous systems such as:
- PostgreSQL.
- Kafka.
- Payment providers.
- Email services.
- Blockchain RPC endpoints.
When a participant cannot join the transaction, the application must rely on stable identities, retries, state machines, and reconciliation.
10. External side effects
An operation cannot be made exactly-once merely by wrapping the local database write in a transaction if its effect occurs in another system.
Payment providers
If a payment API supports idempotency keys, every retry of one logical payment must use the same key:
Idempotency-Key: withdrawal-456plain text
Generating a new key on retry creates a new logical operation and defeats duplicate protection.
The key must therefore be derived from the business operation and persisted with it, not generated at call time. A key produced inside the retry loop is a new key on every attempt.
Blockchain transactions
A client may submit a transaction and lose the RPC response.
The absence of a response does not prove that the transaction failed.
The application should persist:
- Business operation ID.
- Signed transaction hash or signature.
- Submission attempts.
- Confirmation status.
- Finalized outcome.
- Replacement relationship, when supported.
Recovery checks canonical chain state before constructing another logical transaction.
Solana is a useful concrete case, because the chain implements at the protocol layer exactly the mechanism this article recommends at the application layer.
A signed transaction has a deterministic signature, which functions as its stable operation ID. The runtime tracks recently processed signatures and rejects a duplicate, so resubmitting the identical signed transaction is safe: it lands once or it does not land. Retrying the same bytes is not a second logical operation.
That protection is bounded by blockhash expiry. Once the referenced blockhash falls outside the recent window, the transaction can no longer be accepted, and the deduplication state that made retry safe is no longer relevant to it. This is the same constraint as section 5’s retention rule, expressed in block height: the duplicate-suppression window must cover the retry window.
For operations that must remain retryable beyond that window, a durable nonce account replaces the recent blockhash with a nonce value that advances only when a transaction using it is processed. One logical withdrawal can then be signed once and resubmitted for as long as necessary, with the chain guaranteeing it executes at most once.
The general lesson: sign once, store the signature, retry the identical bytes, and check canonical state before ever constructing a replacement. Constructing a new transaction for the same business operation is what creates the double-spend risk, not resubmitting the old one.
A local transaction can guarantee one committed email request in the application database.
It cannot necessarily guarantee that exactly one copy appears in the recipient’s mailbox.
Those are different guarantees:
One committed send request ≠ one displayed emailplain text
Every exactly-once claim must state the endpoint at which the effect is observed.
11. Ordering is a separate property
Exactly-once processing does not imply correct ordering.
Consider:
101: Open long 10 SOL at $100 102: Reduce long by 4 SOL at $110plain text
Processing both events exactly once in reverse order is still incorrect.
For an entity a, the required order is:
e(a,1) → e(a,2) → ... → e(a,n)plain text
A scalable broker usually guarantees order within a partition, not across the entire topic.
The partition key should therefore represent the smallest state boundary that must be serialized.
Examples:
| State model | Partition key |
|---|---|
| Isolated position | account_id + market_id |
| Cross-margin trading account | account_id |
| Bank ledger | bank_account_id |
| Product inventory | product_id |
A consumer can persist:
last_applied_sequence = 101plain text
Then classify incoming events:
| Incoming sequence | Meaning |
|---|---|
| 101 | Duplicate |
| 102 | Next valid event |
| 103 | Gap: 102 is missing |
| 100 | Stale event |
event_id establishes identity.
sequence_number establishes order and detects gaps.
They solve different problems.
What to do about a gap
Detection is the easy half. The policy is the part that has to be decided before the incident.
A duplicate is discarded. A stale event is discarded. A gap is the only case that requires a decision, and there are two defensible answers:
Buffer and wait. Hold event 103 without applying it and wait for 102 to arrive. Under at-least-once delivery, a gap is frequently transient reordering or an in-flight redelivery, and 102 arrives shortly. This preserves correctness at the cost of consumer lag, and it requires a bounded buffer and a timeout, because an unbounded wait converts a missing message into a stalled partition.
Stall and escalate. When the timeout expires, stop applying events for that entity and raise an alert. Section 13 develops this path.
What a consumer must not do is apply 103 and continue. For a projector deriving state from a sequence, skipping an event does not lose one update; it corrupts every value derived after it, silently and permanently. The gap is detectable now. The corruption it produces may not be detectable for weeks.
12. Consumer groups do not remove duplicates
Assume a topic has P partitions and a consumer group has C workers.
At any moment, each partition is assigned to at most one active worker in the group:
For each partition p: number_of_active_owners(p) ≤ 1plain text
If C > P, some workers remain idle.
If P > C, some workers process multiple partitions.
Partition ownership distributes work, but it does not guarantee exactly-once external effects.
During reassignment:
- Worker A receives event 101.
- Worker A commits the database mutation.
- Worker A crashes before committing its broker offset.
- Worker B receives the partition.
- Worker B resumes from the previous offset.
- Event 101 is delivered again.
The inbox uniqueness constraint remains necessary.
The zombie case
The sequence above assumes worker A is dead. The harder failure is the worker that is merely slow.
A long garbage collection pause, a stalled disk, or a partitioned network link can prevent worker A from sending heartbeats. The group coordinator declares it dead and reassigns its partition to worker B. Worker A is still running and still holds the events it was processing. When it resumes, it writes.
Worker A pauses Coordinator reassigns partition to B B processes events 101–110 A resumes and commits its in-flight writeplain text
Kafka fences the offset commit: worker A’s commit is rejected because the group generation has advanced. The database has no such protection. A’s write to PostgreSQL is an ordinary transaction from an ordinary client.
The inbox table contains this for identical events, because A’s attempt to insert event 101 conflicts with the row B already committed. It does not contain it for read-modify-write against shared state, where A’s stale computation may be based on a position that B has since advanced.
The general mechanism is a fencing token: a monotonically increasing number issued with ownership, stored alongside the protected state, and checked on every write.
UPDATE positions SET signed_size = :new_size, owner_epoch = :epoch WHERE account_id = :account_id AND market = :market AND owner_epoch <= :epoch;sql
A write carrying a stale epoch affects zero rows and the transaction aborts. The database enforces ownership rather than trusting the caller to still hold it.
Row-level locking with FOR UPDATE addresses the interleaving but not the staleness: worker A blocks, acquires the lock after B commits, and then writes a value computed from state it read before B’s update. Re-reading inside the locked transaction is required regardless. Epoch checking is what makes the stale owner’s write fail rather than succeed with old data.
13. Poison events and retries
Not every failure should trigger infinite retry.
Transient failures
Examples:
- Database unavailable.
- Network timeout.
- Lock contention.
- Broker interruption.
These should normally use bounded exponential backoff.
Permanent failures
Examples:
- Unsupported event version.
- Malformed payload.
- Arithmetic overflow.
- Impossible state transition.
- Missing invariant.
These events may be moved to a dead-letter queue after bounded retries.
However, a financial projector cannot always skip the failed event and continue.
If sequence 102 fails, applying sequence 103 may corrupt the position derived from both. This is the gap case from section 11 arriving through a different door: there, 102 had not been delivered; here, it was delivered and could not be applied. The consequence is identical.
A safer policy is:
Quarantine affected partition or entity → alert → repair or migrate the event → replay from last valid sequenceplain text
A dead-letter queue is appropriate when events are independent. When events are sequential inputs to derived state, moving one to a dead-letter queue and continuing is a decision to corrupt that state.
Availability must not silently override accounting correctness.
14. Replay and reconciliation
Correct delivery does not imply correct computation.
A consumer can process every event once and still contain a PnL bug.
Derived state should therefore be reconstructible:
Immutable event log → deterministic projector → materialized stateplain text
When projection logic changes:
- Start a new projection version.
- Replay the immutable history.
- Compare old and new outputs.
- Investigate divergence.
- Atomically switch readers to the corrected projection.
Reconciliation checks application state against independent invariants:
Deposits − withdrawals + realized PnL − fees ± funding = custodied balance Funding debits = funding credits + explicit remainder Aggregated positions = recorded open interest Fees charged = protocol fee balances Off-chain projection = on-chain state at the same finalized checkpointplain text
The first invariant is worth stating carefully rather than approximately. A balance identity that omits realized PnL, fees, or funding will not hold on any exchange that charges fees, and an invariant that never holds is indistinguishable from an alert that is always firing. It gets muted, and then it is not an invariant at all.
Four separate questions must be answered:
| Mechanism | Question answered |
|---|---|
| Delivery guarantee | Did the event reach the consumer? |
| Idempotency | Did it affect state more than once? |
| Ordering | Was it applied at the correct point in the sequence? |
| Reconciliation | Did the application derive the correct result? |
A financial system needs all four.
15. Comparison
| Property | At-most-once | At-least-once | Exactly-once within a defined boundary |
|---|---|---|---|
Delivery count D(m) | Zero or one | One or more | One or more |
Committed effect E(m) | Zero or one | One or more without idempotency | One inside the declared boundary |
| Message loss | Possible | Prevented under durability and liveness assumptions | Prevented within the supported fault model |
| Duplicate delivery | Prevented | Expected | Occurs; suppressed before commit |
| Retry behaviour | Limited or deduplicated | Retry until ACK or policy limit | Transaction or checkpoint controlled |
| Consumer idempotency | Often unnecessary | Required for once-only effects | Still required outside the transaction boundary |
| State overhead | Low | Inbox and deduplication state | Transaction logs, fencing, offsets, snapshots |
| Coordination cost | Low | Moderate | Potentially high |
| External database effects | May be lost | May repeat without idempotency | Exactly-once only if database participates |
| Recovery | Can skip work | Replays unacknowledged work | Restores a consistent committed/checkpointed state |
| Typical use | Disposable updates | Financial and business workflows | Closed transactional stream topologies |
The first two rows are the ones that get conflated. Exactly-once does not reduce the delivery count. It reduces the committed effect count, and only inside a boundary that has to be named.
16. Practical architecture
For correctness-sensitive systems whose authoritative state lives in a transactional database:
1. Assign every logical operation a stable event ID. 2. Commit producer state and an outbox row atomically. 3. Publish through a durable at-least-once broker. 4. Partition events by the entity requiring ordered mutation. 5. Consume using an inbox/deduplication table. 6. Commit the inbox row and business mutation atomically. 7. Acknowledge the broker only after commit. 8. Fence ownership with an epoch on contended state. 9. Reuse stable idempotency keys for external APIs. 10. Preserve replayable history. 11. Reconcile projections against independent authoritative state.plain text
This architecture accepts:
D(m) ≥ 1plain text
while enforcing:
E(m) = 1plain text
inside each local transactional boundary.
For external systems that cannot participate in that transaction, it uses:
- Idempotency keys.
- Durable operation state machines.
- Result polling.
- Canonical identifiers.
- Reconciliation.
Conclusion
At-most-once guarantees:
D(m) ≤ 1plain text
and accepts possible loss.
At-least-once guarantees:
D(m) ≥ 1plain text
under explicit durability and liveness assumptions, while accepting duplicate delivery.
Exactly-once does not claim:
D(m) = 1plain text
Transactional systems still redeliver. What they bound is the committed effect inside a declared boundary B:
E_B(m) = 1 with D(m) ≥ 1 permitted throughoutplain text
A claim that omits B is not a claim about anything.
The application generally requires:
E(m) = 1plain text
That property requires more than a broker configuration. It requires:
- Stable operation identities.
- Atomic state transitions.
- Duplicate suppression.
- Correct partitioning and ordering.
- Ownership fencing under reassignment.
- Consistent checkpoint recovery.
- Transactional sinks where possible.
- Idempotent external APIs.
- Replay and reconciliation.
A technically defensible guarantee sounds like this:
Within the database transaction boundary, every uniquely identified event produces one committed state transition. The transport may redeliver. Effects outside that boundary are made idempotent where possible and reconciled where atomicity is unavailable.
That statement is narrower than “our system has exactly-once delivery.”
It is also far more useful.
References
- Andrew D. Birrell and Bruce Jay Nelson, “Implementing Remote Procedure Calls,” ACM Transactions on Computer Systems, 1984.
- Jerome H. Saltzer, David P. Reed, and David D. Clark, “End-to-End Arguments in System Design,” ACM Transactions on Computer Systems, 1984.
- Pat Helland, “Life Beyond Distributed Transactions: An Apostate’s Opinion,” CIDR, 2007.
- Jim Gray and Leslie Lamport, “Consensus on Transaction Commit,” Microsoft Research Technical Report MSR-TR-2003-96, 2003; revised in ACM Transactions on Database Systems, 2006.
- Tyler Akidau et al., “MillWheel: Fault-Tolerant Stream Processing at Internet Scale,” Proceedings of the VLDB Endowment, 2013.
- Paris Carbone et al., “Lightweight Asynchronous Snapshots for Distributed Dataflows,” 2015.
- Apache Kafka, “Design,” current documentation.
- Apache Kafka, “Producer Configuration,” current documentation.