Not Losing Messages (Delivery Guarantees)
When you record how far you've read decides whether you lose messages or process them twice. Commit before doing the work and a crash mid-processing means that message is never read again (at-most-once); commit after and a crash means the same message is read once more (at-least-once). Kafka usually takes the latter and makes the processing side idempotent so duplicates are harmless.
01Not Losing Messages (Delivery Guarantees)
Concept at a GlanceCommit first · you can lose (at-most-once)
Four orders are queued and the consumer is about to read B. Nothing has been committed yet.
This approach commits the moment it reads. It records that B is done, and the next read position becomes C.
But the consumer dies while processing B. Nothing was charged, nothing was saved.
On restart it reads from C, the committed position. Nobody ever processed B, and it is skipped forever.
Process first · you can duplicate (at-least-once)
Same situation, only the order is swapped. Nothing has been committed yet.
This time B is processed first. Nothing is committed, so the next read position stays at B.
Right after the work finishes — but before the commit — the consumer dies.
On restart there is no newer committed position, so B is read once more. Nothing is lost, but it is processed twice.
So this is what production picks, and the work is made idempotent — filtering duplicates by order id, for example.
02 Understand It Simply
For EveryoneWhether you commit or process first decides between loss and duplication. Commit first and a failed message is skipped; process first and the same message is read again.
Commit first and any message you died on is skipped forever (loss).
Process first and a crash before the commit means reprocessing it after restart (duplication).
Removing both requires the work and the commit to be one transaction, which is expensive.
So in practice you pick at-least-once and make processing idempotent — same result whether a message arrives once or twice.
- –Designing flows where loss is unacceptable (payments
- –orders)
- –designing idempotency keys against duplicates
- –and trading commit interval against recovery time
03 Frequently Asked Questions
FAQWhat is Not Losing Messages (Delivery Guarantees)?+
When you record how far you've read decides whether you lose messages or process them twice. Commit before doing the work and a crash mid-processing means that message is never read again (at-most-once); commit after and a crash means the same message is read once more (at-least-once). Kafka usually takes the latter and makes the processing side idempotent so duplicates are harmless.
Where is Not Losing Messages (Delivery Guarantees) used?+
Designing flows where loss is unacceptable (payments, orders), designing idempotency keys against duplicates, and trading commit interval against recovery time.
What's a simple analogy for Not Losing Messages (Delivery Guarantees)?+
Whether you commit or process first decides between loss and duplication. Commit first and a failed message is skipped; process first and the same message is read again.