Not Losing Messages (Delivery Guarantees)
When you record how far you've read decides whether you lose messages or process them twice. Commit before doing the work and a crash mid-processing means that message is never read again (at-most-once); commit after and a crash means the same message is read once more (at-least-once). Kafka usually takes the latter and makes the processing side idempotent so duplicates are harmless.
01 Concept at a Glance
Interactive Step-by-StepFour orders are queued and the consumer is about to read B. ▲ points at B.
Same situation, only the order is swapped. ▲ is pointing at B.
02 Understand It Simply
For EveryoneLike when you sign for a delivery. Sign before opening the box and you can't take it back if it's wrong; check first and sign after, and if you collapse just before signing, the next person checks the same box again.
Commit first and any message you died on is skipped forever (loss).
Process first and a crash before the commit means reprocessing it after restart (duplication).
Removing both requires the work and the commit to be one transaction, which is expensive.
So in practice you pick at-least-once and make processing idempotent — same result whether a message arrives once or twice.
- –Designing flows where loss is unacceptable (payments
- –orders)
- –designing idempotency keys against duplicates
- –and trading commit interval against recovery time
03 Frequently Asked Questions
FAQWhat is Not Losing Messages (Delivery Guarantees)?+
When you record how far you've read decides whether you lose messages or process them twice. Commit before doing the work and a crash mid-processing means that message is never read again (at-most-once); commit after and a crash means the same message is read once more (at-least-once). Kafka usually takes the latter and makes the processing side idempotent so duplicates are harmless.
Where is Not Losing Messages (Delivery Guarantees) used?+
Designing flows where loss is unacceptable (payments, orders), designing idempotency keys against duplicates, and trading commit interval against recovery time.
What's a simple analogy for Not Losing Messages (Delivery Guarantees)?+
Like when you sign for a delivery. Sign before opening the box and you can't take it back if it's wrong; check first and sign after, and if you collapse just before signing, the next person checks the same box again.