Log Compaction
Deleting the oldest data once the retention period passes is the default, but some topics need the *current* state instead. Log compaction removes older records for the same key and keeps only the last value, turning the log itself into an up-to-date snapshot. Read it from the beginning and you can rebuild the current value of every key.
01 Concept at a Glance
Interactive Step-by-StepUser profiles are stored in Kafka. Every time an address changes, a new line is appended — nothing is edited in place.
02 Understand It Simply
For EveryoneLike an address book where you write a new line every time someone moves, then tidy up by keeping only each person's latest address. The record gets shorter, but you can still tell where everyone lives now.
Compaction works per key.
If a key was written several times, only the most recent one survives and the earlier ones are removed.
A record with a null value — a tombstone — means "delete this key", and it disappears along with the key after a while.
Because of this, reading the whole log from the front reconstructs the current state, which makes it a fit for caches, configuration, and profiles.
- –Keeping key-value state in Kafka (CDC
- –config
- –profiles)
- –restoring state on service restart
- –and controlling the size of topics that would otherwise grow forever
03 Frequently Asked Questions
FAQWhat is Log Compaction?+
Deleting the oldest data once the retention period passes is the default, but some topics need the *current* state instead. Log compaction removes older records for the same key and keeps only the last value, turning the log itself into an up-to-date snapshot. Read it from the beginning and you can rebuild the current value of every key.
Where is Log Compaction used?+
Keeping key-value state in Kafka (CDC, config, profiles), restoring state on service restart, and controlling the size of topics that would otherwise grow forever.
What's a simple analogy for Log Compaction?+
Like an address book where you write a new line every time someone moves, then tidy up by keeping only each person's latest address. The record gets shorter, but you can still tell where everyone lives now.