Log Compaction
Deleting the oldest data once the retention period passes is the default, but some topics need the *current* state instead. Log compaction removes older records for the same key and keeps only the last value, turning the log itself into an up-to-date snapshot. Read it from the beginning and you can rebuild the current value of every key.
01Log Compaction
Concept at a GlanceUser profiles are stored in Kafka. Every time an address changes, a new line is appended — nothing is edited in place.
Values for other users land in the same lane. Different keys still go into one partition in order.
user-42 moved to Pangyo. The earlier "Seoul" isn't erased; the new value is appended after it.
Move again and it appends again. Left alone, frequently-changing keys make the log grow without end.
user-7 moved to Daejeon too. Read this log from the front and you can still find the current address — but there's more and more to read.
This is where compaction kicks in. For each key it keeps only the last value and marks the earlier ones for removal.
When the cleanup finishes, one line per key remains. Offsets are not renumbered, so gaps appear — and the current state still reads straight off.
So when a service restarts, reading this topic from the beginning is enough to rebuild every profile.
02 Understand It Simply
For EveryoneOlder records for a key are removed and only the last value is kept. Offsets end up with gaps, but reading from the beginning still reconstructs the current state.
Compaction works per key.
If a key was written several times, only the most recent one survives and the earlier ones are removed.
A record with a null value — a tombstone — means "delete this key", and it disappears along with the key after a while.
Because of this, reading the whole log from the front reconstructs the current state, which makes it a fit for caches, configuration, and profiles.
- –Keeping key-value state in Kafka (CDC
- –config
- –profiles)
- –restoring state on service restart
- –and controlling the size of topics that would otherwise grow forever
03 Frequently Asked Questions
FAQWhat is Log Compaction?+
Deleting the oldest data once the retention period passes is the default, but some topics need the *current* state instead. Log compaction removes older records for the same key and keeps only the last value, turning the log itself into an up-to-date snapshot. Read it from the beginning and you can rebuild the current value of every key.
Where is Log Compaction used?+
Keeping key-value state in Kafka (CDC, config, profiles), restoring state on service restart, and controlling the size of topics that would otherwise grow forever.
What's a simple analogy for Log Compaction?+
Older records for a key are removed and only the last value is kept. Offsets end up with gaps, but reading from the beginning still reconstructs the current state.