Kafka Algorithms

A Kafka topic isn't a queue — it's a log you only ever append to. The topic is split into partitions, each record is appended to the end of one, and the position it lands in becomes its offset. Multiple partitions mean parallel writes and reads, but ordering is only guaranteed inside a single partition. That's exactly why records sharing a key always land in the same one.

A consumer group is how several consumers split one topic between them. Within a group each partition is assigned to exactly one consumer, so adding consumers raises throughput — until you pass the partition count, at which point the extras sit idle. Each consumer commits offsets for the partitions it owns, and a different group reads the same data on its own independent schedule.

When you record how far you've read decides whether you lose messages or process them twice. Commit before doing the work and a crash mid-processing means that message is never read again (at-most-once); commit after and a crash means the same message is read once more (at-least-once). Kafka usually takes the latter and makes the processing side idempotent so duplicates are harmless.

A producer chooses how far a message must be stored before it counts as a success. acks=0 waits for nothing and is fastest but never learns whether a broker received it; acks=1 counts the leader's write as success, so a leader failure can lose it; acks=all needs the replicas to have stored it too — the safest and the slowest.

Each partition has one leader and several followers, and the leader alone handles reads and writes. Followers copy the leader's log, and the set of replicas that have kept up is called the ISR. When the leader dies a new one is elected from the ISR, so being in the ISR is exactly what makes a replica safe to promote.

Whenever a consumer joins or leaves, the partition assignment has to be recalculated. The older protocol (eager) has every consumer give up everything first and then redivide, so partitions that never needed to move stop too. The cooperative protocol revokes only what actually has to move and lets the rest keep reading.

Deleting the oldest data once the retention period passes is the default, but some topics need the *current* state instead. Log compaction removes older records for the same key and keeps only the last value, turning the log itself into an up-to-date snapshot. Read it from the beginning and you can rebuild the current value of every key.