Kafka Algorithms

A Kafka topic isn't a queue — it's a log you only ever append to. The topic is split into partitions, each record is appended to the end of one, and the position it lands in becomes its offset. Multiple partitions mean parallel writes and reads, but ordering is only guaranteed inside a single partition. That's exactly why records sharing a key always land in the same one.

A consumer group is how several consumers split one topic between them. Within a group each partition is assigned to exactly one consumer, so adding consumers raises throughput — until you pass the partition count, at which point the extras sit idle. Each consumer commits offsets for the partitions it owns, and a different group reads the same data on its own independent schedule.