Event storage, distribution, replication, and offsets
Topic & Partition
A topic stores events across partitions. This example uses three partitions.
↔ Drag to explore
Principles and uses
Explore how topic events are distributed across partitions and read in order.
Ordering is per partition, not a total order across partitions.
Keys and partitioning configuration select partitions. Use shared keys for per-key ordering and partitions for parallelism. Retained events can be read by multiple groups.
Use it to plan event keys, partition counts, and ordering requirements.
Consumer Group
① Partition assignments by consumer
A consumer group shares the work across three topic partitions.
↔ Drag to explore
② Save the read position · Offset commits
Three consumers each own one partition. Their read positions start at the beginning.
↔ Drag to explore
③ Independent consumer groups
Group A processes orders and tracks its offsets for each partition.
↔ Drag to explore
Principles and uses
Follow partition assignments and consumption positions within a consumer group.
One consumer owns a partition per group; different groups track independent offsets.
Membership changes trigger rebalancing. New owners resume from committed offsets, potentially repeating work processed after the last commit.
Use it to understand consumption parallelism, recovery, and independent services reading the same events.
Not Losing Messages (Delivery Guarantees)
Commit before processing · Events may be skipped
This example commits the offset before processing event B.
↔ Drag to explore
Commit after processing · Events may be processed twice
This time, processing completes before the offset is committed.
↔ Drag to explore
Principles and uses
Compare failure outcomes when offsets are committed before or after processing.
Committing first can skip unfinished work; committing afterward can repeat completed work.
At-least-once consumers can reread events after a crash before commit. Use identifiers or other idempotency mechanisms to prevent duplicate effects.
Use it to design recovery positions and account for lost or repeated processing.
Producer & acks
acks=0 · Do not wait for acknowledgment
With acks=0, the producer does not wait for a broker acknowledgment.
↔ Drag to explore
acks=1 · Wait for the leader to record
With acks=1, the leader acknowledges after appending the message to its local log.
↔ Drag to explore
acks=all · Wait for ISR replication
With acks=all, acknowledgment waits for the current in-sync replicas to append the message.
↔ Drag to explore
Principles and uses
Compare which acknowledgments the producer waits for under each acks setting.
Acks=0 waits for none, acks=1 for the leader, and acks=all for current in-sync replicas.
Replication acknowledgment can add latency and protect against leader failure. Durability also depends on replication factor and min.insync.replicas.
Use it to assess acknowledgment latency and data durability together.
Replication & ISR
Replicate the partition across three brokers: one leader and two followers.
↔ Drag to explore
Principles and uses
Follow replication from a leader to followers and recovery after leader failure.
The ISR tracks replicas sufficiently synchronized with the leader.
Lagging replicas can leave the ISR. This example elects an in-sync replica that retains the replicated records.
Use it to understand replication lag, leader election, and acknowledgment settings.
Why Rebalancing Stalls
Eager · Revoke all assignments and reassign
A third consumer joins a group where two consumers process three partitions.