Kafka Broker Internals

Overview

This visualization shows how Apache Kafka works at the broker level -- the internal mechanics that make Kafka a reliable, high-throughput distributed streaming platform.

Unlike a high-level pipeline diagram, this focuses on how messages actually flow inside a Kafka cluster: partition leaders and followers, In-Sync Replicas (ISR), the High Watermark mechanism, KRaft consensus, and consumer group coordination.

Architecture

Producers

Three application services (Order, Payment, User) produce messages to different partitions based on key hashing: hash(key) % numPartitions.

Broker Cluster (3 brokers)

Each broker hosts all 3 partitions, but in different roles:

• Broker 1: P0 Leader, P1 Follower, P2 Follower
• Broker 2: P0 Follower, P1 Leader, P2 Follower
• Broker 3: P0 Follower, P1 Follower, P2 Leader

This distribution ensures no single broker failure loses all leaders.

KRaft Controller Quorum

Replaces ZooKeeper (deprecated since Kafka 3.3). Three controller nodes use Raft consensus to manage cluster metadata: broker registration, partition assignments, leader elections.

Consumer Groups

• Group A (3 consumers): Each consumer assigned exactly one partition (1
  mapping)
• Group B (2 consumers): One consumer handles two partitions, showing uneven assignment

Key Concepts Explained

ISR (In-Sync Replicas)

The set of replicas that are fully caught up with the leader. When acks=all, the producer waits for ALL ISR replicas to acknowledge before confirming the write. If a follower falls behind, it's removed from ISR.

High Watermark (HW)

The offset up to which all ISR replicas have replicated. Only messages below HW are visible to consumers. This prevents consumers from reading data that might be lost if the leader fails.

LEO (Log End Offset)

The offset of the last message in a replica's local log. Each replica tracks its own LEO. The leader uses follower LEOs to determine when to advance the High Watermark.

Consumer Group Rebalancing

When consumers join or leave a group, the group coordinator (a specific broker) triggers a rebalance protocol: all consumers stop, commit offsets, receive new partition assignments, and resume.

Scenarios

1. Producer Write (acks=all) -- Full write path with ISR replication and HW advancement
2. Consumer Group Rebalancing -- New consumer triggers partition reassignment
3. Broker Failure + Leader Election -- KRaft detects failure, promotes new partition leaders
4. ISR Replication Deep Dive -- Single message journey with LEO/HW tracking
5. Consumer Offset Commit -- Batch processing and offset management