Kafka beyond the basics

Everybody can write a consumer. Almost nobody can explain a rebalance.

Every Kafka tutorial ends at the same place: produce a message, consume a message, congratulations. That's enough to ship a prototype and nowhere near enough to debug the 3 a.m. page where consumer lag is climbing, throughput is zero, and the logs say nothing except Attempt to heartbeat failed since group is rebalancing over and over.

This post is the layer underneath. Six things that actually determine how Kafka behaves in production — the log on disk, group assignment, rebalancing, replication and the high watermark, compaction, and exactly-once — each with an interactive visualizer you can poke at until the mental model clicks.

Start with the machine at the centre of it all: a producer choosing a partition, and a consumer walking behind it.

Two things in there are worth naming right now. First, the partition is chosen on the client, not the broker — the producer hashes the key and picks. Second, ordering is a per-partition property, never a per-topic one. Every record with key user-42 lands in the same partition and is therefore ordered relative to every other user-42 record. Its ordering relative to order-991 is undefined and always will be.

That single fact is the source of most Kafka design mistakes. If you need two events ordered with respect to each other, they must share a key. If they don't share a key, you cannot get that ordering back downstream, no matter how many consumers you throw at it.

That widget is the whole machine in miniature — a producer picking a partition, a log accepting the write, a consumer walking behind it. The rest of this post opens it up one layer at a time, starting with the thing everything else is built on.

Part 1: The log is the whole product

Kafka is not a queue. A queue's defining move is destructive read — you take a message and it's gone. Kafka never removes a record because someone read it. A partition is an append-only, immutable, ordered sequence of bytes on disk, and reading is just a positioned scan over that file. Ten consumers reading the same partition cost the broker almost nothing extra, because none of them mutate anything.

On disk, a partition directory looks like this:

/var/lib/kafka/data/events-0/
  00000000000000000000.log        # records 0..8191
  00000000000000000000.index      # sparse: relative offset -> byte position
  00000000000000000000.timeindex  # sparse: timestamp -> relative offset
  00000000000000008192.log        # the active segment, still being written
  00000000000000008192.index
  leader-epoch-checkpoint

The filename is the base offset — the offset of the first record in that segment. Only the newest segment is open for writes; the rest are sealed. That single design choice is why retention is cheap: deleting old data means unlinking whole files, never rewriting anything.

An offset is a position, not an ID

This is the most common misreading. An offset is not a primary key handed out by the broker; it's the record's index in its partition. Offset 4192 in events-0 has nothing whatsoever to do with offset 4192 in events-1. And a consumer doesn't ask "give me the message with ID 4192" — it says "position me at 4192 in this partition and stream forward."

Which raises the obvious question: how do you find byte position of offset 4192 in a 1 GB file without reading 1 GB? The .index file. It's sparse — one entry roughly every 4 KB of log, not one per record — so it stays small enough to memory-map. The lookup is a binary search on the index, a seek, and a short forward scan.

Drag the slider and run a lookup. Watch how few records actually get read.

The consequence of this design is the feature everyone eventually needs: re-reading. Because reads don't consume, resetting a consumer group to an earlier offset replays history exactly as it happened.

# replay a bad deploy's worth of events
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group billing-worker --topic events --reset-offsets \
  --to-datetime 2025-08-16T02:00:00.000 --execute

That --to-datetime is served by .timeindex, the same sparse-lookup trick keyed on timestamp instead of offset. Rebuild a corrupted downstream store, backfill a new service, re-run a fixed aggregation — all of it is the same operation: move a number and read again.

Part 2: Consumer groups and who owns what

The mental model most people carry is "consumers subscribe to a topic and Kafka load-balances messages between them." That's wrong in a way that matters, and the correct version is only slightly harder: Kafka balances partitions, not messages. A partition is owned by exactly one consumer in a group at a time. Add a ninth consumer to a group reading eight partitions and it sits idle forever.

Two roles run the show, and they are not the same thing:

• Group Coordinator — a broker. The one hosting the __consumer_offsets partition for hash(group.id) % 50. It tracks membership, receives heartbeats, and decides when a rebalance is needed.
• Group Leader — a consumer. The first member to join. It receives the full member list and runs the assignment algorithm locally, then uploads the result.

Yes: the partition assignment your production cluster is using was computed by one of your own application pods, not by a broker. That's deliberate — it means you can ship a custom assignor without touching the cluster.

The handshake is two phases:

consumer                       coordinator (broker)
   |-- FindCoordinator --------->|
   |-- JoinGroup --------------->|   collects members until all arrive
   |<-- JoinGroup response ------|   one member marked leader, given member list
   |-- SyncGroup (leader: full   |
   |   assignment; others: {}) ->|
   |<-- SyncGroup response ------|   each member gets only its own partitions
   |-- Heartbeat (every 3s) ---->|

Where do the committed offsets themselves live? In __consumer_offsets, an ordinary Kafka topic with 50 partitions — no special storage engine, no database. A commit is just a produce to it, keyed by (group, topic, partition). And because that topic is compacted rather than time-retained, the latest value for every key survives forever while the history is collapsed away. That's why your group's position survives a broker restart, and why the offsets topic doesn't grow without bound despite every consumer writing to it every few seconds. Part 5 covers the mechanism.

Move the sliders below. Watch what happens to the "partitions moved" counter as you switch strategies — that number is the cost of the rebalance you just triggered.

Which assignor to use

• Range (the historical default) — contiguous blocks per topic. With 2 consumers and 2 topics of 3 partitions each, consumer 1 gets 4 partitions and consumer 2 gets 2. Skew is structural, not bad luck.
• RoundRobin — even distribution, but no memory. Every rebalance can reshuffle everything.
• Sticky — even and minimises movement from the previous assignment. Still stop-the-world.
• CooperativeSticky — sticky plus incremental revocation. This is the one you want; see the next section for why.

What lag actually measures

Lag is log end offset − last committed offset, per partition. Two things follow that people get wrong constantly:

1. Lag is measured against the committed offset, not the processed one. If you process records and commit every 30 seconds, your reported lag sawtooths by 30 seconds' worth of traffic while nothing is actually wrong.
2. Total lag hides the failure mode that matters. One stuck partition at 2 M lag and eleven healthy ones averages out to something unalarming. Always alert on max partition lag, not the sum.

And lag that is flat but nonzero is fine. Lag with a positive first derivative is the incident.

Part 3: Rebalancing, the part that bites

A rebalance is Kafka redistributing partitions across a group. It's triggered by:

• a member joining or leaving cleanly (deploy, scale-up)
• session.timeout.ms elapsing with no heartbeat — the consumer process is gone or wedged
• max.poll.interval.ms elapsing between poll() calls — the process is alive and heartbeating, but stuck processing a batch
• partitions being added to a subscribed topic, or a regex subscription matching a new one

That third trigger is the one that causes production incidents, because the heartbeat runs on a background thread. Your consumer keeps telling the coordinator "I'm alive!" while the main thread has been sitting in a 6-minute batch write for the last 6 minutes. Default max.poll.interval.ms is 5 minutes. The coordinator evicts it mid-batch, hands its partitions to somebody else, and your commit fails afterwards with CommitFailedException — those records get processed twice.

Now the important part: how the redistribution happens. Press the button and watch both strategies handle the identical event.

Eager: stop the world

Every member calls onPartitionsRevoked for all its partitions, rejoins, and waits. Between the revoke and the new assignment, the group processes nothing — not the partitions that were moving, not the ones that were staying put. On a group with heavy state or a slow rebalance, that's seconds to minutes of total blackout for a change that affected two partitions.

Cooperative: revoke only what moves

Two rebalances instead of one, but the first revokes nothing. Members compute the new assignment, diff it against what they hold, and only give up the partitions that genuinely change hands. The second rebalance hands those out. Everything else never stops.

partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

One migration caveat that has burned plenty of teams: you cannot flip this in a single rolling restart. Deploy once with both assignors listed (CooperativeStickyAssignor,RangeAssignor), let the whole group land on that build, then deploy again with only the cooperative one. The group negotiates the common protocol; if you jump straight there, members disagree and the group won't stabilise.

The rebalance storm

The pathological loop, and it is genuinely self-sustaining:

1. A consumer takes slightly too long in poll() and gets evicted.
2. Rebalance starts. Every other consumer pauses (eager) and their in-flight work stalls.
3. The pause makes their next poll() late too.
4. More evictions. Another rebalance. Go to 2.

The group burns its entire day rebalancing and processes nothing. Lag goes vertical. The fixes, in order of how often they're the actual answer:

# 1. process less per poll — the single highest-leverage knob
max.poll.records=100

# 2. give slow batches room, but not so much that dead consumers linger
max.poll.interval.ms=600000

# 3. survive a rolling deploy without any rebalance at all (Kafka 2.3+)
group.instance.id=worker-3        # static membership
session.timeout.ms=45000

# 4. wait for stragglers before assigning, so one restart = one rebalance
group.initial.rebalance.delay.ms=3000

Static membership deserves the callout. With group.instance.id set, a consumer that disappears and comes back within session.timeout.ms reclaims exactly its old partitions and no rebalance happens at all. For a StatefulSet doing a rolling restart, that turns N rebalances into zero. Pair it with a session timeout comfortably longer than your pod restart time.

Part 4: ISR, replication and the high watermark

Each partition has one leader replica and some followers. All reads and writes go to the leader; followers are pure fetchers, running an ordinary consumer fetch loop against it.

The ISR (In-Sync Replicas) is the subset of replicas currently caught up — specifically, replicas that have fetched from the leader within replica.lag.time.max.ms (default 30s). Note what that definition is not: it is not "within N records." Kafka dropped the message-count criterion years ago because a legitimate traffic burst would eject every follower at once.

The high watermark is the minimum log-end-offset across the ISR, and it is the visibility boundary: consumers cannot read past it. A record that exists on the leader's disk but hasn't been replicated is simply invisible. That's what stops a consumer from reading a record that a subsequent leader failover would erase.

Produce a few records, then stall a follower and watch the ISR shrink.

What acks actually buys you

Here's the trap: acks=all alone does not make you durable. If the ISR has shrunk to just the leader, then "all ISR members" is one broker, and acks=all degrades silently to acks=1. The setting that closes that hole is broker- or topic-side:

# topic must have >= 2 in-sync replicas or produces are rejected outright
kafka-configs.sh --alter --entity-type topics --entity-name events \
  --add-config min.insync.replicas=2

With replication.factor=3, min.insync.replicas=2 and acks=all, you can lose one broker and keep writing, lose two and start getting NOT_ENOUGH_REPLICAS — which is the correct behaviour. A rejected write is an incident you can see; a silently unreplicated write is one you discover a week later.

Leader epoch: the subtle one

When a leader fails and a follower takes over, the follower's log may be shorter. What happens to the extra records the old leader had?

Pre-0.11, the recovering old leader truncated to its high watermark and refetched — and there were interleavings where that lost committed data or produced divergent logs. The fix is the leader epoch: a monotonic counter bumped on every leader change, stamped into every record batch. A recovering replica now asks the new leader "what's the end offset for epoch 5?" and truncates to precisely that point rather than guessing from its watermark.

You never configure this. You just want it on, which means keeping message.format.version modern and never enabling unclean.leader.election.enable — unclean election lets an out-of-sync replica become leader, which is the one setting that will knowingly discard committed records.

Part 5: Compaction is not retention

Two independent cleanup policies, routinely confused:

• cleanup.policy=delete — drop whole segments older than retention.ms or beyond retention.bytes. Time-based. Doesn't care about content.
• cleanup.policy=compact — keep at least the last value for every key, forever. Key-based. Doesn't care about time.

A compacted topic is a changelog that converges to a table. Read it from offset 0 and you reconstruct the current state of every key — which is exactly how __consumer_offsets works, how Kafka Streams restores its state stores, and how CDC topics stay bounded while remaining complete.

The log cleaner is a background thread that picks the partition with the highest ratio of dirty (uncompacted) to total bytes, builds an in-memory map of key → highest offset over the dirty region, and rewrites the segments keeping only records whose offset matches the map. Two rules do most of the surprising work:

• The active segment is never compacted. Your latest writes stay duplicated until the segment rolls. This is why segment.ms matters on low-traffic compacted topics — a segment that never fills also never rolls, and compaction never runs.
• Offsets are never renumbered. Compaction removes records, leaving gaps. Offset 7 can simply not exist. Any code assuming contiguous offsets is broken code.

Append records with repeating keys, drop a tombstone, run the cleaner.

Tombstones, and the delete window

A record with a null value is a tombstone: it means "this key is deleted." The cleaner keeps it around for delete.retention.ms (default 24h) before purging it, and that window exists for one specific reason — a consumer rebuilding state from offset 0 must observe the deletion. Purge tombstones too aggressively and a slow bootstrap misses the delete entirely, resurrecting rows that should be gone.

kafka-topics.sh --create --topic user-profiles --partitions 12 \
  --config cleanup.policy=compact \
  --config min.cleanable.dirty.ratio=0.1 \   # compact aggressively (default 0.5)
  --config segment.ms=3600000 \               # roll hourly so compaction can run
  --config delete.retention.ms=86400000       # 24h for consumers to see tombstones

You can also set cleanup.policy=compact,delete — keep the latest value per key and drop anything older than the retention window. That's the right choice for a state topic where truly ancient keys are worthless.

Part 6: Exactly-once, honestly explained

"Exactly-once delivery" is impossible in a distributed system — that's the Two Generals problem and no vendor has repealed it. What Kafka provides is exactly-once processing semantics: the observable effects of a record are applied once, even when the delivery underneath was at-least-once. The distinction is not pedantry; it tells you exactly where the guarantee stops (inside Kafka) and where it doesn't (your external database).

The problem: retries are duplicates

A producer sends a batch. The broker writes it. The ACK is lost on the way back. The producer, having no way to distinguish "never arrived" from "arrived, response lost," retries. Now the record is on the log twice — and the producer thinks everything is fine.

Idempotent producer closes this. The producer gets a producerId from the broker and stamps every batch with a monotonic sequence number per partition. The broker keeps the last five sequence numbers per producer per partition and rejects anything it has already seen with DUPLICATE_SEQUENCE_NUMBER — which the client treats as success. Cost: effectively zero. It has been on by default since Kafka 3.0.

Flip the toggle and lose an ACK both ways.

Transactions: atomicity across partitions

Idempotence protects one producer session writing to one partition. Transactions extend that to many partitions, many topics, and the consumer's own offsets, atomically.

// confluent-kafka-go/v2. InitTransactions fences any zombie still holding
// this transactional.id — it must run once, before the loop.
if err := producer.InitTransactions(ctx); err != nil {
    log.Fatalf("init transactions: %v", err)
}

for {
    msgs := drain(consumer, 500*time.Millisecond, 1000)
    if len(msgs) == 0 {
        continue
    }
    if err := processBatch(ctx, producer, consumer, msgs); err != nil {
        log.Printf("batch aborted, will be reprocessed: %v", err)
    }
}

func processBatch(ctx context.Context, p *kafka.Producer, c *kafka.Consumer,
    msgs []*kafka.Message) error {

    if err := p.BeginTransaction(); err != nil {
        return err
    }

    // Every failure past this point must abort, or the transaction hangs open
    // and read_committed consumers stall behind the LSO until it times out.
    abort := func(err error) error {
        p.AbortTransaction(ctx) // outputs AND offsets roll back together
        return err
    }

    for _, m := range msgs {
        if err := p.Produce(enrich(m), nil); err != nil {
            return abort(err)
        }
        if err := p.Produce(audit(m), nil); err != nil {
            return abort(err)
        }
    }

    meta, err := c.GetConsumerGroupMetadata()
    if err != nil {
        return abort(err)
    }

    // The input offsets join the SAME transaction. This is the whole trick.
    // Note nextOffsets returns last-read + 1 — the offset to resume FROM.
    if err := p.SendOffsetsToTransaction(ctx, nextOffsets(msgs), meta); err != nil {
        return abort(err)
    }

    return p.CommitTransaction(ctx)
}

That SendOffsetsToTransaction call is what makes read-process-write atomic. The consumer offset commit is written into __consumer_offsets as part of the transaction, so "I produced the output" and "I marked the input consumed" either both happen or neither does. Without it you have two separate commits and a window between them.

Go's explicit error handling makes the shape of this clearer than the Java equivalent does: every path out of the transaction is either a commit or an abort, never a return. Leaking out without calling one of them leaves the transaction open, and every read_committed consumer on those partitions blocks behind the LSO until transaction.timeout.ms expires.

Mechanically: a Transaction Coordinator (a broker, chosen by hash(transactional.id)) writes state to the internal __transaction_state topic, then writes commit or abort markers into every partition the transaction touched. A read_committed consumer never reads past the LSO (Last Stable Offset) — the first offset of any still-open transaction — so uncommitted records are on disk and invisible, and become visible all at once when the marker lands.

Step through the transaction timeline in the second panel above and watch the LSO hold everything back until the commit.

The four settings, and the one caveat

# producer
enable.idempotence=true
transactional.id=order-enricher-3      # MUST be stable across restarts
transaction.timeout.ms=60000

# consumer
isolation.level=read_committed
enable.auto.commit=false               # non-negotiable — offsets go via the transaction

transactional.id must be stable per logical task and unique across instances. That's what fences zombies: when a new instance calls initTransactions() with an existing id, the coordinator bumps the producer epoch, and the old instance's next write fails with ProducerFencedException instead of corrupting the stream.

The caveat worth stating loudly: the guarantee ends at Kafka's boundary. A transaction covering a Kafka write and a Postgres INSERT does not exist. For that path you still need an idempotent write downstream — a unique key, an upsert, or an outbox table — and read-committed consumption. Exactly-once inside Kafka, idempotence at every edge.

The production checklist

Everything above, compressed into things to actually go and check:

• Alert on max partition lag, not total. And alert on its rate of change, not its value.
• Alert on rebalance rate. consumer-coordinator-metrics:rebalance-rate-per-hour above single digits means something is wrong.
• Set min.insync.replicas=2 on every topic that matters. acks=all without it is a comfortable lie.
• Keep unclean.leader.election.enable=false. Availability is not worth silent data loss.
• Move to CooperativeStickyAssignor — but through a two-step rolling deploy with both assignors listed.
• Set group.instance.id on stateful consumers. Rolling restarts stop triggering rebalances entirely.
• Tune max.poll.records down before you tune max.poll.interval.ms up. Smaller batches fix more incidents than longer timeouts.
• Set segment.ms on low-traffic compacted topics. A segment that never rolls is never compacted.
• Watch UnderReplicatedPartitions and UnderMinIsrPartitionCount. Non-zero for more than a minute is a page.

Summary

Kafka is a log with careful bookkeeping around it. The log gives you cheap fan-out, replay, and retention by file deletion. The bookkeeping — group coordination, ISR tracking, high watermarks, sequence numbers, transaction markers — is what turns that log into something you can build a business on.

Almost every Kafka production surprise traces back to one of six misunderstandings — one per section of this post:

1. Thinking it's a queue. It's a log. Reads don't consume, which is why replay is free and fan-out is cheap.
2. Thinking Kafka balances messages. It balances partitions. That's why your ninth consumer on an eight-partition topic does nothing at all.
3. Not knowing a rebalance stops the world. Under the eager protocol, partitions that were never going to move stop anyway.
4. Thinking acks=all is enough on its own. Without min.insync.replicas, a shrunken ISR silently downgrades it to acks=1.
5. Confusing compaction with retention. One is keyed and keeps the latest value forever; the other is timed and deletes whole segments. They're orthogonal, and you can run both.
6. Expecting exactly-once to cross Kafka's boundary. It doesn't. The moment you write to Postgres, you're back to needing an idempotent write.

Play with the visualizers until each one feels obvious. Then go and look at your consumer group's rebalance rate.

Related: PostgreSQL storage internals — MVCC, dead tuples and the cost-based planner. Also: Raft consensus visualizer, which covers the leader-election machinery Kafka's own KRaft controller is built on, and the consistent hashing visualizer for the partitioning problem one layer down.