Infrastructure Primitives

Design a Message Queue (Kafka clone)

A durable, partitioned append-only log system decoupling producers from consumers with configurable retention and consumer-group semantics.

Core Architectural Components

Per-topic, per-partition sequential storage — the fundamental durability primitive.

Hosts partition leaders and followers, spreading load and replicas across machines.

Batches and routes messages to the correct partition via a key or round-robin.

Manages offset tracking and partition rebalancing across a consumer group.

In-sync-replica tracking that guarantees no acknowledged write is lost on leader failure.

Tracks cluster topology, partition leadership, and broker liveness.

Time/size-based retention or key-based log compaction per topic.