Infrastructure · Advanced

Distributed message queue

A Kafka-like partitioned, replicated log with independent consumer groups and tiered storage.

Every part, and why it is there

Producer services

Publish events such as “order placed”. Each event has a key (e.g. the order id); events with the same key always land in the same partition, so they stay in order.

Bootstrap endpoint

Only used to discover the cluster. After that, clients talk directly to the broker that leads each partition.

Broker cluster

Each broker appends incoming records to the end of partition files and serves reads by offset. Appending and sequential reads are what disks do best, which is why one broker can handle hundreds of MB/s.

Controller quorum (KRaft)

A small Raft group that tracks which broker leads each partition and elects a new leader when one fails. Older Kafka used ZooKeeper for this.

Partition logs (local disk)

Every partition is copied to 3 brokers. A write is acknowledged once all in-sync copies have it (acks=all), so losing one broker loses nothing.

Tiered storage

Older log segments move to cheap object storage, so you can keep weeks of history without buying weeks of broker disk.

orders topic (12 partitions)

What consumers see: an ordered, replayable log. Messages are not deleted when read; they expire by time or size, so a new consumer can replay from the start.

Billing consumer group

Each partition is read by exactly one member of the group, so 12 partitions allow up to 12 parallel consumers. The group commits its offset (how far it got) back to the brokers.

Billing DB

The consumer’s own state. Writes here must be idempotent, because after a crash the consumer re-reads from its last committed offset and may see a message twice.

Analytics consumer group

A second group reading the same topic independently at its own speed. Adding it puts no extra load on producers.

Warehouse

Loads events in large batches for reporting.

Asked in interviews at

  • anchorage-digital
  • roblox
  • amazon
  • ethos
  • opal-security
  • meta

Concepts used here

  • Message queues: Let the fast part of your app reply now and the slow part catch up later. Nothing is lost in a rush, but any job may run twice, so make twice harmless.
  • Raft consensus: Five servers hold copies of the same settings. Raft’s four rules stop them from ever disagreeing, even when the network splits them in two.

Learn more

All system design examples