Scope the requirements
The trap here is treating this as a CRUD problem. The interviewer wants to see whether I know that a queue at this scale is a storage-engine design question — and whether I can state the ordering and delivery guarantees precisely instead of hand-waving them.
Functional — what it must do
- Producers publish messages to named topics; consumers subscribe and consume them.
- Topics split into partitions; ordering is guaranteed within a partition, keyed by a partition key.
- Consumer groups: each partition is consumed by exactly one member of a group; separate groups each see all the data.
- Configurable retention (say two weeks) with replay — a consumer can rewind to any offset it still holds.
- Explicitly cut: message routing rules, priority queues, per-message delay — those are RabbitMQ-shaped features and I'd say so.
Non-functional — the numbers that shape the design
- Throughput target: ~1 GB/s sustained ingest per cluster, messages around 1 KB.
- Durability: no acknowledged message is ever lost — replication factor 3, survive any single broker failure.
- Delivery: at-least-once by default, with at-most-once and effectively-exactly-once as configurable options.
- Latency vs throughput is a dial, not a fixed target — batching trades tens of ms of latency for order-of-magnitude throughput.
- Horizontally scalable: add brokers and partitions without downtime.
The founding fact of this design is that sequential disk writes run at hundreds of megabytes a second even on spinning disks. So I'm not going to build a database — I'm going to build a replicated append-only log and let the filesystem do the work.
Back-of-envelope estimates
The numbers here don't size a database — they size a log. What matters is bytes per second per partition, and how many partitions and disks that implies.
| Quantity | Estimate | How you got there |
|---|---|---|
| Ingest | ~1 GB/s | ~1M messages/s at ~1 KB each; this is the sustained cluster target. |
| Per-partition throughput | ~10 MB/s | one partition is one sequential log on one leader — call it ~10k msg/s comfortably. |
| Partitions needed | hundreds | 1 GB/s ÷ 10 MB/s = 100 minimum; provision several hundred for headroom and hot keys. |
| Retention storage | ~1.2 PB | 1 GB/s × 14 days ≈ 1.2 PB of raw log. |
| With replication ×3 | ~3.6 PB | every partition has a leader and two followers; storage is the real cluster cost, not CPU. |
| Sequential disk write | ~500 MB/s | even HDDs manage hundreds of MB/s sequentially — the number that makes the whole design work. |
One gigabyte a second sounds scary until you divide it: a hundred-odd partitions at ten megabytes a second each, and each partition is a plain sequential file write. The hard part isn't throughput, it's not losing data during failures.
Sketch the API
The API is small and looks deceptively like a function call. The interesting decisions are the acks setting on produce and the fact that consume is a pull, not a push.
acks (0, 1, all).- Consumers pull; the broker never pushes. Pull lets a slow consumer control its own rate — backpressure is free — and makes replay trivial because the consumer owns its offset. The cost is polling overhead, which long-polling (hold the fetch open until data arrives) reduces to nearly nothing. I'd volunteer this before being asked; push-based designs with no backpressure story are the classic failure here.
- The offset is the whole consumption model: the broker deletes nothing on consume, it only ages data out by retention. That single choice is what separates this from RabbitMQ and enables replay, multiple independent groups, and reprocessing after a bug.
Data model
There is deliberately no database here. The partition log is the data model, and saying that clearly is worth more than any schema.
I'd store messages as raw log segments on local disks and refuse the database on principle: sequential appends, the OS page cache serving hot reads, and zero-copy sendfile from file to socket is the entire performance story — a B-tree would throw all three away. If a requirement appeared for months of retention, I'd add tiered storage — old segments offloaded to S3 with brokers serving them on demand — rather than ever reaching for a database.
High-level design
Brokers own partitions; each partition has one leader taking all reads and writes, with followers replicating. A small metadata quorum handles liveness and leader election, and a coordinator manages consumer-group membership.
- A producer fetches cluster metadata once, hashes the partition key, batches messages, and sends the batch to that partition's leader broker.
- The leader appends the batch to its log segment — a sequential write into the page cache — and followers pull the new records to replicate.
- With
acks=all, the leader acknowledges only once every in-sync replica has the batch; the producer's ack means "this survives a broker loss". - Consumers in a group are assigned partitions by the coordinator; each long-polls its leaders with
fetch(offset)and receives batches served largely from page cache via zero-copy. - After processing, the consumer commits its offset to the compacted offsets topic. If a broker dies, the metadata quorum promotes an in-sync follower and clients re-fetch metadata — the log makes recovery a pointer update, not a rebuild.
Deep dives — where the interview is won
The storage engine is a file, and that's the whole trick
Every message store temptation — MySQL, Mongo, even Redis — dies on the same fact: this workload is write-once, read-sequentially, delete-by-age. An append-only segment file matches it perfectly. Writes are sequential (hundreds of MB/s per disk), reads for a caught-up consumer come straight from the OS page cache, and delivery to the socket uses zero-copy sendfile so message bytes never even enter the application's memory. Retention is dropping whole segment files — no vacuuming, no fragmentation.
The offset index is the only structure on top: a sparse file mapping offset → byte position every few KB, so a rewind seeks in two hops. I'd say explicitly that consumers being mostly caught-up is what makes this scream — the hot tail of every partition lives in RAM for free, and the design degrades gracefully to disk reads only for replays.
Durability is a dial: acks and the in-sync replica set
Each partition has a leader and (with RF=3) two followers. The ISR — in-sync replica set — is the subset currently caught up. The producer chooses its guarantee: acks=0 is fire-and-forget for metrics, acks=1 means the leader has it (a small loss window if the leader dies before replication), acks=all means every ISR member has it — durable against any single failure, at the cost of the slowest in-sync follower's latency.
The subtle brilliance of the ISR is what it decouples: "replicated" no longer means "all replicas". A follower that falls behind gets evicted from the ISR, writes continue, and it rejoins when caught up. Without that, one slow disk stalls the whole partition. Leader failure is then a metadata operation: the quorum promotes an ISR member, which by definition has every acknowledged message.
acks=all silently degrades to acks=1 — so you set min.insync.replicas=2 and accept refusing writes over losing them. That's a CAP choice and I'd name it as one: for a durability product, unavailability beats data loss.Exactly-once is at-least-once plus idempotence — say the mechanics
At-least-once falls out naturally: producers retry on timeout, consumers commit offsets after processing, so crashes cause reprocessing, never loss. The duplicates come from two places and each has a distinct fix. Producer-side: a retry after a lost ack re-appends the batch — fixed by an idempotent producer, where each producer gets an ID and per-partition sequence numbers so the broker discards replays. Consumer-side: crash after processing but before commit — fixed either by transactional offset commits (process and commit atomically) or, far more common in practice, an idempotent consumer that dedupes on a business key.
The interview point to land: exactly-once is not a wire protocol property, it's an end-to-end contract, and the pragmatic answer for most systems is at-least-once delivery with idempotent handlers. Claiming exactly-once without saying sequence numbers or dedupe keys is the fastest way to lose credibility here.
Partitions are the unit of everything — ordering, parallelism, and pain
One partition gives you total order, one consumer's worth of parallelism, and one broker's worth of throughput — so partition count is the central capacity decision. Keys with the same value always land in the same partition, which is the ordering guarantee users actually rely on: all events for one user, in order. The two failure modes are hot partitions (one celebrity key taking 10× the traffic — fix by salting the key or accepting per-entity throughput limits) and the fact that adding partitions breaks key→partition stability, reshuffling which partition a key maps to — so you over-provision partitions up front rather than resizing live.
Consumer-group rebalancing is the operational sharp edge: naive rebalance stops the world for every member whenever one joins or dies. I'd mention cooperative/sticky assignment — only the moved partitions pause — as the fix, because rebalance storms are what actually pages people at 3am.
Follow-ups you should expect
When would you pick RabbitMQ over this design?
Can you give me global ordering across the whole topic?
A consumer keeps crashing on one message — what happens?
How do you handle a 50 MB message?
Retention is two weeks but compliance wants six months. Now what?
Where candidates lose the room
- Storing messages in MySQL or MongoDB with random access — it misses the append-only insight the entire question exists to test.
- Promising global total ordering across partitions, which quietly commits you to single-node throughput.
- Designing push-based delivery with no backpressure story — the first slow consumer takes down the pipeline.
- Skipping offset management: no answer for a consumer that crashes mid-batch means no delivery guarantee at all.
- Claiming "exactly-once" as a checkbox without sequence numbers, transactions, or idempotent-consumer mechanics behind it.