Queues & pub/sub

What it is

A message queue decouples producers from consumers: producers hand work to a broker and move on; consumers process it at their own pace. Point-to-point queueing delivers each message to exactly one worker (task distribution); pub/sub delivers each message to every subscribed consumer group (event broadcast). The wins are decoupling, burst absorption, retries, and smoothing the speed mismatch between a fast producer and a slow consumer.

In an interview, the queue is the reflex answer to "this work is slow, bursty, or can fail — take it off the request path": notification fan-out, video transcoding, crawler frontiers, location event streams. The caution I repeat out loud is that queues hide capacity problems rather than solving them — if consumers fall behind forever, the queue is a landfill with a delay, and the real fix is consumer capacity or load shedding.

Say this out loud

I'd put this behind a queue so the request path stays fast — Kafka if I need replay and high-volume fan-out, RabbitMQ or SQS for plain task distribution — and my consumers are idempotent because delivery is at-least-once.

The picture

producers append, the log keeps everything, and each consumer group remembers its own offset — the queue never deletes on read, which is why two groups can read the same stream.

The variants and when to pick each

Kafka — a replayable log, not really a queue

Kafka is a distributed append-only log. Topics split into partitions; consumers track their own offsets, which means they can replay history, and retention is by time or size (7 days default) rather than by consumption. Ordering is guaranteed per partition only — key by user_id and one user's events stay in order. Consumer groups give both semantics at once: within a group it's a queue, across groups it's pub/sub. Throughput comes from sequential disk I/O and zero-copy — clusters do a million-plus messages per second without heroics.

I pick Kafka for event streaming, analytics pipelines, event sourcing, and any fan-out where several independent consumers each need the full stream.

Trade-offThe log model trades broker conveniences for throughput and replay — no per-message acks, priorities or delays; more partitions buy parallelism but ordering survives only within a key.

RabbitMQ and SQS — smart broker, per-message guarantees

RabbitMQ is the inverse philosophy: a smart broker, dumb consumers. Exchanges (direct, topic, fanout, headers) route messages to queues; you get per-message acks, priorities, TTLs, delayed delivery and dead-letter exchanges; a message is deleted once acked. SQS is the managed flavour — visibility timeout instead of acks, effectively unlimited throughput on standard queues, and a FIFO variant capped at 300 messages per second (3,000 batched).

This family wins for job queues, complex routing, and request-like work at moderate volume where per-message behaviour matters more than raw throughput.

Trade-offRich routing and per-message control top out around 10–50k messages per second per node and offer no replay — once acked, the message is gone.

Delivery semantics — exactly-once is engineered, not offered

At-most-once (fire and forget) can lose messages — acceptable for metrics and telemetry. At-least-once (ack after processing) is the default everywhere, and it means duplicates on retry, so consumers must be idempotent — that phrase, said unprompted, is what interviewers are listening for. Exactly-once is at-least-once plus idempotent or transactional processing: Kafka provides it within its own ecosystem via idempotent producers and transactions, but end-to-end across arbitrary external systems it is really "effectively once" via dedupe keys.

Concretely: key each side effect on a message ID and make the handler an upsert or a conditional write, so replaying a message is a no-op.

Trade-offAt-least-once pushes correctness work into every consumer; at-most-once pushes data loss onto the business — pick per message type, and never claim free exactly-once across systems.

The operational vocabulary that signals experience

Consumer lag is the number-one health metric — how far behind the log the consumers are. Dead-letter queues quarantine poison messages after N failed attempts instead of letting one bad payload wedge a partition. Backpressure is the plan for when producers outrun consumers: scale consumers, shed low-value load, or slow producers — never "make the queue bigger". And the ordering-versus-parallelism tension: more partitions means more parallel consumers, but order holds only per key.

Trade-offEvery mitigation costs something — DLQs need someone to drain them, load shedding drops real work, and slowing producers moves the pain upstream; the failure is having no plan, not picking the wrong one.

Numbers worth memorising

Kafka cluster throughput	~1M+ msgs/s comfortably; per-partition ceiling ~10s of MB/s
RabbitMQ per node	~10k–50k msgs/s typical
SQS FIFO	300 msg/s, ~3,000 with batching (standard: effectively unlimited)
Kafka retention default	7 days (configurable, or compacted forever)
Message size sweet spot	≤1 MB (Kafka's default max) — store blobs in S3, queue the pointer

Where the interviewer pushes

The first push: "a consumer crashes halfway through a message — what happens?" The message was never acked (or its visibility timeout lapses), so the broker redelivers it — meaning any side effects already performed will happen again. So the strong answer is idempotency, mechanically: key the side effect on the message ID, write via upsert or a conditional insert into a processed-ids table, and now redelivery is harmless. Answering "it's redelivered" without the duplicate consequence is a half answer.

Second push: "queue depth has been growing for an hour — what do you do?" The trap answer is more retention or a bigger queue; the queue was supposed to absorb a burst, and a permanently growing one means consumers are structurally under-provisioned. The real answer: scale consumers out (up to the partition count — beyond that, repartition), shed or degrade low-value work, apply backpressure to producers, and check for a poison message spinning in a retry loop that a dead-letter queue would have caught.