Scope the requirements
The interviewer wants two things found fast: messages must survive recipients being offline, and the connection count is so large that anything per-connection-per-heartbeat is instantly fatal. Everything else is decoration around those two facts.
Functional — what it must do
- 1:1 and group chats (cap groups at 100 participants — say the cap out loud).
- Send and receive messages in real time, with delivery and read acknowledgements.
- Offline delivery: undelivered messages are stored (30 days) until the recipient reconnects.
- Media attachments (photos, video) in messages.
- Multi-device: one account, up to ~3 clients, all in sync.
- Explicitly cut: stories, calls, payments, server-side search — offer them back as extensions.
Non-functional — the numbers that shape the design
- Delivery latency under 500 ms to online recipients.
- At-least-once delivery — a message accepted by the server must eventually reach every recipient device.
- Scale: billions registered, ~200M concurrent connections.
- Minimal server-side retention — delete on delivery, 30-day cap for the undelivered (privacy is a feature here).
- Survive chat-server crashes and flaky mobile networks without losing messages.
The hard requirement is guaranteed delivery to devices that are offline half the day, at 200 million concurrent connections. I'll design the durability story first and treat the realtime path as an optimisation layered on top of it.
Back-of-envelope estimates
The numbers here don't argue for sharding storage — messages are tiny and short-lived. They argue about connections: the heartbeat arithmetic alone rules out entire classes of design.
| Quantity | Estimate | How you got there |
|---|---|---|
| Concurrent connections | ~200M | 1B daily users, ~20% connected at any moment. |
| Messages | ~40k/s | 1B users × 20 msgs/day ÷ 86,400 s; ~100k row writes/s once group messages fan out into per-recipient inbox entries. |
| Heartbeats | ~20M/s | 200M connections ÷ 10 s interval. This is why heartbeats must never touch a database. |
| Chat servers | ~200–1,000 | a tuned box holds hundreds of thousands of WebSocket connections; the fleet is small, the routing between boxes is the problem. |
| Message storage | ~25 TB rolling | 4B msgs/day × ~200 B × 30-day retention; deleted on delivery, so the working set stays bounded. |
| Media | not through us | presigned upload straight to S3; the message carries a URL, never bytes. |
Forty thousand messages a second is ordinary database traffic — the scary number is twenty million heartbeats a second. So state lives in memory on the chat servers, and the database only ever sees messages, never liveness.
Sketch the API
Chat is bidirectional, so the API is WebSocket messages, not REST — the server initiates as often as the client does. HTTP survives only for login and media presigning.
- The client ack is load-bearing:
newMessagedelivery over the socket proves nothing — the network can drop the frame after the server sends it. Only the explicit ack lets the server delete the inbox row. No ack protocol, no delivery guarantee — this is the single most common structural omission in this question. - WebSocket over long polling or SSE, and I'd say why: chat is genuinely bidirectional at high frequency, which is the one workload where WebSocket's stateful-connection cost is clearly paid back.
Data model
Five small tables, all keyed for exactly one access pattern each. The interesting one is the inbox — it's the durability mechanism, not a mailbox feature.
I'd use DynamoDB (or Cassandra) — every access is a key-value lookup or a partition scan, writes are 100k/s and spiky, and native TTL does the 30-day cleanup for free. There's no relational query anywhere in this design, so Postgres buys me nothing here. If the product grew server-side message search or complex chat administration, that's when I'd bring in a relational or search store beside it — for the new workload, not as a migration.
High-level design
Sender and recipient usually sit on different chat servers, so the design is really about the hop between them: a durable write first, then a fast, lossy pub/sub hop that is allowed to fail because the inbox has it covered.
- Send: sender's device fires
sendMessageover its socket → chat server A writes the message and one inbox row per recipient device to DynamoDB, then acks the sender. Durability is done before anyone hears about the message. - Deliver: chat server A publishes to the Redis Pub/Sub channel for each recipient userId → chat server B, subscribed on behalf of its connected users, pushes
newMessagedown the recipient's socket. - Ack: the recipient device acks → chat server B deletes that device's inbox row. No ack (dropped frame, crashed app) → the row stays, and the message is re-delivered on the next inbox sync.
- Offline: no subscriber on the channel, nothing happens — the inbox row waits. On reconnect the client calls
GET /inbox?since={seq}and drains everything it missed. - Media: client gets a presigned URL, uploads to S3 directly, and sends a message whose body is the S3 key — chat servers never carry bytes.
Deep dives — where the interview is won
Write the inbox first — the pub/sub hop is allowed to fail
Redis Pub/Sub is at-most-once: no subscriber at the instant of publish, message gone. That sounds disqualifying for guaranteed delivery, and it would be — if the bus were the durability mechanism. It isn't. The inbox row is written and acknowledged before the publish, so the invariant is: every undelivered message has a row, and rows are only deleted by client acks. The bus is purely a latency optimisation that turns 'wait for the next sync' into 'arrives in 100 ms'.
Gaps in the fast path still need detecting. Two mechanisms: the client polls its inbox every 30–60 s as a safety net, and every message carries a per-chat sequence number — a client seeing seq 7 after seq 5 knows to sync immediately rather than wait for the poll. Piggyback the latest sequence numbers on heartbeats and gap detection costs nothing extra.
Routing: a Redis channel per user, never a Kafka topic per user
The recipient's socket lives on some other chat server, so I need a way to reach it. The clean answer is a Pub/Sub channel per userId: whichever server holds a user's socket subscribes to that user's channel, and senders publish blind. Nobody maintains a userId→server map that must be kept consistent through crashes and reconnects — subscription is the routing table, and it self-heals when a client reconnects elsewhere.
The trap answer is Kafka with a topic per user, and it's worth killing with arithmetic: a Kafka topic costs roughly 50 KB of cluster metadata, so a billion users is ~50 TB of pure topic bookkeeping before the first message flows. Kafka also persists everything — but I already have durability in the inbox, so I'd be paying for a guarantee twice. Per-user channels beat per-chat channels too: WhatsApp traffic skews heavily 1:1, and per-user means a device's server holds exactly one subscription per connected user regardless of how many chats they're in.
200M connections: liveness is an in-memory problem
TCP won't tell you a phone died — a peer that vanishes behind a dropped cell connection looks identical to a silent one for minutes. So liveness is application-level: ping every 10–30 s, expect a pong within ~5 s, and a missed pong means the server closes the socket and unsubscribes that client's channels; the client reconnects (through the load balancer, to any server) and drains its inbox. That reconnect-and-sync loop is the same machinery as offline delivery, which is the sign of a sound design — failure handling reuses the normal path.
The arithmetic is the discipline: 20M heartbeats/s must terminate entirely in the chat server's memory — a socket map and a timer wheel. The one thing users see derived from liveness is last-seen, and the write pattern is: update on disconnect, plus a lazy conditional write at most once every few minutes while connected. Writing last-seen per heartbeat is a 20M writes/s self-inflicted outage, and interviewers listen specifically for whether you step around it.
Multi-device: the inbox is per client, not per user
One account, three devices, each with its own connection, its own delivery state, and its own gaps. If the inbox were keyed by userId, the first device to ack would delete the row and starve the others. So every message fans out to one inbox row per registered client, and acks are per-client too — the phone acking does nothing to the laptop's copy.
Routing barely changes: publish once to the userId channel, and every chat server holding one of that user's sockets picks it up and pushes to its device. The cost of the cap matters here — ~3 clients per user bounds the inbox write amplification at 3×, which is why the earlier estimate said ~100k row writes/s against 40k messages/s. An unbounded device list would make every send a fan-out of unknown size.
Follow-ups you should expect
How would this change for Telegram-style groups with 100,000 members?
Where does end-to-end encryption change your design?
What happens when a chat server crashes with 500k connections on it?
A Redis Pub/Sub node dies — what do users experience?
How do you order messages within a chat?
Where candidates lose the room
- No client ack protocol — delivery 'guaranteed' by the act of sending a WebSocket frame, which is precisely the guarantee TCP doesn't give you across a dying mobile connection.
- Designing only the online path and bolting on offline storage when prompted — the inbox is the core of this question, not an edge case.
- Kafka with a topic per user for routing — ~50 KB of metadata per topic × 1B users is 50 TB of bookkeeping, and it duplicates a durability guarantee the inbox already provides.
- Writing last-seen (or anything) to a database per heartbeat — 20M writes/s that a socket map in memory handles for free.
- Pushing media bytes through chat servers instead of presigned S3 URLs, turning a message fabric into a file-transfer bottleneck.
- No failure walk-through — what happens on chat-server crash, Redis loss and network partition is where this question's marks actually live.