Scope the requirements
The trap: candidates start from WhatsApp and layer channels on top. Slack's real spine is the channel and the workspace — everything hangs off those two.
Functional — what it must do
- Send messages in channels (public/private) and DMs, with threads under any message.
- Real-time delivery to every online member of a channel; unread markers per (user, channel).
- Presence: online/away/dnd, updated within a few seconds.
- Full-text search scoped to the user's workspace, with permission checks per channel.
- Explicitly cut: video calls, apps/bots, message editing history — offer them back.
Non-functional — the numbers that shape the design
- Delivery latency p95 under 200 ms intra-workspace.
- 500M messages per day across ~5M active workspaces; peak ~30k msg/s.
- 50M concurrent WebSocket connections at peak.
- Strict workspace isolation — a search or fan-out never crosses tenants.
- 99.99% availability on read paths; 99.9% on send.
I'm going to design channel-first, not conversation-first. That single choice fixes fan-out, unread counts and search boundaries at the same time.
Back-of-envelope estimates
The numbers say: fan-out is cheap on average and brutal in one channel, and the search index is 100× the size of the message store.
| Quantity | Estimate | How you got there |
|---|---|---|
| Messages/s | ~6k avg, ~30k peak | 500M/day ÷ 86,400; peak 5× average during working hours. |
| Fan-out per message | median 8, p99 ~2k | most channels are small; #general in a 10k-employee tenant is the tail. |
| Sockets | ~50M | one per active client; each server holds ~50k, so ~1,000 gateway boxes. |
| Storage/day | ~150 GB | 500M × ~300 B (msg + meta), 10× with attachments metadata. |
| Search index | ~15 TB | postings + positions ~100× raw messages; sharded by workspace_id. |
| Presence updates | ~5M/s peak | heartbeat coalesced to 15 s per client; total ≈ concurrent users / 15. |
50M sockets means my gateway tier is the biggest thing on the diagram, and it must not talk to the database on the send path.
Sketch the API
REST for CRUD on channels and history, one WebSocket for realtime. The interesting detail is that send is a REST POST — the socket is only for receiving.
- Send is REST, receive is WS: keeps sends idempotent and retryable, and lets the WS tier be pure fan-out with no write path.
clientMsgIddeduplicates retries so the client can safely resend on network flakiness without double-posting.
Data model
Two hot tables: messages (append-heavy, wide-partitioned) and channel_members. Everything else — search, unread — is derived.
Cassandra for messages because writes are append-only, reads are always slice-by-channel, and one channel maps cleanly to one partition. Postgres holds channels/members where I need transactional joins. If a channel got too hot (a single mega-channel > 100 GB), I'd bucket by (channel_id, month) — say that unprompted.
High-level design
Two tiers with different jobs. A stateless API tier writes and reads. A gateway tier holds all the sockets and fans out via a Redis pub/sub bus — it never touches the database.
- Client POSTs to send API with clientMsgId. API checks membership against Postgres, appends to Cassandra (channel_id, ts), publishes to Redis pub/sub keyed on channel_id.
- Every gateway holds a subscription for every channel one of its sockets is in — a message on that channel lands on a handful of gateway boxes, not all 1,000.
- Each gateway pushes to the exact sockets that are members of that channel — the local socket map is the fan-out oracle.
- The indexer consumes the same events off the bus and writes into a workspace-partitioned Elasticsearch index.
- Search hits the API tier, which queries the caller's workspace index only — the tenant boundary is baked into the shard key, not an application check.
Deep dives — where the interview is won
Fan-out via a channel-keyed bus, not a per-user queue
The naive design gives every user a mailbox and inserts N rows per message. That works for DMs and dies on #general. The stronger move: publish once to a channel topic on Redis pub/sub, and let every gateway that holds a subscriber decide who to push to locally. Message writes stay O(1); fan-out cost is paid in memory, not in database rows.
The gateway maintains an in-memory channel_id → set<socket_id> map, populated on socket connect from Postgres. When a channel event lands, it iterates the set and writes to sockets — pure memory, ~10 µs per delivery. The bus itself needs enough partitions that no single channel's traffic swamps one broker; 128 partitions handles hot channels comfortably.
Unread counts without a per-user counter
The naive answer — increment unread[user][channel] on every fan-out — writes N times per message and locks a hot row in #general. The Slack move: store one number per (user, channel), the user's last_read_ts, and compute unread on demand as count(messages where ts > last_read_ts). That count is bounded (clients don't need exact numbers past 99+) so you can cap the query.
The client updates last_read_ts when it renders new messages — one write per active view, not one write per message. The unread badge shown before the channel opens comes from a lightweight aggregate maintained by the same indexer that feeds search: it emits (channel_id, ts) events, and a per-user rollup keeps counts in Redis with a TTL.
Search with the workspace boundary in the shard key
Multi-tenant search is where junior designs leak data. The right structure: shard Elasticsearch by workspace_id (either as routing key on a shared index or as separate index-per-workspace once a tenant grows). Every query is forced to include workspace_id = X — enforcement lives in a proxy layer that clients can't bypass. Cross-tenant searches become physically impossible, not merely disallowed by application code.
Permission filtering (private channels the user isn't in) then happens as a post-query filter against the caller's channel_members set, cached in Redis. Elasticsearch returns a superset by workspace; the API drops anything the user can't see. The alternative — filtering during ranking — makes the index know about ACLs, and every membership change becomes a reindex.
Follow-ups you should expect
How do you handle a socket reconnect without lost or duplicated messages?
What happens in a workspace with a #general channel of 50k members?
How do threads not blow up your data model?
thread_ts set to the parent ts, in the same channel partition. Reading a thread is a filtered slice within the partition, which is cheap. The channel view shows only messages with null thread_ts unless a client opens a thread.Presence for 50M users — how do you not DDOS yourself?
presence changed event on transitions (online↔away↔offline), not on every ping. The steady-state cost is a boolean update per user per interval, gossiped to interested gateways via the same bus, keyed on friends.How would you add message editing history?
message_edits(channel_id, ts, edit_ts, text). Reads join on demand — most reads never fetch edits, so the extra table costs nothing on the hot path. Never mutate the original row.Where candidates lose the room
- Modelling channels as recipients of DMs — you inherit N-writes-per-message and lose the fan-out win.
- Storing unread as a counter per (user, channel) — hot rows on #general and no way to recompute after a bug.
- Fanning out from the send API directly to sockets — the API tier becomes the socket owner and can't scale independently.
- Filtering search results in application code without workspace-in-shard-key — one bug leaks tenant data.
- Pushing every presence heartbeat as an event instead of only on transitions — 5M/s of noise for no user-visible change.