Scope the requirements
The interviewer is testing whether you can build a reliable async pipeline around unreliable third parties. The hard parts are all failure-shaped: a worker dies mid-send, Twilio throttles you, a retry double-sends — and the design is judged on whether those cases were handled before being asked.
Functional — what it must do
- Send notifications on three channels: mobile push (APNs/FCM), SMS and email — in-app as an extension.
- Any internal service can trigger a send via API — transactional (OTP, receipts), promotional, system alerts.
- Scheduled and delayed delivery — "send this at 9am local time".
- User preferences: per-channel opt-out, quiet hours, and rate caps on promotional messages.
- Delivery status tracking with retries on failure.
- Explicitly cut: building the push infrastructure itself — APNs, FCM, Twilio and SES are the delivery rails; I orchestrate them.
Non-functional — the numbers that shape the design
- Soft real-time: delivered within seconds, tolerating brief delay under load spikes.
- At-least-once delivery — a notification must never be silently lost; dedup makes the user experience near-exactly-once.
- Scale: 50M DAU × 5 notifications/day = 250M/day, ~17k/s at peak.
- Highly available with no single point of failure — this system is upstream of every OTP login in the company.
- Ordering across notifications is not required — I say so explicitly, because it buys enormous freedom.
The contract I'm designing to is at-least-once with dedup on top — true exactly-once through SMS and email providers is impossible, and saying that up front, then engineering to make duplicates invisible, is the whole game.
Back-of-envelope estimates
The volume is modest by pipeline standards — the arithmetic matters because it shows the bottleneck isn't my system, it's the providers I call.
| Quantity | Estimate | How you got there |
|---|---|---|
| Send volume | ~3k/s avg | 250M/day ÷ ~86k s ≈ 3k/s; peak ~17k/s when a marketing campaign fires. |
| Peak absorption | ~17k/s → queue | Providers won't take 17k/s bursts — the queue is what turns a spike into a steady drain. |
| Device tokens | ~150M | 50M users × ~3 devices. Token storage ~50 GB — small, but token hygiene (pruning dead ones) is the real work. |
| Notification log | ~250 GB/day | 250M × ~1 KB status record ≈ 90 TB/yr — time-partitioned store with a retention policy, not a hot table. |
| Preference lookups | ~17k/s at peak | Every send checks opt-outs and caps — this read must come from cache, never the database. |
Seventeen thousand a second at peak is not a scary write rate — the point of the maths is that provider rate limits, not my throughput, are the constraint, which is why everything downstream of the API is queue-buffered and per-channel.
Sketch the API
One endpoint does the heavy lifting. The interesting decisions are the mandatory idempotency key and returning 202 rather than 200 — both are the async contract made visible.
requestId is a required idempotency key.- 202, not 200, and it's a promise: the API acks once the notification is persisted and enqueued — not sent. Callers who need the outcome poll status or subscribe to a callback. Acking before persistence is the classic way to lose notifications; acking after provider send would make the API's latency hostage to Twilio's.
- requestId is mandatory, not optional: retried API calls are inevitable — the caller's timeout fires, they resend, and without an idempotency key the user gets two OTPs. Dedup on requestId at the front door is cheaper than every downstream fix.
Data model
Four small stores with very different temperatures: tokens and preferences are hot reads, the log is a warm append-only stream, templates are nearly static.
Tokens and preferences go to DynamoDB with a Redis cache in front — the access is key-value at 17k/s and I want single-digit-millisecond reads without owning shards. The log goes to Cassandra or time-partitioned Postgres; it's append-heavy, queried by notificationId and by time window, and ages out on a retention schedule. If scale were 10× smaller I'd put everything in Postgres and say so — the queue architecture, not the database, is what this design is about.
High-level design
The shape is a funnel: many producers, one validating front door, then a fan-out into per-channel queues with independent worker pools calling the providers. Every arrow past the API is asynchronous on purpose.
- A service POSTs a notification with a
requestId→ the API dedupes against the log's unique index, checks preferences and promo caps from Redis, and drops invalid or opted-out channels immediately. - The notification is persisted to the log first, then enqueued onto the right per-channel Kafka topics — only then does the caller get its 202. Scheduled sends go to the scheduler, which enqueues when due.
- Channel workers consume their own topic, render the template, and call the provider — APNs/FCM for push, Twilio for SMS, SES for email — under a per-provider rate limit.
- Failures retry with exponential backoff via the queue's redelivery; after ~5 attempts the message lands in a dead-letter queue that pages a human instead of silently vanishing.
- Provider webhooks (delivered, bounced, invalid token) flow into the status updater → the log; invalid-token feedback prunes the devices table so tomorrow's sends don't fail the same way.
Deep dives — where the interview is won
Persist before ack, then dedupe your way to exactly-once-ish
The reliability contract lives in one ordering decision: the API writes the notification to durable storage before returning 202. From that moment the system owes a delivery, and every downstream failure is recoverable — a crashed worker's message reappears via the queue's visibility timeout, an unacked send retries with exponential backoff, and after about five attempts the message parks in a DLQ with an alert rather than evaporating. At-least-once falls out of "never delete until confirmed"; the queue does the bookkeeping.
At-least-once means duplicates, and a duplicate OTP SMS is a user-visible bug — so dedup runs at two levels. At the front door, the unique index on requestId absorbs caller retries. At the worker, each send attempt is recorded against (notificationId, channel) before calling the provider, so a redelivered message that was already sent is dropped, not re-sent. There's an irreducible window — worker sends, then crashes before recording — and I name it honestly: that's why the target is near-exactly-once, and why idempotency on the provider side (where offered) is worth using.
One queue per channel, or a slow SMS provider blocks your OTPs
A single queue for all channels is the design that fails politely in the demo and catastrophically in production: SMS providers throttle at a fraction of email's throughput, so SMS messages back up, and every push and email behind them waits — head-of-line blocking across channels that share nothing but a pipe. Per-channel topics make each channel's backlog its own problem: email workers scale on email lag, SMS workers on SMS lag, and a Twilio brown-out delays only SMS.
The same logic applies within a channel by priority. An OTP and a marketing blast are both "SMS" but have opposite urgency, so transactional and promotional traffic get separate lanes — separate topics or a strict priority tier — and campaign spikes of millions of messages drain behind a rate cap while OTPs cut through in seconds. This is the follow-up interviewers hold in reserve, and volunteering it early is cheap credit.
Providers fail constantly — rate limits, circuit breakers, and failover are the day job
APNs, FCM, Twilio and SES are the parts of this system I don't control, and they misbehave in every way distributed systems can: throttling, brown-outs, full outages, and silent token invalidation. So every worker pool runs a per-provider rate limiter matched to the contracted quota (send at their pace, not mine), a circuit breaker that stops hammering a failing provider and lets messages accumulate in the queue instead, and — for channels where it's viable — a failover provider: Twilio down means SMS shifts to SNS at the breaker, not after a war-room decision.
The quieter failure is token rot. Users reinstall the app, swap phones, revoke permissions — and APNs/FCM tell you, via feedback in the send response. Consuming that feedback to prune the devices table is unglamorous and essential: skipping it means an ever-growing fraction of "sends" burn quota against dead tokens and pollute delivery metrics until nobody trusts the dashboard.
Preferences and caps are checked before enqueue, not before send
Opt-outs, quiet hours and promotional caps sit at the front of the pipeline, in the API's validation step, against a Redis-cached copy of preferences. Filtering early means an opted-out user's notification never consumes queue, worker, or provider quota — at 250M a day, even a 10% opt-out rate is 25M messages of work avoided. Quiet hours convert into a scheduler hand-off: the message is accepted, held, and enqueued at the window's edge, so callers don't have to know user timezones.
Promotional rate caps (say, two marketing pushes a day) are a counter per (user, category, day) in Redis — increment-and-check at enqueue time, which is atomic and cheap. One deliberate exception cuts through all of it: transactional messages — OTPs, security alerts, receipts — bypass caps and quiet hours, because a 2am login code the user requested at 2am must arrive at 2am. Encoding that distinction in the type system of the API, not in caller discipline, is the design decision.
Follow-ups you should expect
Can you guarantee exactly-once delivery?
A marketing campaign wants to notify 30M users at once. What happens?
How do you handle a full Twilio outage?
Do notifications need to arrive in order?
How do you fight notification fatigue?
Where candidates lose the room
- Calling providers synchronously from the API handler — the caller's latency is now Twilio's latency, spikes have nowhere to buffer, and a provider outage becomes your outage.
- One shared queue for all channels — a throttled SMS provider head-of-line blocks every push and email behind it; the per-channel split is the architecture.
- No idempotency key — the first caller retry double-sends an OTP, which is the single most user-visible bug this system can produce.
- Acking before persisting, or deleting from the queue before provider confirmation — both are quiet ways to lose notifications, and 'at-least-once' was the headline requirement.
- Ignoring preferences and token hygiene — sends that burn quota on opted-out users and dead tokens, and delivery metrics nobody can trust.