Scope the requirements
Scope ruthlessly: this question is upload and playback of huge files. Search, recommendations and comments are separate systems — name them and cut them, or the transcoding pipeline you're actually being tested on gets ninety seconds of airtime.
Functional — what it must do
- Upload videos — tens of GB — with title and metadata.
- Stream videos with quality that adapts to the viewer's bandwidth.
- Playback starts fast and stays smooth on poor connections.
- Resumable uploads: a dropped connection at 90% must not restart the transfer.
- Explicitly cut: search, recommendations, comments, monetisation, live streaming — each is its own interview.
Non-functional — the numbers that shape the design
- Playback start under ~1–2 s; no rebuffering on a stable connection at any bandwidth (adaptive bitrate is a requirement, not a feature).
- Availability over consistency — a video appearing a minute late is fine; playback failing is not.
- Scale: ~1M uploads/day, 100M watches/day, global audience.
- Originals stored durably forever — re-uploading is not an option.
- Uploads of 10+ GB must survive flaky client connections.
Watch-to-upload is 100 to 1, but the write path is where the engineering is: getting a 10 GB file in reliably and turning it into something a phone on 3G can play. I'll design the pipeline first, then make the read path a CDN problem.
Back-of-envelope estimates
The request rates are almost embarrassingly small — the numbers that matter are bytes: daily ingest, one upload's transfer time, and the CDN invoice. Those three drive every design decision here.
| Quantity | Estimate | How you got there |
|---|---|---|
| Uploads | ~12/s | 1M/day ÷ 86,400 s. Trivial as a request rate; enormous as bytes. |
| Watches | ~1,200/s | 100M/day; 100:1 watch:upload ratio. Metadata reads, then the CDN takes over. |
| New storage | ~150 TB/day | ~500k daily uploaders × ~300 MB average — call it 50 PB/year before transcoded renditions roughly double it. |
| One 10 GB upload | ~13 min at 100 Mbps | any real connection drops in that window — resumable chunked upload is mandatory, not nice-to-have. |
| CDN egress | ~$150k/day | order-of-magnitude for ~5M daily watchers at CDN per-GB rates — the bill that makes long-tail-from-origin worth designing for. |
| Metadata | ~365M rows/yr | a rounding error next to the blobs; metadata scale is never the problem in this question. |
Twelve uploads a second but 150 terabytes a day — this system is about bytes, not requests. So the architecture keeps video bytes off my application servers entirely: clients talk to S3 and the CDN directly, and my services only ever move metadata.
Sketch the API
The API splits cleanly in two: a control plane my services own, and a data plane where the client talks straight to S3 and the CDN. The interesting decision is that playback never goes through my API at all.
uploading state; bytes never touch this service.- The manifest is the API for playback. After one metadata call, the player deals exclusively with the CDN: fetch manifest, measure bandwidth, pick a rendition, fetch segments, re-decide every few seconds. My services could be down and playback of anything already published keeps working — worth saying out loud, it reframes the availability story.
- Presigned multipart URLs mean upload auth is decided once by my control plane, then S3 enforces it per-part — my servers never proxy a byte in either direction.
Data model
Two tables and a bucket. The metadata row does double duty as the upload and processing state machine, which is what keeps resumability honest.
Cassandra for video metadata: point lookups by videoId, no cross-entity transactions anywhere, and I get multi-region replication and linear write scale for free — availability over consistency is exactly Cassandra's contract. Honestly, at 365M rows a year Postgres would also cope for a long time; if the interviewer prefers it I'd take it and name the multi-region replication story as the thing I'd have to build myself. The blobs were never going in a database either way.
High-level design
Upload is a control-plane handshake followed by a client-to-S3 transfer; an event-driven DAG then fans one video out into dozens of segment-level transcode jobs; playback is a manifest and a stream of segments served almost entirely from CDN edges.
- Upload: client POSTs metadata → video service writes a
uploadingrow to Cassandra and returns presigned multipart URLs → client uploads 5–10 MB parts straight to S3, PATCHing progress as it goes. - On completion, an S3 event kicks the orchestrator: split the original into segments, then fan out one transcode job per segment per rendition — H.264/H.265 × 360p–4K — all in parallel.
- Workers write finished segments back to S3, passing data between steps as S3 URLs; a final step generates the HLS/DASH manifests and flips the Cassandra row to
ready. - Watch: viewer fetches metadata (one API call) → pulls the manifest from the CDN → player measures bandwidth and requests segments at whatever bitrate the last few seconds justify.
- The CDN serves hot segments from edge; misses pull from S3 once and stay cached — segments are immutable, so TTLs are effectively infinite.
Deep dives — where the interview is won
Segment first — the one decision everything else hangs on
Cutting every video into ~4-second segments looks like an encoding detail, but it's the load-bearing decision of the whole design. Segments make transcoding parallel: a two-hour film is ~1,800 independent jobs per rendition, so wall-clock transcode time is minutes on a big worker pool instead of hours on one machine. Segments make playback adaptive: the client can change bitrate at every segment boundary because each segment is independently decodable. And segments make the CDN efficient: the cache unit is a few MB that many viewers share, not a multi-GB file.
The naive alternative — progressive download of one big MP4 — fails the stated requirements one by one: no mid-stream quality switching, so the low-bandwidth requirement dies; no parallel transcode, so publish latency dies; cache granularity of whole files, so the CDN hit rate dies. When an interviewer asks 'why not just serve the file?', this is the answer, requirement by requirement.
The transcoding pipeline is a DAG with an orchestrator, not a script
Post-upload processing is a genuine dependency graph: split → per-segment transcodes (fully parallel, per codec and resolution) → audio and thumbnail branches alongside → manifest generation, which needs every segment done. At a million uploads a day some worker is always dying mid-job, so I run it on a workflow orchestrator — Temporal or Step Functions — which owns the graph state, retries individual failed segment jobs with backoff, and resumes a half-finished video without redoing completed work.
Two rules keep the pipeline sane. Workers exchange data as S3 URLs, never payloads — steps stay stateless and any retry can run on any machine. And every step is idempotent — writing segment 42 at 720p twice must be harmless, because with retries it will happen. Hand-rolling this coordination in application code is the classic way this question goes sideways in the last fifteen minutes.
Resumable upload: multipart with server-verified ETags
Thirteen minutes is the happy-path transfer time for 10 GB, so the design assumption is that every large upload fails at least once. S3 multipart is the mechanism: the control plane initialises an upload and hands out presigned per-part URLs, the client pushes 5–10 MB parts — in parallel, which also fills fat pipes — and records each part's ETag via PATCH. On resume, the client reads the chunk-state array from metadata and skips what's done.
The subtle point that separates candidates: don't trust the client's claim. Before completing the multipart upload, the server verifies the reported parts against S3 (ListParts) — a buggy or malicious client claiming chunks it never sent would otherwise publish a corrupt video. The client's report is an optimisation for progress UX; S3 is the truth.
The CDN is the biggest line item — serve the long tail from origin
At roughly $150k/day of egress, the CDN is where this system's money goes, and view distribution is brutally skewed: a small percent of videos take nearly all traffic while the long tail gets single-digit views. Caching everything at edge pays premium rates to store segments nobody re-requests. The optimisation: route popular content through the CDN and let long-tail requests go to origin (S3 or cheap origin shields) directly — a popularity threshold on the URL the metadata service hands out is enough.
For genuinely hot content the same skew works in my favour: a new release's segments are requested thousands of times per edge node, so hit rates approach 100% and origin barely notices launch traffic. The remaining hot spot is metadata — a viral video's Cassandra partition — absorbed by replication plus a small LRU cache in front, and it's honest to note view-count-style features are exactly why that cache earns its keep.
Follow-ups you should expect
Would you pre-transcode every rendition for every video?
How would you do view counts?
How do you stop people ripping paid or private content?
What changes for live streaming?
A transcode worker dies halfway through a 4K film — what happens?
Where candidates lose the room
- Designing playback as 'download the file, then play' — progressive download fails the adaptive-bitrate requirement and usually signals the candidate hasn't met HLS/DASH.
- Skipping segmentation, then having no answer for parallel transcode, mid-stream quality switching, or CDN cache granularity — three follow-ups lost to one omission.
- Treating transcoding as a magic box ('a service transcodes it') with no DAG, no orchestration, no retry story — the pipeline is the technical core of this question.
- Routing video bytes through application servers on upload or playback instead of presigned S3 and the CDN.
- Claiming resumable upload without chunk-state tracking, or trusting client-reported chunks without verifying ETags against S3.
- No manifest concept — without it there's no coherent story for how the player discovers renditions and switches between them.