Scope the requirements
This is a storage system dressed up as a monitoring product. The interview is about the write path (millions of points/s), the read path (dashboards over hours-to-months), and the killer axis: cardinality — how many distinct time series exist.
Functional — what it must do
- Agents on every host push metrics (counters, gauges, histograms) every 10-60 s.
- PromQL-like queries over time ranges: rate, aggregate by label, over hosts/services.
- Alerts: run a query at intervals, page on threshold breach.
- Retention tiers: raw for 24h, 1-min for a month, 1-hour for a year.
- Explicitly cut: distributed tracing, logs — different products, different systems.
Non-functional — the numbers that shape the design
- Ingest: 100M data points/s across the fleet.
- Ingest lag under 10 s from write to queryable.
- Query p95 under 1 s for dashboards over the last 24 hours.
- Cardinality: support 100M distinct time series without collapsing.
- 99.9% availability on ingest; queries can degrade to older data during incidents.
The number that decides the whole design is cardinality, not points-per-second. A stupid label choice ("user_id as a label") explodes the series count into the billions and kills the storage engine — call that out early.
Back-of-envelope estimates
Compression is the reason this works at all — raw storage would be untenable. Gorilla compresses to ~1.5 bytes per point in practice.
| Quantity | Estimate | How you got there |
|---|---|---|
| Points/s | ~100M | 1M hosts × 100 series/host × 1 point / 10 s = 10M/s per fleet, ×10 for a large product. |
| Distinct series | ~100M | host × metric × label combinations. The knob that most needs guardrails. |
| Raw storage/day | ~13 TB | 100M pts/s × 86400 s × ~1.5 B (compressed). Uncompressed would be ~70 TB. |
| Monthly rollup storage | ~200 GB | 1-minute buckets × 100M series × 30 days × 8 B. |
| Query fanout | up to ~1M series | a broad dashboard aggregation ('rate(http_requests) by service') can span this many series. |
| Alerts | ~10k rules × 1 min | each rule = one query per minute; ~170 QPS steady-state, spiky during incidents. |
1.5 bytes per point is not a typo — Gorilla and its descendants make time-series data 10× smaller than a naive representation. Without that, we don't fit.
Sketch the API
Push and query. Push is a firehose; query is the interactive path that must stay fast even at the tail.
- Ingest is agent-push, not central pull at this scale — a scraper trying to poll 1M hosts is a nightmare and scales worse. Prometheus-style pull is fine at small scale; at 100M points/s the push model wins.
- Queries never scan raw storage for windows longer than a day — the query planner picks the coarsest rollup that satisfies the requested step.
Data model
Two data structures do all the heavy lifting: an inverted index of labels → series-ids, and a per-series compressed block of (timestamp, value) points.
This is a purpose-built time-series database — Prometheus TSDB, InfluxDB, VictoriaMetrics — not a general-purpose store. The reason: general databases don't support Gorilla-style compression, don't have label-inverted indexes tuned for time-range scans, and their write-amplification kills the ingest rate. Say "I would use or build a time-series database" — the interviewer wants you to know this is a distinct category.
High-level design
Ingest tier fans out across shards keyed on series_id. Each shard owns some subset of the total series. Query is scatter-gather across shards, with a query planner that picks the right rollup tier.
- Agent batches ~5 s worth of points and POSTs to ingest LB. Ingester hashes each series to a shard (consistent hashing on series_id).
- Ingester appends to a shard's in-memory buffer + WAL. Every ~2h, the buffer is compacted into an immutable block: sorted, Gorilla-compressed, indexed by labels.
- Blocks older than 24h are uploaded to S3 as rollup Parquet; local disk keeps only recent hot data. Query planner knows which tier holds which range.
- Query arrives → planner resolves label filter to a series_id set → determines the coarsest rollup satisfying step → scatters to shards holding those series → shards return partial results → planner merges + applies aggregators.
- Alerts service polls the query API on each rule's schedule; a positive result generates a notification event → pager routes to the right on-call.
Deep dives — where the interview is won
Gorilla compression: 16 bytes per point becomes 1.5
Naive storage per point is 8 bytes timestamp + 8 bytes value = 16 B. Gorilla's insight: consecutive points are highly correlated. Timestamp deltas between samples are almost always constant, so encode delta-of-delta — a stream of small integers, often zeros, that Elias-Gamma or similar bit-packs to ~1-2 bits each. Values are floats that change slowly; XOR consecutive values and the result has long runs of zero leading bits — encode the meaningful nibble and its position, and you get ~10 bits per value on typical monitoring data.
The whole scheme is designed for the read pattern: sequential scan of a time-series segment. Random access to point N inside a block requires decoding from the block's start, but that's ~10 µs for a 2h block, and dashboards never ask for a single point. Compression ratios are ~10-20× versus naive, and the CPU cost of decode is minimal compared to disk or network.
The cardinality bomb and how to defuse it
The single biggest failure mode: someone puts a high-cardinality field into labels. user_id, request_id, trace_id, email — any of these turns one metric into millions of distinct series, each with its own compressed block, its own index entry, its own memory. The system doesn't crash gracefully — it slowly gets slower until queries time out and the on-call is paged.
Defences run at three layers. Agent-side: SDKs cap the number of distinct label values a client can emit (http_requests with endpoint allowed, endpoint=/user/123 normalised to /user/:id). Ingest-side: reject series that would push a metric past a cardinality budget, with alerts to the team owning that metric. Query-side: enforce a hard limit on series count per query so a runaway PromQL doesn't take down the shard fleet. All three are necessary — one alone leaks eventually.
Query planner: pick the tier before you scan
A dashboard asks for the last 30 days of rate(http_requests[5m]) by service. If you scanned raw 10-second data, that's a billion points read and reduced. The right move: precompute rollups (1-minute and 1-hour) at write time. The planner reads the query step and range and picks the coarsest tier that satisfies. For a 30-day range at 1-hour resolution, the 1-hour rollup answers in milliseconds.
The planner also decides scatter-gather scope. If the query is by service and there are 100 services, it hits every shard that holds any matching series. If the query names specific labels (service=api), the series index resolves to ~1 shard's worth and only that shard is queried. The interview answer that lands: 'the planner is the whole query-latency story; without it, every query is a scan.'
p99 latency) need pre-aggregated histograms because you can't compute a percentile from a mean. Choose your aggregators at write time or accept a subset of queryable functions on long ranges.Follow-ups you should expect
Prometheus pull vs push?
How do you avoid overwriting history when clocks drift?
How do you handle alert storms?
What breaks when a shard dies?
Downsampling on read vs write?
Where candidates lose the room
- Storing metrics in Postgres or Elasticsearch — they cost 10-50× the disk and can't sustain the write rate.
- Treating cardinality as "just a number" rather than the number that decides survival — leaks turn a fast system into a slow one over weeks.
- Not building rollups — a month-range dashboard on raw data is a full block scan of 10 TB of compressed points.
- Pull-based scrapes at fleet scale — the moment a datacenter reschedules 10k containers, the scrapers can't keep up.
- Alerting from the raw ingest tier instead of the same query planner — alerts silently drift out of sync with dashboards showing the same query.