What it is
Stream processing is what you reach for when the natural shape of the input is 'events that keep arriving' and the natural shape of the output is 'a value that keeps changing'. Fanning out to a table isn't a hack — it's the whole point of Flink, Kafka Streams, Spark Structured Streaming. The primitives are windows, joins and state.
The interview shows up whenever the design has to serve fresh aggregate numbers (dashboards, anomaly detection, real-time fraud). Interviewers score on knowing the exactly-once guarantees, the watermark model, and where stream processing fits vs a nightly batch.
I'll use Flink for the stream layer with tumbling windows and watermarks — that gives me exactly-once with about 5 minutes of latency, and I'll fall back to a nightly batch for anything long-tail or ML-heavy.
The variants and when to pick each
Windows: tumbling, sliding, session
Tumbling windows are contiguous and non-overlapping — 'clicks in each 1-minute bucket'. Sliding windows overlap — 'clicks in the last 5 minutes, updated every 1 minute'. Session windows are gap-defined — 'group activity until 30 minutes of silence.' The three cover almost every business-metric shape you'll be asked about.
The interview point: pick the smallest window that answers the question. Sliding windows are expensive because a point belongs to many windows simultaneously; use them only when you need continuous freshness.
Watermarks — deciding when a window is 'done'
Events arrive out of order (network jitter, mobile buffering). The stream processor tracks the maximum event timestamp it has seen; the watermark is that max minus a slack. A window closes when the watermark passes its end. Late-arriving events either update the closed window (allowed lateness) or fork to a side output (dropped from the primary).
Watermark slack is the freshness/completeness knob. 5 minutes is a common default; live systems can go tighter with a small correctness cost, batch-anchored systems can go looser.
Exactly-once via checkpoints
Flink's mechanism (Chandy-Lamport algorithm): the job periodically injects checkpoint barriers into every stream. When operators see a barrier, they snapshot their state to durable storage and pass the barrier along. On failure, restart from the last checkpoint — the input source rewinds, state restores, and every event gets processed exactly once.
The catch is 'exactly-once for state-modifying operations' — a side effect like an HTTP call cannot be undone. Exactly-once outputs require either idempotent sinks or transactional two-phase commit (Kafka transactions do this).
Stream vs batch: the eternal debate
For any given number, you can compute it in a stream (fresh, complex code) or in a batch (delayed, simple SQL). The right call depends on latency needs and reconciliation. Streaming shines at seconds-to-minutes freshness with tolerable approximation; batch wins for anything requiring joins with slowly changing dimensions, historical corrections, or exact arithmetic (billing, financial close).
Modern practice runs both when correctness matters: the stream gives dashboards their number, the batch computes the audited daily number, and drift between them is monitored.
Numbers worth memorising
| Flink throughput per task slot | ~100k-500k events/s (record dependent) |
| Checkpoint interval | 10 s to 1 min, tuned for recovery target |
| Watermark slack | 1-5 min typical for user events; ms for market data |
| State backend size per operator | GB to TB (RocksDB local, S3 for checkpoints) |
| End-to-end latency for a tumbling window | window size + watermark slack + checkpoint interval |
Where the interviewer pushes
First push: 'why not just batch every minute — same effect, simpler.' It's a real question and often the right call for smaller scales. Streams win when you need incremental joins, session-based windows, or state that carries across batch boundaries. If a nightly job answers your question, don't build a Flink job to do it every minute.
Second push: 'what if a stream job goes down for an hour?' Kafka retention buffers the input; when the job restarts from its last checkpoint, it consumes the backlog. Windows already committed to the sink stay committed; in-flight ones resume from checkpointed state. The user visible effect is a delay, not corruption — that's the whole promise of exactly-once state processing.