Blueprint
Fundamentals

Replication & failover

Copies of your data on multiple nodes buy you read scale and survival — and a new physics problem: the copies are never quite the same at the same moment.

01

What it is

Replication keeps the same data on multiple nodes for three distinct reasons: read scaling (spread queries across replicas), availability (a replica takes over when the leader dies), and durability (the data survives a disk). It's orthogonal to sharding — each shard is typically itself a replicated group — and it's the standard answer to both "the database is a SPOF" and "reads are 100× writes".

The interview scoring is on the failure windows, not the happy path. Replication lag and what it does to a user who just wrote; what exactly happens in the seconds after a leader dies; whether async replication can silently lose acknowledged writes. Candidates who can walk the failover timeline beat candidates who can list the topologies.

Say this out loud

One leader taking writes, two replicas across availability zones serving reads with async replication — and because there's replica lag, anything the user needs to read back immediately after writing goes to the leader.

02

The picture

Writer Readers Leaderall writes land here Follower 1lag ~50 ms Follower 2lag ~80 ms WAL stream
writes go to one leader; the dashed lines are the replication stream, always slightly behind. reads scale by fanning out to followers — and inherit that lag.
03

The variants and when to pick each

Leader-follower — the default, and its lag anomaly

All writes go to one leader; followers replay its log and serve reads. Simple, no write conflicts, and read capacity scales by adding replicas — which is why it's the default in nine designs out of ten.

The weakness everyone gets pushed on: replication lag. A user posts a comment, the write commits on the leader, they refresh, the read hits a lagging replica — their comment is gone. Fixes: read-your-own-writes routing (the writer reads from the leader for a short window), session pinning, or waiting for the replica to reach the write's log position. And the leader is still the write ceiling — replication scales reads only.

Trade-offSimplicity and conflict-free writes, paid for with lag anomalies on the read path and a single-node ceiling on writes.

Sync vs async — the durability dial

Synchronous replication means the leader waits for replicas before acknowledging: no committed write can be lost in failover, but write latency becomes the slowest replica and a dead replica can block writes entirely. Asynchronous acknowledges immediately: fast, always-available writes, but the seconds of un-replicated commits vanish if the leader dies.

The grown-up middle is semi-sync: at least one replica must ack. Same-region that costs a millisecond or two per write and caps loss at near-zero. Cross-region sync would cost 30–150 ms per write, which is why cross-region replication is async almost everywhere.

Trade-offSync trades write latency and availability for zero loss; async trades a window of recent writes for speed — for a payments ledger that window is unacceptable, for a feed it's invisible.

Multi-leader and leaderless — when one leader isn't enough

Multi-leader accepts writes in several regions: local write latency everywhere and survival of a region loss, but now two leaders can accept conflicting writes and something must merge them — last-write-wins (which silently discards data), CRDTs, or application-level merge. It earns its complexity for multi-datacenter, collaborative editing, and offline-first apps.

Leaderless (Dynamo-style: Cassandra, DynamoDB's lineage) lets any replica take writes, with read/write quorums for consistency and read repair plus anti-entropy to converge. There's no failover pause because there's no leader to fail — the trade is consistency machinery on every single read and write.

Trade-offBoth remove the single write bottleneck by making conflicts your problem — take conflict resolution only when geography or availability genuinely demands it.

Failover mechanics — walk the timeline

The leader dies. Detection: heartbeats time out, typically 10–30 seconds — too aggressive and a GC pause triggers a false failover. Election: a replica is promoted, via Raft/Paxos or an orchestrator, and it should be the least-lagged replica or recent async writes are lost twice over. Repointing: clients and proxies discover the new leader. Total: commonly 10–60 seconds of unavailable or degraded writes.

The disaster to name unprompted is split brain: the old leader was only partitioned, not dead, and now two nodes accept writes. Prevention is quorum-based election plus fencing — the promoted leader invalidates the old one's authority (fencing tokens, STONITH) before taking traffic.

Trade-offFaster failure detection means faster recovery but more false failovers — each of which is a self-inflicted outage with a risk of lost async writes.
04

Numbers worth memorising

Async replication lagsub-second healthy; seconds–minutes under load
Automated failover time~10–60 s, detection to repointed clients
Standard topology1 leader + 2 replicas across 3 AZs
Semi-sync write overhead~1–2 ms same-region (one extra RTT)
Cross-region sync penalty30–150 ms per write — why cross-region is async
05

Where the interviewer pushes

The most common push is the lag anomaly: "a user posts a comment, refreshes, and it's gone — why, and what do you do?" The answer names the mechanism — the read hit a replica that hasn't replayed the write yet — then picks a fix with its cost: route that user's reads to the leader for a few seconds after a write (cheap, mildly stateful), pin the session to one replica (breaks on replica failure), or wait-for-log-position (precise, adds read latency). Saying "the user sees their own write; everyone else can lag" shows I know exactly which consistency I'm buying.

The second push is "sync or async replication for a payments ledger?" This is a values question wearing a config flag. Async means an acknowledged payment can vanish in failover — indefensible. So: semi-sync minimum within the region, so no commit is acknowledged until it exists on two nodes, and async cross-region for disaster recovery with the RPO stated out loud. If the interviewer pushes to fully-sync cross-region, that's a 30–150 ms write tax — I'd argue the semi-sync-plus-async-DR posture is what real payment systems run.

Shows up in

Found a bug or want more?

Tell me what's broken, missing, or wrong. Every message lands in my inbox.