Scope the requirements
Everyone reaches for Redis SETNX first. That's the trap — it works until it doesn't, and "doesn't" is a data corruption incident. The interview is asking you to know precisely when Redis is enough and when you need Zookeeper.
Functional — what it must do
- acquire(key, holderId, ttl) → boolean, non-blocking.
- release(key, holderId): only the holder can release.
- extend(key, holderId, newTtl): a live holder can prolong its lease.
- Fencing token: a monotonically increasing number handed out on acquire.
- Explicitly cut: waiting queues, priority acquisition — leave callers to retry.
Non-functional — the numbers that shape the design
- Correctness under network partitions: at most one holder per key at any time.
- acquire latency under 10 ms p95.
- 10k acquire/release ops per second on the cluster.
- Auto-release on holder death — lease expiry within 2× the TTL worst-case.
- 99.9% availability on acquire; caller retries handle brief outages.
The whole question is one word: correctness. "Fast" is easy — Redis single-node does that. The hard part is what "one holder at a time" means when clocks drift and processes pause.
Back-of-envelope estimates
The load is small compared to a normal database — a lock service handles thousands of ops per second, not millions. The design constraint is correctness, not throughput.
| Quantity | Estimate | How you got there |
|---|---|---|
| Ops/s | ~10k | acquire + release, dominated by short-lived locks (job leases, leader election). |
| Live locks | ~1M | in-memory hash map; 100 bytes each = 100 MB, trivially fits on any node. |
| TTL median | ~30 s | long enough to survive a GC pause, short enough to auto-release on crash. |
| Nodes in quorum | 3 or 5 | an odd number for majority; 5 tolerates 2 failures at the cost of write latency. |
| Consensus latency | ~5-10 ms | Raft round-trip within a region; the reason a global lock service is a bad idea. |
| Fencing tokens | monotonic 64-bit | handed out per lock; never reused, even after lock is released. |
I want three or five nodes running Raft, not ten. Consensus latency scales with the majority — bigger clusters are slower, not more reliable.
Sketch the API
A tiny surface. The interesting field is the fencing token — most naive designs forget it and pay for it in a corruption incident.
- Every acquire returns a fencing token that the client must include with every I/O to the resource being protected. Storage systems reject writes with tokens lower than the last-seen token. This is the correctness backbone.
- release is a no-op on mismatch — a client whose lease expired and got taken by someone else should not be able to release the successor's lock.
Data model
One in-memory table replicated by Raft. Persistence is the Raft log itself, not a separate database.
The whole state fits in RAM on every node. Postgres is the wrong tool because we're doing this to avoid Postgres — the point is to make coordination decisions fast and outside the transactional store. Raft's log is our durable store, and it's designed for exactly this.
High-level design
A cluster of odd-numbered nodes running Raft (etcd, Zookeeper or a custom implementation). All writes go through the leader; reads can be linearised via readIndex or served from local state with a small lag.
- Client A sends acquire(K, ttl=30s) to any node; router forwards to the current leader.
- Leader assigns a monotonic fencing token, appends an acquire command to its Raft log, replicates to followers, waits for a majority to ack.
- On quorum ack, the leader applies to state machine:
locks[K] = (A, now+30s, token=42). Returns success and token=42 to A. - A now writes to the protected resource including token=42. The resource stores the last-seen token per key; a write with a token lower than the stored one is rejected.
- If A crashes without releasing, the lease expires at
expires_at. Next acquirer B gets a higher token (43), and any late writes from A still carrying token 42 are refused by the resource.
Deep dives — where the interview is won
Fencing tokens: why Redis SET NX PX alone isn't enough
The failure mode looks like this: client A acquires the lock, then its process pauses for 40 seconds (GC, stop-the-world, virtualisation freeze). The 30-second lease expires. Client B acquires the lock and starts writing to the protected resource. Then A wakes up, still thinks it holds the lock, and writes too. Two writers. Corruption. No amount of clever Redis scripting fixes this — the fundamental issue is that A cannot know it has been paused past its lease.
The fencing token flips the problem: the resource itself validates every write against a monotonically-increasing token. A carries token 42, B carries token 43. When A finally writes, the resource sees 42 < 43 and rejects. The lock service doesn't need to prevent A from thinking it's still the holder — it just needs to make sure A's writes are stale and refused. This is Martin Kleppmann's canonical argument for why 'Redlock' is not safe by itself.
Consensus vs Redis: pick your correctness bar
Redis single-node with SET NX PX is fast and correct as long as the Redis node doesn't fail during a critical window. Failover to a replica can silently promote a node that hasn't seen the latest SET — for a brief window, two clients can both hold the 'same' lock. This is fine for cache coordination, cron deduplication, feature-flag guards. It is not fine for anything that touches money or user data.
Raft-backed services (etcd, Zookeeper) commit acquire only after a majority of nodes have persisted the write. A failover is safe because the new leader has the log. The cost is 5-10 ms per acquire versus <1 ms on Redis, and the operational overhead of running a quorum service. The interview move: name the axis. Redis for cheap coordination; Zookeeper/etcd for anything you'd be woken up for at 3 am.
Lease extension without a thundering herd of renewals
Long-running jobs need to hold locks for minutes or hours, but the TTL should stay short (30-60 s) so a dead holder releases fast. The solution is periodic extension: the holder sends extend(key, holderId, newTtl) well before expiry — say at 1/3 of the lease elapsed. If the extend fails (leader unreachable, holder no longer matches), the job self-cancels rather than continuing under a lock it may have lost.
The design pitfall: a naive extender fires exactly at expiry-minus-epsilon. If 10k jobs share a schedule, they all extend at the same instant and DDoS the lock service. Jitter the extension time (±25%) and stagger acquire times when many jobs start together. This isn't unique to locks — it's the same pattern as cache TTL and cron randomisation.
Follow-ups you should expect
Why not use a Postgres row with SELECT ... FOR UPDATE as a lock?
How do you deal with clock drift?
What breaks under a network partition?
How would you support waiting on a lock?
Redlock — is it safe?
Where candidates lose the room
- Believing Redis SETNX is a lock — it's a mutex-in-happy-path, not a lock in the distributed-systems sense.
- Not handing out a fencing token, or handing one out but not requiring the resource to check it.
- Building a global lock service — a single cluster in one region is fine; cross-region locks pay a WAN round trip per acquire.
- Long TTLs to "cover the job" — you can't shorten the failover window later; short TTL with extension is the pattern.
- Firing all extensions on a fixed schedule — thundering herd against the leader every N seconds.