What it is
Resilience patterns are the standard defences that stop one failure from cascading through a distributed system: timeouts so a slow dependency can't exhaust your thread pools, retries with backoff so transient failures heal without storms, circuit breakers so a dying service gets room to recover instead of a hammering, and backpressure so overload gets rejected early instead of buffered into an outage. Alongside them sits observability — metrics, logs and traces — because you can't defend against what you can't see.
This is where senior and staff candidates separate from mid-level ones. The happy-path design is table stakes; the deep dive always ends at "what happens when X is slow or down?", and the scoring is on whether you have a systematic answer — timeout, retry policy, breaker, fallback, in that order — rather than an improvised one. Payment flows, notification systems and any microservices fan-out are the classic stages for it.
Every remote call gets a timeout set from the callee's p99, retries are three attempts with exponential backoff and jitter on idempotent operations only, and each dependency sits behind a circuit breaker with a defined fallback — let me walk the cascade that setup prevents.
The picture
The variants and when to pick each
Timeouts — the pattern everything else depends on
The default of every network library is 'wait forever', and that default turns one slow dependency into thread-pool exhaustion two layers up: threads pile up waiting, the caller's callers queue behind them, and a latency problem in service C becomes an outage in service A. Every remote call gets an explicit timeout, set from the callee's observed p99 — roughly p99 × 1.5 is a defensible starting point.
The senior addition is deadline propagation: the entry point sets a total budget (say 2 s for the user request) and passes the remaining budget down the call chain, so a retry three layers deep can't spend time the user's SLA no longer has. gRPC ships this natively; over REST you pass a deadline header.
Retries — helpful in small doses, catastrophic in stacks
Retry only transient failures, only on idempotent operations, with exponential backoff and jitter: base 100 ms, doubling, capped around 10 s, with full jitter so a thousand clients don't retry in synchronised waves. Cap attempts at 3. Without jitter, retries synchronise into exactly the load spike that keeps the recovering service down.
The killer follow-up is retry amplification: if three layers each retry three times, the bottom service sees up to 27× load precisely when it's least able to serve it — retries turn a partial outage into a total one. The fix is policy: retry at one layer only (usually the edge), and use retry budgets — retries capped at ~10% of request volume — rather than per-request counts.
Circuit breakers — fail fast and give the patient room
A per-dependency state machine. Closed: normal operation, counting failures over a window. Open: failure rate crossed the threshold — say over 50% across at least 20 requests — so calls fail immediately without touching the dependency; no threads blocked, no load added to a dying service. Half-open: after a cool-down (~30 s) a few probe requests go through; success closes the circuit, failure re-opens it.
A breaker without a fallback is a fancier error page, so pair each one with a degradation story: serve the cached copy, return a sensible default, hide the widget. Recommendations down shouldn't take the product page with it. Name-drops that land: resilience4j, Envoy outlier detection, Hystrix as the historical reference.
Backpressure — an unbounded queue is an outage on layaway
When a consumer can't keep up, the system must push back rather than buffer forever. Unbounded buffering converts an overload into an outage with extra steps: the queue absorbs the excess, latency climbs unboundedly, memory grows, and when it finally falls over you've lost everything in flight. Bounded queues plus explicit load shedding — reject early with 429/503 and a Retry-After — keep the system serving some users well instead of all users terribly.
The supporting cast to mention once: bulkheads (separate thread and connection pools per dependency, so one slow neighbour can't drain the shared pool), health checks driving LB ejection, and dead-letter queues so poison messages don't block the stream. On observability: RED metrics per service (rate, errors, duration), tracing with propagated correlation IDs, and alert on p99 — at fan-out, the slowest of 100 shards sets the user's latency, so averages hide exactly the failures that matter.
Numbers worth memorising
| Availability table | 99.9% = 8.8 h down/yr · 99.99% = 53 min · 99.999% = 5 min |
| Serial dependencies multiply | two 99.9% services in series ≈ 99.8% |
| Retry defaults | 3 attempts, 100→200→400 ms backoff + full jitter |
| Circuit breaker defaults | open at >50% failures over ≥20 reqs; half-open after ~30 s |
| Retry amplification | 3 retries × 3 layers = up to 27× load at the bottom |
Where the interviewer pushes
The push: "retries made the outage worse — explain." Two mechanisms, name both. Amplification: retries stacked at multiple layers multiplied load on the struggling service — 3 × 3 × 3 is 27× at the worst possible moment. Synchronisation: no jitter, so every client that failed together retried together, and the recovering service got hit by a wall of traffic each backoff interval and never got the quiet it needed. The fix is structural, not tuning: retry at one layer, full jitter, retry budgets, and a circuit breaker that stops retrying entirely once failure is systemic.
Second push: "the circuit is open — what does the user actually see?" The weak answer is an error message. The strong answer is a degradation ladder: serve stale from cache if there's a copy, a default if there's a sensible one (empty recommendations row, hidden widget), and only then an honest failure with Retry-After — while the breaker's probes decide when to resume. Then close with the availability arithmetic: each 99.9% dependency in series costs you, five of them ≈ 99.5%, so past a point the answer isn't more retries, it's fewer synchronous dependencies on the critical path.