Designing distributed systems that survive real-world failure at scale — failure modes, resilience patterns, consensus, data consistency, capacity planning, and the operational discipline that keeps all of it running.
Most system design material stops at the point where the architecture diagram is drawn: services, a database, a cache, a queue, arrows between them. This course starts where that material ends. Every one of those arrows is a network call that can hang, a dependency that can degrade without dying, a peer that can disagree about what time it is. The difference between a system that is merely architected and one that is resilient is not the boxes on the diagram — it is everything this course covers: what happens in the first 500 milliseconds after a downstream dependency starts timing out, how a retry policy behaves when ten thousand clients hit it simultaneously, whether a partition reshard can happen without a maintenance window, and whether the on-call engineer holding the pager at 3 a.m. has a runbook or a Slack scrollback.
This is written for engineers who already know the vocabulary — load balancers, sharding, eventual consistency, Kafka partitions, Redis eviction policies — and want to go past the vocabulary into the mechanics: the actual state machine of a circuit breaker, the actual formula behind jittered backoff, the actual arithmetic of an error budget, the actual trade-offs a funds-transfer saga has to make when step three of five fails. The six labs embedded below are not decorative animations; they are small, mechanically correct simulations you can perturb — toggle jitter on and off and watch synchronized retry storms actually form, fail a saga step and watch compensations actually fire in reverse order, skew a partition key and watch one shard actually absorb the load the others don't.
Three words that get used interchangeably in casual conversation and mean three different, precisely measurable things once you're the one on the hook for an SLA.
Availability measures whether a system responds to a request at all, within an acceptable time, right now. It is a point-in-time property, usually expressed as the fraction of requests (or time) served successfully out of the total attempted — uptime / (uptime + downtime). Reliability measures whether a system keeps behaving correctly over a duration, under the conditions it was designed for — it is closer to "does it keep working the way it's supposed to for as long as it's supposed to" than "is it up right now." Durability is narrower still: it measures whether data, once acknowledged as written, survives failures — disk death, node loss, region loss — independent of whether the system serving that data is currently reachable. These are orthogonal. A single-node in-memory cache can be extremely available (sub-millisecond responses, no network hop, no dependency to fail) while being completely non-durable (a process restart erases everything it holds). An object store with synchronous multi-region replication can be extremely durable (eleven nines of data survival is a real, published class of guarantee from major cloud object stores) while occasionally being unavailable during a control-plane incident, without losing a single byte of already-committed data. Conflating these three is not a pedantic nitpick: it produces designs where a team ships aggressive caching to "improve reliability" and discovers, during an incident, that they've actually improved availability while quietly weakening durability, because the cache now masks writes that never made it to the durable store.
The error budget exists to turn availability from a moral aspiration ("we should be reliable") into an engineering constraint with arithmetic behind it. You start from a Service Level Objective — a target such as 99.9% availability, measured over a rolling window, usually 28 or 30 days. Everything below that line, for that window, is budget: time (or request volume) the system is permitted to be unavailable without anyone owing an apology. At 99.9% over a 30-day month, the arithmetic is direct: (1 - 0.999) × 30 × 24 × 60 = 43.2 minutes. That's the entire allowance, for everything — deploys that go wrong, a dependency having a bad day, a bad config push, a datacenter blip. The moment you can state that number, you can use it as a governance lever instead of a vibe: if the budget is nearly exhausted, velocity slows — freeze risky deploys, prioritize the reliability backlog. If the budget is barely touched, the team has earned room to ship faster and take on more risk, because the SLO has slack. This is the mechanism Google's SRE discipline formalized, and it is the single most useful idea in this entire course for turning "reliability" from a subjective argument between engineering and product into a number both sides can look at.
The instinct to chase "the system must never fail" is understandable and, for any distributed system built from more than a handful of independent components, mathematically doomed. If a system depends on N independent components each with uptime p, and all of them must be up for the system to be up, the system's own availability is p^N — and it degrades fast. Ten components each individually at a very respectable 99.9% combine, in a naive series dependency, to roughly 99.0% — an order of magnitude worse than any single piece. This is not a hypothetical: it is the default outcome of wiring services together without deliberately engineering around it, and it is why large systems with dozens or hundreds of services in their critical path cannot simply inherit reliability from their parts being individually reliable. Redundancy (parallel, independent paths where only one of several needs to succeed) fights this multiplication; naive chaining makes it worse.
The correct design goal, given that failure at scale is a statistical certainty rather than an edge case, is not "never fail" but fail safely, fail small, and recover fast. Fail safely means a failure degrades behavior in a bounded, predictable, non-destructive way rather than corrupting state or cascading. Fail small means the blast radius of any one failure is contained to the smallest possible slice of users, requests, or data — a concept you should treat as a first-class design variable, not an afterthought, the same way you'd size a database index. Recover fast means the system is built so that once a failure is detected, restoring healthy behavior takes seconds, not hours — through automated failover, fast-failing dependencies, and operational readiness, rather than through heroics. A system optimized this way will still fail regularly; it will simply fail in ways that are cheap, contained, and short, which is the actual, achievable version of "reliable" at scale.
None of this is abstract for a payments or ledger system specifically. A funds-transfer service that is briefly unavailable but never loses or double-applies a transaction is in a defensible, recoverable state. A funds-transfer service that stays available by skipping durability guarantees under load is not — it has traded a visible, bounded failure (an error response, a retry) for an invisible, unbounded one (an incorrect balance). Every design decision in the sections that follow is, at bottom, a decision about which axis — availability, reliability, or durability — you are willing to spend budget on, and which one you are not.
Availability: can you get a response right now. Reliability: does the system keep behaving correctly over time, under design conditions. Durability: does committed data survive failures, independent of whether the system is currently reachable. A system can score high on one and low on another — they are not substitutes for each other.
budget = (1 − SLO) × window. At a 99.9% SLO over a 30-day month, the allowed downtime is 0.001 × 43200 min = 43.2 min. At 99.99% it drops to 4.32 min. Every incident, bad deploy, and dependency blip is debited against this number until it resets with the next window.
MTBF (mean time between failures) measures how often things break; MTTR (mean time to recovery) measures how fast you fix them once they do. At scale, MTTR is the more controllable, higher-leverage number — you cannot engineer failure to zero, but you can engineer detection and recovery to be very fast.
For N independent components each with uptime p chained in series, naive composed availability is p^N, which falls quickly as N grows. Chasing zero failures ignores this arithmetic and this statistical reality; the achievable target is bounding how failures manifest and how fast they're undone, not preventing them outright.
Blast radius is the maximum scope — users, requests, tenants, regions — that a single failure can affect. It should be sized deliberately through sharding, cell-based isolation, canary rollouts, and bulkheads, the same way you'd deliberately size a cache TTL or an index. An undersized blast radius turns a single bad deploy into a full outage.
Operational reality: teams that treat their SLO as a target to hit rather than a budget to spend tend to either over-invest in reliability work nobody asked for, or discover their real risk tolerance only during an incident. Publish the budget, and let it govern the deploy-freeze decision instead of a Slack argument.
Interactive · Lab 1 of 6
Pick an SLO and an observed error rate and see exactly how fast the monthly budget burns — and how many days remain before it's exhausted. A burn rate above 1.0× means the budget runs out before the window resets.
Burn rate is observed_error_rate ÷ (1 − SLO): a burn rate of 6× means the team is spending the monthly budget six times faster than the SLO allows, and will exhaust it in window ÷ burn_rate days.