Distributed Systems Series — Part 4.9: Fault Tolerance & High Availability The Gap Between Designed Resilience and Actual Resilience Parts 4.1 through 4.8 have established the complete fault tolerance and high availability engineering stack. Post 4.1 defined the failure taxonomy. Posts 4.2 and 4.3 established the fault tolerance and redundancy foundations. Post 4.4 covered failure … Read more
Distributed Systems Series — Part 4.7: Fault Tolerance & High Availability Failures Are Inevitable — Outages Are Not Every large distributed system experiences component failures continuously. Nodes crash, networks degrade, downstream services slow, disks fill, processes run out of memory. The engineering discipline is not preventing these failures — that is impossible at scale — … Read more
Distributed Systems Series — Part 4.6: Fault Tolerance & High Availability High Availability Is a System-Level Property The availability nines defined in Post 4.2 — 99.9%, 99.99%, 99.999% — are measurements of an outcome. This post is about the architecture that produces that outcome. High availability cannot be achieved by adding redundancy to one layer … Read more
Distributed Systems Series — Part 4.5: Fault Tolerance & High Availability Detection Is Not Recovery Post 4.4 established how distributed systems detect failures — through heartbeats, timeouts, phi accrual detectors, and gossip protocols. Detection is the prerequisite. But detecting that a node has failed solves nothing by itself. The system must then do something about … Read more
Distributed Systems Series — Part 4.3: Fault Tolerance & High Availability Redundancy Is the Foundation, Not the Solution Post 4.1 established the taxonomy of failures — crash-stop, crash-recovery, omission, timing, gray, Byzantine, and correlated. Post 4.2 established the distinction between fault tolerance (correctness under failure) and high availability (uptime). This post addresses the structural mechanism … Read more
Distributed Systems Series — Part 4.2: Fault Tolerance & High Availability Two Goals That Sound the Same and Are Not Fault tolerance and high availability are the two most frequently conflated concepts in distributed systems engineering. Engineers use them interchangeably in architecture discussions, design documents, and system reviews. This conflation is not just imprecise — … Read more