Distributed Systems Series — Part 4.6: Fault Tolerance & High Availability High Availability Is a System-Level Property The availability nines defined in Post 4.2 — 99.9%, 99.99%, 99.999% — are measurements of an outcome. This post is about the architecture that produces that outcome. High availability cannot be achieved by adding redundancy to one layer … Read more
Distributed Systems Series — Part 4.5: Fault Tolerance & High Availability Detection Is Not Recovery Post 4.4 established how distributed systems detect failures — through heartbeats, timeouts, phi accrual detectors, and gossip protocols. Detection is the prerequisite. But detecting that a node has failed solves nothing by itself. The system must then do something about … Read more
Distributed Systems Series — Part 4.4: Fault Tolerance & High Availability The Question That Has No Perfect Answer When a distributed node stops responding, other nodes face a question that cannot be answered with certainty: has this node failed, or is it merely slow? In a single-machine system, this question does not exist. The operating … Read more
Distributed Systems Series — Part 4.3: Fault Tolerance & High Availability Redundancy Is the Foundation, Not the Solution Post 4.1 established the taxonomy of failures — crash-stop, crash-recovery, omission, timing, gray, Byzantine, and correlated. Post 4.2 established the distinction between fault tolerance (correctness under failure) and high availability (uptime). This post addresses the structural mechanism … Read more
Distributed Systems Series — Part 4.2: Fault Tolerance & High Availability Two Goals That Sound the Same and Are Not Fault tolerance and high availability are the two most frequently conflated concepts in distributed systems engineering. Engineers use them interchangeably in architecture discussions, design documents, and system reviews. This conflation is not just imprecise — … Read more
Distributed Systems Series — Part 4.1: Fault Tolerance & High Availability Why Failure Vocabulary Matters Before Failure Mechanisms Part 4 covers how distributed systems survive failures. But before designing survival mechanisms — redundancy, failure detection, circuit breakers, chaos engineering — engineers must be precise about what kinds of failures they are designing for. A retry … Read more