Distributed Systems Series Home
System Models, Network Behaviour and Time: The Three Foundations
Most engineers learn distributed systems backwards. They encounter a production incident — a mysterious data inconsistency, a cascading failure that started with one slow node, a retry storm that brought down a service — and work backwards to understand why it happened. The learning is real but expensive. Every incident could have been anticipated by someone who understood the foundations.
Part 1 of this series establishes those foundations before anything else. Not because they are abstract theory, but because they are the operating environment that every distributed system lives in. Every decision in Parts 2 through 5 — How to structure retries, Which consistency model to use, When to use consensus, How to design for fault tolerance — is only comprehensible if you first understand the three realities Part 1 describes.
Those three realities are:
Networks are unreliable. Messages between nodes can be lost, delayed, duplicated or delivered out of order. A sender cannot know whether its message was received, whether the receiver processed it or whether the response was lost on the way back. Silence is ambiguous — it could mean failure, slowness or partition. Every communication pattern in distributed systems exists because of this reality.
Nodes fail independently and partially. In a single-machine system, failure is obvious — the process either runs or it does not. In a distributed system, a node can be running while failing to serve useful work. It can respond to health checks while its database connection pool is exhausted. It can hold a distributed lock past its expiry while paused by garbage collection. These partial, ambiguous failures are harder to handle than clean crashes and produce the majority of production incidents in distributed systems at scale.
There is no global clock. Physical clocks on different machines drift at different rates, can be adjusted backward by NTP synchronisation, and cannot be trusted to establish the order of events across nodes. An event with a later timestamp did not necessarily happen later. This makes ordering — which feels trivial on a single machine — a fundamental coordination problem in distributed systems.
These three realities are not edge cases to be handled carefully. They are the normal operating environment. A Distributed System that is not designed around all three will fail in production in confusing, hard-to-diagnose ways. One that is designed around all three will fail predictably, recover automatically and give engineers the information they need to understand what happened.
Part 1 establishes the vocabulary and mental models that every subsequent part depends on. Engineers who skip it and jump to replication or consensus typically find those topics harder to understand — not because the algorithms are complex, but because the problem those algorithms are solving is not yet clear. Start here.
Part 1 Posts — Full Reading List
1.1 — What Is a Distributed System (Really)?
The correct definition of a distributed system — not “Multiple machines working together” but “Independent components coordinating over unreliable communication while still making progress.” Covers partial failure as the default, Why communication is inherently unreliable, Why time is not what you think it is and Why Consistency and Availability are always trade-offs. Includes the banking transaction example and how Amazon’s architecture was reshaped by exactly these realities.
Key concepts: Partial failure, Network unreliability, Clock uncertainty, CAP theorem introduction, Coordination under uncertainty
Read this if: you are new to Distributed Systems or want to rebuild your mental model from the correct starting point
1.2 — System Models: How Distributed Systems See the World
A system model is a simplified set of assumptions about how a Distributed System might behave — not how it actually behaves, but what it is allowed to assume. These assumptions determine what guarantees are possible. The three core models are the network model (How messages behave), the node model (How processes fail), and the time model (How ordering works). Covers safety vs liveness, Why most production outages trace back to violated model assumptions and Why Google Chubby’s correctness guarantees collapsed when its assumptions were violated.
Key concepts: Network model, Node model, Time model, Safety, Liveness, Crash-Stop vs Crash-Recovery, Synchronous vs Asynchronous Networks
Read this if: you want to understand Why Distributed System algorithms have preconditions and What happens when those preconditions are violated
1.3 — Network Model: Latency, Loss and Partitions
The network does not behave the way engineers intuitively expect. Covers latency variability (Why tail Latency matters more than Average Latency), Why timeouts introduce ambiguity rather than resolving it, message loss and duplication as normal behaviours and network partitions as inevitable rather than rare. Includes the five reasons a service call might return no response and Why all five look identical to the caller. Grounded in the 2012 AWS US-East-1 outage as a production-scale partition example.
Key concepts: Latency Variability, Tail Latency, Timeouts, Message Loss, Message Duplication, Network Partitions, Synchronous vs Asynchronous vs Partially Synchronous Networks
Read this if: you want to understand Why “the network is reliable” is the most dangerous assumption in Distributed Systems
1.4 — Node & Failure Model: Crashes, Slow Nodes and Partial Failure
Nodes fail in ways that single-machine thinking does not prepare engineers for. Crash-stop (a node fails and stops) is theoretically clean but rare in practice. Crash-recovery (a node fails, restarts and rejoins) is the real-world model and significantly harder to handle. The most dangerous failure mode is the slow node — running, responding to health checks, but processing requests far too slowly to be useful, holding locks past their expiry and creating cascading failures. Includes the GitHub 2012 GC pause incident as a documented production example of exactly this failure mode.
Key concepts: Crash-Stop, Crash-Recovery, Partial Failure, Slow Nodes, Health Checks vs Readiness, Failure Detection, Phi Accrual Detector
Read this if: you want to understand Why Distributed Systems can be simultaneously up and broken and Why slow nodes are worse than crashed ones
1.5 — Time Model: Why Ordering Is Harder Than It Looks
There is no global clock in a Distributed System. Physical clocks drift, NTP adjustments can move time backward and network delays mean that arrival order does not equal execution order. Timestamps cannot be trusted to determine which event happened first across nodes. The solution is logical time — reasoning about causality (Which event could have influenced which other event) rather than physical time. Covers the Happens-Before relationship, Logical time vs Physical time and the real-world consequences of trusting timestamps, including Cassandra’s Last Write Wins data loss incidents caused by clock skew.
Key concepts: Clock Drift, Clock Skew, NTP limitations, Happens-Before, Causality, Physical time vs Logical time, Why timestamps lie
Read this if: you want to understand Why time-based ordering is unreliable across nodes and what Distributed Systems use instead
What Part 1 Prepares You For
After Part 1, three things become clear that were not clear before.
First, every retry, timeout and idempotency mechanism in Part 2 exists specifically because of the network model established in Post 1.3. Retries exist because messages can be lost. Idempotency exists because retries can produce duplicates. Timeouts exist because silence is ambiguous. The mechanisms make no sense without the model that motivated them.
Second, every consistency model in Part 3 exists because of the combination of the network model and the time model. Eventual consistency is not a weak compromise — it is the natural consequence of allowing replicas to accept writes independently over an unreliable network with no global clock. Strong consistency is not the default — it is an expensive guarantee that requires coordination to overcome exactly the realities Post 1.3 and Post 1.5 describe.
Third, every fault tolerance pattern in Part 4 exists because of the node model established in Post 1.4. Redundancy exists because nodes fail independently. Failure detection exists because crash-stop is not the real failure model. Bulkheads exist because partial failure is the norm and isolation prevents it from spreading.
The foundations are not optional reading before the interesting parts. They are the explanation for why the interesting parts exist at all.
Continue the Series
Series home: Distributed Systems — Concepts, Design & Real-World Engineering
Next: Part 2 — Communication & Coordination
Part 2 takes the three constraints established in Part 1 and examines how distributed systems cope with them — through communication patterns, retry strategies, service discovery, distributed locks, logical clocks and coordination services.