Distributed Systems Series — Part 1.2: Foundations
“Distributed systems are hard not because computers are slow or engineers lack skill, but because they fail in ways that seem normal — until they suddenly don’t.”
Why System Models Exist
After understanding What a Distributed System really is, the next question is this, how do we reason about something that is unpredictable, unreliable and constantly changing? The answer is System Models.
A System Model is not a description of how the world actually behaves. It is a simplified set of assumptions about how a distributed system might behave — assumptions that allow engineers to design, analyse and reason about correctness. Every distributed system is built on a model, whether explicitly stated or silently assumed. The danger is not choosing the wrong model. The danger is not knowing which model you are operating under.
Real distributed systems are messy. Networks delay messages unpredictably. Nodes crash, restart or slow down. Clocks drift and disagree. Failures overlap in unexpected ways. If engineers tried to design systems that assumed all possible behaviours at all times, nothing would ship. System models reduce infinite real-world behaviours into manageable assumptions, allowing reasoning about two fundamental properties:
Safety — nothing bad happens. No double-spend. No data corruption. No two nodes simultaneously believe they are the leader.
Liveness — something good eventually happens. Requests eventually complete. Elections eventually resolve. The system makes progress.
In short, models give us a way to think clearly in an uncertain world.
Models Are Assumptions, Not Guarantees
This is the most important and most commonly missed point about system models.
A system model does not describe what will happen. It describes what the system is allowed to assume about failures, timing, and communication. If reality violates the model, every guarantee built on top of it collapses. Most production outages are not caused by bad algorithms — they are caused by hidden model assumptions being broken in production.
The three most common violated assumptions in production:
Assuming bounded network latency when the network occasionally stalls for seconds. Assuming nodes fail fast (crash-stop) when they fail slowly — remaining alive, responding to health checks, but processing requests incorrectly. Assuming clocks are synchronised when they drift by seconds rather than milliseconds.
Understanding models is not academic. It is the difference between an engineer who can predict failure modes before they occur and one who discovers them through incidents.
When a Model Assumption Broke Google’s Chubby
Google’s Chubby lock service — the distributed coordination system that underpins Bigtable, GFS, and many other Google infrastructure components — was designed under a partially synchronous network assumption. The algorithm assumed that while timing was unreliable most of the time, it would eventually stabilise enough for the system to make progress.
In practice, Chubby clients occasionally held locks far longer than expected during periods of heavy garbage collection pauses or network instability. The node model assumed crash-recovery behaviour, but some nodes behaved like crash-stop from the outside while still holding resources internally. The gap between the assumed model and real behaviour produced exactly the class of bugs described above: systems that appeared healthy while quietly diverging from correct state.
Mike Burrows, who designed Chubby, noted in the original paper that the system’s correctness guarantees rested entirely on the accuracy of its failure assumptions. When those assumptions were violated — even briefly — the system required careful human intervention to recover safely.
This is the production lesson: the algorithm is only as strong as the assumptions underneath it. Know your model. Know where it breaks.
The Three Core System Models
System models are described along three dimensions. Together they define the rules of the game the distributed system is playing. Each dimension has its own dedicated post in this series — here the goal is the complete picture.
Network model: how messages move or do not
The network model defines assumptions about communication between nodes — specifically whether messages can be delayed, lost, or duplicated, and whether the network can partition. Three common network models:
Synchronous — messages arrive within a known, fixed time bound. No real production network behaves this way, but some algorithms are correct only under this assumption.
Asynchronous — no timing guarantees whatsoever. Messages may be arbitrarily delayed or lost. The FLP impossibility theorem applies in this model — consensus cannot be guaranteed to terminate. More on this in Post 3.6.
Partially synchronous — timing is unreliable but eventually stabilises. This is the model that describes real production networks most accurately and the model under which Raft and Paxos operate. They guarantee safety always and liveness only during periods of sufficient stability.
The full treatment of the network model — latency variability, packet loss, the ambiguity of timeouts, and the 2012 AWS US-East-1 outage as a production partition example — is in Post 1.3.
Node model: how components fail
The node model describes how processes behave when things go wrong. Three common failure assumptions:
Crash-stop — a node fails and never recovers. The cleanest model for algorithm design but rare in production. A Kubernetes pod killed by the OOM killer approximates crash-stop from the cluster’s perspective.
Crash-recovery — a node can fail and later rejoin with potentially stale or incomplete state. This is the model that production systems actually exhibit. Engineers often design for crash-stop while implicitly assuming crash-recovery — this mismatch creates subtle bugs during restarts, retries, and message replay.
Byzantine — a node behaves arbitrarily or maliciously. Requires BFT algorithms (PBFT, Tendermint) that tolerate up to f Byzantine nodes in a cluster of 3f+1. Used in blockchain and adversarial environments. Almost never assumed in enterprise distributed systems.
The particularly dangerous case is the slow node — a node that has not crashed, still responds, but responds too late to be useful. From the perspective of other nodes using fixed timeouts, a slow node is indistinguishable from a failed one. This ambiguity is the root cause of false leader elections, unnecessary retries, and cascading failures. The full treatment is in Post 1.4.
Time model: how events are ordered
The time model defines what can be assumed about clocks and event ordering. There is no global clock in a distributed system. Clock synchronisation via NTP is approximate — nodes can disagree by milliseconds to seconds, and NTP can adjust clocks backward. When two events appear to happen at the same time, what that means is: we do not know which happened first, and we may never know.
The result is that systems frequently rely on logical time — causality — instead of physical time. Lamport timestamps, vector clocks, and hybrid logical clocks (used by CockroachDB and MongoDB) establish ordering based on which events could have caused which other events, without trusting wall-clock timestamps. The full treatment — including how Cassandra’s timestamp-based conflict resolution caused silent data loss — is in Post 1.5. Logical clocks in production systems are covered in Post 2.5.
Safety vs Liveness: The Hidden Trade-off
System models also shape what is possible for the two fundamental correctness properties.
Safety means nothing bad ever happens. In a financial system: no double-charge. In a leader election: no two leaders simultaneously. In a database: no write is lost after being acknowledged. Safety violations are the catastrophic failures — data corruption, split-brain, incorrect billing.
Liveness means something good eventually happens. Requests eventually complete. Leaders are eventually elected. Replicas eventually converge. Liveness violations are the availability failures — the system is correct but stuck.
Many distributed system problems are unsolvable unless either safety or liveness is weakened, depending on the model. The FLP impossibility theorem proves this formally for consensus in asynchronous systems. Raft and Paxos respond by guaranteeing safety always and liveness only in partially synchronous conditions. During a network partition, they may pause (sacrifice liveness) rather than risk committing conflicting log entries (sacrifice safety). This is the right trade-off — a brief pause is recoverable, split-brain is not.
The CAP theorem makes the same trade-off explicit for distributed data systems during network partitions — covered in Post 3.4.
Why This Matters Every Time You Write Production Code
Every time an engineer adds a timeout, retries a request, replicates data, or elects a leader, they are making model-based decisions — whether they realise it or not. Engineers who understand system models:
Design fewer brittle systems because they know which assumptions their design depends on. Debug outages faster because they can identify which model assumption was violated. Choose the right trade-offs because they understand what is mathematically possible under their operating conditions rather than what feels intuitively correct.
The retries and idempotency patterns that make retries safe rather than dangerous are covered in Post 2.2. The consensus algorithms that operate under specific model assumptions are covered in Post 3.7. Everything in this series builds on the model assumptions established here.
Key Takeaways
- A system model is a simplified set of assumptions about how a distributed system behaves — not a description of reality, but a foundation for reasoning about correctness
- Models are assumptions, not guarantees — when reality violates the model (bounded latency, crash-stop failures, synchronised clocks), every guarantee built on that model collapses
- The three core dimensions of system models are the network model (how messages behave), the node model (how processes fail), and the time model (how events are ordered)
- Most production outages are not caused by algorithmic bugs — they are caused by hidden model assumptions being violated, especially slow nodes misclassified as crashed and network stalls exceeding assumed latency bounds
- Safety (nothing bad happens) and liveness (something good eventually happens) are the two fundamental correctness properties — many distributed systems problems require weakening one to achieve the other
- The partially synchronous network model describes real production systems most accurately — algorithms like Raft guarantee safety always and liveness only during periods of network stability
- Engineers who understand system models make fewer accidental trade-offs and debug outages faster than those who operate under implicit, unexamined assumptions
Frequently Asked Questions (FAQ)
What is a system model in distributed systems?
A system model is a simplified set of assumptions about how a distributed system behaves — specifically how messages move (network model), how processes fail (node model), and how time and ordering work (time model). System models are not descriptions of reality but frameworks for reasoning about correctness. Every distributed system algorithm is correct only under specific model assumptions. When reality violates those assumptions, the algorithm’s guarantees break down regardless of how correctly it is implemented.
Why do most production outages trace back to violated model assumptions?
Because engineers design systems assuming best-case model behaviour and production delivers worst-case reality. The most common violations: assuming bounded network latency when the network occasionally stalls for seconds (triggering false failure detection and unnecessary leader elections), assuming crash-stop failures when nodes actually fail slowly (creating the split-brain window where a GC-paused node is declared failed while still holding a lock), and assuming synchronised clocks when drift produces incorrect causal ordering. The Chubby example in this post illustrates exactly this dynamic at Google scale.
What is the difference between crash-stop and crash-recovery failure models?
In the crash-stop model, a failed node permanently stops and never sends another message. This is the cleanest model for algorithm design — failed nodes can be safely ignored. In the crash-recovery model, a failed node may restart and rejoin the cluster, potentially with stale state from before its failure. This requires algorithms to handle a node that believes it is entitled to a role it no longer holds. Crash-recovery is the model that production systems actually exhibit — Kubernetes pods restart, VMs reboot, services crash and recover. Most distributed systems bugs in production occur when engineers design for crash-stop but operate in crash-recovery conditions.
Why can’t distributed systems use a single global clock?
Physical clocks on different machines drift at different rates and are synchronised only approximately via NTP. NTP synchronisation can move a clock backward. Network delays mean a timestamp from one machine cannot be reliably compared to a timestamp from another machine to determine which event happened first. Even with GPS-based time (as Google Spanner’s TrueTime uses), there is always a bounded uncertainty interval — not zero. This is why distributed systems use logical clocks (Lamport timestamps, vector clocks) that establish causal ordering based on message passing rather than wall-clock time.
What is the difference between safety and liveness in distributed systems?
Safety is the property that nothing bad ever happens — no data corruption, no two simultaneous leaders, no acknowledged write is lost. Safety violations are catastrophic and typically unrecoverable without manual intervention. Liveness is the property that something good eventually happens — requests complete, elections resolve, the system makes progress. Liveness violations are availability failures — the system is stuck but not incorrect. Most distributed systems algorithm design involves trading liveness for safety during adverse conditions: Raft pauses during leader elections (sacrificing liveness briefly) rather than risking split-brain (sacrificing safety permanently).
Which network model describes real production systems?
Partially synchronous. Real production networks are not synchronous (messages do not arrive within a fixed bound — networks experience congestion, routing changes, and packet loss) and not purely asynchronous (messages are not arbitrarily delayed forever — the network eventually stabilises). The partially synchronous model captures this: timing is unreliable most of the time but eventually stabilises enough for the system to make progress. Raft and Paxos both operate under this assumption, which is why they guarantee safety always but liveness only when the network is sufficiently stable.
The post I think about most when something breaks at 3am
Of all the posts in Part 1, this is the one I come back to most often — not when building, but when debugging. When something is wrong in a distributed system and the logs show nothing obviously broken, the question I ask first is: which model assumption did reality just violate?
At IDFC First Bank, we have a financial platform where the cost of a model assumption breaking is concrete and immediate. A write that was assumed committed but was not. A leader that was assumed unique but was not. A timestamp that was assumed to reflect causal order but was not. Each of these has happened — not because the algorithms were wrong, but because the environment in which they ran briefly violated the assumptions they were designed for.
The pattern I have seen most often is the slow node that looks like a crashed node. A disk stall, a network hiccup — the node is alive, it is processing, but it is not responding within the timeout window. The system declares it failed. It elects a new leader. The paused node resumes and believes it is still the leader. For a window of seconds, two nodes are both issuing decisions. Whether that produces a correctness violation depends entirely on whether fencing is implemented at the resource — which, in my experience, it frequently is not.
That failure mode is not exotic. It is the GitHub 2012 MySQL GC pause incident. It is the 2013 etcd split-brain. It is the class of production incident that system model awareness is specifically designed to prevent — or at minimum, to recognise quickly when it occurs.
Read this post before you read anything else in the series. Then read Post 1.3 on the network model and Post 1.4 on the node failure model. By the time you reach Part 3 on consensus and consistency, the trade-offs will feel inevitable rather than arbitrary — because you will understand the environment the algorithms are operating in.
Series home: Distributed Systems — Concepts, Design & Real-World Engineering
Part 1 — Foundations
- 1.1 — What Is a Distributed System (Really)?
- 1.2 — System Models: How Distributed Systems See the World
- 1.3 — Network Model: Latency, Loss and Partitions
- 1.4 — Node & Failure Model: Crashes, Slow Nodes and Partial Failure
- 1.5 — Time Model: Why Ordering Is Harder Than It Looks
Previous: ← 1.1 — What Is a Distributed System (Really)?
Next: 1.3 — Network Model: Latency, Loss and Partitions →
Once you have worked through all five Foundation posts, Part 2 begins with how distributed systems cope with these realities in practice →