Distributed Systems Series — Part 3.1: Replication, Consistency & Consensus
Replication Is Not an Optimisation
Most engineers first encounter replication as a performance feature. Add read replicas to your database, serve more queries, reduce load on the primary. That is a legitimate use of replication — but it is the least important one.
Replication exists because a single copy of data on a single node is a liability in a distributed system. Not a risk to be managed. A liability to be eliminated.
In Parts 1 and 2 of this series we established three uncomfortable truths that every distributed system must operate within. Nodes fail independently — they crash, restart, slow down, and fail partially in ways that are hard to detect. Networks are unreliable — messages are lost, delayed, duplicated, and reordered. Time is uncertain — there is no perfect global clock, and event ordering is ambiguous.
Once you accept these three realities, storing data on a single node is not a simplification. It is a single point of failure waiting to manifest. Replication is the first and most fundamental response to that reality.
Replication exists because reality is hostile.
What Replication Actually Means
Replication means maintaining multiple copies of the same logical data across different nodes. That sentence sounds simple. The word “maintaining” is where all the difficulty lives.
A replica is not a backup that sits inert until needed. It is an independent process, running on a different machine, communicating with other replicas over an unreliable network, potentially serving live traffic at the same moment a write is happening elsewhere. Replicas can be temporarily inconsistent with each other — and that inconsistency is not a bug. It is an unavoidable consequence of the network unreliability we established in Part 1.
This is why replication is not just a storage decision. It is a coordination problem. The moment you have two copies of the same data, you must answer questions that do not exist with one copy: which copy is authoritative? What happens when they disagree? Which copy does a read see? Can a write succeed if only one copy is reachable?
These questions have no universally correct answers. They depend on trade-offs — trade-offs that the rest of Part 3 will explore in depth.
Five Reasons Replication Is Necessary
Availability: Serving Requests Despite Failures
In a distributed system, failure is not an edge case. Machines crash. Containers are evicted. Availability zones go down. Entire cloud regions have had multi-hour outages. This is not theoretical — it has happened to AWS, Google Cloud, and Azure, to every major cloud provider, repeatedly and publicly.
Without replication, a single node failure means total unavailability for whatever data that node owned. With replication, other replicas continue serving requests while the failed node is repaired or replaced. The system degrades but does not stop.
The key insight is that availability is not achieved by preventing failure. It is achieved by assuming failure will happen and designing the system to continue operating when it does. Replication is the first concrete step toward that assumption.
Amazon S3 stores each object across multiple availability zones by default, explicitly because the assumption is that any individual zone can fail at any time. Google Spanner replicates data across multiple datacenters across multiple geographic regions for the same reason. These are not cautious overengineering choices — they are the baseline required to deliver the availability guarantees those systems promise.
Fault Tolerance: Surviving Partial Failure
Distributed systems rarely fail all at once. More commonly, one node is slow, one disk corrupts data silently, one network link starts dropping packets intermittently. These partial failures are harder to handle than complete failures because they are ambiguous — the system is not clearly up or clearly down.
Replication allows systems to mask individual component failures. If one replica is returning errors, read traffic can be routed to another. If one replica falls behind on writes, the system can continue operating while it catches up. If one replica is lost entirely, a replacement can be provisioned and brought back up to date from the surviving replicas.
This is fault tolerance — continuing correct operation in the presence of faults. Replication does not eliminate faults. It absorbs them, converting what would be a complete outage into a degraded but functional state.
Cassandra’s design philosophy makes this explicit: it is designed to survive the loss of any single node, any single rack, or any single datacenter, depending on how replication is configured. The system continues operating — with potentially degraded consistency — rather than stopping to wait for a failed component to return.
Durability: Protecting Data from Permanent Loss
Data stored on a single node is vulnerable in ways that go beyond temporary unavailability. Disks fail permanently. Hardware is destroyed in fires, floods, and power surges. Software bugs can corrupt data silently over days or weeks before the corruption is detected. Accidental deletions propagate instantly.
Replication reduces the probability of permanent data loss by storing redundant copies on independent hardware. Losing one copy does not mean losing the data — the other copies survive. This is durability: the guarantee that committed data persists across failures.
However, durability through replication is not free. Replicas must agree on what the data actually is. A write that propagates to one replica but not others before a failure creates a gap — and closing that gap requires coordination. Durability pushes us directly into consistency problems. You cannot guarantee that data is durably stored without also asking: durably stored in which state, and visible to which readers, when?
This is why PostgreSQL’s synchronous replication mode exists: a write is not acknowledged to the client until it has been written to the primary and confirmed by at least one synchronous standby. The client pays a latency cost in exchange for a durability guarantee that survives primary failure.
Performance: Scaling Reads Across Replicas
Replication is frequently introduced for performance reasons, particularly in read-heavy workloads. Multiple replicas can serve reads in parallel, increasing overall read throughput. Reads can be routed to geographically nearby replicas, reducing latency. The primary node is relieved of read load and can focus on writes and coordination.
MySQL read replicas, PostgreSQL hot standbys, and DynamoDB global tables all exploit this pattern. A single primary cannot serve unlimited read traffic — but ten replicas can serve roughly ten times the read load, with reads distributed across them.
The performance benefit comes with a constraint: it only materialises if the consistency requirements of the application allow reads to be served from replicas that may be slightly behind the primary. An application that requires reading the most recent write cannot benefit from read replicas — it must always read from the primary. The more replicas you have, and the more aggressively you exploit them for read throughput, the harder it becomes to keep them consistent with each other and with the primary.
Replication improves performance only if your consistency requirements allow it.
Geographic Distribution: Serving Users Across Regions
Modern systems are global by default. Users in Mumbai, São Paulo, and Stockholm expect the same latency as users in Virginia. Regulatory requirements in some jurisdictions require that certain data never leaves specific geographic boundaries. Regional failures — a natural disaster, a submarine cable cut, a cloud region outage — should not take down a globally deployed service.
Replication across geographic regions addresses all three. Data stored close to users produces lower read latency. Data replicated within a jurisdiction satisfies residency requirements. Data replicated across regions survives regional failures.
But geographic replication magnifies every distributed systems challenge rather than reducing them. Network delays across regions are orders of magnitude larger than within a datacenter — tens to hundreds of milliseconds rather than sub-millisecond. Network partitions across regions are more common and longer-lasting. Keeping replicas consistent across regions requires either accepting high write latency (to wait for cross-region acknowledgement) or accepting that different regions may temporarily see different states.
CockroachDB’s geo-partitioning feature exists precisely to navigate this tension: data can be pinned to specific regions for latency and compliance reasons while still replicating across regions for durability. The configuration is complex because the underlying problem is genuinely complex.
Why Replication Is Hard
The surface-level description of replication — copy the data to multiple machines — sounds straightforward. The implementation reveals a cascade of questions that have no simple answers.
When a write happens, which replicas must acknowledge it before the write is considered complete? If only one replica must acknowledge, writes are fast but a single node failure after the write could lose it. If all replicas must acknowledge, the write is durable but the system cannot accept writes when any replica is unreachable. What do you choose when one replica is slow but not failed — do you wait indefinitely or time out and accept the write with reduced replication?
What happens when replicas disagree? Two clients write to the same key concurrently on different replicas. Both writes succeed locally. Now the replicas disagree about what the current value is. Which write wins? The later one by timestamp — but whose clock do you trust? The one from the primary — but what if the primary just changed? This is not a hypothetical. It is the everyday reality of systems like DynamoDB, Riak, and Cassandra, which is precisely why they invested in vector clocks, quorum reads, and conflict resolution strategies.
What does it mean for a read to be correct? A client writes a value, then immediately reads it back. Should the read always return the value just written? In a single-node system this is trivially yes. In a replicated system it depends entirely on which replica the read goes to and how far behind that replica is.
None of these questions have universally correct answers. They have trade-offs. Replication forces you to confront those trade-offs explicitly rather than ignoring them.
Replication vs Backups: A Critical Distinction
Replication and backups are frequently confused, and conflating them is dangerous. A team that has replication but no backups may believe they are protected against data loss when they are not. A team that has backups but no replication may believe they have availability when they do not.
They serve fundamentally different purposes:
| Replication | Backups |
|---|---|
| Improves availability | Improves recoverability |
| Serves live traffic | Used offline, after a failure |
| Requires active coordination | No coordination required |
| Failures move into logic | Simple point-in-time snapshots |
| Helps you stay up | Helps you recover later |
The critical failure mode that exposes this distinction is a software bug or accidental deletion. If a bug corrupts data and that corruption propagates to all replicas within seconds, replication provides no protection — every copy is equally corrupted. A backup taken before the corruption occurred is the only recovery path.
Conversely, replication protects against hardware failure and node unavailability in ways that backups cannot. A backup stored on S3 does not help you serve traffic while your primary database node is being replaced. A replica does.
Serious production systems require both. They are not alternatives.
Replication Creates New Failure Modes
One of the most important — and counterintuitive — aspects of replication is that it introduces failure modes that do not exist in single-node systems. Adding replicas does not only make a system more resilient. It also makes a system more complex, and complexity introduces new ways to fail.
Split-brain occurs when a network partition causes replicas on each side to accept writes independently, without knowledge of what the other side is writing. When the partition heals, the replicas have diverged and must reconcile conflicting state. This is not a hardware failure — it is a logical failure caused by the coordination overhead of replication itself.
Stale reads occur when a client reads from a replica that has not yet received the most recent write. The data returned is technically valid — it reflects a real state the system was in — but it is not the current state. Applications that cannot tolerate stale reads must be designed specifically to avoid them, typically by routing reads to the primary or using quorum reads.
Write amplification occurs when a single logical write must be replicated to multiple nodes, each of which may need to acknowledge it, retry it, or resolve conflicts with concurrent writes. The work of a single write multiplies with the replication factor and the complexity of the consistency protocol.
Replication lag — the delay between a write being applied on the primary and appearing on replicas — creates windows during which the system is in an inconsistent state. In synchronous replication, this window is minimised at the cost of write latency. In asynchronous replication, the window can grow during periods of high load or connectivity problems, and a primary failure during that window can result in data loss.
These are not bugs in specific implementations. They are consequences of distributed reality. Understanding replication means accepting that some failures move from hardware into logic — and logical failures are harder to detect, harder to diagnose, and harder to recover from than hardware failures.
Replication Is the Door Into Part 3
Replication answers one question: How many copies of the data should exist? It immediately raises questions that are harder and more consequential.
How consistent should those copies be? This is the consistency models question — strong consistency, eventual consistency, causal consistency, and the spectrum between them. What happens when a network partition forces a choice between consistency and availability? This is the CAP theorem, correctly understood. How do replicas agree on the order of writes when they cannot communicate reliably? This is the consensus problem, which Paxos and Raft were built to solve.
Each post in Part 3 takes one of these questions and answers it as precisely as it can be answered — which means acknowledging that the answers are always trade-offs, never absolutes.
Replication is the door. Consistency and consensus are what you find behind it.
Key Takeaways
- Replication is not an optimisation — it is a mandatory response to the fundamental unreliability of nodes, networks, and time in distributed systems
- The five motivations for replication are availability, fault tolerance, durability, performance, and geographic distribution — each carries a distinct cost
- Replication converts hardware failures into coordination problems — a replicated system must answer questions about consistency, ordering, and conflict resolution that do not exist with a single copy
- Replication and backups serve different purposes and are not substitutes — replication keeps systems available, backups allow recovery from logical failures like bugs and accidental deletions
- Replication introduces new failure modes — split-brain, stale reads, write amplification, and replication lag — that must be understood and designed around explicitly
- Every replicated system must make explicit trade-offs — there is no universally correct replication strategy, only strategies that fit specific consistency, availability, and latency requirements
Frequently Asked Questions (FAQ)
What is replication in distributed systems?
Replication means maintaining multiple copies of the same logical data across different nodes in a distributed system. Each copy is called a replica. Replicas are independent processes that communicate over unreliable networks and can be temporarily inconsistent with each other. Replication improves availability, fault tolerance, and durability — but requires coordination to keep replicas consistent, which introduces its own complexity and failure modes.
Why is replication necessary?
Because a single copy of data on a single node fails when that node fails. In a distributed system, node failures are not exceptional — they are normal operating conditions. Replication ensures that other copies of the data remain available to serve requests when one copy becomes unavailable. Without replication, any individual node failure causes total unavailability for the data that node owned.
What is the difference between replication and backups?
Replication maintains live, active copies of data that serve traffic continuously. It improves availability and allows the system to continue operating when a node fails. Backups are point-in-time snapshots stored separately, used to recover from logical failures like software bugs, data corruption, or accidental deletion. Replication does not protect against a bug that corrupts all replicas simultaneously — only a backup taken before the corruption can recover from that. Production systems require both.
What is replication lag?
Replication lag is the delay between a write being applied on the primary node and that write appearing on replica nodes. In asynchronous replication, the primary acknowledges the write immediately and replicates in the background — lag can grow during high load or connectivity problems. In synchronous replication, the primary waits for confirmation from replicas before acknowledging the write — lag is minimised but write latency increases. A primary failure during an asynchronous replication lag window can result in the loss of writes that were acknowledged to clients but not yet replicated.
What is split-brain in replication?
Split-brain occurs when a network partition causes two groups of replicas to each accept writes independently, without knowledge of what the other group is writing. Each group believes it is the authoritative copy. When the partition heals, the replicas have diverged and hold conflicting state that must be reconciled. Split-brain is not a hardware failure — it is a logical failure caused by the coordination overhead of replication, and it is one of the most dangerous failure modes in replicated systems.
Does replication guarantee no data loss?
No. Replication reduces the probability of data loss significantly but does not eliminate it. In synchronous replication, a write is not acknowledged until it has been confirmed by a quorum of replicas — the window for data loss is very small. In asynchronous replication, writes acknowledged by the primary may not yet have reached replicas when the primary fails — those writes can be lost. Additionally, replication does not protect against logical failures that propagate to all replicas before detection, such as a software bug that corrupts data. Backups are required alongside replication to protect against these scenarios.
Continue the Series
Series home: Distributed Systems — Concepts, Design & Real-World Engineering
Part 3 — Replication, Consistency & Consensus
- 3.1 — Why Replication Is Necessary
- 3.2 — Replication Models: Leader-Based, Multi-Leader and Leaderless
- 3.3 — Consistency Models: Strong, Eventual, Causal and Session
- 3.4 — The CAP Theorem Correctly Understood
- 3.5 — Quorums and Voting in Distributed Systems
- 3.6 — Why Consensus Is Hard in Distributed Systems
- 3.7 — Paxos vs Raft: Consensus Algorithms Compared
- 3.8 — Performance Trade-offs in Replicated Systems
- 3.9 — Engineering Guidelines for Replication, Consistency and Consensus
Not read Part 1 or Part 2 yet?
Next: 3.2 — Replication Models: Leader-Follower, Multi-Leader and Leaderless