Redundancy Patterns and Strategies in Distributed Systems

Home » Distributed Systems » Fault Tolerance & High Availability » Redundancy Patterns and Strategies in Distributed Systems

Distributed Systems Series — Part 4.3: Fault Tolerance & High Availability

Redundancy Is the Foundation, Not the Solution

Post 4.1 established the taxonomy of failures — crash-stop, crash-recovery, omission, timing, gray, Byzantine, and correlated. Post 4.2 established the distinction between fault tolerance (correctness under failure) and high availability (uptime). This post addresses the structural mechanism that enables both: redundancy.

Redundancy is the deliberate duplication of components, data, and network paths so that failures do not disrupt system operation. It is the foundational mechanism — without it, fault tolerance and high availability are impossible to achieve. But redundancy is not sufficient on its own. The critical lesson that engineers learn expensively in production: Redundancy without coordination creates new failure modes rather than eliminating existing ones.

A system with three database replicas that have no coordination protocol may end up with three different versions of the truth after a network partition — more dangerous than having a single node, because the divergence is invisible until it produces incorrect results. Redundancy must be paired with the right coordination mechanisms, deployed across genuinely independent failure domains, and matched to the RTO and RPO requirements of the system it protects.

This post explains the four primary redundancy patterns, the synchronous vs asynchronous replication trade-off that cuts across all of them, RTO and RPO as the business constraints that drive redundancy decisions, and how to choose the right pattern for specific requirements.

RTO and RPO: The Business Constraints That Drive Redundancy Decisions

Before examining redundancy patterns, the two metrics that determine which pattern is appropriate for a given system must be precise. These are the most important concepts in disaster recovery and business continuity planning — and they are searched for constantly by engineers making redundancy decisions.

Recovery Time Objective (RTO) is the maximum acceptable time between a failure occurring and the system returning to normal operation. It answers: how long can the system be unavailable? An RTO of zero means the system must continue serving traffic without interruption — no failover gap is acceptable. An RTO of four hours means the system can be completely unavailable for up to four hours before the business impact becomes unacceptable.

Recovery Point Objective (RPO) is the maximum acceptable amount of data loss measured in time. It answers: how much data can be lost? An RPO of zero means no data loss is acceptable — every committed write must survive any failure. An RPO of one hour means the system can lose up to one hour of writes — data written in the last hour before a failure may not be recoverable.

RTO target	RPO target	Required redundancy pattern	Typical use case
Zero (continuous)	Zero (no data loss)	Active-active with synchronous replication	Payment processing, trading systems
Seconds	Zero	Active-passive with synchronous replication	Financial databases, coordination services
Minutes	Seconds to minutes	Active-passive with asynchronous replication	Standard SaaS databases, most web applications
Hours	Hours	Backup and restore, cold standby	Dev environments, archival systems

The relationship between RTO, RPO, and cost is direct and non-linear. Moving from an RTO of minutes to an RTO of zero requires active-active deployment across multiple regions — which increases infrastructure cost by two to three times and introduces the write conflict problem that active-active systems must solve. Moving from an RPO of minutes to an RPO of zero requires synchronous replication — which increases write latency by the network round-trip time to the farthest required replica, which may be tens to hundreds of milliseconds for cross-region deployments.

Misaligning redundancy investment with actual RTO and RPO requirements is one of the most common and expensive mistakes in distributed system design. Systems are over-engineered for availability targets they do not actually need, and under-engineered for data durability targets they do.

Synchronous vs Asynchronous Replication: The Foundational Trade-off

Before examining the structural redundancy patterns, the replication timing mode that applies to all of them must be understood. This is covered in depth in Post 3.2, but its significance for redundancy decisions deserves emphasis here.

Synchronous replication means the primary node waits for acknowledgement from the required replicas before confirming a write to the client. The write is only considered complete when the required number of replicas have durably stored it. This guarantees zero data loss — if the primary fails immediately after acknowledging a write, at least one replica has the data and can take over. The cost is write latency: every write waits for a network round trip to each synchronous replica. In a single datacenter, this adds milliseconds. Across regions, this adds tens to hundreds of milliseconds.

Asynchronous replication means the primary acknowledges the write to the client immediately and replicates to replicas in the background. Write latency is lower — the client does not wait for replica acknowledgement. But there is a durability window: writes acknowledged to clients may not yet have reached replicas. If the primary fails during this window, those writes are lost. The RPO is not zero — it is the replication lag at the moment of failure, which is typically milliseconds under normal conditions but can grow to seconds or minutes under high load or connectivity problems.

PostgreSQL’s synchronous standby configuration demonstrates the synchronous trade-off in production. With one synchronous standby, every write must be confirmed by both the primary and the standby before the client receives acknowledgement. If the standby becomes unavailable, PostgreSQL can be configured to either wait indefinitely for the standby to return (preserving RPO=0 at the cost of availability) or fall back to asynchronous replication (preserving availability at the cost of the zero-loss guarantee). The operator must choose which failure mode is acceptable — neither option is free.

Pattern 1: Active-Passive Redundancy

Active-passive redundancy designates one node as the primary — handling all client traffic — and one or more nodes as standbys that replicate from the primary but serve no client traffic during normal operation. When the primary fails, one standby is promoted to primary and begins accepting traffic.

Active-passive is the simplest and most widely deployed redundancy pattern. It is used by PostgreSQL streaming replication, MySQL primary-replica, Redis Sentinel, and most traditional relational database deployments. Its appeal is simplicity: there is one authoritative source of truth at all times, write conflicts are impossible because only one node accepts writes, and the consistency model is straightforward — reads from the primary are always current, reads from replicas reflect a slightly delayed state.

Failover is the critical operational challenge. When the primary fails, three things must happen in sequence: the failure must be detected, a standby must be elected as the new primary, and clients must be redirected to the new primary. Each step takes time and introduces risk.

Failure detection depends on heartbeat timeouts — the monitoring system must wait for enough missed heartbeats to conclude that the primary has failed rather than merely slowed. As established in Post 4.1, a slow node is indistinguishable from a crashed node during a timeout window. Aggressive timeout settings reduce the detection window but increase false positives — unnecessary failovers triggered by transient slowness that promotes a standby while the primary is still alive. This is the split-brain risk: two nodes believing they are the primary simultaneously.

Fencing — the mechanism introduced in Post 2.4 — prevents the old primary from accepting writes after a new primary is elected. Without fencing, a primary that is merely partitioned from the monitoring system (but still reachable by some clients) may continue accepting writes that the new primary does not know about, producing diverged state that must be reconciled when the partition heals.

With asynchronous replication, active-passive failover can produce data loss: writes acknowledged by the old primary but not yet replicated to standbys are lost when the standby is promoted. The RPO is the replication lag at the moment of failure. With synchronous replication, active-passive failover preserves all committed writes — but the standby must be reachable for the primary to accept any writes, which reduces availability during standby failures.

N+1 redundancy is a specific application of active-passive at the infrastructure level. Rather than one primary and one standby, N+1 deploys N active instances with one spare — for every N instances handling traffic, one additional instance is on standby ready to absorb failures. This is the standard pattern for stateless service redundancy: a Kubernetes deployment with 3 replicas can lose one and continue serving with 2, while the scheduler replaces the lost instance. For stateful services, N+1 combines with the active-passive replication model to provide both component redundancy and data redundancy.

Pattern 2: Active-Active Redundancy

Active-active redundancy runs multiple nodes simultaneously, all serving client traffic. There is no standby — all nodes are active. When one node fails, the remaining nodes absorb its traffic without a promotion step. From a user perspective, there is no failover gap.

Active-active achieves the RTO=0 target that active-passive cannot — traffic is rerouted immediately because the surviving nodes are already serving traffic. It also improves resource utilisation: in active-passive, standby nodes consume infrastructure cost while serving no traffic. In active-active, all nodes are productive.

The cost is the write conflict problem. When multiple nodes accept writes to the same data simultaneously, concurrent writes to the same key on different nodes will inevitably occur. Both writes succeed locally. When the writes propagate to each other, both nodes discover they have conflicting versions of the same data. The system must resolve the conflict — and the resolution strategy determines whether the outcome is correct.

Three conflict resolution strategies are used in production systems, each with distinct correctness properties:

Last-write-wins (LWW) resolves conflicts by keeping the write with the higher timestamp and discarding the other. It is the simplest strategy and the most dangerous. As established in Post 1.5, clocks in distributed systems cannot be trusted for ordering. A write with a higher timestamp from a node with a clock running slightly ahead will silently overwrite a later write from a node with a clock running slightly behind. LWW produces silent data loss that is difficult to detect and impossible to recover from. It is acceptable only for use cases where losing some writes is acceptable — analytics events, log aggregation — and is wrong for any use case where all writes must be preserved.

Conflict-free replicated data types (CRDTs) are data structures designed so that concurrent updates can always be merged automatically without producing conflicts. A counter that only increments can always be merged by summing all increments across all nodes. A set where elements can be added but not removed (a grow-only set) can always be merged by taking the union. CRDTs work correctly for specific data shapes — counters, sets, registers with specific merge semantics — but cannot handle arbitrary mutations to the same key. They are the right solution for collaborative editing, distributed counters, and shopping carts, and the wrong solution for account balances, inventory counts, or any value that must be exactly correct.

Application-level merging exposes both conflicting versions to the application code and requires the application to decide which wins or how to merge them. Amazon’s original Dynamo design used this approach for shopping carts — both cart versions were presented to the client application, which merged them by taking the union of items. This is the most flexible approach and the most complex to implement correctly. It requires the application to be designed with conflict resolution in mind from the start.

Active-active is the right pattern for geographically distributed systems where users in different regions must write with low latency and cross-region write latency is unacceptable. CouchDB’s multi-master replication, Cassandra in multi-datacenter mode, and active-active database deployments across AWS regions all use this pattern. It requires accepting that conflicts will occur and investing in conflict detection and resolution before choosing this pattern — not after discovering conflicts in production.

Pattern 3: Quorum-Based Redundancy

Quorum-based redundancy distributes both reads and writes across multiple nodes without designating a single active primary, using the W+R>N overlap guarantee to ensure consistency. Covered in depth in Post 3.5, the quorum mechanism deserves re-examination here through the lens of redundancy design.

In a quorum system with N replicas, a write requires W replicas to acknowledge it and a read queries R replicas. When W+R>N, at least one replica that received the write will always respond to the read — guaranteeing that reads see the most recent write. A cluster of N=3 with W=2 and R=2 tolerates one replica failure for both reads and writes while maintaining the overlap guarantee.

Quorum-based redundancy is the pattern that provides the most flexibility for tuning the RTO/RPO/consistency trade-off. By adjusting W and R, the same cluster can be configured for different operations with different requirements. A financial audit query might use W=3, R=3 — maximum durability and consistency, accepting the highest latency. A product recommendation read might use W=2, R=1 — acceptable staleness, minimum latency. Cassandra’s per-operation consistency level configuration implements exactly this flexibility.

The key limitation: quorum-based systems do not automatically provide linearisability, as established in Post 3.3. Concurrent writes that each achieve quorum on overlapping but different replica sets can produce conflicts. Operations that require linearisability — distributed locks, unique constraint enforcement — need additional coordination beyond quorums alone.

Geographic Redundancy: Failure Domain Independence

All three redundancy patterns above provide protection only to the extent that the replicas they use are in independent failure domains. This is the connection to the correlated failure class introduced in Post 4.1: redundancy across the same failure domain provides no protection against failures that affect the entire domain simultaneously.

Cloud providers structure failure domains hierarchically. AWS defines availability zones (physically separate datacenters within a region, connected by low-latency high-bandwidth links), regions (geographically separated groups of availability zones), and global edge locations (CDN and DNS infrastructure distributed worldwide). Google Cloud defines zones, regions, and multi-regions. Azure defines availability zones, regions, and geography pairs.

The independence guarantees at each level differ significantly. Two instances in different availability zones of the same region share the same regional network infrastructure, the same regional control plane, and are subject to correlated failures from regional-level events — power grid failures, large-scale networking incidents, regional cloud provider outages. Two instances in different regions share only the global internet and the cloud provider’s inter-region network. Two instances in different cloud providers share nothing except the global internet.

The 2017 AWS US-EAST-1 S3 outage illustrates the importance of failure domain independence. The outage affected the entire US-EAST-1 region — including all availability zones within it. Systems that had deployed “multi-AZ” redundancy within US-EAST-1 were not protected because both zones experienced the same regional control plane failure simultaneously. Systems that had deployed cross-region redundancy to US-WEST-2 continued operating.

Geographic redundancy across regions introduces latency costs that must be factored into RTO and RPO decisions. Cross-region synchronous replication adds the inter-region network round-trip time to every write — typically 50 to 100 milliseconds for US-US cross-region, 80 to 200 milliseconds for US-EU cross-region. Systems that require RPO=0 and geographic failure domain independence must accept these latency costs or use specialised hardware (Google’s TrueTime, leased-line connections) to reduce them.

Stateless vs Stateful Redundancy

The three redundancy patterns above apply differently to stateless and stateful components, and the difference in complexity is significant enough to drive architectural decisions.

Stateless component redundancy is straightforward. A stateless web server, API gateway, or compute function holds no persistent state between requests — every request is self-contained. When a stateless instance fails, a replacement instance can be started immediately from the same container image or deployment package. There is no data to synchronise, no replication lag to worry about, and no split-brain risk. Kubernetes deployments, AWS Auto Scaling Groups, and Google Cloud Managed Instance Groups implement stateless redundancy with health checks and automatic replacement — the replacement instance is equivalent to the failed one from the moment it starts.

Stateful component redundancy is where all the complexity lives. A database, message queue, distributed cache, or coordination service holds state that must be preserved across instance failures. Replacing a failed stateful instance requires either restoring from a replica (which requires the replica to be current) or restoring from a backup (which introduces RPO equal to the backup interval). All three redundancy patterns are primarily concerned with stateful redundancy — the challenge of keeping multiple copies of state consistent while tolerating individual copy failures.

This asymmetry is why distributed systems architectures deliberately separate stateless and stateful components. The stateless tier can be scaled and made redundant with simple N+1 infrastructure patterns. The stateful tier requires replication protocols, consensus algorithms, and careful consistency management. Mixing stateful and stateless concerns in the same component makes both redundancy and scaling significantly harder.

Redundancy Does Not Eliminate the Need for Coordination

The most important lesson from production experience with redundancy: adding replicas without adding coordination protocols creates new failure modes rather than eliminating existing ones.

Three specific failure modes that redundancy without coordination produces. Split-brain: during a network partition, multiple nodes believe they are the active primary and accept writes independently, producing diverged state that must be reconciled when the partition heals. Stale reads: a client reads from a replica that has not yet received recent writes, receiving data that is correct as of a past state but incorrect for the current state. Lost writes: with asynchronous replication, a primary acknowledges writes that replicas have not received; when the primary fails, those writes are lost without the client knowing.

Coordination protocols address each failure mode. Consensus-based leader election (Raft, Paxos) prevents split-brain by ensuring only one node can be elected leader at a time and that the old leader cannot accept writes after a new leader is elected. Quorum-based reads prevent stale reads by requiring that reads consult enough replicas to guarantee at least one has the most recent write. Synchronous replication prevents lost writes by requiring replica acknowledgement before client acknowledgement.

The connection between redundancy patterns and coordination protocols is direct: active-passive redundancy requires a leader election protocol to safely promote a standby. Active-active redundancy requires a conflict detection and resolution protocol to handle concurrent writes. Quorum-based redundancy uses the W+R>N overlap as its coordination mechanism. None of these are optional additions to redundancy — they are the mechanisms that make redundancy provide the fault tolerance guarantee rather than merely the appearance of it.

Choosing the Right Redundancy Pattern

The choice follows from RTO, RPO, and consistency requirements — not from technology familiarity or trend.

Choose active-passive with synchronous replication when RPO must be zero and write conflicts must be impossible. The failover gap (seconds to minutes) is acceptable. The use case requires strong consistency. PostgreSQL, MySQL with semi-synchronous replication, and etcd are the standard implementations. This is the right default for most production databases.

Choose active-active when RTO must be zero and geographic distribution requires low-latency writes from multiple regions. Accept that write conflicts will occur and invest in conflict resolution before choosing this pattern. CouchDB, Cassandra multi-datacenter, and active-active database deployments across cloud regions are the production implementations. This is the right pattern for globally distributed systems where cross-region write latency is a hard constraint.

Choose quorum-based replication when per-operation consistency tuning is required and the workload benefits from distributing both reads and writes across nodes. Cassandra and DynamoDB are the production implementations. This is the right pattern for high-throughput, geographically distributed data stores where different operations have different consistency requirements.

Choose N+1 redundancy for stateless components always. There is no meaningful trade-off for stateless redundancy — adding one spare instance to every group of N instances provides failure tolerance at minimal cost and zero coordination complexity.

Key Takeaways

Redundancy is the foundational mechanism for fault tolerance and high availability — but redundancy without coordination creates new failure modes including split-brain, stale reads, and lost writes
RTO (recovery time objective) and RPO (recovery point objective) are the business constraints that determine which redundancy pattern is appropriate — choosing a pattern without defining these targets produces either over-engineering or under-protection
Synchronous replication achieves RPO=0 at the cost of write latency — every write waits for replica acknowledgement; asynchronous replication provides lower latency but a durability window where acknowledged writes can be lost on primary failure
Active-passive provides strong consistency and simple conflict-free writes at the cost of a failover gap during primary failure — automated fencing prevents split-brain during the promotion window
Active-active eliminates the failover gap and improves resource utilisation but makes write conflicts inevitable — conflict resolution strategy (LWW, CRDTs, application-level merge) determines whether the outcome is correct
Quorum-based redundancy provides tunable consistency per operation and tolerates partial failures without a designated primary — but does not automatically provide linearisability without additional coordination
Geographic redundancy requires replicas in genuinely independent failure domains — multi-AZ within the same region does not protect against regional-level failures; multi-region deployment does

Frequently Asked Questions (FAQ)

What is redundancy in distributed systems?

Redundancy is the deliberate duplication of components, data, and network paths so that system failures do not disrupt operation. In distributed systems, redundancy applies at multiple layers: compute (multiple application instances), storage (multiple data replicas), network (multiple paths and load balancers), and geography (multiple datacenters and regions). Redundancy alone is not sufficient — it must be paired with coordination protocols (consensus, leader election, conflict resolution) to prevent new failure modes like split-brain and divergent replicas.

What is the difference between RTO and RPO?

RTO (recovery time objective) is the maximum acceptable time a system can be unavailable after a failure — how long before it must be back online. RPO (recovery point objective) is the maximum acceptable data loss measured in time — how much data written before the failure can be lost. An RTO of zero requires active-active deployment with no failover gap. An RPO of zero requires synchronous replication with acknowledgement from replicas before confirming writes to clients. Both targets together require active-active deployment with synchronous replication — the most expensive and complex configuration, appropriate for payment systems and financial databases.

What is the difference between active-passive and active-active redundancy?

In active-passive, one node handles all traffic while others remain on standby. When the primary fails, a standby is promoted — introducing a failover gap of seconds to minutes. There are no write conflicts because only one node accepts writes. In active-active, multiple nodes handle traffic simultaneously. When one fails, others absorb its traffic immediately — RTO is effectively zero. But all nodes accept writes, making write conflicts inevitable and requiring conflict resolution. Active-passive is simpler and provides stronger consistency. Active-active eliminates failover downtime at the cost of conflict resolution complexity.

What is N+1 redundancy?

N+1 redundancy deploys one additional instance beyond the minimum required to handle normal load. For every N instances serving traffic, one spare instance is available to absorb failures. This is the standard pattern for stateless service redundancy — a Kubernetes deployment with 3 replicas handling the load of 2 has N+1 redundancy. If one replica fails, the remaining 2 continue serving while Kubernetes replaces the failed instance. For stateless components, N+1 redundancy is straightforward and inexpensive. For stateful components, N+1 must be combined with a replication protocol to ensure the spare has current data.

Why does redundancy without coordination cause split-brain?

Split-brain occurs when two or more nodes simultaneously believe they are the authoritative primary and accept writes independently. Without a coordination protocol to prevent this, a network partition can cause two groups of nodes to each elect their own primary. Both primaries accept writes from clients they can reach. When the partition heals, the two primaries discover they have accepted conflicting writes that must be reconciled. Coordination protocols prevent split-brain by requiring that a node receive acknowledgement from a majority before declaring itself primary — ensuring at most one primary can exist at any time.

How does geographic redundancy differ from multi-AZ redundancy?

Multi-AZ redundancy places replicas in different availability zones within the same cloud region. Availability zones are physically separate but share the same regional network infrastructure, control plane, and power grid at the regional level. This protects against individual datacenter failures but not against regional-level failures that affect all zones simultaneously — as demonstrated by the 2017 AWS US-EAST-1 S3 outage. Geographic redundancy across regions places replicas in entirely separate geographic locations with independent infrastructure, protecting against regional-level failures. The cost is higher cross-region network latency, which increases write latency for synchronous replication and widens the RPO window for asynchronous replication.

Continue the Series

Series home: Distributed Systems — Concepts, Design & Real-World Engineering

Part 4 — Fault Tolerance & High Availability Overview

Previous: ← 4.2 — Fault Tolerance vs High Availability

Next: 4.4 — Failure Detection: Heartbeats, Timeouts and the Phi Accrual Detector →

Related posts from earlier in the series:

3.2 — Replication Models: Leader-Based, Multi-Leader and Leaderless — the deep treatment of synchronous vs asynchronous replication and conflict resolution
3.5 — Quorums and Voting in Distributed Systems — W+R>N explained in full with worked examples
2.4 — Coordination and Distributed Locks — fencing tokens and split-brain prevention
3.4 — The CAP Theorem Correctly Understood — the formal framework for consistency vs availability during partitions