Geo-Distribution and Multi-Region Design in Distributed Systems

by
Home » Distributed Systems » Scalability & Performance » Geo-Distribution and Multi-Region Design in Distributed Systems

Distributed Systems Series — Part 5.9: Scalability & Performance

When Single-Region Is No Longer Sufficient

Every scalability mechanism covered so far in Part 5 — partitioning, load balancing, caching, backpressure, indexing, autoscaling — operates within a single geographic region. These mechanisms collectively allow a system to handle enormous load within one region. But they cannot address two fundamental constraints that emerge as systems grow globally: physics and compliance.

Physics imposes a minimum latency on every network request bounded by the speed of light. A request from a user in Tokyo to a server in Virginia must travel approximately 11,000 kilometres. At the speed of light in fibre optic cable (approximately 200,000 km/s), the minimum one-way transit time is 55ms. A round trip is 110ms minimum — before any processing, any queuing, any congestion. No engineering optimisation can reduce this below the physical limit. A user in Tokyo will always experience higher latency than a user in Virginia when both are served from Virginia.

Compliance imposes geographic constraints on where data can be stored and processed. GDPR requires that European Union residents’ personal data not be transferred outside the EU without specific legal mechanisms. Healthcare regulations in many countries require patient data to remain within national borders. Financial regulations in some jurisdictions require transaction records to be stored domestically. These constraints are not engineering problems — they are legal requirements that determine where infrastructure must exist, regardless of where it would be most efficient to place it.

Geo-distribution addresses both constraints by deploying infrastructure in multiple geographic regions. This post covers the three multi-region deployment models and their trade-offs, the physics-bounded latency floor, GeoDNS and Anycast for global traffic routing, data residency compliance architecture, the data gravity problem, and how Google Spanner and CockroachDB achieve global consistency despite geographic latency.

The Physics-Bounded Latency Floor

Before discussing architecture patterns, the physical reality that constrains all of them deserves precise treatment. As established in Post 3.8, the speed of light is the non-negotiable floor on inter-region latency. The real-world round-trip times that production systems observe are:

US-East to US-West: 60-80ms. US-East to EU-West: 80-100ms. US-East to AP-Southeast: 170-200ms. EU-West to AP-Southeast: 150-180ms. These numbers are the minimum achievable with direct fibre routing — actual production numbers are typically 10-20ms higher due to routing overhead, network equipment processing, and the fact that fibre paths follow cable routes rather than straight-line distances.

These latencies have direct architectural implications. Synchronous cross-region replication — the primary write waiting for acknowledgement from a replica in another region — adds the full round-trip latency to every write. A payment system requiring synchronous replication from US-East to EU-West acknowledges no write in less than 80ms, regardless of how fast the local storage is. This is the fundamental cost of global durability, and it is why most globally distributed systems use asynchronous cross-region replication with a bounded staleness window rather than synchronous cross-region replication.

The consensus implications are equally significant. A Raft cluster with nodes in US-East, EU-West, and AP-Southeast cannot complete a consensus round in less than the longest round-trip time between any two nodes — approximately 200ms for US-East to AP-Southeast. A system that runs global consensus for every write will acknowledge writes in 200ms minimum. This is why global consensus systems (Google Spanner, CockroachDB multi-region) are expensive and why most globally distributed databases shard consensus by region rather than running a single global consensus group.

Three Multi-Region Deployment Models

Three deployment patterns exist for multi-region systems, each making fundamentally different trade-offs between write latency, consistency, operational complexity, and failover speed.

Active-Passive: Simple, Bounded RPO, Slower Failover

In active-passive deployment, one region (the active region) serves all writes. One or more passive regions receive replicated data from the active region and serve reads only. When the active region fails, one passive region is promoted to active — this is the failover event.

Active-passive with synchronous replication achieves RPO=0 — no committed writes are lost on active region failure, because every write was confirmed by at least one passive replica before being acknowledged to the client. The cost is write latency equal to the active-to-passive round-trip time. With a passive replica in EU-West and writes acknowledged only after EU-West confirmation, US-East writes take 80-100ms minimum.

Active-passive with asynchronous replication achieves lower write latency — writes are acknowledged from local storage within milliseconds — at the cost of a non-zero RPO. Writes acknowledged to clients but not yet replicated to the passive region are lost if the active region fails before they replicate. The RPO is the replication lag at the moment of failure — typically milliseconds to seconds under normal conditions, but potentially minutes under network degradation.

Failover in active-passive requires promoting a passive region to active. This involves changing DNS or load balancer routing to point at the new active region, verifying data consistency between the new active and other passive replicas, and potentially running recovery procedures to apply any writes that were in transit during the failure. Failover time in production is typically 2-10 minutes for automated failover, longer for manual failover. During this window, writes are unavailable — the system is consistent (no conflicting writes are being accepted) but not available for writes.

Active-passive is the correct choice for disaster recovery architectures, read scaling (passive regions serve read traffic), and systems where write conflicts are unacceptable and write latency can be bounded by the active-to-passive distance.

Active-Active: Zero RTO, Write Conflicts Inevitable

In active-active deployment, multiple regions simultaneously accept both reads and writes. Traffic is distributed across all active regions. When one region fails, traffic continues to the surviving regions without any failover step — RTO is effectively zero.

The fundamental challenge of active-active is write conflicts. When two users in different regions simultaneously write to the same data — updating the same record, decrementing the same counter — both writes succeed locally but produce conflicting versions that must be reconciled. As established in Post 3.2, three conflict resolution strategies exist: last-write-wins (dangerous — silent data loss for the losing write), CRDTs (correct for specific data shapes), and application-level merge logic (correct but requires explicit design).

Active-active write conflicts are not rare edge cases — they are regular occurrences in any system with meaningful write volume across multiple regions. A system serving 10,000 writes per second across two regions will experience approximately 10 conflicts per second for data with 0.1% conflict rate — 864,000 conflicts per day requiring resolution. The conflict resolution strategy must be designed before choosing active-active, not discovered after.

Netflix’s global active-active deployment uses a combination of conflict avoidance (partitioning data by user so each user’s data is owned by one region) and conflict resolution (for shared data, using CRDTs and eventual consistency). Cloudflare’s global network is active-active at the CDN layer — every edge location serves traffic and the stateless nature of CDN caching means there are no write conflicts for cache operations.

Regional Isolation: Compliance-First Architecture

Regional isolation pins each user’s data to a specific region — a European user’s data is stored and processed only in the EU region, an American user’s data only in the US region. Regions operate independently for their assigned users. A small global metadata layer (itself highly available across regions) stores the mapping of users to their home regions.

Regional isolation elegantly satisfies data residency requirements — EU user data never leaves the EU region, satisfying GDPR. It eliminates cross-region write conflicts — each region owns its users’ data exclusively, so concurrent writes from different regions cannot conflict for the same user’s data. Write latency is local to the user’s home region — there is no cross-region replication on the critical path.

The complexity is in cross-region user scenarios: a user who travels from Europe to Asia and accesses their account from Asia. Their data is in the EU region. The AP region must proxy the request to EU, or the user must accept higher latency. Most regional isolation architectures accept this trade-off — the cross-region case is the minority, and the benefits for the majority (local latency, data residency compliance) outweigh the degradation for the minority (cross-region latency for travellers).

Shopify uses regional isolation for merchant data. Each merchant is assigned to a primary region and all their data operations are local to that region. Stripe uses a similar model for payment processing — payment data is stored in the region where the transaction originates, satisfying data residency requirements across multiple jurisdictions simultaneously.

GeoDNS: Routing Traffic to the Nearest Region

GeoDNS returns different DNS records based on the geographic origin of the DNS resolver making the query. A user in Europe resolving the service hostname receives the IP address of the EU-West load balancer. A user in Asia receives the AP-Southeast load balancer IP. Routing to the nearest region happens at DNS resolution time, before any TCP connection is established.

AWS Route 53’s latency-based routing policy implements GeoDNS with latency measurement rather than pure geography — it routes each query to the region with the lowest measured latency from the query origin, which accounts for actual network conditions rather than just geographic distance. Route 53’s geolocation routing policy routes based on the country or continent of the DNS resolver, enabling exact geographic routing for data residency compliance.

The fundamental limitation of GeoDNS is DNS TTL caching. DNS responses are cached by resolvers for the duration of the TTL. If the EU-West region fails and the Route 53 health check removes it from the GeoDNS pool, clients that have cached the EU-West IP will continue sending traffic to the failed region until their cached record expires. With a TTL of 60 seconds, failover completes in approximately 60 seconds plus propagation time. With a TTL of 300 seconds, failover takes 5 minutes. Short TTLs reduce failover time but increase DNS query load — every client re-queries more frequently.

Route 53 supports health check integration — when a region’s health check fails, Route 53 removes it from the DNS response and routes traffic to healthy regions. Combined with a short TTL (30-60 seconds for production systems requiring fast failover), this provides automatic multi-region failover without application changes.

Anycast: Network-Layer Global Routing

Anycast announces the same IP address from multiple geographic locations simultaneously. BGP routing ensures that each client’s packets are routed to the nearest announcement point — the nearest location advertising that IP address. A client connecting to the service’s Anycast IP automatically connects to the nearest region without any DNS involvement.

The advantages over GeoDNS are failover speed and routing accuracy. When a region fails and withdraws its BGP announcement, BGP reconverges — typically in seconds — and traffic is rerouted to the next-nearest announcement point. There is no TTL to wait for. BGP routing also accounts for actual network topology rather than just geographic distance — a region that is geographically nearby but connected through a congested path may route to a more distant but better-connected region.

Cloudflare, Fastly, and most major CDN providers use Anycast for their edge networks — every edge location announces the same IP prefix, and user traffic is automatically routed to the nearest edge location by BGP. AWS Global Accelerator uses Anycast to route traffic to the nearest AWS edge location, which then routes over the AWS backbone to the healthiest regional endpoint. This combines Anycast’s fast global routing with health-aware regional failover.

Anycast requires control of BGP routing, which means it is practical for organisations with their own Autonomous System Numbers (ASNs) and significant network infrastructure, or for organisations using CDN providers that offer Anycast as a service. Most application teams will access Anycast through CDN or global load balancer products rather than implementing it directly.

CDN Architecture: Caching at the Edge

Content Delivery Networks extend the caching layer from Post 5.5 to global scale. CDN edge nodes cache responses at points of presence (PoPs) distributed worldwide — hundreds of locations in major cities. When a user requests cacheable content, the CDN serves it from the nearest PoP rather than the origin server. Latency drops from the origin round-trip time (80-200ms across regions) to the edge round-trip time (5-20ms to a nearby PoP).

CDN architecture introduces a specific invalidation challenge at global scale: purging cached content from hundreds of edge nodes simultaneously. A product price update must propagate to all edge nodes quickly to prevent users from seeing stale prices. CDN providers offer global purge APIs that propagate invalidation to all edge nodes, but propagation takes time — typically seconds to minutes depending on the CDN and the purge scope. During this window, some edge nodes serve stale content.

Cache-Control headers with short max-age values limit staleness but increase origin load. Cache-Control: max-age=60 means edge nodes re-validate with the origin every 60 seconds. At 1,000 edge nodes and 60-second TTL, the origin receives approximately 1,000 re-validation requests per minute just for cache maintenance — proportional to the number of edge nodes, not to user traffic. Stale-while-revalidate (SWR) mitigates this by allowing edge nodes to serve stale content while revalidating in the background — the user gets a fast response, the edge node updates asynchronously.

Fastly and Cloudflare provide real-time purge APIs (sub-second propagation at Fastly, 150ms globally at Cloudflare) that enable short TTLs combined with fast invalidation — the best of both worlds for content that changes frequently but where stale-while-revalidate is not acceptable.

Data Gravity: Why Computation Moves to Data

Data gravity is the principle that data attracts computation — as a dataset grows, the cost and complexity of moving it increases, making it progressively more practical to move computation to the data rather than data to the computation.

At small scale, data movement is cheap. A 1GB dataset can be transferred between regions in seconds. Processing it anywhere is practical. At production scale, a 100TB dataset transferred between AWS regions costs approximately $2,000 per transfer at standard egress pricing and takes hours. A 1PB dataset costs $20,000 per transfer. Moving this data to where the computation is becomes impractical — the computation must go to where the data is.

Data gravity has direct implications for multi-region architecture. Analytics workloads that process large datasets should run in the same region as the data rather than transferring data to a centralised analytics cluster. Machine learning training that requires large training datasets should run where the training data is stored, not where the training cluster happens to be located. Database read replicas should be co-located with the application tier that queries them rather than in a different region.

Cloud providers have recognised data gravity as a primary architectural driver. AWS’s Local Zones extend AWS infrastructure to metropolitan areas close to large user populations, reducing both latency and data movement costs for latency-sensitive applications. AWS Outposts brings AWS infrastructure on-premises, allowing computation to run adjacent to on-premises data that cannot be moved to the cloud. The pattern in all cases is the same: move the computation to the data.

Data Residency and GDPR Compliance Architecture

GDPR requires that personal data of EU residents be processed lawfully, which includes requirements for data transfers outside the EU. The simplest compliance architecture is data residency: store and process EU residents’ data exclusively within EU infrastructure, never transferring it to non-EU regions. This eliminates the legal complexity of cross-border data transfers entirely.

Implementing data residency requires partitioning data by user geography and enforcing that each partition’s data stays within its designated region. The regional isolation architecture described above implements this directly — EU users’ data is stored in EU regions, US users’ data in US regions. The global metadata layer stores only non-personal data (user ID to home region mapping) that can be globally replicated without residency constraints.

The engineering challenges of data residency compliance are in the details that are easy to overlook. Log data containing user identifiers must be routed to the user’s home region log store, not to a centralised global log aggregation system. Analytics dashboards that aggregate user behaviour must run per-region, not on a global dataset that combines EU and non-EU data. Backup systems must store EU backups in EU regions. Third-party services that process EU data — analytics, monitoring, fraud detection — must operate from EU infrastructure or have appropriate data transfer mechanisms in place.

GDPR compliance is not an architecture choice that can be retrofitted after a global system is built — it requires deliberate design from the start, with data residency enforcement at the infrastructure level rather than relying on application code to enforce geographic boundaries.

Global Consistency: Spanner and CockroachDB

Google Spanner and CockroachDB are the two production database systems that provide strong consistency (linearisability) across globally distributed nodes — achieving the seemingly impossible combination of global distribution and strong consistency at the cost of higher write latency.

Google Spanner uses TrueTime — a globally synchronised clock system based on GPS receivers and atomic clocks deployed across Google’s data centres. TrueTime provides a bounded clock uncertainty interval: the current time is guaranteed to be within [earliest, latest] where the interval width is typically less than 10ms. Spanner uses this bounded uncertainty to implement external consistency — transactions are assigned commit timestamps that guarantee that a transaction which commits later in real time has a higher commit timestamp. Cross-region reads are served by waiting until TrueTime confirms that the read timestamp is safe — typically adding 5-10ms to read latency. Cross-region writes use two-phase commit with Paxos groups, adding the cross-region round-trip latency (80-200ms) to write latency.

CockroachDB achieves global consistency without specialised time hardware by using a hybrid logical clock (HLC) — a combination of physical time and logical counters that maintains causal ordering without requiring perfectly synchronised clocks. CockroachDB’s multi-region deployment model uses a “home region” concept for each row — reads from a row’s home region are served locally (low latency), writes to any row require consensus with the row’s home region (write latency includes cross-region round-trip if writing from a non-home region). The leaseholder — the Raft group member that serves reads for a range — can be configured to be in the same region as the majority of reads for that data, minimising read latency while maintaining global consistency.

Both systems demonstrate that global strong consistency is achievable but expensive: write latency is bounded below by the cross-region round-trip time, and reads must either be served from the data’s home region or pay a similar latency penalty. For use cases where strong consistency is non-negotiable — financial transactions, inventory management, compliance audit logs — this cost is justified. For use cases where bounded staleness is acceptable, regional replication with eventual consistency is dramatically cheaper and faster.

Key Takeaways

  1. Physics sets a non-negotiable minimum latency floor — US-East to EU-West is 80-100ms round-trip, and no engineering optimisation can reduce cross-region write latency below this physical constraint
  2. Active-passive provides simple consistency and no conflicts at the cost of cross-region write latency and minutes-long failover; active-active provides zero RTO and local write latency at the cost of inevitable write conflicts requiring explicit resolution strategies
  3. Regional isolation pins each user’s data to a home region — it satisfies data residency requirements, eliminates cross-region conflicts, and provides local write latency for users in their home region, at the cost of higher latency for cross-region access scenarios
  4. GeoDNS routes traffic to the nearest region at DNS resolution time with failover bounded by TTL; Anycast routes at the network layer via BGP with second-scale failover — use Anycast through CDN or global load balancer products for the fastest global routing and failover
  5. Data gravity means computation should move to data rather than data to computation — as datasets grow, the cost of moving data between regions becomes prohibitive, and analytics, machine learning, and processing workloads should run in the same region as their data
  6. GDPR and data residency compliance requires deliberate architecture from the start — log routing, analytics, backups, and third-party services must all enforce geographic data boundaries, not just the primary database
  7. Global strong consistency (Spanner, CockroachDB) is achievable but expensive — write latency is bounded below by cross-region round-trip time; for workloads that tolerate bounded staleness, regional replication with eventual consistency is dramatically cheaper and faster

Frequently Asked Questions (FAQ)

What is the difference between active-passive and active-active multi-region deployment?

In active-passive, one region handles all writes while passive regions replicate data and serve reads only. Failover requires promoting a passive region to active — typically 2-10 minutes. There are no write conflicts because only one region accepts writes at a time. In active-active, all regions simultaneously accept both reads and writes. When one region fails, traffic continues to survivors with no failover step — RTO is effectively zero. The cost is write conflicts: concurrent writes to the same data from different regions produce conflicting versions that must be resolved using last-write-wins, CRDTs, or application-level merge logic. Choose active-passive when consistency is critical and write latency can be bounded. Choose active-active when zero RTO is required and conflict resolution can be designed into the data model.

What is data residency and how does it affect distributed systems architecture?

Data residency is the requirement that specific data be stored and processed within a defined geographic boundary — typically a country or region. GDPR requires EU residents’ personal data to be processed within the EU without requiring separate legal mechanisms for cross-border transfer. Data residency requirements force architecture decisions that would otherwise be purely technical: which region stores which users’ data, how logs are routed, where analytics runs, and which third-party services are used (only those operating from compliant regions). Data residency is most cleanly implemented through regional isolation architecture, where each user’s data is pinned to their home region and never transferred to other regions.

What is data gravity and why does it matter for geo-distribution?

Data gravity is the principle that large datasets attract computation — as data grows, moving it becomes costly enough that it becomes more practical to move computation to the data than data to the computation. At cloud scale, transferring petabytes of data between regions costs tens of thousands of dollars and takes hours. Analytics, machine learning training, and batch processing workloads should run in the same region as their data rather than centralising computation in a single region. Data gravity influences multi-region architecture by making each region increasingly self-sufficient for compute-intensive workloads as regional data volumes grow.

What is the difference between GeoDNS and Anycast?

GeoDNS returns different DNS records based on the geographic origin of the DNS resolver, routing clients to their nearest regional endpoint at DNS resolution time. Failover is bounded by DNS TTL — clients cache the DNS record for the TTL duration and continue routing to a failed region until the cache expires. Anycast announces the same IP address from multiple locations simultaneously; BGP routes each client’s packets to the nearest announcement point automatically at the network layer. Failover happens in seconds as BGP reconverges when a location withdraws its announcement. GeoDNS is simpler and widely available through managed DNS services. Anycast provides faster failover and more accurate routing but requires BGP control through own ASN or CDN providers.

How do Google Spanner and CockroachDB achieve global consistency?

Google Spanner uses TrueTime — GPS and atomic clock-based globally synchronised time with a bounded uncertainty interval (typically less than 10ms) — to assign commit timestamps that guarantee external consistency across globally distributed nodes. Reads wait until TrueTime confirms the timestamp is safe, adding 5-10ms. Writes use cross-region Paxos consensus, adding the full cross-region round-trip latency. CockroachDB uses hybrid logical clocks to maintain causal ordering without specialised time hardware. Both systems guarantee linearisability globally at the cost of write latency bounded below by cross-region round-trip time — 80-200ms depending on region distances. This cost is justified for workloads requiring global strong consistency; for workloads tolerating bounded staleness, regional replication with eventual consistency is far less expensive.

When should I choose regional isolation over active-active?

Choose regional isolation when data residency compliance is required (GDPR, healthcare regulations, financial regulations), when your data model naturally partitions by user or tenant geography (each user’s data is accessed primarily from their home region), and when cross-region write conflicts would be difficult to resolve correctly for your use case. Regional isolation provides local write latency for users in their home region, no cross-region conflicts, and clean compliance boundaries. Choose active-active when you need zero RTO for any regional failure, when your users are globally distributed without clear geographic affinity, and when your data model can be designed to avoid or cleanly resolve write conflicts.

Continue the Series

Series home: Distributed Systems — Concepts, Design & Real-World Engineering

Part 5 — Scalability & Performance

Previous: ← 5.8 — Autoscaling Distributed Systems

Next: 5.10 — Cost and Capacity Planning at Scale →

Related posts from earlier in the series:

Discover more from Rahul Suryawanshi

Subscribe now to keep reading and get access to the full archive.

Continue reading