From Constraints to Communication

by
Home » Distributed Systems » Communication & Coordination » From Constraints to Communication

Distributed Systems Series — Part 2.0: Communication & Coordination

From Understanding the Problem to Engineering the Solution

Communication and Coordination in Distributed Systems are the mechanisms that allow systems to make progress despite the three constraints Part 1 established. Part 1 focused on something many engineers skip too quickly — the rules of the world in which distributed systems operate.

Networks are unreliable. Messages can be delayed, lost, duplicated, or delivered out of order. Silence is always ambiguous — a node that stops responding might have crashed, might be slow, or might be separated by a partition. Every retry mechanism, every idempotency key, every circuit breaker exists because of this constraint.

Nodes fail in partial and slow ways. Crash-recovery is the real-world failure model. Slow nodes are more dangerous than crashed ones. Gray failures pass health checks while producing incorrect results. Every redundancy pattern, every failure detector, every leadership lease exists because of this constraint.

Time cannot be trusted for ordering. There is no global clock. Timestamps reflect local clock readings, not causal truth. Events that appear simultaneous may be causally related in ways no timestamp can reveal. Every logical clock, every consistency model, every conflict resolution strategy exists because of this constraint.

These are not edge cases. They are normal operating conditions. Part 1 established what they are. Part 2 addresses the practical question: how do distributed systems cope with them?

Coping Mechanisms, Not Silver Bullets

The mechanisms covered in Part 2 — retries, idempotency, service discovery, distributed locks, logical clocks, coordination services — are frequently presented as features or engineering tools. They are better understood as coping mechanisms: responses to constraints that cannot be eliminated.

Retries exist because messages are lost and nodes are sometimes slow. Idempotency keys exist because retries produce duplicate deliveries. Circuit breakers exist because retrying into a failing service amplifies load rather than recovering it. Service discovery exists because network topology changes constantly and static IP configuration breaks continuously. Distributed locks exist because coordinating shared resources across independent nodes requires an explicit protocol. Logical clocks exist because physical timestamps cannot establish causal ordering.

None of these mechanisms make the underlying constraints disappear. The network is still unreliable after a retry. The node is still potentially slow after a circuit breaker opens. The clock is still unsynchronised after a Lamport timestamp is applied. What these mechanisms do is allow systems to make progress despite the constraints — to serve users correctly most of the time, degrade gracefully when they cannot, and recover automatically when conditions improve.

Distributed systems design is not about eliminating failures. It is about making progress despite them.

Why Communication Comes First

Before a distributed system can replicate data, elect a leader, enforce consistency, or tolerate failures, its components must first communicate. Communication is the foundation on which every higher-level distributed systems property is built.

The choice of communication pattern determines how failures propagate. A synchronous RPC call that blocks waiting for a response propagates the latency of every slow downstream dependency directly to the caller. An asynchronous message that is buffered in a queue decouples the producer’s availability from the consumer’s — but introduces the ordering and delivery semantics questions that Post 2.2 addresses. An event stream that is retained and replayable allows new consumers to catch up on history — but requires consumers to handle duplicate events correctly.

Communication also determines how failures are detected, or missed. A service that calls its dependencies synchronously will detect their failures through timeouts — but as Post 1.3 established, timeouts surface ambiguity rather than resolving it. A service that communicates through a message queue may not detect consumer failures for minutes or hours, depending on how queue depth is monitored.

Understanding the communication model — what guarantees it provides, what failures it can and cannot detect, what happens when the underlying network violates its assumptions — is the prerequisite for reasoning about everything that Part 2 covers.

The Eight Fallacies in Practice

L. Peter Deutsch’s eight fallacies of distributed computing, documented at Sun Microsystems in 1994, remain the most precise catalogue of the assumptions that cause distributed communication to fail. The first two fallacies — the network is reliable and latency is zero — are the direct subject of Post 1.3. Fallacies three through eight (bandwidth is infinite, the network is secure, topology doesn’t change, there is one administrator, transport cost is zero, the network is homogeneous) all manifest in communication design decisions.

Fallacy five — topology doesn’t change — is the direct motivation for service discovery. In a Kubernetes cluster, pod IPs change on every restart. In an auto-scaling group, instances are created and destroyed continuously. Any communication pattern that depends on stable addresses breaks the moment topology changes, which in production is constant. Post 2.3 covers how service discovery addresses this.

Fallacy six — there is one administrator — explains why distributed coordination is expensive. When multiple teams own multiple services with different deployment schedules, no single authority can coordinate all changes. Distributed locks, consensus protocols, and coordination services exist precisely because there is no central administrator who can simply serialise all operations. Post 2.4 and Post 2.6 cover the mechanisms that replace central coordination with distributed protocols.

The complete eleven fallacies — including the three Richards and Ford additions from 2020 — are at The Eight Fallacies of Distributed Computing.

What Part 2 Covers

Eight posts cover the complete Communication and Coordination stack, each addressing a specific constraint or mechanism that builds on what came before.

Post 2.1 — Communication Fundamentals establishes the vocabulary: RPC vs REST vs async messaging, the eight fallacies in practice, and how a single slow downstream service cascades latency through a synchronous call chain.

Post 2.2 — Reliability and Retries addresses the network unreliability constraint directly. Retries are dangerous without exponential backoff and jitter. Idempotency keys are the production standard for safe retries. Circuit breakers prevent retry storms from collapsing failing services. The 2019 AWS SQS retry storm is the production incident that grounds these mechanisms in real consequence.

Post 2.3 — Naming and Service Discovery addresses the topology-changes constraint. Client-side vs server-side discovery, DNS-based discovery, Kubernetes Services and kube-dns, and how health-check-driven routing prevents traffic from reaching unhealthy instances.

Post 2.4 — Coordination and Distributed Locks addresses the shared resource problem. Lease-based locks, fencing tokens that prevent stale lock holders from writing after their lease expires, and why distributed locks are harder to implement correctly than they appear — the split-brain risk that the GitHub 2012 GC pause incident illustrated.

Post 2.5 — Logical Clocks and Time addresses the untrusted time constraint in production systems. Lamport clocks, vector clocks, and Hybrid Logical Clocks — building on the theoretical foundation from Post 1.5 and showing how CockroachDB, MongoDB, and Cassandra implement them.

Post 2.6 — Coordination Services covers ZooKeeper, etcd, and Consul — the production systems that implement distributed coordination correctly so that applications do not have to. What they provide (leader election, distributed configuration, service registry), what they cost (consensus overhead, write throughput ceiling), and when to use each.

Post 2.7 — Engineering Guidelines synthesises Part 2 into ten practical principles and a design review checklist that engineers can apply immediately in architecture reviews and production system design.

Key Takeaways

  1. Part 2 shifts from understanding the constraints of distributed systems to engineering coping mechanisms — retries, service discovery, coordination, and logical clocks are responses to constraints that cannot be eliminated, not features added on top of a working system
  2. Communication is the foundation of all higher-level distributed systems properties — before a system can replicate data, elect a leader, or enforce consistency, its components must first communicate reliably despite an unreliable network
  3. Every Part 2 mechanism exists because of a specific Part 1 constraint — retries because networks lose messages, idempotency because retries duplicate deliveries, service discovery because topology changes, distributed locks because coordination requires explicit protocols without a central authority
  4. The eight fallacies of distributed computing catalogue the communication assumptions that cause production systems to fail — particularly fallacy five (topology does not change) and fallacy six (there is one administrator), which are the direct motivation for service discovery and distributed coordination
  5. Distributed systems design is not about eliminating failures but about making progress despite them — correct behaviour most of the time, graceful degradation when that is not possible, and automatic recovery when conditions improve
  6. Coordination is expensive and should be avoided wherever possible — every coordination step reduces the system’s maximum throughput ceiling through Amdahl’s Law, and the design goal is to coordinate only where genuinely necessary
  7. Part 2’s eight posts build sequentially — communication fundamentals before retries, retries before service discovery, service discovery before coordination, coordination before coordination services, all before the engineering guidelines that synthesise the complete picture

Frequently Asked Questions (FAQ)

What is the difference between Part 1 and Part 2 of this series?

Part 1 establishes the environment — the three unavoidable constraints that every distributed system operates within: unreliable networks, partial node failures, and untrusted time. Part 2 establishes the coping mechanisms — the communication patterns, retry strategies, service discovery approaches, coordination protocols, and logical clock implementations that allow distributed systems to make progress despite those constraints. Part 1 answers “why is this hard?” Part 2 answers “what do we do about it?”

Why does Part 2 start with communication rather than consistency or consensus?

Because communication is the prerequisite for everything else. A distributed system cannot replicate data without communication. It cannot detect failures without communication. It cannot elect a leader, enforce a lock, or agree on a value without communication. Every higher-level distributed systems property — consistency, fault tolerance, coordination — is built on communication primitives. Understanding what guarantees communication provides (and does not provide) is the foundation for understanding why consistency models and consensus algorithms are designed the way they are.

What does “coping mechanism” mean in the context of distributed systems?

A coping mechanism is an engineering response to a constraint that cannot be eliminated. Retries are a coping mechanism for message loss — they increase the probability of delivery without eliminating the underlying unreliability. Idempotency keys are a coping mechanism for duplicate delivery — they allow retries to be safe without preventing duplication from occurring. Circuit breakers are a coping mechanism for cascading failures — they limit damage without preventing the underlying service from failing. None of these mechanisms make the network reliable, nodes stable, or clocks synchronised. They make systems behave correctly despite those permanent conditions.

Is coordination always necessary in distributed systems?

No — and avoiding unnecessary coordination is one of the most important scalability principles in distributed systems. Coordination requires multiple nodes to agree, which requires communication rounds, which adds latency and creates availability dependencies. Every coordination step reduces the system’s maximum throughput ceiling (Amdahl’s Law). The design goal is to use coordination only where it is genuinely required — for mutual exclusion, leader election, distributed transactions — and to use coordination-free designs everywhere else. Post 2.7’s engineering guidelines cover when coordination is necessary and when it can be replaced with idempotent, eventually consistent alternatives.

What production incidents ground the Part 2 mechanisms?

Each post in Part 2 is anchored in a named production incident. Post 2.2 covers the 2019 AWS SQS retry storm — a retry pattern without backoff that amplified load rather than recovering it. Post 2.4 covers the 2012 GitHub MySQL GC pause — a slow node holding a leadership lock past its useful expiry, creating a split-brain window when it resumed. Post 2.6 covers the 2013 etcd split-brain incident — a subtle bug in leader transition that allowed two nodes to simultaneously believe they were leader. These are not hypothetical failure modes. They are documented production failures at organisations with mature engineering practices.

How does Part 2 connect to Parts 3, 4, and 5?

Part 2 establishes the communication and coordination primitives that Parts 3, 4, and 5 build on. Part 3’s replication models depend on the message delivery semantics from Post 2.2 and the logical clock foundations from Post 2.5. Part 3’s consensus algorithms (Raft, Paxos) are built on the coordination concepts from Posts 2.4 and 2.6. Part 4’s failure detection builds on the heartbeat mechanisms introduced in Part 2. Part 5’s backpressure and load balancing extend the retry and service discovery patterns from Posts 2.2 and 2.3. Part 2 is not a standalone section — it is the mechanism layer that everything else in the series depends on.

Continue the Series

Before you start Part 2 — one honest observation

The gap between reading about these mechanisms and using them correctly in production is wider than most engineers expect. Not because the concepts are hard to understand — they are not, once the mental model clicks. But because the failure modes only surface under the specific combination of load, timing, and component interaction that you cannot fully reproduce in development or staging.

I have seen retry storms cause outages at organisations that understood retry theory perfectly. I have seen service discovery misconfigurations route traffic to failed instances for minutes after the instance was gone — because the health check was a liveness check masquerading as a readiness check.

The posts in Part 2 are grounded in these failure modes because understanding the mechanism is not enough. Understanding where it breaks — and why it breaks under the precise conditions you have not planned for — is what separates a system that is reliable from one that is reliable until it matters.

Series home: Distributed Systems — Concepts, Design & Real-World Engineering

Part 1 — Foundations

Part 2 — Communication & Coordination

Next: 2.1 — Communication Fundamentals in Distributed Systems →

Part 2 overview and reading guide: Communication & Coordination

Discover more from Rahul Suryawanshi

Subscribe now to keep reading and get access to the full archive.

Continue reading