Distributed Systems Series — Part 2.6: Communication & Coordination
Why Coordination Services Exist
Coordination services in distributed systems — ZooKeeper, etcd, and Consul — exist because of a problem that Post 2.4 made precise: distributed locks and leader election are hard to implement correctly, and dangerous when implemented wrong. The failure modes — split-brain, stale locks, clock skew, partial writes under partition — require deep expertise and extensive testing to handle correctly. Re-implementing these mechanisms inside every application that needs them would be expensive, error-prone, and almost certainly wrong in subtle ways that only surface under production failure conditions.
A coordination service is a purpose-built, highly reliable distributed system whose sole job is to help other distributed systems coordinate safely. It absorbs the complexity of consensus, replication, and failure handling so application teams can use proven primitives rather than build them. By now, Part 2 has built a clear picture of what those primitives must handle: unreliable communication, retry-induced duplication, untrustworthy time, eventually accurate discovery, and the precise failure modes of distributed locks. Coordination services are the engineering response to all of it, packaged into a system you can operate rather than prove.
What All Coordination Services Provide
Despite their different histories and designs, every serious coordination service provides the same foundational guarantees. These guarantees are what justify their operational cost and what make them useful for coordination workloads where eventual consistency is simply not acceptable.
Strong consistency. All clients observe updates in the same order. When a write succeeds, every subsequent read — from any client on any node — will see that write. There is no stale state, no replica lag, no eventual catch-up. This is the property that makes coordination services useful for locks and leader election: when you acquire a lock, every other node in the system must see you as the lock holder immediately, not eventually.
Strong consistency is achieved through leader-based replication with consensus protocols. Every write goes through a leader, which replicates the write to a quorum of followers before acknowledging it. No write is acknowledged until a majority of nodes have durably recorded it. This is fundamentally different from eventually consistent systems like DynamoDB or Cassandra in their default configuration, which acknowledge writes before all replicas have them.
Linearisable writes. Once a write is acknowledged, all future reads will reflect it. A write that succeeds will never be lost — not through a leader crash, not through a follower falling behind. This is specifically what makes coordination services safe for critical operations. If you store the address of the current primary database in a coordination service, you need that address to be correct the moment you read it, not eventually correct after replicas converge.
Fault tolerance through quorum replication. Coordination services replicate their state across an odd number of nodes — typically three or five. A cluster of three tolerates one node failure. A cluster of five tolerates two. As long as a majority of nodes are reachable, the service continues operating. When a node fails and recovers, it catches up from the other nodes. The application using the coordination service sees no interruption.
Consistency over availability during partitions. When a network partition occurs, a coordination service chooses consistency. The minority partition — the group that cannot reach a quorum — stops serving writes rather than risk serving incorrect data. This is the correct behaviour: brief unavailability is recoverable, split-brain is not. Engineers who expect a coordination service to remain fully available during a partition are misunderstanding its design contract. This is the CAP theorem’s CP choice, covered in Post 3.4, applied deliberately.
Apache ZooKeeper: The Original
ZooKeeper was developed at Yahoo in the mid-2000s and open-sourced through the Apache Software Foundation. It was the first widely adopted coordination service in production distributed systems, and its design shaped how the industry thinks about the problem.
ZooKeeper’s data model is a hierarchical namespace of znodes — nodes in a tree structure similar to a filesystem. Each znode stores a small amount of data and can have children. The path /services/payments/leader might store the address of the current payments service leader. /config/database/host might store the current primary database address.
Two znode types make ZooKeeper particularly powerful for coordination. Ephemeral znodes exist only as long as the client session that created them remains active. When a client disconnects — whether cleanly or through a crash — its ephemeral znodes are automatically deleted. This is the natural mechanism for leader election: a candidate creates an ephemeral znode to claim leadership. When the leader fails, its session expires, the ephemeral znode disappears, and other candidates can detect the vacancy and compete. Watches allow clients to register for notifications when a znode changes — without polling. A client watching /services/payments/leader is notified immediately when that znode is created, modified, or deleted.
ZooKeeper uses the ZAB protocol (ZooKeeper Atomic Broadcast) — a consensus protocol similar to Paxos — to replicate writes across its ensemble. All writes go through the elected ZooKeeper leader, which broadcasts them to followers. A write is only acknowledged once a quorum of followers have persisted it.
ZooKeeper in production: Apache Kafka used ZooKeeper for broker registration, topic metadata, and partition leader election from its initial release until KRaft was introduced. Apache HBase uses it for master election and region server coordination. Apache Hadoop uses it for NameNode HA. For much of the 2010s, if you ran a serious Apache ecosystem deployment, you ran ZooKeeper.
ZooKeeper’s limitations became apparent as cloud-native architectures emerged. Its watch semantics are one-time — a watch fires once and must be re-registered, which introduces a subtle race condition between the watch expiry and re-registration during which events can be missed. Its hierarchical data model is more complex than most coordination use cases actually need. And its operational overhead — Java heap tuning, ensemble sizing, GC pause sensitivity — adds maintenance cost that lighter alternatives avoid. For new systems with no Apache ecosystem dependency, etcd or Consul are almost always the better starting point today.
etcd: Built for the Cloud-Native Era
etcd was developed at CoreOS in 2013 as the configuration backbone for their Container Linux operating system. It was later adopted as the primary data store for Kubernetes, which is the context in which most engineers encounter it today.
Where ZooKeeper uses a hierarchical namespace, etcd uses a flat key-value model with prefix-based range queries. Keys are arbitrary byte strings. The key /registry/pods/default/my-app stores the serialised state of a Kubernetes pod. Range queries on key prefixes allow the Kubernetes API server to efficiently watch all pods, all services, or all resources of any type — a pattern that ZooKeeper’s hierarchical model supports less efficiently.
etcd uses the Raft consensus algorithm for replication — specifically designed to be understandable and implementable correctly, in contrast to Paxos which is notoriously difficult to implement without subtle bugs. The full treatment of Raft is in Post 3.7. The key operational difference: Raft’s design makes leader election, log replication, and membership changes well-specified and auditable — properties that matter enormously when diagnosing etcd behaviour under a 3am Kubernetes incident.
etcd’s watch API is more robust than ZooKeeper’s. Watches in etcd are persistent and revision-based — a client can watch from a specific revision, ensuring it does not miss events that occurred between its last watch and the current state. This eliminates the race condition inherent in ZooKeeper’s one-time watch model and is the reason etcd’s watches are considered production-grade for event-driven coordination.
etcd exposes leases for lock and leader election patterns. A client creates a lease with a TTL and attaches keys to it. When the lease expires — because the client stopped renewing it — all attached keys are automatically deleted. This is etcd’s equivalent of ZooKeeper’s ephemeral nodes, and it serves the same coordination purpose with the fencing token integration that Post 2.4 established as essential.
etcd in production: every Kubernetes cluster runs etcd as its authoritative state store. All Kubernetes objects — pods, services, deployments, config maps, secrets — are stored in etcd. The Kubernetes API server is essentially a strongly consistent cache in front of etcd with an access control layer. At any given moment, millions of production Kubernetes clusters worldwide are running on etcd. Patroni — the most widely deployed PostgreSQL high-availability solution — uses etcd (or Consul or ZooKeeper) for its distributed leader election, making it the coordination service of choice for database HA in cloud-native environments.
etcd’s limitations are primarily around scale. It is designed for coordination workloads — small values, infrequent writes, strong consistency — not high-throughput data storage. Kubernetes clusters with tens of thousands of objects and high object churn can stress etcd significantly, requiring dedicated hardware, careful tuning, and regular compaction of revision history to maintain performance.
HashiCorp Consul: Coordination Meets Service Discovery
Consul was developed by HashiCorp and released in 2014. It occupies a distinct position from ZooKeeper and etcd: rather than being a pure coordination service, Consul combines service discovery, health checking, and key-value storage in a single system. For environments where service discovery and coordination are tightly coupled — which describes most non-Kubernetes microservices deployments — Consul eliminates the need to run and operate two separate systems.
Consul’s service catalog is its primary abstraction. Services register with Consul and Consul tracks their addresses, ports, tags, and health status. Services query the catalog through DNS (payments.service.consul) or the HTTP API. Only instances passing health checks are returned by default — the real-time, health-filtered discovery that Post 2.3 established as the standard for dynamic distributed systems.
Consul’s health checking is more sophisticated than etcd’s lease-based approach — HTTP checks, TCP checks, script checks, and gRPC checks, each running at configurable intervals with passing/warning/critical states. Its key-value store serves the same coordination role as etcd for distributed locks and configuration, though with less robust watch semantics — no revision-based watch history.
Consul uses Raft for its server cluster with the same quorum properties as etcd. Its multi-datacenter support — each datacenter runs its own Raft cluster, cross-datacenter queries federated through WAN gossip — is a capability neither ZooKeeper nor etcd provides natively, making Consul the natural fit for globally distributed deployments where service discovery needs to span regions.
Consul in production: HashiCorp Vault uses Consul as its default storage backend and for HA coordination. Large non-Kubernetes microservices deployments at organisations including Cloudflare, Barclays, and Citrix have been publicly documented. Consul Connect provides a service mesh implementation built on Consul’s service catalog as its control plane.
How Coordination Services Use Consensus
Every coordination service described above is built on a consensus algorithm. Consensus is what makes strong consistency possible across multiple nodes — it is the mechanism by which a group of nodes agrees on a single value despite failures and message delays. The full treatment of consensus — why it is hard, what FLP impossibility means in practice, and how Raft and Paxos achieve it — is in Post 3.6 and Post 3.7.
The key point here: coordination services are only possible because consensus algorithms exist to make strong consistency achievable across multiple nodes in the presence of failures. Raft — used by etcd and Consul — structures consensus around an explicit elected leader who accepts all writes, replicates them to a quorum, and acknowledges only after majority confirmation. ZAB — used by ZooKeeper — is conceptually similar but predates Raft and differs in its recovery protocol during leader failover. Both provide the same fundamental guarantee: a write acknowledged by the coordination service will survive any single node failure.
Choosing Between ZooKeeper, etcd, and Consul
The choice is usually determined by your existing infrastructure rather than abstract capability comparison. Understanding the trade-offs helps evaluate that fit clearly.
Choose etcd if you are running Kubernetes or building infrastructure that integrates with it. etcd is already present in every Kubernetes cluster, is well-understood by the Kubernetes community, and its revision-based watch API is the most robust of the three for event-driven coordination. For new cloud-native systems built on Kubernetes, etcd is the natural default — and if you are running Patroni for PostgreSQL HA, etcd is already the recommended coordination backend.
Choose Consul if you need service discovery and coordination together in a non-Kubernetes environment, or if you operate across multiple datacenters and need native multi-region support. Consul’s integrated health checking and DNS interface make it operationally simpler than running etcd plus a separate service discovery system.
Choose ZooKeeper if you are operating in the Apache ecosystem — Kafka before KRaft, HBase, Hadoop — where ZooKeeper is already a dependency and its operational patterns are well-understood by your team. For new systems with no Apache ecosystem dependency, ZooKeeper’s operational complexity rarely justifies choosing it over etcd or Consul.
The one rule that applies regardless of choice: treat your coordination service as critical infrastructure. Run a minimum of three nodes (five for higher availability requirements). Monitor quorum health actively. Perform regular backup and restore testing. A coordination service that becomes unavailable can cascade into a platform-wide outage — every system depending on it for leader election or configuration cannot function correctly without it.
When to Use Coordination Services — and When Not To
Coordination services are powerful but expensive. Used correctly, they centralise complexity. Used incorrectly, they become a bottleneck.
Good use cases: leader election for database primaries, job schedulers, and stream processors; distributed locks for critical mutual exclusion with fencing tokens at the resource; configuration management for small, infrequently changing values that all nodes must see consistently; cluster membership tracking; service discovery in non-Kubernetes environments.
Poor use cases: high-volume application data — coordination services are not databases; low-latency request paths — a coordination service round-trip adds tens of milliseconds; frequently changing large datasets — etcd and ZooKeeper have practical limits on value size and write throughput that application data routinely exceeds.
A useful rule: if your application requires a coordination service call on every request, your architecture has a problem. Coordination should happen at system startup, during failure events, and during configuration changes — not in the hot path of request handling.
Key Takeaways
- Coordination services exist to centralise the complexity of distributed coordination — strong consistency, fault-tolerant replication, and safe primitives for locks and leader election — so application teams use proven building blocks rather than build their own incorrect ones
- All serious coordination services provide the same core guarantees: strong consistency, linearisable writes, quorum-based fault tolerance, and consistency over availability during partitions — these guarantees are what make them useful for coordination workloads where eventual consistency is unacceptable
- ZooKeeper pioneered distributed coordination with ephemeral znodes and watches, and remains the foundation of the Apache ecosystem — etcd is the Kubernetes-native choice with the most robust revision-based watch semantics — Consul integrates service discovery and coordination for multi-datacenter non-Kubernetes environments
- All three are built on consensus algorithms — ZAB for ZooKeeper, Raft for etcd and Consul — which make strong consistency possible across multiple nodes in the presence of failures; the full Raft and Paxos treatment is in Part 3
- Coordination services are for coordination workloads — small values, infrequent writes, failure events and configuration changes — not for high-throughput application data or low-latency per-request paths
- Treat your coordination service as critical infrastructure — three or five nodes minimum, active quorum monitoring, regular backup and restore testing — its failure cascades to every system that depends on it for leader election or configuration
- The choice between ZooKeeper, etcd, and Consul is primarily determined by existing infrastructure — etcd for Kubernetes environments, Consul for non-Kubernetes multi-datacenter deployments, ZooKeeper for Apache ecosystem dependencies
Frequently Asked Questions (FAQ)
What is a coordination service and why do I need one?
A coordination service is a purpose-built distributed system that provides strong consistency, fault-tolerant replication, and safe coordination primitives — distributed locks, leader election, configuration management, and cluster membership tracking. You need one because implementing these correctly from scratch requires deep expertise in consensus algorithms and distributed systems failure modes that most application teams do not have and should not need to develop. ZooKeeper, etcd, and Consul are the result of years of research and production hardening. Using them is almost always better than building equivalent functionality yourself, where the failure modes will only surface under the precise production conditions you have not tested for.
What is the difference between ZooKeeper, etcd, and Consul?
ZooKeeper is the oldest — a hierarchical namespace with ephemeral nodes and one-time watches, built on the ZAB consensus protocol, widely used in the Apache ecosystem (Kafka pre-KRaft, HBase, Hadoop). etcd is the Kubernetes-native choice — a flat key-value store with persistent revision-based watches, built on Raft, optimised for cloud-native coordination workloads. Consul combines service discovery and coordination in a single system with native multi-datacenter support, making it the natural choice for non-Kubernetes microservices environments where discovery and coordination are tightly coupled. All three provide strong consistency and quorum-based fault tolerance — the choice is determined primarily by your existing infrastructure.
Why do coordination services choose consistency over availability during partitions?
Because the consequences of inconsistency in coordination workloads are catastrophic. If a coordination service returns stale data during a partition, two nodes might simultaneously believe they are the leader — split-brain. Two processes might simultaneously believe they hold the same lock — data corruption. In a financial transaction system, split-brain in a payment processing coordinator is not a data quality issue — it is a compliance event and a potential double-charge. For these use cases, brief unavailability is recoverable. Incorrect behaviour may not be. This is the CAP theorem’s CP choice, applied deliberately to the specific requirements of coordination.
Can I use etcd or ZooKeeper as a general application database?
No. Coordination services are designed for small values, infrequent writes, and strong consistency — not high-throughput data storage. etcd recommends keeping individual values under 1.5MB and total data under 8GB. ZooKeeper limits znode data to 1MB. Both are optimised for read-heavy, write-rare patterns of coordination workloads. Using them for application data will hit throughput limits quickly, increase operational complexity significantly, and make both the application and the coordination infrastructure harder to manage. The distinction matters: your Kubernetes cluster state lives correctly in etcd, your application’s user records do not.
What is the difference between ZooKeeper watches and etcd watches?
ZooKeeper watches are one-time — they fire once when the watched znode changes and must be explicitly re-registered to continue watching. This creates a race condition: events that occur between the watch firing and the client re-registering can be missed. etcd watches are persistent and revision-based — a client can watch from a specific revision number and receive all events from that point forward, even if the watch was temporarily disconnected. The client will never miss an event that occurred between reconnections. This makes etcd’s watch semantics significantly more reliable for event-driven coordination patterns that cannot tolerate missed events.
What is Raft and why do etcd and Consul use it instead of Paxos?
Raft is a consensus algorithm designed specifically for understandability and correct implementation. It structures consensus around an explicit elected leader, a replicated log with clear ordering rules, and well-specified procedures for leader election and membership changes. Paxos — the earlier consensus algorithm — is notoriously difficult to implement correctly; the original papers leave many practical details underspecified, and every real Paxos implementation diverges from the theoretical description in ways that are hard to audit. etcd and Consul use Raft because its correctness properties are well-specified, its behaviour under failure is predictable, and its state is directly observable — all of which matter when diagnosing a production incident at 3am. The full comparison of Raft and Paxos is in Post 3.7.
What running etcd in production actually taught me
The first time I truly understood what a coordination service was doing — not intellectually but operationally — was during an etcd performance investigation at IDFC First Bank. We were running a Kubernetes cluster with a significantly higher than average object churn rate: financial services processing means pods being created, modified, and deleted at a rate that most workloads do not approach. etcd’s revision history was growing faster than our compaction schedule could manage. Watch events were backing up. The Kubernetes API server was seeing increased latency on list operations. Nothing was broken, but the trajectory was clear.
The investigation revealed something that the documentation mentions but does not emphasise: etcd is a coordination service, not a database. The moment you start treating it like a high-throughput data store — even indirectly, through the volume of Kubernetes object updates it is recording — you have misused it. We adjusted our compaction schedule, tuned our resource quotas to reduce unnecessary pod churn, and moved some workloads to a dedicated etcd cluster. The lesson stayed with me: know what a coordination service is for, and do not push it beyond that boundary.
At Jio Platforms, the coordination problem was at a different layer — service registration and health-check propagation at the scale. The moment a service instance failed, downstream services needed to stop routing to it. The gap between “the health check failed” and “the coordination service has propagated the failure and callers have received the update” is not zero, and at scale it is not negligible. We instrumented that gap carefully. The mean was acceptable. The p99 was not. Reducing the p99 required understanding exactly how Consul’s gossip propagation worked under load — not just that it worked.
The practical thing I take from both experiences: treat your coordination service as the most important single piece of infrastructure in your distributed system. Not because it handles the most traffic — it handles the least. But because every system that depends on it for leader election, configuration, or discovery cannot function correctly without it. Its availability budget should be tighter than anything else you operate. Its failure modes should be the ones you have tested most thoroughly. And the moment you start wondering whether you could use it for something it was not designed for — application data, high-throughput writes, per-request coordination — stop and ask whether you actually need a database instead.
Part 3 is where the foundations from Parts 1 and 2 get applied to the hardest problem in distributed systems: how do you store and replicate data correctly across multiple nodes, with strong consistency guarantees, at scale? Post 3.1 starts from first principles — why replication is necessary at all — and the part builds through consistency models, CAP theorem, quorums, consensus algorithms, and performance trade-offs. Everything in Part 2 was about how systems communicate and coordinate. Part 3 is about what they agree on and why that agreement is so hard to achieve.
Series home: Distributed Systems — Concepts, Design & Real-World Engineering
Part 2 — Communication & Coordination (complete)
- 2.0 — From Constraints to Communication
- 2.1 — Communication Fundamentals in Distributed Systems
- 2.2 — Reliability and Retries in Distributed Systems
- 2.3 — Naming and Service Discovery
- 2.4 — Coordination and Distributed Locks
- 2.5 — Logical Clocks and Time
- 2.6 — Coordination Services: ZooKeeper, etcd and Consul
- 2.7 — Engineering Guidelines: Communication and Coordination
Previous: ← 2.5 — Logical Clocks and Time in Distributed Systems
Next: 2.7 — Engineering Guidelines: Communication and Coordination →
Coming in Part 3: Replication, Consistency and Consensus — Where these coordination principles are applied at scale across distributed data systems.
Not read Part 1 yet? Start with 1.1 — What Is a Distributed System (Really)?