Reliability and Retries in Distributed Systems

by
Home » Distributed Systems » Communication & Coordination » Reliability and Retries in Distributed Systems

Distributed Systems Series — Part 2.2: Communication & Coordination

Reliability Is Not the Absence of Failure

Reliability and retries in distributed systems are the mechanisms that allow systems to make progress despite the network unreliability established in Post 1.3 and the communication uncertainty established in Post 2.1. Networks drop packets. Nodes slow down or restart. Requests time out. Reliability is not the property of a system that never fails — it is the property of a system that makes correct progress despite failures.

The most common response to a failed request is simple: retry it. At first glance, retries seem harmless. If something failed, try again. In distributed systems, retries are not just a recovery mechanism — they are a design decision with system-wide consequences that must be coordinated across client retry policies, server-side idempotency, and infrastructure-level circuit breaking. This post covers all three layers and how they must work together.

Why Retries Exist — and Why They Are Dangerous

Retries exist because of the fundamental ambiguity that Post 2.1 established: when a request times out, the caller cannot determine whether the request never reached the server, the server processed it but the response was lost, or the server is still processing it. All three cases look identical from the caller’s perspective. Retrying is a way to make progress despite this uncertainty — but uncertainty cuts both ways.

Consider what happens under load. A service slows down. Clients time out and retry. Retries increase load on the already-degraded service. The service slows further. More clients time out. More retries fire. This positive feedback loop is a retry storm — and it turns a slow degradation into a complete outage. Each individual client is behaving rationally. The collective behaviour is catastrophic.

How AWS SQS Experienced a Retry Storm in 2019

In 2019, Amazon Simple Queue Service experienced a service disruption in US-EAST-1 that illustrated retry amplification precisely. An increase in error rates from one internal component caused clients — both internal AWS services and external customers — to begin retrying. The retries increased load on the already-degraded component. The increased load produced more errors. The errors produced more retries.

What began as a partial degradation in one component became a cascading failure across multiple services that depended on SQS. Each individual client was behaving rationally — retrying a failed request is correct behaviour in isolation. The collective behaviour was catastrophic because retry policies were not coordinated with the health of the system being called.

Amazon’s post-incident review identified two root causes beyond the initial trigger. Retry policies across client libraries were not using exponential backoff with jitter — they retried at fixed intervals, synchronising the retry waves and maximising load spikes. And there were insufficient circuit breakers between dependent services to shed load before the cascade propagated. This incident is why retry design is a system-level concern, not a client-level one.

Exponential Backoff: The First Required Component

The retry storm has a direct mechanical cause: retries firing at fixed intervals. When thousands of clients all retry at the same cadence, their requests arrive in synchronised waves that hammer a degraded service at exactly the moment it is trying to recover.

Exponential backoff breaks this by making each successive retry wait twice as long as the previous one:

wait_time = min(base_delay × 2^attempt, max_delay)

Where base_delay is the starting interval (commonly 100–500ms), attempt is the retry count starting from 0, and max_delay is a ceiling that prevents waits from growing unbounded (commonly 30–60 seconds). The delay grows fast enough that a service under pressure gets progressively more recovery time with each retry round. A retry budget — a maximum number of attempts — sits alongside this to ensure clients eventually give up rather than retrying indefinitely.

Jitter: The Second Required Component

Exponential backoff alone is not sufficient. When a service restarts and becomes available, every client that timed out during the outage has been waiting — and every client’s backoff timer fires at roughly the same moment, because they all started failing at the same time. The first wave of retries hits the freshly recovered service simultaneously. If that wave exceeds capacity, the service degrades again immediately. This is the thundering herd problem — a specific variant of the retry storm that occurs at recovery time rather than at degradation time.

Jitter solves it by randomising the wait time within each backoff window:

wait_time = random_between(0, min(base_delay × 2^attempt, max_delay))

With full jitter, clients that all started failing simultaneously retry at different moments, spreading recovery load across a time window rather than concentrating it in a single spike. AWS published a detailed analysis in 2015 comparing fixed retry, exponential backoff, and full jitter across simulated workloads. Full jitter consistently produced the lowest server load during recovery and the fastest stabilisation time. AWS SDK clients, gRPC’s built-in retry policy, and most production HTTP client libraries apply full jitter by default.

The Complete Production Retry Policy

A production-ready retry policy requires four components working together. Exponential backoff with full jitter as described above. A retry budget — a maximum number of attempts, typically 3 to 5 for synchronous calls, after which the caller returns an error rather than continuing to retry. Retryable error classification — only retry errors that indicate temporary unavailability (network timeouts, connection refused, HTTP 503, HTTP 429 after honouring the Retry-After header) and never retry errors that indicate a permanent problem with the request (HTTP 400, 401, 403, 422) or definitive server failures. And idempotency at the server, covered in the next section — without server-side idempotency, retries risk duplicate execution regardless of how well the client retry policy is designed.

Without all four components, a retry strategy that appears robust in testing will fail under exactly the production conditions where reliability is most needed.

Idempotency: Making Retries Safe at the Server

Because retries can cause duplicate execution, systems rely on idempotency — the property that executing an operation multiple times produces the same result as executing it once. As Martin Kleppmann emphasises in Designing Data-Intensive Applications, idempotency is one of the most important tools for building reliable distributed systems. It does not prevent retries from occurring. It limits the damage they can cause.

Operations fall into two categories. Read operations are naturally idempotent — reading the same data twice produces no side effects. State-changing operations — writes, payments, sends, allocations — are not naturally idempotent and require explicit design to become so.

Idempotency Keys: The Production Implementation

The standard production technique for making state-changing operations safe to retry is the idempotency key: a unique identifier generated by the client and attached to every request. The server uses this key to deduplicate requests — if it has already processed a request with that key, it returns the stored result rather than executing the operation again.

The pattern works as follows. The client generates a unique key before making the request — commonly a UUID or a hash of the operation’s inputs. The client includes this key in the request (typically as an Idempotency-Key header). The server checks its deduplication store before processing — if the key exists, it returns the stored response immediately. If the key does not exist, the server processes the request, stores the result against the key, and returns the response. The client retries on timeout using the same key — the server deduplicates safely.

Stripe’s payment API is the canonical production example. Every charge request accepts an Idempotency-Key header. If a client times out after submitting a charge and retries with the same key, Stripe returns the result of the original charge rather than charging the card twice. The key is stored for 24 hours — long enough to cover any realistic retry window. This single mechanism protects against duplicate charges across millions of transactions per day without requiring the client to know whether its first request succeeded.

Three implementation details matter significantly in production. Key scope — the key must be scoped to a specific operation type; the same UUID used for a charge should not be reused for a refund. Storage durability — the deduplication store must be durable; an in-memory cache that restarts clears all keys, allowing duplicates during the restart window; Redis with persistence or a database table with a TTL index are the standard approaches. Transactional atomicity — idempotency key storage must happen inside the same transaction as the operation itself; if the server processes the operation but crashes before storing the key, the next retry will be processed again, which means idempotency key deduplication must be atomic with the operation or the operation must be naturally idempotent regardless.

Retries also interact with ordering. A retried request can arrive after a newer request, out of order, or while a previous attempt is still processing. Because there is no global clock — as established in Post 1.5 — systems cannot rely on timing alone to determine which request is authoritative. Retry logic must be designed with loose ordering assumptions, explicit versioning where ordering matters, and clear state transitions that are correct regardless of arrival order.

Circuit Breakers: Protecting Against Sustained Failures

Retries with exponential backoff handle brief, transient failures — a momentary network hiccup, a brief service restart, a short-lived resource contention. For sustained failures — a downstream service degraded for minutes or hours — retries continue sending load to a service that cannot recover under that load. The circuit breaker pattern solves this.

Named deliberately after the electrical component that cuts power when current exceeds safe limits, a circuit breaker wraps calls to a downstream service and monitors their failure rate. When failures exceed a configured threshold, the circuit opens — subsequent calls are rejected immediately without attempting to reach the downstream service. After a timeout, it probes for recovery with a limited number of test requests.

A circuit breaker operates in three states. Closed — normal operation. Calls pass through to the downstream service. The circuit breaker tracks success and failure rates in a rolling window. As long as the failure rate stays below the configured threshold (commonly 50% in a 10-second window), the circuit remains closed. Open — fast failure mode. When the failure rate exceeds the threshold, the circuit opens. All calls are immediately rejected with an error, without network contact. This serves two purposes: it protects the caller’s thread pool from being exhausted waiting for timeouts, and it gives the downstream service a complete rest from incoming load — the condition it most needs to recover. The caller must have a fallback for the open state — a cached response, a degraded response with reduced features, or a clear error the user interface can handle gracefully. Half-Open — recovery probe. After a configurable timeout (commonly 30 to 60 seconds), the circuit allows a small number of test requests through. If they succeed, the circuit closes and normal operation resumes. If they fail, the circuit reopens and the timeout resets. The half-open state prevents returning to full load before the downstream service is stable, and prevents staying open indefinitely after it has recovered.

Netflix’s Hystrix library, released in 2012, made circuit breakers a mainstream pattern in distributed systems engineering. Hystrix tracked metrics per command in a rolling window, opened circuits automatically when failure rates exceeded thresholds, and required engineers to define a fallback at the point of use — forcing degraded behaviour to be considered at design time rather than at incident time. Hystrix is now in maintenance mode. Its successor patterns are implemented in Resilience4j for Java, Polly for .NET, and as built-in features of service meshes like Istio and Envoy, which apply circuit breaking at the infrastructure layer without requiring application code changes.

Circuit breakers address the call-level failure mode. Bulkheads — covered in Post 4.7 — address the resource exhaustion failure mode that circuit breakers alone cannot prevent: a slow downstream service that does not yet meet the circuit breaker threshold but is still holding enough threads open to degrade the caller. The two patterns are complementary and should be deployed together.

Timeouts, Retries, and Circuit Breakers as a Coordinated System

These three mechanisms are not independent choices. They form a coordinated defence layer that must be designed together, because each addresses the failure mode the others cannot.

Timeouts ensure a single slow call does not hold a thread indefinitely — without timeouts, circuit breakers cannot function because a call that never returns cannot be counted as a failure in the rolling window. Retries with backoff and jitter give transient failures a chance to resolve without amplifying load — without retries, transient failures produce unnecessary errors that a simple retry would have resolved. Circuit breakers protect against sustained failures by stopping retry attempts entirely and giving downstream services recovery time — without circuit breakers, retries under sustained failure produce the retry storm that collapsed the 2019 AWS SQS service.

A system with all three configured correctly will fail fast, recover gracefully, and avoid amplifying failures into cascades. A system with only one or two in place has gaps that will surface under the production conditions where reliability is most needed.

Reliability in distributed systems is a system property, not a client property. The client must retry correctly. The server must support idempotency. The infrastructure must support circuit breaking. The team must agree on retry budgets across service boundaries. None of these can be substituted for the others.

In distributed systems, doing something again is often more dangerous than doing nothing. The goal is not to retry aggressively — it is to retry intentionally, with clear limits, idempotent operations, and full awareness of system behaviour under stress.

Key Takeaways

  1. Retries are a system-level design decision, not a client-level default — every retry policy requires a budget, exponential backoff with full jitter, and error classification that distinguishes retryable from non-retryable failures
  2. Exponential backoff reduces load amplification during degradation; full jitter prevents the thundering herd at recovery time — both are required and neither is sufficient alone
  3. Idempotency keys are the production standard for making state-changing operations safe to retry — the key must be scoped to the operation type, stored durably, and committed atomically with the operation
  4. Circuit breakers protect call chains from sustained downstream failures through three states — closed (normal), open (fast failure with fallback), and half-open (recovery probe) — and require a defined fallback to be operationally useful
  5. Timeouts, retries, and circuit breakers are a coordinated defence layer — each addresses the failure mode the others cannot, and all three must be present for a robust reliability layer
  6. Bulkheads complement circuit breakers by isolating resource pools per downstream dependency — a slow service that has not yet tripped the circuit breaker can still exhaust a shared thread pool without bulkhead isolation
  7. Reliability is a system property — the client retries correctly, the server supports idempotency, the infrastructure supports circuit breaking, and the team agrees on retry budgets across service boundaries — none of these substitutes for the others

Frequently Asked Questions (FAQ)

What is the difference between exponential backoff and jitter in retry strategies?

Exponential backoff increases the wait time between retries geometrically — each attempt waits twice as long as the previous — reducing the rate at which failing clients send load to a degraded service. Jitter adds a random offset to each wait time, preventing all clients that started failing simultaneously from retrying at the same moment. Without jitter, even perfectly implemented exponential backoff produces synchronised retry waves that hit a recovering service simultaneously. AWS’s 2015 analysis of backoff strategies found that full jitter — randomising the entire wait within the backoff window — consistently produced the lowest server load during recovery and the fastest overall stabilisation time.

What is an idempotency key and how is it implemented?

An idempotency key is a unique identifier generated by the client and attached to every state-changing request, allowing the server to deduplicate retried requests. If the server has already processed a request with that key, it returns the stored result rather than executing the operation again. The client generates the key before the first attempt (commonly a UUID) and includes it unchanged in every retry. The server stores the key and result durably — in Redis with persistence or a database table with a TTL index — and commits the key storage atomically with the operation. Stripe’s payment API is the canonical implementation: every charge request accepts an Idempotency-Key header and protects against duplicate charges regardless of how many times the client retries.

What are the three states of a circuit breaker and what happens in each?

Closed is normal operation — calls pass through to the downstream service and the circuit breaker tracks success and failure rates in a rolling window. Open is fast failure mode — triggered when the failure rate exceeds a configured threshold, all calls are rejected immediately without network contact, protecting the caller’s thread pool and giving the downstream service recovery time without load; the caller must have a defined fallback. Half-open is the recovery probe — after a timeout, a small number of test requests are allowed through; if they succeed the circuit closes and normal operation resumes, if they fail the circuit reopens and the timeout resets. The threshold to open is commonly 50% failures in a 10-second window, tuned per service based on its baseline error rate.

Which errors should and should not be retried?

Retry transient errors that indicate temporary unavailability: network timeouts, connection refused, HTTP 503 (service unavailable), HTTP 429 (rate limited — after honouring the Retry-After header if present). Never retry errors that indicate a permanent problem with the request: HTTP 400 (bad request), HTTP 401 (unauthorised), HTTP 403 (forbidden), HTTP 422 (unprocessable entity). Retrying a 400 will always produce another 400 — it wastes resources and delays the caller receiving the error it needs to act on. HTTP 500 requires judgment: some 500s are transient (the server crashed mid-request and restarted), others indicate bugs that retrying will not resolve. A safe heuristic is to retry 500s once with backoff and treat a second 500 as non-retryable.

What is the thundering herd problem and how does jitter prevent it?

The thundering herd occurs when a large number of clients, all of which started failing at approximately the same time, complete their backoff windows simultaneously and send a synchronised wave of retries to a recovering service. If that wave exceeds the service’s capacity, it causes immediate re-degradation — the service never gets a stable recovery window. Jitter prevents this by randomising each client’s wait time within the backoff window, spreading retries across time rather than concentrating them. With full jitter, a service that recovers at time T receives retries distributed across the backoff window rather than in a single spike at T.

Why do circuit breakers require a defined fallback to be useful?

Without a fallback, an open circuit simply converts a slow failure into a fast one — which reduces resource exhaustion but produces an error for every request. The purpose of a circuit breaker is to degrade gracefully under failure, not just to fail faster. A defined fallback — a cached version of the last successful response, a default value, a reduced-feature response, or a clear error the user interface can handle — allows the system to continue serving users in a degraded but functional state while the downstream service recovers. The design constraint that Hystrix enforced — requiring a fallback to be defined at the point of use — forced engineers to think about degraded behaviour at design time rather than at incident time, which is the correct place to make that decision.

What I wish someone had told me earlier

I have seen retry storms take down services that were running perfectly fine. Not because the engineers were careless — but because retries felt like a safe default and nobody questioned it. The assumption was: if something fails, try again. Reasonable. Dangerous at scale.

The moment that changed how I think about this was reading the AWS SQS post-incident review. Every client was doing the right thing individually. The collective behaviour was catastrophic. That gap — between individually correct and collectively dangerous — is where most distributed systems problems actually live. It is not a bug. It is an emergent property of how independent systems interact under stress.

If there is one thing I would have wanted someone to tell me before my first production incident involving retries, it is this: implement idempotency before you implement retries. Not after. Not as a follow-up task. Before. Because a retry without idempotency is not a reliability mechanism — it is a duplication mechanism with optimistic timing.

The next post — Post 2.3 on Service Discovery — is where a lot of this becomes concrete in a different way. You cannot retry your way to reliability if your service registry is serving stale addresses that no longer exist. The two problems are more connected than they appear at first.

Later in the series, Post 4.7 on Bulkheads is the post I would have wanted during my first incident involving a cascading failure. Circuit breakers stop calls to a failing service. Bulkheads keep a failing service contained — they prevent it from consuming the shared resources that every other service in your fleet depends on. Both matter. Most systems I have reviewed have one, not both. Usually they have the circuit breaker, because it is the pattern that gets talked about. The bulkhead is the one that would have actually saved them.

And if idempotency keys are new to you — the Stripe implementation in this post is worth studying in detail. Not because Stripe is unusual, but because they got it exactly right and documented it publicly. Key scope, storage durability, atomic commits with the operation. Most retry bugs I have encountered in production trace back not to the retry policy but to an idempotency assumption that was wrong in exactly one edge case nobody thought to test.

Series home: Distributed Systems — Concepts, Design & Real-World Engineering

Part 2 — Communication & Coordination

Previous: ← 2.1 — Communication Fundamentals in Distributed Systems

Next: 2.3 — Naming and Service Discovery

Not read Part 1 yet? Start with 1.1 — What Is a Distributed System (Really)?

Discover more from Rahul Suryawanshi

Subscribe now to keep reading and get access to the full archive.

Continue reading