Backpressure and Overload Management in Distributed Systems

by
Home » Distributed Systems » Scalability & Performance » Backpressure and Overload Management in Distributed Systems

Distributed Systems Series — Part 5.6: Scalability & Performance

The Difference Between a System That Degrades and One That Collapses

Traffic is never uniform. Every distributed system experiences load spikes — viral events, marketing campaigns, coordinated usage patterns, or simply the natural rhythm of peak hours. The engineering question is not whether spikes will occur but whether the system degrades gracefully when they do or collapses catastrophically.

A system without overload protection collapses under excess load through a predictable sequence. Incoming requests fill available threads. Thread pool exhaustion causes new requests to queue. Queue depth grows. Memory pressure increases. Response latency rises. Clients that receive no response retry. Retries amplify load. The system spends more CPU managing overload than processing requests. Throughput collapses. The system becomes completely unavailable — not because of hardware failure but because it attempted to serve more traffic than it could handle and the attempt produced worse outcomes than rejecting the excess would have.

Rate limiting and backpressure are the two complementary mechanisms that prevent this collapse. Rate limiting controls traffic at the entry point — defining how much demand the system will accept. Backpressure propagates capacity signals upstream — allowing components to communicate their load to callers so producers slow down before queues overflow. Together they ensure that the system degrades to a reduced but functional state under overload rather than collapsing entirely.

This post builds on the load shedding introduced in Post 4.6 and extends it with the algorithmic detail, protocol implementations, and production patterns that make overload management work in practice.

Rate Limiting vs Backpressure: Precise Definitions

Rate limiting is a proactive control applied at the entry point to a system. It enforces a maximum request rate — rejecting or queuing requests that exceed a configured threshold before they consume system resources. Rate limiting answers: how much traffic will this system accept? It is applied based on predefined policies (100 requests per second per API key, 1000 requests per minute per IP address) rather than in response to observed system state.

Backpressure is a reactive signal propagated upstream when a component approaches or exceeds its capacity. It answers: what should upstream components do when this component cannot keep up? Backpressure is not a rejection — it is a communication: the downstream is overloaded, the upstream should slow its production rate. It is applied in response to observed system state (queue depth, CPU utilisation, response latency) rather than based on predefined policies.

The two mechanisms operate at different points in the system and address different problems. Rate limiting protects the system from external overload at its boundary. Backpressure protects internal components from overload caused by internal imbalances — a fast producer overwhelming a slow consumer, a batch job saturating a database, a service receiving more internal traffic than its downstream can handle.

Both are necessary. A system with rate limiting but no backpressure will reject external excess but may still collapse when internal components produce faster than their dependencies can consume. A system with backpressure but no rate limiting will manage internal flow but will be vulnerable to external traffic spikes that exceed total system capacity.

Rate Limiting Algorithms: Four Production Implementations

Fixed window counter counts requests in a fixed time window — for example, a 60-second window that resets at the start of each minute. If the count exceeds the limit within the window, requests are rejected. Simple to implement, cheap to store (one counter per key per window), and easy to reason about.

The boundary burst problem: if the limit is 100 requests per minute and a client sends 100 requests at 00:59 and 100 more at 01:01, the client has sent 200 requests in two seconds — double the intended rate — while satisfying the fixed window limit. Both windows show exactly 100 requests. The rate limit has not been violated in any technical sense, but the intent has been completely circumvented. Fixed window counters are appropriate for coarse rate limiting where boundary bursts are acceptable. They are inappropriate for rate limiting where the burst behaviour matters.

Sliding window log records the timestamp of every request in a sorted set. On each new request, it counts the timestamps in the log within the rolling window (current time minus window duration) and compares against the limit. If under the limit, the request proceeds and its timestamp is added. This provides accurate rate limiting without boundary bursts — the window truly slides.

The cost is memory: storing a timestamp per request for every active client. At high traffic rates with many clients, the memory cost is significant. Redis sorted sets implement the sliding window log efficiently — ZREMRANGEBYSCORE removes expired timestamps and ZCARD counts current ones. The memory cost limits sliding window logs to scenarios where accuracy is critical and memory is not the primary constraint.

Token bucket maintains a bucket per client that fills at a constant rate R (tokens per second) up to a maximum capacity B (the bucket capacity). Each request consumes one token. If a token is available, the request proceeds. If no token is available, the request is rejected or queued.

The token bucket allows controlled bursts. A client that has not made any requests for 10 seconds accumulates up to B tokens and can spend them in a burst. A client with a 100 requests/second rate (R=100) and a bucket capacity of 500 (B=500) can burst to 500 requests instantaneously, then sustains at 100 requests/second as the bucket refills. This models the actual behaviour that legitimate API clients exhibit — occasional bursts followed by normal sustained traffic — while still enforcing an average rate limit.

Token buckets are O(1) in memory per client (store only the current token count and the timestamp of the last refill) and O(1) in computation (calculate tokens accumulated since last check using elapsed time × rate). Redis implements token buckets efficiently using a Lua script that reads the current count, calculates the token accumulation, and atomically updates and checks — all in a single round trip. This is the algorithm used by Stripe, Twilio, and most major API providers for per-customer rate limiting.

Leaky bucket processes requests at a constant output rate regardless of the input rate. Excess requests queue up to a maximum queue depth. When the queue is full, excess requests are rejected. The output is smooth — requests leave at a constant rate regardless of how bursty the input is.

The difference from token bucket: token bucket allows bursts at the output (a client can consume accumulated tokens quickly), while leaky bucket smooths the output to a constant rate (requests exit the queue at the same rate regardless of how many are waiting). Leaky bucket is appropriate for traffic shaping scenarios where the downstream requires constant, smooth input — audio/video streaming, network traffic shaping, database write rate limiting. It is inappropriate for APIs where bursty but bounded client behaviour should be accommodated.

Sliding window counter combines the memory efficiency of fixed windows with the accuracy of sliding window logs. It maintains two fixed window counters — the current window and the previous window — and interpolates between them based on how far through the current window the request arrives. If the request arrives 40% through the current window, it counts 60% of the previous window’s count plus 100% of the current window’s count against the limit.

This approximation handles boundary bursts without the memory cost of the full sliding window log. Redis, Nginx’s limit_req module, and most production rate limiting implementations use this approach. The approximation error is bounded and small in practice — the interpolation slightly over-counts or under-counts at window boundaries but never by more than the rate limit itself.

TCP Backpressure: The Foundation Everyone Uses

The most widely deployed backpressure mechanism is one engineers rarely think about explicitly: TCP’s flow control and congestion control.

TCP’s receive window is a backpressure signal. Every TCP ACK packet includes a receive window size — the number of bytes the receiver is willing to accept before requiring acknowledgement. When the receiver’s buffer fills (because the application is not reading data fast enough), the receive window shrinks. When the window reaches zero, the sender stops transmitting. The receiver’s inability to process data propagates as a backpressure signal to the sender through the TCP window mechanism, without any application-level code.

TCP’s congestion control (CUBIC, BBR, RENO variants) implements backpressure at the network layer. When packet loss is detected (a proxy for network congestion), the sender reduces its transmission rate. The congestion window shrinks. The network’s overload signal propagates upstream to the sender as a reduction in allowed transmission rate. This is backpressure implemented entirely within the transport layer, transparent to the application.

Understanding TCP backpressure matters for distributed systems design because it sets the floor on backpressure propagation. If a downstream service is overloaded and stops reading from its TCP receive buffers, the buffers fill, the TCP window advertised to upstream services shrinks, and upstream services slow their transmission rate automatically — even if no application-level backpressure is implemented. This provides a safety net but not a complete solution — TCP backpressure is coarse and slow-reacting compared to application-level backpressure.

gRPC Flow Control: Backpressure for Service Calls

gRPC implements explicit flow control at the application layer on top of HTTP/2. HTTP/2’s stream-level and connection-level flow control windows operate identically to TCP’s receive window — a receiver advertises how many bytes it is willing to accept, and the sender limits transmission to within that window.

gRPC’s flow control is bidirectional — both the client and the server can apply backpressure to each other. A server that is overloaded stops consuming messages from the gRPC stream, its HTTP/2 receive window fills, and the client’s send call blocks. The client experiences backpressure as a blocked write rather than a rejected request — the gRPC library holds the message until the server has capacity to receive it.

This makes gRPC naturally backpressure-aware. A gRPC service chain where each service calls the next with gRPC propagates overload upstream through flow control windows — a slow downstream causes an upstream to block, which causes its upstream to block, propagating the backpressure signal all the way to the request source. No application code is required to implement this — it is a property of the HTTP/2 and gRPC protocol implementation.

The practical implication: gRPC service chains with bounded concurrency limits and flow control enabled will slow together under load rather than overloading individual services while others run idle. This is the correct failure mode — bounded degradation rather than one service collapsing while upstream services continue sending traffic at full rate.

Kafka Backpressure: Consumer-Controlled Reading

Kafka implements backpressure through its pull-based consumer model. Producers write messages to Kafka partitions at whatever rate they produce. Consumers read messages from partitions at whatever rate they can process. The producer never pushes messages to the consumer — the consumer pulls when it is ready.

This consumer-controlled pull model is inherently backpressure-aware. A consumer that is overloaded simply stops pulling. Messages accumulate in the Kafka partition — Kafka retains messages for a configurable retention period regardless of whether they have been consumed. The producer continues writing at its natural rate, completely unaffected by the consumer’s processing rate. Consumer lag — the difference between the latest produced offset and the latest consumed offset — grows as the consumer falls behind.

Consumer lag is the primary metric for Kafka backpressure monitoring. Growing consumer lag indicates that the consumer cannot keep up with the producer and that backpressure is accumulating. Lag-based autoscaling uses consumer lag as the scaling signal — when lag grows beyond a threshold, additional consumer instances are added to reduce the lag. This is the integration between backpressure monitoring (lag) and autoscaling (adding consumers) that Post 5.7 will cover in depth.

The limitation: Kafka’s pull model decouples producers from consumers at the cost of unbounded message accumulation if consumers fall too far behind. A consumer that is permanently slower than its producer will accumulate lag indefinitely until either the consumer is scaled or the partition reaches its storage retention limit and begins discarding old messages. Kafka’s backpressure model handles temporary overload gracefully but requires explicit intervention for sustained producer-consumer rate mismatches.

Netflix’s Adaptive Concurrency Limiting

Fixed rate limits — “this service accepts 1000 requests per second” — have a fundamental problem: the right limit depends on the system’s current state, which changes continuously. A service that can handle 1000 requests per second at normal latency may only be able to handle 200 requests per second when a downstream dependency is slow. A fixed limit of 1000 will overload the service during the slow period; a fixed limit of 200 will underutilise it during normal operation.

Netflix’s concurrency limiting approach, described in their 2018 engineering blog post, uses an adaptive algorithm based on TCP’s Additive Increase Multiplicative Decrease (AIMD) congestion control. The algorithm continuously adjusts the concurrency limit based on observed request latency and inflight request count.

The mechanism: the service maintains a concurrency limit — the maximum number of requests it will process simultaneously. When a request completes successfully within the expected latency, the limit is slightly increased (additive increase). When a request exceeds the latency threshold or is rejected, the limit is reduced significantly (multiplicative decrease). The limit continuously probes upward to find the maximum throughput point, and retreats quickly when overload signals appear.

This adaptive approach produces a rate limit that tracks the system’s actual capacity rather than a fixed configuration. During a slow downstream period, the limit reduces automatically without operator intervention. When the downstream recovers, the limit increases automatically. The service self-calibrates to its environment.

Netflix’s Concurrency Limits library implements this in Java. The Envoy proxy implements a similar adaptive algorithm through its adaptive concurrency filter. This approach is increasingly preferred over fixed rate limits for service-to-service communication where the appropriate limit varies with system state.

Queue Depth as a Backpressure Signal

Many backpressure implementations use queue depth as the primary signal. When a service’s request queue exceeds a configured depth, it signals backpressure to callers — stopping new requests from being added rather than allowing the queue to grow without bound.

Unbounded queues are the failure mode that transforms overload from a temporary problem into a cascade. A service that accepts every incoming request into an unbounded queue appears healthy by many metrics — it is accepting requests, not returning errors. But the queue grows as requests arrive faster than they are processed. Memory pressure increases. Response latency for queued requests increases — requests queued for 30 seconds before processing appear as 30-second latency to the client, which has likely already timed out and retried. The retries add more load. The queue grows faster. Eventually the service OOMs or the client-side timeouts produce a wave of retries that overloads every upstream service.

Bounded queues with explicit rejection on overflow prevent this. When the queue reaches its maximum depth, new requests are rejected immediately with an explicit error (HTTP 429 or 503). The client receives an immediate rejection rather than a 30-second timeout. The rejection can be handled with backoff and retry. The queue does not grow unboundedly. The service remains responsive for requests it can serve.

The queue depth limit is a design choice that determines the trade-off between latency and throughput. A small queue (zero to a few requests) produces fast rejection of excess traffic but lower burst absorption. A large queue absorbs bursts but produces high latency for queued requests that must wait behind a large backlog. For latency-sensitive services, prefer small queues or zero queuing (reject immediately on thread pool exhaustion). For throughput-oriented services where latency variance is acceptable, larger queues provide burst absorption.

Retry Storms: When Rate Limiting Without Coordination Fails

Rate limiting rejects excess requests with HTTP 429. Clients that receive 429 should back off and retry later. The problem arises when many clients receive 429 simultaneously and retry simultaneously — the retry storm.

A traffic spike causes 90% of requests to be rejected with 429. All rejected clients wait a fixed retry interval (say, 1 second) and retry simultaneously. The second wave is identical in volume to the first. 90% are rejected again. All wait 1 second and retry again. The system is permanently under spike load from the retries of the spike, even if the original spike was a one-time event.

Exponential backoff with full jitter — covered in depth in Post 2.2 — prevents retry storms by randomising retry timing. Each client waits a random duration (between 0 and the exponential backoff ceiling) before retrying. Clients desynchronise naturally. The retry load spreads across time rather than arriving as a simultaneous wave. The server recovers from the initial spike without being immediately re-spiked by synchronised retries.

The Retry-After HTTP header communicates when the client should next attempt a request. A rate limiter that returns 429 with Retry-After: 5 tells the client to wait 5 seconds before retrying. Without the header, clients must implement their own backoff logic. With it, the server controls the retry timing and can stagger retry windows across clients by varying the Retry-After value.

Key Takeaways

  1. Rate limiting and backpressure are complementary mechanisms — rate limiting controls entry point traffic based on predefined policies, backpressure propagates capacity signals upstream based on observed system state; both are required for complete overload protection
  2. Token bucket is the right default rate limiting algorithm for APIs — it allows controlled bursts up to bucket capacity B while enforcing a sustained rate R, is O(1) in memory and computation, and models legitimate client behaviour correctly
  3. TCP flow control and congestion control are the foundational backpressure mechanism that every distributed system already uses — understanding them clarifies why application-level backpressure is needed for fast-reacting, coarse-grained control that TCP’s window mechanism cannot provide
  4. gRPC’s HTTP/2 flow control propagates backpressure naturally through service chains — a slow downstream blocks an upstream’s send call, which blocks its upstream, propagating the overload signal to the request source without application code
  5. Kafka’s pull-based consumer model is inherently backpressure-aware — consumers read when ready, consumer lag is the backpressure metric, and lag-based autoscaling connects backpressure monitoring to capacity management
  6. Netflix’s adaptive concurrency limiting tracks the system’s actual capacity rather than a fixed configuration — adjusting limits based on observed latency and inflight requests, automatically reducing under load and recovering when conditions improve
  7. Bounded queues with explicit rejection are essential — unbounded queues convert temporary overload into cascading failures by accumulating backlog that produces high latency, triggers client timeouts, and generates retry storms that compound the original overload

Frequently Asked Questions (FAQ)

What is the difference between rate limiting and backpressure?

Rate limiting is a proactive control applied at the entry point based on predefined policies — it enforces a maximum request rate regardless of current system state. Backpressure is a reactive signal propagated upstream based on observed system state — it communicates that a component is approaching capacity so that producers slow their rate. Rate limiting protects the system from external overload at its boundary. Backpressure protects internal components from overload caused by internal rate mismatches between producers and consumers. Both are necessary: rate limiting alone leaves internal components vulnerable, backpressure alone leaves the entry point unprotected.

What is the token bucket algorithm and when should I use it?

The token bucket algorithm maintains a bucket per client that fills at a constant rate R (tokens per second) up to a maximum capacity B. Each request consumes one token. If a token is available, the request proceeds. If no token is available, the request is rejected (HTTP 429) or queued. The bucket capacity B determines the maximum burst size — a client that has not made requests for 10 seconds accumulates up to B tokens and can spend them immediately. Use the token bucket for API rate limiting where you want to allow controlled bursts from legitimate clients while enforcing an average rate limit. It is O(1) in memory and computation, making it efficient at scale.

What is a retry storm and how do I prevent it?

A retry storm occurs when many clients receive rate limit rejections (429) simultaneously and retry simultaneously, producing a second wave of traffic identical to the first. The server is continuously overloaded by the retries of the overload. Prevention requires two mechanisms. Exponential backoff with full jitter randomises retry timing — each client waits a random duration before retrying, desynchronising the retry wave. The Retry-After HTTP header allows the server to specify when each client should retry, enabling staggered retry windows across clients. Without these mechanisms, rate limiting alone may produce retry storms that make the overload condition permanent.

How does Kafka implement backpressure?

Kafka implements backpressure through its pull-based consumer model. Producers write to Kafka partitions at their natural rate. Consumers pull messages when they are ready to process them — the consumer controls the read rate, not the producer. An overloaded consumer stops pulling; messages accumulate in the partition without affecting the producer. Consumer lag — the difference between the latest produced offset and the latest consumed offset — grows when the consumer cannot keep up. Monitoring consumer lag provides the backpressure signal. Lag-based autoscaling uses growing lag as the trigger to add consumer instances, connecting the backpressure signal to capacity management.

What is adaptive concurrency limiting?

Adaptive concurrency limiting continuously adjusts the maximum number of simultaneous requests a service will process based on observed latency and inflight request count, rather than using a fixed configured limit. The algorithm uses additive increase (incrementally raising the limit when requests complete within expected latency) and multiplicative decrease (significantly reducing the limit when latency exceeds threshold or requests are rejected). This produces a rate limit that tracks the system’s actual capacity — automatically reducing during downstream slowdowns and recovering when conditions improve. Netflix’s Concurrency Limits library and Envoy’s adaptive concurrency filter implement this approach.

Why are unbounded queues dangerous under overload?

Unbounded queues convert temporary overload into cascading failures. A service that accepts every request into an unbounded queue appears healthy by traffic metrics — it is accepting requests and not rejecting them — but the queue grows as arrivals exceed processing capacity. Queued requests experience latency equal to their wait time in the queue, which grows as the queue deepens. Clients that configured short timeouts receive no response and retry, adding more requests to the queue. Memory pressure increases. Eventually the service OOMs or the client retry wave overloads upstream services. Bounded queues with explicit rejection on overflow prevent this — immediate rejection of excess requests produces fast failures that clients can handle with backoff, rather than slow failures that clients cannot detect until timeout.

Continue the Series

Series home: Distributed Systems — Concepts, Design & Real-World Engineering

Part 5 — Scalability & Performance

Previous: ← 5.5 — Caching Trade-offs in Distributed Systems

Next: 5.7 — Indexing and Query Optimisation in Distributed Databases →

Related posts from earlier in the series:

Discover more from Rahul Suryawanshi

Subscribe now to keep reading and get access to the full archive.

Continue reading