Distributed Queues and Async Processing in Distributed Systems

by
Home » Distributed Systems » Scalability & Performance » Distributed Queues and Async Processing in Distributed Systems

Distributed Systems Series — Part 5.11: Scalability & Performance

Why Synchronous Communication Does Not Scale

Every communication pattern covered in Post 2.1 falls into one of two categories: synchronous (the producer waits for the consumer to respond before proceeding) or asynchronous (the producer sends a message and continues immediately without waiting). Synchronous communication is the default — it is how function calls work, how HTTP request-response works, and how most engineers instinctively design service interactions.

Synchronous communication has a fundamental scalability limitation: the producer’s throughput is bounded by the consumer’s throughput. If a producer can generate 10,000 requests per second but the consumer can only process 1,000 per second, the producer is bottlenecked — it must wait for the consumer to respond before sending the next request, limiting it to 1,000 per second regardless of its own capacity. Worse, when the consumer slows down under load, the producer slows down with it, even if the producer itself has no resource constraints. This tight coupling between producer and consumer throughput is what makes synchronous communication hard to scale independently.

Distributed queues break this coupling. A queue sits between the producer and consumer — the producer writes messages to the queue at its natural rate and returns immediately without waiting for the consumer. The consumer reads from the queue at its own rate. When the producer is faster than the consumer, messages accumulate in the queue. The queue absorbs the rate mismatch. The producer and consumer are decoupled — each can scale independently based on its own resource constraints, not on the other’s throughput.

This post covers the delivery semantics that determine correctness guarantees, the major queue implementations and their trade-offs, event streaming with Kafka as the architecture that extends queuing to persistent replayable logs, the dead letter queue pattern for handling failures, the outbox pattern for transactional message publishing, and the fan-out patterns that connect queuing to the scalability mechanisms of the rest of Part 5.

Delivery Semantics: The Three Levels

The correctness guarantee a queue provides determines what the producer and consumer must do to build correct systems on top of it. Three delivery semantics exist, and choosing the wrong one produces either data loss or duplicate processing — both of which produce incorrect application behaviour.

At-most-once delivery means the queue delivers each message zero or one times — never more than once. Messages may be lost (sent but not delivered) but are never duplicated. This is the simplest and cheapest to implement — the producer fires the message and forgets it, with no acknowledgement required. At-most-once is appropriate for use cases where message loss is acceptable: telemetry events where losing 0.1% of metrics does not affect the statistical accuracy of dashboards, fire-and-forget notifications where a missed notification is acceptable, and logging pipelines where occasional log loss is preferable to pipeline backpressure.

At-least-once delivery means the queue delivers each message one or more times — never zero times. Messages are never lost but may be delivered multiple times (when the consumer processes a message but fails before acknowledging it, the queue redelivers the message). At-least-once is the most widely used delivery semantic in production distributed systems because it is achievable without distributed transactions — the queue simply redelivers any message that has not been acknowledged within a timeout window.

At-least-once delivery requires that consumer handlers be idempotent — applying the same message multiple times must produce the same result as applying it once. This is the same idempotency requirement from Post 2.2, applied to message processing rather than HTTP retries. An idempotent payment processor that uses the message ID as a deduplication key can safely receive the same payment message twice — the second processing is detected as a duplicate and discarded. A non-idempotent payment processor that processes every message unconditionally will charge the customer twice if the message is delivered twice.

Exactly-once delivery means each message is delivered and processed exactly once — never lost, never duplicated. This is the strongest semantic and the most expensive to implement. True exactly-once delivery requires coordination between the queue and the consumer’s data store — the message acknowledgement and the consumer’s state update must be atomic. If the consumer updates its state and then acknowledges the message, a crash between the two operations produces a duplicate (the message is redelivered but the state was already updated). If the consumer acknowledges and then updates state, a crash between the two loses the update.

Kafka’s exactly-once semantics (introduced in Kafka 0.11) implement this through idempotent producers (each message has a unique sequence number, and the broker deduplicates retransmissions) and transactional APIs (the consumer’s state update and offset commit happen in the same transaction). This requires both the producer and consumer to use the Kafka transactions API and limits the consumer’s state store to systems that participate in the Kafka transaction. In practice, most systems use at-least-once delivery with idempotent consumers rather than exactly-once delivery — the implementation complexity of exactly-once is significant and at-least-once with idempotency achieves the same correctness guarantee at lower cost.

Queue Implementations: Choosing the Right System

Apache Kafka is a distributed event streaming platform that models the queue as a persistent, ordered, replayable log. Messages (called events) are written to partitions within topics. Consumers read events by tracking their position in the log (the offset) — they can replay events by resetting their offset to an earlier position. Events are retained for a configurable retention period (days to weeks) rather than being deleted on consumption.

Kafka’s log model provides properties that traditional message queues do not. Multiple independent consumer groups can read the same topic simultaneously — each group maintains its own offset, so a new consumer group can start reading from the beginning of the topic and process all historical events without affecting other consumers. This enables event sourcing architectures where the log is the authoritative record and any derived state can be rebuilt by replaying the log.

Kafka’s ordering guarantee is per-partition — events in the same partition are delivered in the order they were written, with no ordering guarantee across partitions. Ordering within a partition is maintained by routing related events (events for the same user, order, or entity) to the same partition using a consistent hash of the entity ID as the partition key. This connects directly to the partitioning concepts from Post 5.4 — Kafka partitioning and database partitioning address the same distribution problem with the same techniques.

Kafka is the correct choice for event streaming at scale, audit logs, event sourcing, and any workload where replay capability or multiple independent consumers are required. Its throughput at scale is exceptional — a well-tuned Kafka cluster can sustain millions of events per second. Its operational complexity is non-trivial — Kafka requires ZooKeeper (or KRaft in newer versions) for cluster coordination, careful partition management, and consumer group offset management.

Amazon SQS is a fully managed queue service with two modes. SQS Standard provides best-effort ordering and at-least-once delivery with very high throughput (virtually unlimited). SQS FIFO provides strict ordering and exactly-once delivery within a message group at lower throughput (up to 3,000 messages per second with batching). SQS Standard is the correct choice for high-throughput task queues where message ordering is not required and consumers are idempotent. SQS FIFO is the correct choice for order-sensitive workflows where strict sequencing is required.

SQS’s visibility timeout is its primary mechanism for at-least-once delivery. When a consumer reads a message, the message becomes invisible to other consumers for the visibility timeout duration. If the consumer processes and deletes the message before the timeout, the message is gone. If the consumer fails or times out, the message becomes visible again and is redelivered to another consumer. The visibility timeout must be set longer than the maximum expected processing time — if a consumer regularly takes longer than the visibility timeout to process a message, the message is redelivered while still being processed, creating duplicate processing.

RabbitMQ is a general-purpose message broker implementing the AMQP protocol. It supports complex routing through exchanges — direct routing (message goes to a specific queue), fanout (message goes to all bound queues), topic routing (message goes to queues matching a pattern), and header-based routing. RabbitMQ’s acknowledgement model supports at-least-once delivery with manual acknowledgements — the consumer explicitly acknowledges each message after successful processing, and unacknowledged messages are redelivered on consumer failure.

RabbitMQ is the correct choice for complex routing requirements, request-reply patterns (RPC over messaging), and workloads requiring per-message acknowledgement with fine-grained control. It is less appropriate than Kafka for high-throughput event streaming because RabbitMQ deletes messages on consumption — there is no replay capability.

Google Cloud Pub/Sub and AWS EventBridge are managed event bus services that decouple event producers from event consumers through a subscription model — producers publish events, multiple subscribers consume them independently. These are appropriate for event-driven architectures where many services need to react to the same events without tight coupling between them.

The Dead Letter Queue Pattern

A dead letter queue (DLQ) is a secondary queue that receives messages that cannot be successfully processed after a configured number of attempts. Without a DLQ, a message that repeatedly causes processing failures either blocks the queue (if ordering is maintained) or is silently discarded after the retry limit (losing the message permanently). Both outcomes are unacceptable for production systems.

The DLQ pattern: configure a maximum receive count on the primary queue (for SQS) or a maximum delivery count on the consumer (for Kafka, implemented in the consumer code). When a message fails processing N times, move it to the DLQ rather than discarding it or blocking. Operations teams monitor the DLQ — messages in the DLQ require investigation. After investigating and fixing the root cause (a bug in the consumer, a malformed message, a dependency outage), messages in the DLQ can be replayed to the primary queue for reprocessing.

DLQs are the safety net for poison messages — messages that cannot be processed due to data corruption, unexpected format, or logic errors in the consumer. Without a DLQ, a poison message that causes the consumer to crash on every processing attempt will continuously block or restart the consumer, preventing it from processing any subsequent messages. With a DLQ, the poison message is isolated after N failures, and the consumer continues processing the remaining messages in the queue.

DLQ monitoring is a critical observability requirement. A DLQ that is filling with messages indicates a systematic problem — either a consumer bug or a pattern of malformed messages from a producer. The four golden signals from Post 4.8 apply: DLQ depth is a saturation metric that should be alerted on when it exceeds zero for a sustained period.

The Outbox Pattern: Transactional Message Publishing

A common correctness problem in event-driven systems: an application writes to a database and publishes a message to a queue in sequence — but these two operations are not atomic. If the application crashes after writing to the database but before publishing the message, the database update is committed but the message is never published. Downstream consumers never learn about the change. The system is in an inconsistent state.

The outbox pattern solves this by writing the message to an outbox table in the same database transaction as the business data update. Both the data update and the outbox entry are committed atomically — either both succeed or both fail. A separate outbox relay process reads from the outbox table and publishes to the message queue. After successful publication, the outbox entry is marked as published.

This guarantees that every database update eventually produces a corresponding message, with at-least-once delivery semantics (if the relay crashes after publishing but before marking as published, the message is published again — requiring idempotent consumers). The outbox pattern is the standard solution for the dual-write problem in event-driven microservices architectures.

Debezium, an open-source change data capture (CDC) tool, implements a variant of the outbox pattern using database transaction logs. Rather than writing to an explicit outbox table, Debezium reads the database’s transaction log (PostgreSQL WAL, MySQL binlog) and publishes every database change as an event to Kafka. This produces the same at-least-once delivery guarantee without requiring application code changes to write to an outbox table.

Fan-Out Patterns: Connecting Queues to Scalability

Fan-out patterns use queues to distribute work across many consumers simultaneously — the same message is delivered to multiple independent consumers, each processing it independently. This is the queue equivalent of the fan-out read pattern from Post 5.2, applied to write processing rather than read serving.

The SNS + SQS fan-out pattern (Publish-Subscribe via AWS) is the standard architecture for fan-out on AWS. An SNS topic receives the message and fans it out to multiple SQS queues — one per consumer group. Each consumer group has its own queue with its own backlog and its own consumer fleet that autoscales independently based on queue depth. A new consumer group can subscribe to the SNS topic and receive all future messages without any changes to the producer or other consumers.

Fan-out enables independent scaling of heterogeneous workloads. An e-commerce order event might be consumed by five independent consumer groups: order fulfilment (high throughput, latency-sensitive), email notification (moderate throughput, latency-tolerant), analytics pipeline (very high throughput, latency-insensitive), fraud detection (moderate throughput, latency-sensitive), and inventory update (lower throughput, must be strongly consistent). Each consumer group can scale based on its own backlog depth and processing requirements, using the autoscaling signal (queue depth or consumer lag) from Post 5.8 independently.

Async Processing Patterns: When to Use Each

Three async processing patterns address different workload characteristics:

Task queues distribute independent, idempotent tasks across a pool of workers. Each task is self-contained — it reads its required data, processes it, writes results, and acknowledges. The task queue provides load balancing across workers and retry logic for failed tasks. SQS Standard with multiple consumer instances is the canonical task queue implementation. Celery (Python), Sidekiq (Ruby), and BullMQ (Node.js) are application-level task queue libraries that implement this pattern on top of Redis or SQS.

Event streaming processes an ordered stream of events as they occur, maintaining per-entity ordering. Kafka is the canonical implementation. Event streaming is appropriate for use cases where the order of events for a specific entity matters — a user’s session events must be processed in order, financial transactions for an account must be processed in order, state machine transitions must be applied in sequence. The partition key determines which events are ordered relative to each other — events with the same partition key are always delivered in order to the same consumer.

Work queues with priority process tasks in priority order rather than FIFO. High-priority tasks (payment completions, security alerts) are processed before low-priority tasks (report generation, cache warming) even when the queue has a large backlog. RabbitMQ supports priority queues natively. Redis sorted sets can implement priority queues — tasks are stored with a priority score and consumed in sorted order.

Backpressure Through Queue Depth

As established in Post 5.6, queue depth is a backpressure signal — when the queue is filling faster than it is being drained, either the consumer needs more capacity or the producer needs to slow down. Queues implement natural backpressure through their bounded capacity: when a queue is full, new writes are rejected or blocked, signalling to the producer that it must slow down.

Queue depth monitoring connects queue-based backpressure to the autoscaling mechanism from Post 5.8. When queue depth grows beyond a threshold, KEDA (for Kubernetes) adds consumer replicas. When queue depth falls, KEDA removes consumer replicas. The queue itself communicates the capacity requirement — no application-level backpressure logic is needed. The consumer fleet sizes itself automatically to drain the queue at approximately the rate it fills.

The correct queue depth threshold for autoscaling depends on the acceptable message processing latency. If messages must be processed within 30 seconds, and each consumer processes 100 messages per second, the queue depth threshold should trigger scale-up before the queue reaches 3,000 messages (30 seconds × 100 messages/second). Scaling up when the queue reaches 1,000 messages provides 10 seconds of headroom before the processing latency SLO would be violated.

Key Takeaways

  1. Distributed queues decouple producer and consumer throughput — the producer writes at its natural rate and returns immediately, the queue absorbs rate mismatches, and the consumer scales independently based on queue depth rather than being bounded by the producer’s rate
  2. At-least-once delivery is the practical standard — never loses messages and achieves correctness when consumers are idempotent; exactly-once delivery is more expensive and is typically replaced by at-least-once with idempotent consumers in production systems
  3. Kafka’s log model enables replay and multiple independent consumer groups — each consumer group maintains its own offset, can start from any historical position, and processes events independently without affecting other consumers
  4. Dead letter queues are mandatory in production — without a DLQ, poison messages that fail processing repeatedly either block the queue or are silently discarded; with a DLQ, they are isolated for investigation and can be replayed after fixing the root cause
  5. The outbox pattern solves the dual-write problem — writing the message to an outbox table in the same database transaction as the business data update guarantees that every data change eventually produces a corresponding message, without requiring distributed transactions
  6. Fan-out patterns (SNS + SQS, Kafka consumer groups) allow multiple independent consumers to process the same event, each scaling independently based on their own queue depth or consumer lag
  7. Queue depth is the correct autoscaling signal for async workloads — KEDA uses queue depth or consumer lag as the scaling trigger, automatically sizing the consumer fleet to drain the queue within the message processing latency SLO

Frequently Asked Questions (FAQ)

What is the difference between at-least-once and exactly-once delivery?

At-least-once delivery guarantees that each message is delivered one or more times — messages are never lost but may be duplicated when the consumer fails after processing but before acknowledging. Correctly handling at-least-once requires idempotent consumers — processing the same message multiple times produces the same result as processing it once. Exactly-once delivery guarantees each message is delivered and processed exactly once — never lost, never duplicated. It requires atomic coordination between the message acknowledgement and the consumer’s state update, typically through distributed transactions. Most production systems use at-least-once with idempotent consumers because it achieves the same correctness at lower implementation complexity than exactly-once.

What is Kafka and how does it differ from traditional message queues?

Kafka is a distributed event streaming platform that models the queue as a persistent, ordered, replayable log. Unlike traditional message queues that delete messages on consumption, Kafka retains messages for a configurable period (days to weeks). Consumers track their position (offset) in the log and can replay events by resetting their offset. Multiple independent consumer groups can read the same topic simultaneously, each maintaining its own offset. Traditional message queues (SQS, RabbitMQ) delete messages after consumption and support only one consumer per queue (or load-balanced consumers for the same queue). Kafka is appropriate for event streaming, event sourcing, and workloads requiring replay; traditional queues are appropriate for task distribution and simple producer-consumer decoupling.

What is a dead letter queue and when is it triggered?

A dead letter queue (DLQ) receives messages that cannot be successfully processed after a configured maximum number of attempts. When a message fails processing N times (the maximum receive count in SQS, the max delivery count in RabbitMQ), it is moved to the DLQ rather than being silently discarded or blocking the queue. DLQs prevent poison messages — messages that cause the consumer to crash on every attempt — from blocking processing of subsequent messages. Operations teams monitor the DLQ and investigate failed messages. After fixing the root cause, messages in the DLQ can be replayed to the primary queue. DLQ depth should be alerted on when it exceeds zero for a sustained period.

What is the outbox pattern and why is it needed?

The outbox pattern solves the dual-write problem: an application that writes to a database and publishes a message to a queue in sequence risks losing the message if it crashes between the two operations. The pattern writes the message to an outbox table within the same database transaction as the business data update — both succeed or both fail atomically. A separate relay process reads the outbox table and publishes to the message queue. This guarantees at-least-once delivery of every database change as a message, without distributed transactions. Debezium implements a variant using change data capture from the database transaction log, producing the same guarantee without requiring application code changes.

How does queue depth connect to autoscaling?

Queue depth (for traditional queues like SQS) and consumer lag (for Kafka) are the correct autoscaling signals for async workloads. When the queue fills faster than consumers drain it, the depth or lag grows — signalling that more consumer capacity is needed. KEDA (Kubernetes Event-Driven Autoscaling) uses queue depth or consumer lag as the scaling trigger, automatically adding consumer replicas when the metric exceeds a threshold and removing them when it falls. The threshold should be set to trigger scale-up before the processing latency SLO would be violated — if messages must be processed within 30 seconds and each consumer processes 100 messages/second, scale-up should trigger before the queue reaches 3,000 messages.

What is the fan-out pattern and when should I use it?

Fan-out delivers the same message to multiple independent consumers simultaneously, each processing it independently with no knowledge of the others. The canonical AWS implementation is SNS + SQS: an SNS topic fans out to multiple SQS queues, one per consumer group. Each consumer group has its own backlog and its own autoscaling consumer fleet. Fan-out is appropriate when multiple services need to react to the same event independently — an order event consumed by fulfilment, notification, analytics, and fraud detection services simultaneously. Each consumer scales based on its own queue depth, processes at its own rate, and can fail or be redeployed without affecting other consumers.

Continue the Series

Series home: Distributed Systems — Concepts, Design & Real-World Engineering

Part 5 — Scalability & Performance

Previous: ← 5.10 — Cost and Capacity Planning at Scale

Next: 5.12 — Engineering Guidelines for Scalability and Performance →

Related posts from earlier in the series:

Discover more from Rahul Suryawanshi

Subscribe now to keep reading and get access to the full archive.

Continue reading