Autoscaling Distributed Systems: Reactive, Predictive and Production Patterns

by
Home » Distributed Systems » Scalability & Performance » Autoscaling Distributed Systems: Reactive, Predictive and Production Patterns

Distributed Systems Series — Part 5.8: Scalability & Performance

Why Manual Scaling Fails at Production Scale

Post 5.1 established that horizontal scaling — adding instances to handle growing load — is the foundation of distributed systems scalability. Posts 5.3 through 5.7 covered the mechanisms that make horizontal scaling possible: partitioning, load balancing, caching, backpressure, and indexing. This post addresses the operational dimension: how does a production system add and remove capacity automatically in response to actual demand, without human intervention for each scaling event?

Manual scaling — an engineer receives an alert, decides to add instances, provisions them, and verifies the fix — fails at production scale for three reasons. Traffic patterns change faster than human response time allows. Production systems at scale experience hundreds of scaling events per day — peak hours, viral spikes, geographic demand shifts, downstream slowdowns — none of which can wait for manual response. The cost of over-provisioning to absorb unexpected spikes without autoscaling is prohibitive — maintaining enough spare capacity to handle a 10× spike manually means running at 10% utilisation on average, paying ten times the necessary infrastructure cost. And human scaling decisions introduce inconsistency — different engineers make different judgments about when and how much to scale, producing unpredictable capacity behaviour.

Autoscaling automates these decisions by continuously measuring demand signals and adjusting capacity to match. Done correctly, it reduces both cost (by releasing capacity when load drops) and risk (by adding capacity before load spikes cause user-visible degradation). Done incorrectly, it produces oscillation, cold starts, and scale-in data loss that create incidents worse than the overload it was designed to prevent.

Reactive vs Predictive Autoscaling

Two fundamentally different approaches to autoscaling exist, and the right choice depends on the predictability of the demand pattern.

Reactive autoscaling monitors a metric signal and scales when the signal crosses a threshold. When CPU utilisation exceeds 70% for two consecutive minutes, add two instances. When CPU drops below 30% for ten minutes, remove one instance. The scaling decision is made in response to observed current state — the system reacts to load as it arrives.

Reactive autoscaling is simple, well-understood, and correct for unpredictable demand patterns — viral events, traffic anomalies, unexpected load spikes that no forecast could have predicted. Its fundamental limitation is the provisioning lag — the time between the scaling decision and the new instance serving traffic. On AWS, a new EC2 instance takes 1-3 minutes to boot. A Kubernetes pod starts in seconds but may take 30-90 seconds to warm up (JVM startup, cache warming, connection pool establishment). During this lag, the system is under the load that triggered scaling but has not yet received the additional capacity. If the lag exceeds the timeout budget of incoming requests, users experience failures during the scaling window.

Predictive autoscaling uses historical patterns to forecast future demand and scales in advance of anticipated load. If traffic spikes every weekday at 9am when the European market opens, predictive scaling provisions the additional capacity at 8:45am — before the spike arrives, eliminating the provisioning lag entirely.

AWS Predictive Scaling uses machine learning trained on historical CloudWatch metrics to forecast demand and schedule scaling actions in advance. Kubernetes’ Vertical Pod Autoscaler has a mode that sets resource requests based on historical usage patterns. KEDA (Kubernetes Event-Driven Autoscaling) supports scheduled scaling rules that provision capacity at configured times regardless of current metrics.

Predictive scaling works well for demand patterns with strong temporal regularity — business hours traffic, daily/weekly cycles, known events like sales campaigns. It works poorly for genuinely unpredictable demand where the historical pattern does not predict future load. The correct production architecture combines both: predictive scaling for anticipated regular patterns, reactive scaling as a safety net for anomalous demand that the predictor did not anticipate.

Scaling Metrics: What to Measure

The metric that drives autoscaling decisions is one of the most consequential configuration choices in the autoscaling system. The wrong metric produces late scaling (the metric rises slowly while users are already experiencing degradation) or oscillation (the metric is too sensitive, causing constant scale-up and scale-down cycles).

CPU utilisation is the most commonly used scaling metric and often the wrong one for latency-sensitive services. CPU utilisation rises when the service is compute-bound — spending most of its time executing code. Many services are I/O-bound — spending most of their time waiting for database responses, cache reads, or downstream service calls. An I/O-bound service can be severely overloaded with users experiencing high latency while CPU sits at 20%. Scaling on CPU for an I/O-bound service adds instances that are also 20% CPU but equally I/O-bound — the bottleneck is not compute, and adding compute does not help.

Request latency (p99) is a more direct user-experience metric. When p99 latency exceeds the SLO threshold, the service needs more capacity — regardless of whether the bottleneck is CPU, I/O, memory, or downstream slowness. The challenge is that latency-based scaling is harder to tune without oscillation — latency spikes are often brief and self-correcting, and scaling in response to every brief spike wastes resources and destabilises the cluster.

Requests per second (RPS) or concurrent connections is the most predictive metric for stateless services. The relationship between RPS and resource consumption is relatively stable for a given service — doubling RPS approximately doubles CPU and I/O consumption. Scaling on RPS allows capacity to be added before CPU or latency thresholds are breached, proactively matching capacity to demand.

Queue depth and consumer lag are the correct metrics for asynchronous workloads. A message processing service should scale based on how many messages are waiting in the queue — not based on the CPU of existing consumers. When queue depth grows, add consumers. When it falls, remove consumers. KEDA implements this pattern for Kubernetes workloads — it scales deployments based on external metrics from Kafka consumer lag, SQS queue depth, Redis list length, or any custom metric.

Custom business metrics are the right choice when the infrastructure metrics above do not accurately represent user-facing load. An e-commerce checkout service should scale based on active checkout sessions in progress — not CPU, not generic RPS. A machine learning inference service should scale based on pending inference requests — not memory utilisation. The correct metric is whatever most directly represents the work the service needs to do.

The Provisioning Lag Problem

Every reactive autoscaling system has a provisioning lag — the time between the scaling decision and new capacity serving traffic. This lag is the fundamental limitation of reactive autoscaling and the primary source of user-visible degradation during scaling events.

The provisioning lag has multiple components that stack. Detection lag: the monitoring system collects metrics at a configured interval (typically 1 minute for CloudWatch, 15 seconds for Kubernetes metrics server) and the threshold breach must persist for a configured evaluation period before triggering scaling (typically 2-3 periods to avoid scaling on transient spikes). Total detection lag: 2-5 minutes in typical production configurations. Decision lag: the autoscaler evaluates the breach and issues the scaling command. Typically seconds. Provisioning lag: the time for a new instance to boot, the time for the application to start, and the time for the application to warm up to full capacity. For containerised workloads: 30 seconds to 2 minutes. For VM-based workloads: 2-5 minutes. Health check lag: the load balancer must confirm the new instance is healthy before routing traffic to it. Typically 30 seconds to 2 minutes depending on health check configuration.

Total end-to-end lag from threshold breach to new instance serving traffic: 4-10 minutes in typical production configurations. For a traffic spike that lasts 5 minutes, reactive autoscaling may not deliver new capacity until the spike is already subsiding — the scaling event is useless for the spike that triggered it but correctly sized for the next spike.

The solutions are pre-warming and floor configuration. Pre-warming runs instances in a standby state that can be activated quickly — they are booted and have completed application startup but are not yet receiving traffic. Activation takes seconds rather than minutes. AWS’s warm pools for Auto Scaling Groups implement this — instances pre-warm to a stopped or running state and can be quickly activated when scaling out. Kubernetes’ PodDisruptionBudgets and slow scale-down policies keep instances running for a configured period after scale-down is triggered, providing a warm pool of recently used instances.

Scale-Up vs Scale-Down Asymmetry

Scaling up and scaling down are not symmetric operations and should not be treated as such. Scale-up has a clear urgency signal — users are experiencing degradation — and benefits from being fast and aggressive. Scale-down has no urgency signal and benefits from being slow and conservative.

The asymmetry in timing: scale-up should trigger as soon as the metric exceeds threshold, with a short evaluation period (1-2 minutes) to avoid false positives from transient spikes. Scale-down should trigger only after an extended cooldown period (10-30 minutes) confirming that load has genuinely dropped and is not about to spike again. A system that scales down aggressively will oscillate — it removes instances as soon as load drops, the metric rises again as remaining instances absorb the load, scale-up triggers, new instances are added, the metric drops, scale-down triggers, and the cycle repeats.

Cooldown periods prevent oscillation by adding a mandatory wait after each scaling event before the next scaling decision is made. AWS Auto Scaling Groups have a cooldown period (default 300 seconds) after each scaling activity. Kubernetes HPA has a stabilisation window for scale-down (default 5 minutes) during which scale-down decisions are stabilised against metric fluctuations.

The scale-down safety problem: stateful instances may hold data or connections that must be drained before the instance is terminated. A web server with active connections must drain those connections before shutdown — closing them abruptly produces errors for the affected clients. A database read replica must flush its write-ahead log buffer before shutdown — terminating abruptly may leave data in an inconsistent state. Kubernetes implements graceful termination through pod lifecycle hooks and the terminationGracePeriodSeconds setting — the pod receives a SIGTERM, has a configured period to drain connections and complete in-flight requests, and is forcefully terminated after the period expires if it has not shut down cleanly.

Cold Start: The Stateful Autoscaling Problem

Stateless services scale cleanly — a new instance starts, accepts requests immediately, and produces correct responses from its first request. No state needs to be transferred, no warm-up is required. The new instance is immediately equivalent to existing instances.

Stateful and warm-up-dependent services have a cold start problem — a new instance cannot serve requests at full performance immediately after startup. The cold start period is the time from instance startup to full performance capacity, and during this period, the instance may serve requests incorrectly or slowly.

JVM-based services experience cold starts from JVM startup (loading classes, initialising the runtime) and JIT compilation warm-up (the JVM compiles hot code paths to native code during the first few minutes of operation — before this compilation, code runs in interpreted mode, which is 2-10× slower). A new Java service instance may serve requests correctly from the first request but at 20-30% of its steady-state throughput for the first 2-3 minutes until JIT compilation has optimised the hot paths.

Cache-warmed services experience cold starts when the cache is empty. A service that relies on an in-process LRU cache to avoid database round trips for frequent queries starts with an empty cache — every query goes to the database, producing high latency and high database load until the cache fills with hot data. During this warm-up period, the new instance is a net burden on the database rather than a capacity relief.

The mitigations depend on the cold start cause. For JVM warm-up, use GraalVM native compilation (eliminates JIT warm-up by pre-compiling to native binary at build time — used by Quarkus and Micronaut in Java, and standard in Go and Rust). For cache warm-up, pre-populate caches during startup by replaying recent request logs or by copying cache contents from an existing warm instance before routing traffic to the new one. For connection pool warm-up, establish the minimum connection pool size during startup before the first request is served rather than on demand.

Kubernetes HPA: Production Configuration

Kubernetes’ Horizontal Pod Autoscaler (HPA) is the standard autoscaling implementation for containerised workloads. It monitors metrics from the Kubernetes Metrics Server (for CPU and memory) or from custom metric adapters (for application-level metrics) and adjusts the replica count of a Deployment or StatefulSet.

The HPA algorithm calculates the desired replica count as:

desiredReplicas = ceil(currentReplicas × (currentMetricValue / desiredMetricValue))

If the current CPU utilisation is 80% and the target is 50%, the desired replica count is ceil(currentReplicas × 80/50) = ceil(currentReplicas × 1.6). With 5 current replicas, the HPA scales to ceil(5 × 1.6) = ceil(8) = 8 replicas.

Critical configuration parameters that determine HPA behaviour in production. minReplicas sets the floor — the minimum number of pods that will always be running regardless of metric values. Set this to at least 2 for any production service (1 replica means a single pod failure takes down the service during the time the replacement pod is starting). For services with cold start times, set minReplicas high enough that the steady-state load can be served without adding instances — adding instances during a spike is fine, but the baseline capacity should never fall below the level needed to serve normal load from cold instances. maxReplicas sets the ceiling — the maximum number of pods that can be provisioned regardless of metric values. Set this to prevent runaway scaling from metric anomalies, and size it against actual cluster capacity. scaleDown.stabilizationWindowSeconds (default 300) is the cooldown period for scale-down — HPA will not scale down until the metric has been below threshold for this duration. Increase this for services where brief metric drops should not trigger scale-down. behavior.scaleUp.policies controls how aggressively HPA scales up — policies can limit scale-up to a maximum of N pods per minute or a maximum percentage increase per period, preventing runaway scale-up that could exhaust cluster capacity.

HPA scales on CPU and memory by default. For most production workloads, custom metrics provide better scaling signal. KEDA extends HPA with a rich set of external metric sources — Kafka consumer lag, SQS queue depth, Prometheus queries, Datadog metrics, and many others. A KEDA ScaledObject on Kafka consumer lag scales the consumer deployment based on how many messages are waiting to be processed — exactly the right metric for an asynchronous processing service.

AWS Auto Scaling Groups: Production Configuration

AWS Auto Scaling Groups (ASGs) manage fleets of EC2 instances, scaling based on CloudWatch metrics or scheduled actions. The configuration structure is analogous to Kubernetes HPA — minimum and maximum instance counts, scaling policies that define the metric and threshold, and cooldown periods between scaling actions.

Target tracking scaling policies maintain a target metric value by automatically calculating the scaling adjustment needed. A target tracking policy with CPU target of 50% continuously adjusts the instance count to maintain average CPU near 50%. This is simpler to configure than step scaling (which requires explicit threshold brackets and adjustment values) and produces smoother scaling behaviour.

AWS Predictive Scaling analyses up to 14 days of CloudWatch metric history and forecasts the next two days of demand. It creates scheduled scaling actions that match the predicted demand, ensuring capacity is available before the predicted load arrives. Predictive Scaling can be combined with reactive Target Tracking — the predictive policy handles anticipated regular patterns while reactive policies handle anomalous demand.

Warm pools pre-warm instances by keeping them in a stopped or running state before they are needed. An ASG with a warm pool of 5 instances and a minimum instance count of 10 maintains 10 active instances and 5 pre-warmed instances. When scale-out is triggered, the 5 warm pool instances are activated (taking 30-60 seconds) rather than booting cold instances (taking 3-5 minutes). After activation, the warm pool refills by booting replacement instances in the background.

Autoscaling Failure Modes

Oscillation is the most common autoscaling failure. It occurs when scale-up and scale-down trigger in rapid succession — the system never reaches a stable state. Causes: cooldown period too short (scale-down triggers before the system has stabilised after scale-up), scaling target too aggressive (a target of 40% CPU on a service that idles at 35% means normal load triggers scale-up, which drops CPU to 20%, which triggers scale-down, which raises CPU to 35% again), or metric with high variance (brief CPU spikes that do not represent sustained load trigger unnecessary scale-up). Fix: increase cooldown periods, adjust scaling targets to have wider margins, use metric smoothing (average over multiple evaluation periods) rather than instantaneous values.

Scale-in data loss occurs when a stateful instance is terminated during scale-down before its state has been persisted. A database replica terminated abruptly may lose buffered writes. A session store terminated without draining may lose active sessions. A message processor terminated mid-processing may lose in-flight messages. Fix: implement graceful shutdown with connection draining and state persistence, configure terminationGracePeriodSeconds long enough for in-flight work to complete, and use at-least-once processing semantics with idempotent handlers so that reprocessing a message that was in-flight during termination produces correct results.

Under-scaling during rapid spikes occurs when traffic rises faster than the provisioning lag allows. A viral event that takes a service from normal load to 10× load in 30 seconds cannot be handled by reactive autoscaling with a 5-minute provisioning lag. Fix: combine predictive scaling with pre-warming for known event types, set higher minimum instance counts for services with high cold start times, and implement backpressure and load shedding as the safety mechanism that protects the system during the provisioning window.

Key Takeaways

  1. Reactive autoscaling responds to observed metric signals — it is simple and handles unpredictable demand but suffers from provisioning lag that can leave systems under-resourced during fast-rising spikes
  2. Predictive autoscaling forecasts future demand from historical patterns and provisions capacity in advance — it eliminates provisioning lag for predictable demand but requires regular temporal patterns and falls back to reactive scaling for anomalous demand
  3. The scaling metric must match the actual bottleneck — CPU is wrong for I/O-bound services, latency is right but oscillation-prone, RPS is predictive for stateless services, and queue depth or consumer lag is correct for asynchronous workloads
  4. Scale-up and scale-down are asymmetric — scale-up should be fast and aggressive (users are degraded), scale-down should be slow and conservative (oscillation risk); cooldown periods prevent the scale-down-triggers-scale-up oscillation loop
  5. Cold starts are the primary scaling challenge for stateful and warm-up-dependent services — JVM JIT warm-up, empty cache, and empty connection pool all mean new instances serve at reduced capacity for minutes after startup
  6. Kubernetes HPA with KEDA custom metrics is the production standard for containerised autoscaling — HPA manages the scaling mechanism, KEDA provides the right metric sources (Kafka lag, SQS depth, Prometheus queries) for application-specific scaling signals
  7. Scale-in data loss is prevented by graceful shutdown — SIGTERM handling, connection draining, terminationGracePeriodSeconds configuration, and at-least-once processing with idempotent handlers ensure that instances can be safely terminated without losing state or in-flight work

Frequently Asked Questions (FAQ)

What is the difference between reactive and predictive autoscaling?

Reactive autoscaling monitors a metric signal and scales when the signal crosses a threshold — it responds to load as it arrives. Predictive autoscaling uses historical patterns to forecast future demand and provisions capacity before the anticipated load arrives. Reactive autoscaling handles unpredictable demand correctly but suffers from provisioning lag — the delay between the scaling trigger and new capacity serving traffic. Predictive autoscaling eliminates this lag for demand with regular temporal patterns (business hours, daily cycles, known events) but cannot handle genuinely unpredictable spikes. Production systems combine both: predictive for regular patterns, reactive as a safety net for anomalous demand.

Why is CPU utilisation often the wrong autoscaling metric?

CPU utilisation accurately represents load only for compute-bound services — services that spend most of their time executing code. Many production services are I/O-bound, spending most of their time waiting for database responses, cache reads, or downstream service calls. An I/O-bound service can be severely overloaded with users experiencing high latency while CPU sits at 20% — adding CPU capacity does not help because the bottleneck is I/O throughput, not compute. Better metrics for I/O-bound services are request latency (p99), requests per second, queue depth, or downstream connection pool wait time — metrics that directly represent the capacity constraint rather than a resource that is not constrained.

What is the cold start problem in autoscaling?

The cold start problem occurs when a new instance cannot serve requests at full performance immediately after startup. JVM-based services run 2-10× slower for the first 2-3 minutes while the JIT compiler optimises hot code paths. Cache-warmed services experience high database load until their in-process caches fill with hot data. Connection-pooled services experience higher latency until connection pools are fully established. During the cold start period, new instances are less efficient than existing instances and may not provide the expected capacity relief. Mitigations include GraalVM native compilation (eliminates JIT warm-up), cache pre-population during startup, warm pools of pre-warmed instances, and minimum replica counts high enough to absorb baseline load without relying on cold instances.

What is autoscaling oscillation and how do I prevent it?

Oscillation occurs when autoscaling triggers scale-up and scale-down in rapid succession — the system never reaches a stable instance count. It is caused by cooldown periods that are too short (scale-down triggers before the system has stabilised after scale-up), scaling targets with insufficient margins (target CPU of 40% on a service that idles at 35% triggers constant scaling), or metric variance (brief spikes trigger scale-up, which drops the metric below the scale-down threshold, which triggers scale-down, which raises the metric). Prevention: increase cooldown or stabilisation window periods, set scaling targets with comfortable margins above the expected idle metric value, use metric smoothing over multiple evaluation periods rather than instantaneous values, and configure separate scale-up and scale-down policies with different aggressiveness levels.

How does Kubernetes HPA work?

Kubernetes Horizontal Pod Autoscaler continuously monitors metrics from the Kubernetes Metrics Server or custom metric adapters. When the observed metric value differs from the target, HPA calculates the desired replica count as ceil(currentReplicas × currentMetric / targetMetric) and adjusts the Deployment’s replica count accordingly. Key configuration parameters are minReplicas (floor — always running regardless of metric), maxReplicas (ceiling — never exceed), scaleDown.stabilizationWindowSeconds (cooldown for scale-down decisions), and behavior.scaleUp.policies (limits on how aggressively HPA scales up per period). KEDA extends HPA with external metric sources — Kafka consumer lag, SQS queue depth, Prometheus queries — enabling application-specific scaling signals beyond CPU and memory.

What is scale-in data loss and how do I prevent it?

Scale-in data loss occurs when a stateful instance is terminated during scale-down before its in-flight state has been persisted or transferred. A message processor terminated mid-processing loses the in-flight message. A database replica terminated abruptly loses buffered writes. A session store terminated without draining loses active sessions. Prevention requires three mechanisms working together: graceful shutdown (the instance handles SIGTERM by completing in-flight work before exiting), sufficient terminationGracePeriodSeconds (Kubernetes waits this long for graceful shutdown before force-terminating), and at-least-once processing with idempotent handlers (if a message is reprocessed after being in-flight during termination, the handler produces the same result and does not create duplicates).

Continue the Series

Series home: Distributed Systems — Concepts, Design & Real-World Engineering

Part 5 — Scalability & Performance

Previous: ← 5.7 — Indexing and Query Optimisation in Distributed Databases

Next: 5.9 — Geo-Distribution and Multi-Region Design →

Related posts from earlier in the series:

Discover more from Rahul Suryawanshi

Subscribe now to keep reading and get access to the full archive.

Continue reading