Cost and Capacity Planning at Scale in Distributed Systems

by
Home » Distributed Systems » Scalability & Performance » Cost and Capacity Planning at Scale in Distributed Systems

Distributed Systems Series — Part 5.10: Scalability & Performance

Scalability Without Cost Awareness Is Not Sustainable

Every scalability mechanism in Part 5 — partitioning, load balancing, caching, autoscaling, geo-distribution — has a cost. Partitioning requires running multiple database nodes rather than one. Load balancers add infrastructure. Caches require memory. Autoscaling provisions capacity that may not always be fully utilised. Multi-region deployment runs each layer of the stack in multiple locations simultaneously. These costs are individually justifiable. Collectively, without deliberate cost management, they compound into infrastructure bills that grow faster than revenue — which is not a scalability success, regardless of how the technical metrics look.

The engineers who build the most performant and reliable distributed systems are also the ones who understand that cost is an engineering constraint, not just a finance concern. A system that achieves 99.99% availability at $10 million per month when a well-designed system could achieve the same at $2 million per month has made a poor engineering decision — not just a poor business decision. Infrastructure cost is the result of engineering choices, and engineering teams own it.

This post covers the four cost dimensions of distributed systems infrastructure, unit economics as the framework for understanding cost at scale, capacity planning methodology, the cost implications of technical design decisions (consistency choices, replication factor, caching strategy), the optimisation ladder for reducing cost without reducing reliability, and how to build a cost model that makes trade-offs visible to both engineering and business stakeholders.

The Four Cost Dimensions

Distributed systems infrastructure costs fall into four dimensions, each with different characteristics and different optimisation strategies. Understanding which dimension dominates a system’s cost determines which optimisation approaches will have the most impact.

Compute cost is proportional to vCPU-hours — the number of virtual CPU cores multiplied by how long they run. Compute is typically the largest cost category for CPU-bound or memory-intensive workloads. The key characteristic of compute cost is that it scales with provisioned capacity, not with actual utilisation. An instance running at 10% CPU costs the same as an instance running at 90% CPU. Over-provisioning compute — running instances at low average utilisation — is the most common source of compute cost waste.

Cloud providers offer three compute pricing tiers. On-demand pricing charges the full hourly rate with no commitment — flexible but expensive. Reserved instances (AWS) or committed use discounts (GCP) offer 30-60% discounts in exchange for 1 or 3-year commitments — appropriate for the baseline compute that the system always needs regardless of traffic. Spot instances (AWS) or preemptible VMs (GCP) offer 60-90% discounts for compute that can be interrupted with short notice — appropriate for fault-tolerant, stateless, or batch workloads that can survive instance termination.

Storage cost is proportional to GB-months — the amount of data stored multiplied by how long it is stored. Storage costs are predictable (they grow with data volume at a known rate) and manageable through tiering. Cloud storage has multiple tiers with dramatically different pricing: AWS S3 Standard charges approximately $0.023 per GB per month for frequently accessed data, S3 Infrequent Access charges approximately $0.0125 per GB (with a retrieval fee), and S3 Glacier charges approximately $0.004 per GB for archival storage retrieved over hours or days.

Data lifecycle management — automatically transitioning data from hot to warm to cold tiers as it ages — is the primary storage cost optimisation. Application logs older than 30 days are rarely accessed; transitioning them to Infrequent Access at 30 days and Glacier at 90 days reduces storage cost by 80-90% for log data with no impact on typical operations. S3 Lifecycle Policies and GCP Object Lifecycle Management automate these transitions.

Network egress cost is charged for data transferred out of a cloud region — from cloud to internet or from one region to another. AWS charges approximately $0.09 per GB for data transferred from US regions to the internet, scaling down for higher volumes. Cross-AZ data transfer within the same region is charged at $0.01 per GB. Cross-region transfer is charged at $0.02 per GB between US regions and higher for international transfers.

Network egress is the cost dimension most commonly overlooked during system design and most surprising when first encountered at scale. A system transferring 1 PB per month of data from AWS to the internet pays approximately $90,000 per month in egress alone — before compute or storage. CDN offloading reduces this dramatically: CDN providers negotiate deeply discounted egress rates with cloud providers and serve cached content from edge nodes without routing traffic through the origin. A system that offloads 90% of read traffic to a CDN reduces egress cost by 90% for that traffic.

Managed service cost covers the per-request, per-unit, or per-feature pricing of cloud-managed services — DynamoDB charges per read capacity unit (RCU) and write capacity unit (WCU), Lambda charges per invocation and per GB-second of execution, API Gateway charges per million API calls, SQS charges per million requests. Managed service costs are invisible at small scale — a few dollars per month — and become significant at production scale.

DynamoDB’s pricing illustrates managed service cost at scale. One RCU allows one strongly consistent read per second for items up to 4KB. A read-heavy application serving 100,000 reads per second of 1KB items requires 100,000 RCUs — at $0.00013 per RCU-hour (on-demand pricing), this costs approximately $114 per hour, $82,000 per month. Switching to eventually consistent reads halves the RCU cost. Switching to a global table adds replication costs. Understanding the DynamoDB pricing model before designing the access patterns avoids architecture decisions that produce unexpected bills.

Unit Economics: The Right Way to Think About Infrastructure Cost

Raw infrastructure cost — total monthly spend on AWS — is not a useful engineering metric. It grows with business scale regardless of efficiency. The useful metric is unit economics: cost per business unit, normalised to the thing the business cares about.

Cost per request is the total infrastructure cost divided by the total number of requests served. As the business grows and request volume increases, cost per request should decrease (economies of scale, improved efficiency) or at worst stay flat. If cost per request is increasing as the business scales, the architecture is getting less efficient as it grows — a sign that a scalability bottleneck is being addressed by throwing infrastructure at it rather than solving it architecturally.

Cost per active user (monthly active user, MAU) is the total infrastructure cost divided by MAU. This is an investor-facing metric and an engineering metric simultaneously — it tells investors whether the unit economics improve as the business scales, and it tells engineers whether infrastructure is scaling efficiently with user growth. A SaaS business with $5 infrastructure cost per MAU at 100,000 users that is still at $5 per MAU at 1,000,000 users has achieved efficient horizontal scaling. A business at $5 per MAU at 100,000 users and $8 per MAU at 1,000,000 users has a scalability efficiency problem.

Cost per transaction is appropriate for transactional systems — payments, orders, bookings. It normalises infrastructure cost to the business value being created. For a payment processor, infrastructure cost per transaction should be measurable in fractions of a cent. If it is measurable in cents, the architecture is too expensive for the business model.

Unit economics make cost optimisation conversations concrete. “Our infrastructure bill is $500,000 per month” is hard to evaluate without context. “Our cost per request is $0.0005, and the industry benchmark for similar services is $0.0001” provides a clear optimisation target. “Our cost per MAU increased 40% this quarter despite only 20% user growth” identifies a scalability efficiency problem that engineering must address.

Capacity Planning Methodology

Capacity planning answers the question: how much infrastructure is needed to serve expected demand at the required performance level with sufficient headroom for uncertainty? Under-capacity produces user-visible degradation. Over-capacity wastes money. The goal is sufficient capacity with minimum waste.

The capacity planning formula: Required capacity = peak demand × (1 + safety margin) / target utilisation.

Peak demand is the maximum load the system must handle. This is not average load — systems must be sized for peak, not average. If the system serves 200 requests per second on average but 1,000 requests per second during the daily peak, the system must be sized for 1,000 RPS. Average utilisation of a system sized for peak will be low (200/1000 = 20% average utilisation), which is the correct and expected outcome for a peak-sized system.

Safety margin is the additional capacity beyond peak demand to handle unexpected spikes. A 20% safety margin provisions for 20% more than the measured peak. Safety margin accounts for measurement uncertainty (the measured peak may not represent the true worst case), growth between planning cycles (traffic may grow between now and the next capacity planning review), and the provisioning lag covered in Post 5.8 (autoscaling cannot respond faster than its provisioning lag allows, so baseline capacity must absorb spikes during the scaling window).

Target utilisation is the average utilisation level the system is designed to sustain. As established in Post 5.2, latency rises non-linearly as utilisation approaches 100% due to queuing effects. Latency-sensitive systems target 60-70% utilisation to maintain acceptable latency headroom. Batch and throughput-oriented systems can target 80-90%. The unused capacity (30-40% for latency-sensitive systems) is not waste — it is the latency headroom budget.

An example: a service with 1,000 RPS measured peak, 20% safety margin, and 70% target utilisation requires provisioned capacity for 1,000 × 1.2 / 0.70 = 1,714 RPS. If each instance handles 100 RPS at 70% utilisation, the service needs 18 instances (rounding up from 17.14). The 18 instances provide 1,800 RPS capacity against 1,200 RPS expected peak (1,000 × 1.2), for approximately 67% utilisation at peak — within the target range.

Capacity planning must account for lead time — the time required to provision additional capacity. For cloud infrastructure with autoscaling, lead time is minutes. For hardware that must be ordered, delivered, and racked, lead time is weeks to months. Capacity planning must initiate at lead time before the capacity is needed, not when the need becomes apparent. A system running at 80% utilisation with 3-month hardware lead time needs to order more hardware now — not when it hits 100%.

The Cost Implications of Technical Design Decisions

Every significant technical design decision in a distributed system has cost implications that should be understood before the decision is made. Four decisions with particularly large cost impact:

Consistency choice. Strong consistency (linearisability) requires synchronous replication — every write waits for acknowledgement from multiple replicas before completing. This adds latency to each write and reduces write throughput, requiring more instances to serve the same write rate. The compute cost of strong consistency versus eventual consistency can be 2-5× for write-heavy workloads. For data that does not require strong consistency — analytics events, social feed updates, product view counts — choosing eventual consistency reduces infrastructure cost without correctness consequences.

Replication factor. A replication factor of 3 (three copies of every byte of data) uses three times the storage of a replication factor of 1. For a petabyte-scale database, the difference between replication factor 2 and replication factor 3 is hundreds of terabytes of additional storage per month. Replication factor decisions should be made with explicit RPO and fault tolerance requirements — replication factor 3 tolerates two simultaneous node failures while factor 2 tolerates only one. For development and staging environments, replication factor 1 or 2 is appropriate. Production systems should justify their replication factor against their specific availability requirements rather than defaulting to 3 everywhere.

Caching strategy. Effective caching dramatically reduces infrastructure cost by serving read traffic from memory rather than compute and storage. A cache hit rate of 90% means 90% of read requests never reach the database — the database serves 10% of reads at a fraction of the cost that would otherwise be required. The cost of the cache tier (Redis or Memcached instances) is typically an order of magnitude less than the database capacity that would be required without caching. The cost optimisation case for caching is straightforward: calculate the cost of serving all reads from the database, calculate the cost of the cache tier plus the reduced database tier, and size the cache to achieve the target hit rate.

Data retention policy. Data that is kept indefinitely accumulates storage cost indefinitely. Most production data has a natural obsolescence curve — recent data is accessed frequently, older data is accessed rarely, and data beyond a certain age is almost never accessed. Explicit data retention policies — delete data older than 2 years, or transition to cold storage after 90 days — prevent storage costs from growing unboundedly. Compliance requirements may mandate specific retention periods; engineering teams should know these requirements and set retention policies to match exactly rather than retaining indefinitely “to be safe.”

The Cost Optimisation Ladder

Cost optimisation should follow a specific order. The most impactful optimisations first, the more complex ones after. Doing them out of order wastes effort — optimising reserved instance coverage before right-sizing means committing to the wrong instance types.

Step 1: Right-size. Identify over-provisioned resources and reduce them to match actual usage. An instance running at 15% average CPU is almost certainly over-provisioned — a smaller instance type could serve the same workload at lower cost. AWS Cost Explorer, GCP Recommender, and similar tools identify right-sizing opportunities automatically. Right-sizing should be done before any other optimisation because subsequent steps (reservations, spot) lock in pricing for the current instance types. Right-sizing first, then committing to pricing, avoids committing to the wrong configuration.

Step 2: Reserve or commit. For the baseline compute and managed service capacity that the system always needs regardless of traffic — the minimum instance count that is never scaled below — purchase reserved instances or committed use discounts. The 1-year savings are 30-40% for most instance types; 3-year savings are 50-60%. Reserved instances should cover only the stable baseline — not the peak capacity that autoscaling adds temporarily. Over-reserving (reserving peak capacity) pays the reservation cost for capacity that is only needed occasionally.

Step 3: Use spot or preemptible for fault-tolerant workloads. Spot instances and preemptible VMs are 60-90% cheaper than on-demand pricing and are appropriate for stateless, fault-tolerant, or interruptible workloads. Batch processing jobs that can be restarted, machine learning training that uses checkpointing, stateless application tiers behind load balancers that can absorb instance loss — all are candidates for spot instances. The risk is interruption: spot instances can be reclaimed by the cloud provider with 2 minutes notice (AWS) or 30 seconds (GCP). Applications must be designed to handle interruption gracefully — checkpoint state, drain connections, complete in-flight work within the notice period.

Step 4: Optimise data transfer. Reduce network egress costs through CDN offloading (serving reads from edge nodes rather than origin), data co-location (placing compute in the same region as the data it processes), and API response compression (reducing payload size reduces transfer volume). Cross-AZ data transfer costs can be reduced by designing services to communicate primarily within the same AZ rather than across AZs — consistent hashing in load balancers that routes requests to the instance nearest the client’s AZ reduces cross-AZ traffic.

Step 5: Implement data lifecycle management. Configure automated transitions from hot to warm to cold storage tiers based on data age and access patterns. Delete data that has reached the end of its retention period. Compress data before archival. These steps have no impact on user experience for data that is rarely accessed but can reduce storage costs by 60-80% for systems with large data volumes and typical access patterns skewed toward recent data.

Building a Cost Model

A cost model is a spreadsheet or tool that maps technical decisions to cost outcomes — allowing engineers to see the cost impact of architecture choices before making them rather than discovering them in the monthly bill.

A useful cost model has five components. The demand model captures current and projected traffic: requests per second, data volume, active users, and their growth trajectories. The resource model maps traffic to infrastructure: how many instances at what size for each service tier, storage volume per data type, network transfer volume. The pricing model applies cloud pricing to the resource model: on-demand vs reserved vs spot pricing for compute, tiered pricing for storage, egress pricing for network. The unit economics model derives cost per business unit from total cost and business metrics. The optimisation model shows the cost impact of specific optimisation decisions: what does moving from on-demand to reserved instances save, what does increasing cache hit rate from 80% to 90% save, what does adding a CDN save on egress.

The cost model serves two audiences. For engineering teams, it makes the cost impact of technical decisions visible before they are made — enabling cost-aware architecture decisions rather than post-hoc cost discoveries. For business stakeholders, it shows the infrastructure cost implications of business decisions — a marketing campaign that drives 3× normal traffic for a week has a specific infrastructure cost that can be calculated before the campaign runs.

Maintaining the cost model requires discipline — it must be updated when architecture changes, when traffic patterns shift, and when cloud providers change pricing. Teams that treat the cost model as a one-time exercise rather than a living document find that it quickly becomes inaccurate and stops being used. Monthly cost model reviews — comparing actual cost to modelled cost and investigating significant deviations — keep the model accurate and surface optimisation opportunities as they emerge.

Key Takeaways

  1. Infrastructure cost is an engineering responsibility — it is the result of technical design decisions, and engineering teams who understand cost dimensions and unit economics make better architecture decisions than those who treat cost as purely a finance concern
  2. The four cost dimensions — compute, storage, network egress, and managed services — each require different optimisation strategies; network egress is the most commonly overlooked at design time and the most surprising at production scale
  3. Unit economics (cost per request, cost per MAU, cost per transaction) normalise infrastructure cost to business outcomes and make efficiency trends visible — a cost per unit that increases as the business scales signals a scalability efficiency problem that engineering must address
  4. Capacity planning requires sizing for peak demand with a safety margin, divided by target utilisation — the unused capacity headroom is not waste but latency budget and autoscaling lag buffer
  5. Technical design decisions have direct cost implications — strong consistency versus eventual consistency can produce 2-5× compute cost differences for write-heavy workloads, and replication factor choices directly multiply storage costs
  6. The cost optimisation ladder — right-size first, then reserve, then spot, then data transfer, then data lifecycle — must be followed in order; reserving the wrong instance types before right-sizing locks in the wrong configuration
  7. A living cost model that maps technical decisions to cost outcomes enables cost-aware architecture decisions and makes infrastructure cost implications of business decisions calculable before they are committed to

Frequently Asked Questions (FAQ)

What are the main cost dimensions of distributed systems infrastructure?

Four cost dimensions determine distributed systems infrastructure cost. Compute cost (vCPU-hours) scales with provisioned capacity and is the largest category for CPU-bound workloads — optimised through right-sizing, reserved instances, and spot instances. Storage cost (GB-months) grows with data volume and is managed through tiered storage and data lifecycle policies. Network egress cost (GB transferred out) is charged for data leaving a cloud region and is often the most surprising cost at scale — CDN offloading is the primary optimisation. Managed service cost (per-request, per-unit pricing) is invisible at small scale and significant at production scale — DynamoDB, Lambda, and API Gateway costs require understanding the pricing model before designing the access patterns.

What is unit economics for distributed systems?

Unit economics normalises infrastructure cost to business outcomes rather than measuring raw monthly spend. Cost per request (total infrastructure cost divided by monthly request volume) should decrease or stay flat as the business scales — if it increases, the architecture is becoming less efficient. Cost per monthly active user (MAU) is an investor-facing and engineering metric that shows whether infrastructure scales efficiently with user growth. Cost per transaction is appropriate for transactional systems — it should be measurable in fractions of a cent for high-volume payment or order processing systems. Unit economics make cost optimisation targets concrete and make infrastructure cost visible to business stakeholders in terms they understand.

How do I calculate how much infrastructure capacity I need?

Required capacity = peak demand × (1 + safety margin) / target utilisation. Size for peak demand, not average — if the system sees 200 RPS average but 1,000 RPS peak, provision for 1,000 RPS. Add a safety margin (typically 20%) for measurement uncertainty and growth between planning cycles. Divide by the target utilisation (60-70% for latency-sensitive services, 80-90% for batch) to determine provisioned capacity. The unused capacity headroom is the latency budget and autoscaling lag buffer — it is not waste. Account for provisioning lead time: capacity planning must begin at lead time before the capacity is needed, not when utilisation reaches 100%.

What is the cost difference between strong and eventual consistency?

Strong consistency (linearisability) requires synchronous replication — every write waits for acknowledgement from multiple replicas before completing. This reduces write throughput and increases write latency, requiring more instances to serve the same write rate. The compute cost difference between strong and eventual consistency is typically 2-5× for write-heavy workloads. Additionally, strongly consistent reads in DynamoDB cost twice as many read capacity units as eventually consistent reads. For data that does not require strong consistency — analytics events, recommendation data, social feed updates — choosing eventual consistency reduces cost without correctness consequences. Only use strong consistency where correctness genuinely requires it.

What is the cost optimisation ladder and why does order matter?

The cost optimisation ladder is a sequence of optimisation steps ordered by impact and dependency. Step 1: right-size — identify over-provisioned resources and reduce them. Step 2: reserve — purchase reserved instances or committed use discounts for the stable baseline. Step 3: spot — use spot or preemptible instances for fault-tolerant workloads. Step 4: data transfer — reduce egress costs through CDN offloading and data co-location. Step 5: data lifecycle — automate transitions from hot to cold storage tiers. Order matters because later steps build on earlier ones: right-sizing must happen before reserving (to avoid committing to the wrong instance types), and reserving only the stable baseline (not peak capacity) requires understanding autoscaling patterns first.

What is a cost model and how do I maintain one?

A cost model maps technical decisions to cost outcomes — allowing engineers to see the cost impact of architecture choices before making them. It consists of a demand model (traffic projections), a resource model (infrastructure required per unit of traffic), a pricing model (cloud pricing applied to resource model), a unit economics model (cost per business unit), and an optimisation model (impact of specific cost reduction decisions). Maintain it as a living document by updating it monthly when architecture changes, when traffic patterns shift, or when cloud providers change pricing. Monthly reviews comparing actual cost to modelled cost surface optimisation opportunities and keep the model accurate. Teams that treat the cost model as a one-time exercise find it quickly becomes inaccurate and unused.

Continue the Series

Series home: Distributed Systems — Concepts, Design & Real-World Engineering

Part 5 — Scalability & Performance

Previous: ← 5.9 — Geo-Distribution and Multi-Region Design

Next: 5.11 — Distributed Queues and Async Processing →

Related posts from earlier in the series:

Discover more from Rahul Suryawanshi

Subscribe now to keep reading and get access to the full archive.

Continue reading