What is Little's Law?

Little's Law states that the average number of items in a system (N) equals the arrival rate (λ) multiplied by the average time each item spends in the system (W): N = λ × W. For HTTP services, N is concurrent requests, λ is RPS, and W is average response time in seconds.

Why use 80% target CPU utilization?

Running at 100% CPU leaves no room for traffic bursts, GC pauses, or background tasks. 70–80% is a widely used target that balances cost efficiency with resilience. For latency-sensitive services, 60–70% is safer.

What should I set for "max concurrent requests per instance"?

This depends on your runtime. A Node.js server handling async I/O can often sustain 100–500 concurrent requests, while a CPU-bound Python/Java service may only handle 10–50. Check your framework's thread pool or event loop limits.

Does this account for connection overhead or warm-up time?

No — this is a steady-state estimate using Little's Law. Add at least 1–2 extra instances beyond the recommendation to cover rolling restarts, health-check failures, and cold-start latency.

How is "capacity headroom" calculated?

Headroom = ((recommended instances × concurrency per instance × target CPU) / steady-state concurrency − 1) × 100%. It shows how much extra RPS your recommended fleet can absorb before hitting the utilization ceiling.

Load Balancer Sizing Calculator

Name: Load Balancer Sizing Calculator
Availability: InStock
Author: Nham Vu

Estimate how many backend instances you need behind a load balancer given requests per second, average response time, and target CPU utilization.

Traffic & Instance Parameters

Peak Requests per Second (RPS)

Total incoming requests the load balancer distributes each second.

Average Response Time (ms)

Mean end-to-end latency per request, including network and processing.

Max Concurrent Requests per Instance

Thread pool, worker count, or async concurrency limit per instance.

Target CPU Utilization: 80%

10%Conservative 60%Recommended 80%100%

Sizing Estimate

Fill in the parameters and click Calculate.

Traffic Spike Scenarios

Results will appear here after calculation.

Traffic Multiplier	RPS	Instances Needed

Formula: Concurrency = RPS × (latency_ms / 1000) | Instances = ceil(Concurrency / (max_concurrent × cpu_target))

Recommended = Instances × 1.20 (ceiling). Add extra instances for rolling restarts and cold starts.

Summary

Estimate how many backend instances you need behind a load balancer given requests per second, average response time, and target CPU utilization.

How it works

Enter the peak requests per second (RPS) your service receives.
Enter the average response time in milliseconds for a single request.
Set the maximum number of concurrent requests each instance can handle.
Set the target CPU utilization (80% is a safe default to leave headroom).
The calculator derives steady-state concurrency via Little's Law (N = RPS × latency in seconds), then divides by concurrency per instance and CPU target to get the required instance count.
Results show minimum instances, recommended instances with headroom, and total throughput capacity.

Use cases

Plan the number of Node.js or Go service pods in a Kubernetes deployment.
Estimate EC2 instance count behind an AWS Application Load Balancer.
Validate whether an existing fleet can absorb a 2× traffic spike.
Compare instance counts across different response time SLA targets.
Right-size a staging environment to mirror production load.
Calculate required concurrency when migrating to a new instance type.

Frequently Asked Questions

Last updated: 2026-07-22 · Reviewed by Nham Vu