What is Little's Law?

Little's Law states that in a stable system, the average number of items present (N) equals the average arrival rate (λ) multiplied by the average time each item spends in the system (W): N = λ × W. For a web service, N is the concurrent workers needed, λ is the request rate, and W is the average latency.

Should I always provision exactly N = λ × W threads?

That is the minimum for a fully-loaded system. In practice, add a safety buffer (10–30%) so that latency spikes or garbage-collection pauses do not instantly saturate the pool and cause queuing. The tool lets you dial in a buffer percentage.

What happens if my pool is smaller than the Little's Law result?

Requests queue waiting for a free thread. Queuing latency adds to the average W, which raises N further — a feedback loop that compounds under sustained load. The symptom is P99 latency growing much faster than P50.

What is the CPU-bound ceiling?

Even with unlimited threads, the CPU can only execute so many instructions per second. The CPU ceiling is cores / (latency_in_seconds × CPU_fraction). For I/O-heavy work where threads spend most time waiting on network or disk, the CPU ceiling is very high and the Little's Law result dominates. For CPU-heavy compute, the ceiling is the binding constraint.

Does this apply to async / event-loop runtimes?

Little's Law still holds for event-loop systems — the "N" becomes in-flight async tasks rather than OS threads. The key difference is that async runtimes can reach high concurrency with far fewer OS threads, so memory overhead per concurrent request is lower.

How does database connection pool size relate to this?

Apply the same formula to the DB layer: λ_db (DB queries per second) × W_db (average query latency). That gives the minimum connection count. Database servers typically cap connections at 100–500 due to memory limits, so this calculation helps you negotiate that ceiling against your service's needs.

Concurrency Calculator

Name: Concurrency Calculator
Availability: InStock
Author: Nham Vu

Enter target throughput, average latency, and CPU core count to calculate the ideal thread pool size using Little's Law (N = λ × W).

Service Parameters

Little's Law: N = λ × W

Target Throughput (λ) — req/s

Requests per second the service must sustain at steady state.

Average Latency (W) — ms

End-to-end time per request including DB, network, and downstream calls.

CPU Cores

Logical cores (vCPUs) available to the service process.

Advanced

Safety buffer (%)

Extra headroom above the Little's Law minimum. Recommended: 20–30%.

CPU active fraction

Fraction of each request's time spent on CPU (not waiting). I/O-heavy: 0.05–0.20. CPU-heavy: 0.70–1.0.

Little's Law Result

Min Concurrency (N)

—

threads / workers

Recommended Pool

—

with 25% buffer

Latency Impact on Pool Size

How pool size changes if average latency drifts from your baseline.

Latency	N (min)	N (recommended)

Set your parameters and click Calculate

Summary