LLM Serving Capacity: KV Cache, Concurrent Users and Queue

Memory: $M_{KV}=uM_{VRAM}-M_W-M_R$. Raw conventional cache bytes/token are $2L H_{KV}d_hb$; compressed MLA uses $L(d_c+d_R)b$ (DeepSeek-V2, attached s3, Sections 2.1.2–2.1.4). Apply replication and metadata factors, cap sliding layers at the window, and round each retained length to complete blocks. The rolling-window storage cap follows Mistral 7B (s10 p.2). Divide the available pool by the reserved per-sequence footprint and round down. PagedAttention (s1 p.4) provides the independent OPT-13B memory example; GQA (s2 p.2) explains shared KV heads.

Roofline: one generated token uses approximately $2P+4LH_Qd_hT$ operations. Each decode iteration takes the larger of compute time and weight-plus-KV bandwidth time, plus launch overhead. For each prefill chunk with $n$ new tokens after $p$ prompt tokens, operation demand is $2Pn+4LH_Qd_h(np+n^2/2)$. Bandwidth demand includes weights plus one read of the retained preceding KV cache, including replication and cache overhead. The retained prefix is capped separately for sliding layers (Mistral 7B, s10 p.3). The chunk time is the larger compute/bandwidth demand plus launch overhead; prefill sums those chunk times. Arithmetic-series sums split at the sliding-window and roofline crossovers, so even one-token chunks need no long simulation loop. The compute input is in TFLOP/s: one operation/ps equals one trillion operations/s, converted explicitly to operations/s. These are our dense-decoder work estimates, following the arithmetic approach of kipply, not fitted performance curves. The service proxy adds the first-token timeline and the remaining $O-1$ decode iterations; completed throughput is $c/S(c)$. It treats prefill and decode resource demands as serialized. Orca (s5) motivates iteration-level admission; the calculator does not reproduce its scheduler.

The batch TTFT screen charges $ct_{prefill}$. When $c>1$ and $O>1$, it also charges $K-1$ decode interleaves before the first token. Completion uses exactly that first-token timeline followed by $O-1$ decode iterations. For one-token responses, no decode interleaves or output decode steps are charged: TTFT and completion coincide. This is a deliberately simple batch-interference proxy, not a scheduler simulation. Decode-only ITL is $t_{decode}$; an interleaved chunk adds the longest chunk duration, illustrating Sarathi-Serve's chunking tradeoff (s4 pp.3–4). For measured mode, $S(c)=c/R_{measured}$; TTFT and ITL use user measurements for multi-token outputs. For a one-token output, first-token time is the same inferred occupancy, and redundant TTFT/ITL inputs are hidden. Below calibrated concurrency, the graph conservatively holds measured slot occupancy constant.

Queue: $\rho=\lambda S/c$. An Erlang B recurrence gives numerically stable Erlang C, followed by $W_q=C S/[c(1-\rho)]$, only for $\rho<1$. Response time is $W_q+S$ and Little's law gives $L=\lambda(W_q+S)$. This is a fixed operating-point M/M/c surrogate for shared batching.

Agents: every stage is a fork of identical calls followed by a join. Local waves respect both requested parallelism and effective server slots. The dependency-only bound is one call per stage plus interstage delays. API launch spacing is the reciprocal of the available minimum of RPM and TPM divided by charged tokens/call. A deterministic schedule enforces that spacing globally, worker completion times and every stage barrier. Rental cost uses graph execution time; API token cost uses input and output tokens, independent of parallelism.

All requests have the same representative prompt and output length. Peak memory reserves output in advance; no prefix sharing, offload, beam branching or speculative draft cache is credited. The overhead factor includes cache quantization scales; weight quantization is already reflected in resident weights.

One balanced replica group is modeled. Cache replication is an explicit multiplier; memory and bandwidth cannot be pooled freely across unrelated replicas or imbalanced pipeline stages. The roofline assumes a dense decoder with equal key/value dimensions. Select measured mode for MoE and MLA. Its full-context attention FLOPs are conservative for sliding-window models.

The server queue is an open Poisson arrival stream with independent exponential service at a fixed target-batch calibration. Actual continuous batching violates the independent-server assumption. Slot count is capped by memory; the displayed server slot value is only meaningful after memory conditions pass.

Agent comparison uses a dedicated local replica and excludes the server queue's background traffic. The API has constant queue-excluded service time, a common combined token bucket, smooth pacing, no burst credit and no retries. For separate input/output buckets, convert each to calls/minute and enter the tightest equivalent combined budget. Tool delay occurs only between stages. Every agent call uses the workload token counts.

All price and quota defaults are illustrative editable assumptions dated 2026-10-27, not provider offers. Local rental excludes idle time before/after execution, billing minimums, energy and operations; API cost excludes caching discounts, tools and failed calls.

Sliding-window storage assumes an ideal rolling buffer rounded to blocks; allow extra overhead if the backend keeps additional boundary or look-ahead blocks.

Prefill bandwidth counts one preceding-KV read per chunk and assumes new-chunk KV reuse within the kernel. It does not model extra memory rereads across query tiles; use measured calibration when kernel tiling, cache writes or communication dominate. The ITL stall screen uses the longest chunk, including a shorter final chunk if present.

Use for screening capacity and identifying which parameter to benchmark next. A memory ceiling is not a safe production concurrency setting or a latency SLO. Saturated throughput is held fixed while arrival rate is swept; light-load batch changes and scheduler feedback are not modeled.

There is no finite steady-state queue result at or above capacity: time cards read “Unbounded”. Near saturation, small calibration errors produce large wait errors. The TTFT/interleave screen uses a serialized batch-interference timeline and is not a percentile guarantee. In measured mode, roofline-only table columns remain labeled as such and are not used for queue or graph throughput.

Graphs are homogeneous layered fork-and-join workloads with deterministic calls. They do not represent arbitrary DAGs, asynchronous agent reasoning, stragglers, token-length distributions, cancellations or shared provider traffic beyond the quota fraction. A concurrency ceiling is not a token-bucket burst allowance.

Validate on the exact server configuration and traffic mix. Vidur (s6) is a reference for calibrated simulator-level studies; this calculator makes no simulator-grade accuracy claim. Quotas and prices must be checked against the user's account and date of use.