LLM VRAM and Throughput Calculator

Stored and executed parameters

The pinned model table separates checkpoint total, text execution pool, and active parameters per token. Stored weight bytes equal retained parameters times effective bytes/weight. FP16/FP32 use 2/4 bytes; GGML Q4_0 and Q8_0 blocks use 18/32 and 34/32 bytes/value. Generic 4/8-bit options include the editable metadata allowance. Mixed unquantized tensors and alignment can make an actual file larger.

For Qwen3.5-4B revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, the two safetensors headers sum to 4,205,751,296 text, 333,514,240 vision, and 120,599,552 MTP parameters. Together they give the Hub total 4,659,865,088. Ordinary text execution uses only the text pool. The retained-weight control follows an explicitly documented runtime baseline or an exact-file override: MLX LM filters vision/MTP; the vLLM multimodal baseline retains vision but its main-model loader skips MTP; llama.cpp text export excludes the separate vision projector and can retain MTP unless exported without it. Verify the actual export and runtime build.

Layer-aware cache and placement

Ordinary attention stores H_KV × (d_key + d_value) values per layer and retained token. Full attention retains the whole context; optimized sliding attention retains min(context, window). Optimized MLA replaces that coefficient by latent rank plus positional width: DeepSeek V2 Lite uses 512 + 64 values/layer; expanded K/V use 16 × (192 + 128). Hybrid recurrent layers receive no context-scaled KV allocation. Their fixed state remains separate from KV precision.

The Qwen3.5 row has eight full-attention and 24 recurrent layers, ordered RRR F repeatedly. Nominal fixed state is 24 × [32 × 128 × 128 × 4 + 8192 × 4 × 2] bytes: FP32 recurrent matrices and BF16 convolution history. Its allocation is runtime dependent.

CPU offload rounds the requested GPU percentage down to whole transformer layers. The last layers execute on the GPU, matching llama.cpp's layer-device cache allocation; preceding layers execute on the host. Full/sliding cache counts, fixed recurrent state and attention computation follow those layer devices. Standard rows supply explicit F/S/R order. State is distributed equally among the identical recurrent layers. Manual hybrid order is explicit; a custom row can describe another order.

Weight and linear-work shares default to the layer fraction. Unequal stored layer weights can use a separately entered measured GPU weight share; linear work still assumes equal work per transformer layer. Device memory is (device weights + device KV/state + device workspace + device runtime reserve + draft allowance) × planning margin. Host memory is (host weights + host KV/state + extra host buffers) × the same margin. Apple/CPU budgets apply the usable-memory share. Bisection finds the largest context satisfying both budgets and the model limit.

Decode traffic and work

Read weight traffic is b_weight × min(text execution pool, batch × active parameters), a conservative MoE expert-union bound. For each device, decode traffic adds its KV history read, new KV write, recurrent-state read and recurrent-state write. Recurrent allocation is one copy; per-step state traffic is two copies. Inactive vision/MTP tensors are never read as ordinary text work, even if retained in memory.

Active linear work is approximately 2 × active parameters × batch. Attention work is 2 × query heads × (d_key + d_value) × batch × retained attention positions on that device. GPU time is max(device traffic / effective bandwidth, device work / effective compute). CPU-layer time adds max(host traffic / sustained DDR bandwidth, host work / effective CPU compute), including host attention. These serial times form the decode step. There is no PCIe weight streaming in this model.

Prefill, compute provenance and energy

Prefill sums triangular causal pairs in full layers and window-capped pairs in sliding layers, separately on CPU/GPU. Prompt weight traffic uses the text execution pool, not unused auxiliary weights. Minimum modeled writes include KV updates and final recurrent state; activation traffic, padding and prefill chunking can add cost.

Manual compute is scoped to a named device. Hardware candidates never inherit another device's rate. Effective-compute row values are zero when not supplied; enter a source and precision/runtime/backend notes with any supplied rate. Without a compute assumption, prefill is unavailable and decode is marked with ≤ as the conditional bandwidth-model ceiling. This is not a statistical confidence interval or a hardware speed guarantee.

A measured net speculative factor adjusts decode only, already including draft/MTP/verification work. Additional resident draft memory is explicit. Average system power divided by aggregate decode rate gives joules/token. Derived speed/time/energy cards are suppressed when the workload is infeasible.

Compatibility and matched evidence

Format × platform restrictions are evaluated before architecture or placement caveats. The documented vLLM matrix rejects AWQ/GPTQ on AMD GPUs and GGUF on x86 CPU; a caveat never turns a definite unsupported result into a conditional one.

The matched-workload tables actually calculate Llama 2 Q4_0 pp512 and tg128. Prompt processing uses 512 input tokens. Generation averages modeled step times over contexts 0 through 127 from empty context; the displayed rate is the reciprocal mean time. KV storage is FP16, batch is one, and speculation is disabled. The comparison exposes discrepancies with the dated original runs; no utilization or formula is fitted to force agreement.

• Reference snapshot checked 26 October 2026. Config revisions, checkpoint totals, sources and check dates are on each row. Qwen3.5 component counts were summed from the pinned safetensors headers without downloading tensor payloads.
• Inference uses identical lengths per sequence, no shared prefixes, and memory-efficient attention. Workspace and runtime reserves are supplied independently of tensor storage.
• Default device workspace 1 GiB, device runtime reserve 0.75 GiB, margin 10%, bandwidth utilization 60%, usable unified/CPU share 75% and system power 300 W are planning scenarios. Additional host buffers default to zero and must be supplied when needed.
• The manual compute scenario is 100 numerical TFLOP/s with 35% utilization, scoped initially to RTX 4090 desktop. It is an assumption, not a vendor rating or benchmark. Unknown hardware-row compute values are deliberately unset. THz denotes the identical numerical scale of 10^12 operations/s; multiply and add count separately.
• CPU layer offload assumes the last whole transformer layers on GPU, normal KV offload enabled, and no overlapping CPU/GPU layer execution. Linear-work and default stored-weight fractions assume equal-sized layers. Use a measured weight fraction when actual tensor placement differs; nonuniform layer compute remains approximate.
• Qwen3.5 fixed-state dtype and convolution-history length follow the stated Transformers layout. Other runtime state packing, paged-cache padding and export choices can change memory. Vision/MTP retention is independent of whether their computation executes.
• GiB is binary; bandwidth is decimal GB/s. CPU DDR5-5600, two 64-bit channels, gives a theoretical 89.6 GB/s for the stated 64 GiB scenario. The offload DDR input asks for measured sustained bandwidth instead.
• The metric/US switch preserves information counts, time and energy semantics. This is static sizing, so no animation is needed.

This is a text-inference planning model for one selected device, optionally with serial CPU layer offload. Both device and host memory, plus model context, must be feasible before speed is shown as available. Fit is independent of runtime support and measured-data coverage. Detailed calculations retain analytical rates for diagnosis; visible rates and curves obey the shared feasibility predicate.

The compact fit codes are Yes (device, host and context all fit), Dev (device memory), RAM (host memory) and Ctx (model context). A dash in prefill means no device-specific compute assumption is supplied or the workload does not fit. A ≤ decode value is only a bandwidth-model ceiling at the chosen utilization. Unknown compute cannot rank real prefill performance across hardware. Enter effective rates for the actual precision, backend and runtime, or use the scoped manual profile.

Speed remains approximate: kernel launches, dequantization, router work, recurrent arithmetic, activation movement, CPU/GPU transfers, output projection placement and scheduling are simplified. Attention and cache/state placement follow the layer order, but unequal parameter/work distributions across layers need more detailed profiling. Context capacity holds temporary workspace fixed. Add CPU buffers to the host allowance and verify peak prefill memory.

MLA, sliding-cache eviction and hybrid-state packing depend on runtime support. Choose conservative cache allocation or exact-file retention overrides when applicable. The vLLM compatibility rules follow its checked hardware matrix; future versions can differ. Runtime baselines describe the cited loader/export paths, not every frontend or conversion option.

Matched benchmark comparisons reset the workload to pp512 or tg128 instead of silently comparing different contexts. Their analytical rates need not match measurements. Current-workload ratios against a nearby reference remain explicitly labelled as potentially different workloads. No measurements or compute profiles are invented, and no llmfit community dataset is redistributed.

Excluded: training, optimizer states, image/video execution, a predictive speculative-acceptance model, disk swapping, multi-GPU communication and PCIe weight streaming. Measure memory, sustained DDR bandwidth, kernel throughput and average system power for the actual deployment.