Multi-GPU LLM Calculator: VRAM Split and RAM Offload

Memory placement

Weights use the pinned Hugging Face safetensors parameter count and packed bytes per value: FP16 = 2; Q4_0 = 18/32; Q8_0 = 34/32 (GGML block definitions, s19). Other 4/8-bit formats use editable metadata overhead. An actual loaded tensor allocation replaces the estimated stored bytes. Effective bytes/parameter = overridden bytes ÷ retained parameters; the same density scales executed dense, routed-expert and output weight traffic. Unused vision/MTP tensors occupy memory but have no ordinary text traffic. An override alone cannot identify unequal tensor precisions; allocation among components is proportional to their parameter counts.

For full attention, KV bytes are $B\,C\,L\,H_{KV}(d_K+d_V)b_{KV}$. GQA uses KV heads, not query heads. Compressed MLA uses the stored latent + rotary dimensions; expanded layout uses the ordinary per-head tensors. Sliding layers retain min(context, window) only in optimized allocation. Hybrid F/S/R order assigns full attention, sliding attention and fixed recurrent state to the correct contiguous blocks. The Qwen3.5 state count follows the shared #770 tensor-dimension derivation.

Each device requires headroom × (weights + KV/state) + runtime reserve + draft allocation. System RAM holds CPU weights/cache, optional extra model copy and host reserve. Apple/CPU capacity is installed memory × usable share, one pool only. Automatic layer allocation fills later device slots first and spills a contiguous prefix into CPU RAM. Input embeddings, final projection/norm, non-executed auxiliaries and decoder-block weights are separate pools. Vocabulary × hidden size and tie_word_embeddings come from each pinned config; a tied input/output matrix is counted once in the output pool. Remaining block weight mass is approximated as uniform. Input lookup reads only touched rows. Exact quantized tensor sizes and allocator layout still require runtime logs.

Decode and prefill

For a serial layer split, $t_{decode}=\sum_i\max(D_i/(\eta_i BW_i),F_i/C_i)+t_{CPU}+t_{link}$. Tensor mode replaces the GPU sum with the slowest concurrent shard. CPU bandwidth is channels × transfers/s × 8 bytes × efficiency; DDR5 subchannels are not counted twice. CPU experts execute on CPU, rather than streaming complete expert weights over PCIe every token. For batch tokens T, each assigned routed-expert pool reads min(assigned storage, assigned active-expert bytes × T). CPU and GPU fractions are capped independently. Shared dense block weights and the output matrix are read once per batch, not once per sequence; compute remains proportional to executed tokens. Routed expert pools are derived from pinned config dimensions; shared experts belong to the dense pool. Prefill uses T = batch × prompt tokens.

Link time is bytes / sustained one-way bandwidth + message count × latency. Tensor mode assumes two reductions per GPU layer and a ring-equivalent per-rank traffic factor $2(n-1)/n$. Layer mode transfers hidden activations across device boundaries. GPU-to-GPU host staging pays two serial PCIe legs. CPU-to-GPU layer/expert activations use a separate one-leg PCIe path, even when the GPUs communicate over NVLink. This is a transparent topology approximation, not a collective implementation simulator.

Prefill includes $2P_{active}BC_{prompt}$ operations and full/sliding attention pairs. Historical Q4_0 profiles infer effective operations/s as 2 × 6.738B × original measured pp512; scaling to another architecture, precision or backend is an extrapolation. Profiles apply only to llama.cpp Q4_0. Other precisions/runtimes require manual or custom-row effective compute; the model scan otherwise shows a bandwidth ceiling and leaves prefill blank. Manual or custom-row effective compute overrides are available. Unknown compute gives a decode bandwidth ceiling and no prefill prediction. Cache traffic uses the active sliding window independently of reserved capacity. MLA access uses the selected compressed/expanded tensor layout; recurrent state is read and written once per iteration. Decode is per sequence at final context; prefill is aggregate across sequences. Net speculative gain affects decode only; energy = assumed whole-system watts / aggregate decode tokens/s.

The benchmark table compares the bandwidth model at the midpoint (64 tokens) of a reference tg128 run with its measured rate. It fixes batch 1, Q4_0 and FP16 KV. Prefill profiles are calibrated from these same measurements, so prefill agreement is not independent validation. The nearest reference remains separate from the user's split verdict.

Pinned llama.cpp placement convention

The loader at 961c376a2daa42220ec09663b9bde32c5fba1f70 (https://github.com/ggml-org/llama.cpp/blob/961c376a2daa42220ec09663b9bde32c5fba1f70/src/llama-model.cpp) (26 October 2026) uses i_gpu_start = max(n_layer_all + 1 − n_gpu_layers, 0). Thus N offloaded decoder blocks require N+1 including the output layer. The generated command uses -ngl all for all blocks, -ngl N+1 for a partial split, and -ngl 0 for CPU-only. Layer proportions include the output-layer position. Its weights are reserved on the last active GPU; tensor mode shards them. Input embeddings and retained auxiliary tensors stay in CPU RAM on discrete rigs. Unified systems charge everything to one pool. Custom rows must provide input/output component counts. Other runtime loaders and optional predictor blocks need separate log verification.

Reading the compact cards

Devices = all decoder blocks fit; host input/auxiliary tensors can remain. CPU blocks / CPU experts indicate the relevant offload. Runtime 'Verify' is provisional; 'Unsupported' suppresses the speed. 'Nearest Rn' is a nearby single-device reference, not a matched workload. Speed cards show per-sequence decode and aggregate prefill in tokens/s. Rows use source IDs Hn, Mn and Rn, whose full links appear below the calculator. Scroll the table vertically for additional rows.

Original measurement conditions

• R1: M2 Ultra 76-core; metal; 2023-11-22; build 8e672ef. ggerganov original M2 Ultra 76-core result; Metal; Llama 2 7B 6.74B; 3.56 GiB; pp512; tg128 from empty context; ngl99; historical baseline.
• R2: RTX 4090 desktop; cuda; 2025-08-06; build 224145325. lhl; CUDA 12.9; driver575.64.03; Llama 2 7B; 3.56 GiB; pp512/tg128; ngl99; FA off.
• R3: RX 7900 XTX desktop; vulkan; 2026-09-10; build 304665f. Diablo-D3 Powercolor Red Devil; Vulkan; Adrenalin26.8.1; Windows11; Llama 2 7B; pp512/tg128; ngl100; FA off; r100.
• R4: RX 7900 XTX desktop; rocm; 2026-09-10; build 304665f. Diablo-D3 Powercolor Red Devil; HIP/ROCm; Adrenalin25.10.2; Windows11; Llama 2 7B; pp512/tg128; ngl100; FA off; r100.
• R5: RTX 3090 desktop; cuda; 2025-08-01; build c76b420. m18coppola original comment; CUDA; no FA; Llama 2 7B; pp512/tg128; single GPU, not a dual-3090 measurement.
• R6: RTX 5090 desktop; cuda; Report date not shown; scoreboard checked 2026-10-27; build 8cf6b42. totaldev selected no-FA scoreboard row; pp512/tg128; ngl99; 3.56 GiB; original report date unavailable.
• R7: M4 Max 40-core; metal; Report date not shown; checked 2026-10-27; build 8e672ef. Selected historical Metal row, Llama 2 7B; pp512/tg128; memory configuration not reported.
• R8: M3 Ultra 80-core; metal; Report date not shown; checked 2026-10-27; build 8e672ef. Selected historical Metal row, Llama 2 7B; pp512/tg128. Vendor BW 819; discussion rounds it to 800.

• Snapshot checked 26–27 October 2026. Hardware/model records reuse #770 revision 638 schema, row keys, pinned HF commits and source fields, with 3090/5090, newer Mac configurations and larger dense models added here. The second calculator is not modified.
• GB/s is decimal bandwidth; memory allocations use binary GiB. Vendor-labelled memory is interpreted as binary installed capacity; confirm OS-reported bytes.
• The default 65% memory/DDR efficiency, 75% link efficiency, 15 µs latency, 75% usable unified share, 1.5 GiB device reserve, 5% payload headroom, 2 GiB host reserve and 650 W wall draw are editable planning assumptions, not measured specifications.
• Weight-size overrides apply one effective byte density to retained/executed component counts; nonuniform quantization and extra tensor copies require manual budgeting. The roofline and partition equations are derived here. They count traffic and work; they do not fit efficiency coefficients to a benchmark.
• Simultaneous sequences reuse dense weights once per batch. Distinct routed-expert reads use a capped linear union approximation, separately bounded by CPU- and GPU-assigned storage. Actual routing overlap may be lower. No prefix sharing, KV deduplication or operating-system swap.
• The context sweep recomputes whole-layer allocation. Jumps represent a layer moving to CPU RAM. It is a capacity-first allocation, not an automatic speed optimizer.
• Only eight individually cited llama.cpp measurements are selected. No llama.cpp scoreboard or llmfit community dataset is mirrored. No llmfit code/data or Epoch AI data is redistributed, so their licences are not invoked.

Runtime and interconnect limits

Memory fit, runtime support and measured evidence are independent. Unsupported configurations suppress displayed speeds; a 'verify' verdict is provisional. llama.cpp GGUF, vLLM, MLX and ExLlama formats are checked conservatively against maintainer documentation. CPU layer execution is different from vLLM weight swapping; confirm the actual offload mechanism. Quantized KV, MLA and hybrid adapters require the exact build's kernels.

llama.cpp tensor mode is experimental, requires flash attention and currently documents non-quantized KV. DeepSeek2 is on its unsupported architecture list. Other tensor cases require validating whole-head sharding, actual split ratios and collectives. Approximate fractional layer-equivalents do not guarantee a legal shard. Deprecated row mode is not modelled.

3090 NVLink requires a compatible bridge and software path; 4090/5090 do not have NVLink. Never assume CUDA P2P works just because PCIe slots exist. Driver, IOMMU/ACS, OS, NCCL/RCCL, NUMA placement and chipset sharing can change both correctness and speed. Enter negotiated lanes and the bottleneck generation, not connector size. Test transfers on your actual machine. Metal's recommended working-set limit is not necessarily the chosen planning share.

The planner does not predict kernel launch overhead beyond editable link latency, exact per-block tensor placement, quantization quality, heterogeneous kernel efficiency, thermal/power limits, server scheduling, disk paging, multi-node RPC, loading time or total cost. Prefill workspace is reserved rather than simulated; enlarge it for long contexts and large batches. A draft multiplier does not establish draft compatibility or output acceptance.

Historical measured rows differ in build, driver and sometimes flash attention. Dates missing from the original selected scoreboard row are marked unavailable and given a check date instead. The matched tg128 midpoint approximation is not a confidence interval. The 2×3090/large-model/RAM scenarios have no matched measurement in this selected sample; benchmark before making a purchase.

The common 65% device-bandwidth assumption overpredicts some Apple reference rows. The visible matched estimate/measured ratios show why this factor is not a universal accuracy claim. Calibrate each platform before comparing close results.

Sliding cache allocation is independent of attention access: Mistral documentation (https://huggingface.co/docs/transformers/v4.38.0/en/model_doc/mistral). A large conservative cache may force CPU spill, which can legitimately change speed; it does not increase per-device attention reads while the same layers remain resident. NVLink capability is an explicit hardware field; unknown custom configurations must use PCIe or provide verified link capability. A physical NVLink bridge and working runtime path are still required on supported 3090s.