Home » Calcs » Fine-Tuning VRAM and Time Calculator: Full, LoRA and QLoRA

Fine-Tuning VRAM and Time Calculator: Full, LoRA and QLoRA

Estimate inference and training VRAM, maximum sequence length and microbatch, runtime and cost for full fine-tuning, LoRA and QLoRA. Compare GPU sharding, checkpointing and editable MFU assumptions.

Wired White

Published 28. October 2026 · version 1

#fine-tuning #FSDP #GPU-cost #LoRA #MFU #QLoRA #VRAM #ZeRO

A model that loads for inference can still run out of memory during its first training step. This fine tuning VRAM calculator compares full fine-tuning, LoRA and QLoRA for a dense decoder language model. It is intended for engineers, students and small teams planning a local workstation run or a GPU rental.

Start with the checkpoint's parameter count and architecture: layers, hidden width, feed-forward width, query heads, KV heads and vocabulary. The defaults describe a rounded Llama-2 7B example on one 24 GiB GPU, using QLoRA at 2,048 tokens and one sequence per microbatch. Replace these values for your model; a model's advertised “7B” size is only an approximation.

Choose the sequence length and microbatch per GPU. Then set gradient accumulation to reach the effective global batch you need. Accumulation processes microbatches in succession, so activation memory is unchanged. With FSDP deferred synchronization (no_sync), however, full-model gradients remain on every GPU until synchronization. Select the synchronization policy your loop actually uses and match its stored gradient precision; FSDP normally uses FP32 gradients. The comparison tables show the three methods under the same architecture and memory settings, including maximum sequence length at the current microbatch and maximum microbatch at the current sequence. A zero means that combination cannot fit. These two maxima describe separate choices; do not combine them into one proposed run.

Enter VRAM for each GPU, not the sum across devices. Replicated training keeps a complete model on every GPU. ZeRO and FSDP distribute specified model states, while activations remain local. If you are investigating LoRA VRAM for 70B, this distinction is essential: simply adding a second card does not pool its memory. Full sharding of quantized weights requires a compatible software stack and still needs temporary layer gathers and unsharded gradient buffers. Choose the backward prefetch policy to estimate which weight units coexist.

The inference result includes weights, a 16-bit KV cache at the entered batch and context, and a prefill workspace allowance. It assumes one complete inference model per GPU, even when training uses sharding. Training includes gradients, Adam state, activations, forward/backward loss buffers and explicit framework allowances. Standard head buffers follow materialized Hugging Face/PyTorch cross-entropy; a custom or fused loss requires a measured workspace coefficient. This makes the QLoRA memory calculator useful for understanding where the extra memory goes.

For the training time calculator, enter the dense BF16 arithmetic peak, processed tokens and an MFU appropriate to your actual implementation. The three MFU defaults are planning assumptions. A pilot run is the best calibration: compilation, quantization kernels, communication and checkpointing can change throughput substantially. The displayed time range is an MFU scenario range, not a statistical confidence interval.

Finally, enter your own GPU-hour rental rate or local electricity tariff. Prices are planning inputs dated to your quote or bill, not live offers. Memory fit is an estimate rather than an OOM guarantee. CPU offload, tensor parallelism, mixture-of-experts models, multimodal inputs, reinforcement learning and training quality are outside this tool.

How it is calculated

Persistent model state

Following ZeRO (https://arxiv.org/abs/1910.02054), attached PDF pp.8–11, mixed-precision Adam uses 2 bytes for weights, 2 for 16-bit gradients, 4 for master weights and 8 for its two moments: 16 bytes per trainable parameter with 16-bit gradients; selecting FP32 gradients adds 2 bytes. The 8-bit option replaces the two moments with approximately 2 bytes in total, retaining the master copy; small tensors and scale metadata are covered by allowances. Full training updates the base. LoRA/QLoRA freeze it and allocate trainable state only for adapters.

LoRA (https://arxiv.org/abs/2106.09685), §4.1 and p.20: each targeted matrix contributes $r(d_{in}+d_{out})$ parameters. Sum these across the selected projections and all layers. KV width is $d H_{KV}/H$ . A gated MLP has gate, up and down projections; a dense MLP has two. Embeddings and the output head are excluded from adapter targets.

QLoRA (https://arxiv.org/abs/2305.14314), p.5: NF4 needs $4+8/64+32/(64\cdot256)=4.126953125$ bits per quantized parameter with double quantization, or 4.5 bits without it. Embeddings, the head and normalization weights remain 16-bit. A temporary BF16 buffer for the largest linear matrix is added conservatively.

Activations and peak VRAM

Korthikanti et al. (https://arxiv.org/abs/2205.05198), pp.4 and 9, gives $BSdL(34+5HS/d)$ bytes for a conventional MHA transformer. Here the linear coefficient is generalized to $18d+4d_{ff}$ for GeLU, recovering $34d$ when $d_{ff}=4d$ . The gated-MLP extension $18d+6d_{ff}$ is an engineering allowance derived here, not a claim from that paper. GQA keeps this conservative MHA buffer allowance. Memory-efficient attention omits the quadratic score-storage term.

Without checkpointing, internal activations are retained for all layers. With layer checkpointing, store $2BSdL$ boundary bytes plus one layer's working activations. Add small rank-dependent adapter buffers and the selected loss/head workspace. The activation multiplier applies to the boundary and layer estimate; logits and adapter buffers are separate.

The standard head follows Transformers ForCausalLMLoss (https://github.com/huggingface/transformers/blob/main/src/transformers/loss/loss_utils.py) and PyTorch cross-entropy (https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/LossNLL.cpp): BF16 logits, their FP32 upcast and FP32 log-softmax can coexist, totaling 10 bytes per vocabulary cell in the forward pass. During log-softmax backward, retained BF16 logits, saved FP32 log-softmax, its incoming gradient and its outgoing gradient total 14 bytes per cell. The standard estimate uses this larger peak. An independent CPU PyTorch 2.6 profiler run with 64 × 2,048 cells measured 14.004 bytes/cell including small label/scalar tensors; it checks tensor lifetimes, not CUDA allocator behavior. The custom/fused option uses the entered measured coefficient instead of assuming a particular kernel's saving.

ZeRO-1 shards optimizer/master state; ZeRO-2 also shards gradients; full-shard ZeRO-3/FSDP also shards weights. Activations remain local. With sharded gradients, deferred synchronization and accumulation greater than one retain full-model gradients on each GPU; per-microbatch synchronization keeps gradient shards. See PyTorch FSDP 2.14 (https://docs.pytorch.org/docs/2.14/fsdp.html#torch.distributed.fsdp.FullyShardedDataParallel.no_sync).

Gather units include adapters and normalization tensors and use the larger block or embedding/head unit. BACKWARD_PRE retains two weight units; POST/None use one. Separately add the unsharded part of the current gradient unit above its resident shard. With deferred synchronization, reserve one complete new gradient/reduction unit beside the retained full gradients. This conservative workspace sum follows PyTorch's prefetch lifetime description (https://docs.pytorch.org/docs/2.14/fsdp.html#torch.distributed.fsdp.BackwardPrefetch); real allocation timing may reduce coincident peaks.

NF4 metadata remains replicated, following Answer.AI (https://www.answer.ai/posts/2024-03-06-fsdp-qlora). The estimate is $(persistent+temporary+activation)\,(1+allowance)+fixed\ reserve$ , compared with installed VRAM times $(1-safety)$ .

Runtime and cost

PaLM (https://arxiv.org/abs/2204.02311), Appendix B, p.66, supplies the dense-transformer convention: full model work is approximately $6N+12LdS$ FLOPs per processed token. LoRA needs base forward and input backpropagation, approximately $4N$ , plus $6A$ for trainable adapters and the same attention term. QLoRA has the same useful matmul work; its kernel overhead changes MFU.

Compute seconds = total processed tokens × useful FLOPs/token ÷ (GPUs × dense peak FLOP/s × MFU). PaLM MFU excludes rematerialization from useful work: do not add checkpoint recomputation again when using measured MFU. Setup/evaluation time is added once. Rental cost uses allocated GPU-hours. Electricity uses facility-adjusted GPU plus host power and elapsed hours.

EleutherAI's Transformer Math 101 (https://github.com/EleutherAI/website/blob/main/content-blog/transformer-math.md), attached text p.3, cross-checks the 12-byte Adam and 6-byte 8-bit optimizer/master-state convention; its sharding expressions also include ZeRO-3 live parameters. Its compact inference-weight heuristic is extended here with explicit context-dependent KV cache.

Assumptions

Dense, decoder-only transformers with uniform blocks, conventional head width, 16-bit compute/activations and Adam, with explicitly selected 16-bit or 32-bit stored gradients. Biases and small norm tensors are approximate. Default dimensions follow Meta's Llama-2 7B architecture; the base count is rounded to 7 billion.
All GPUs are identical and form one data-parallel group. No tensor, pipeline, sequence parallelism or CPU/NVMe offload. FSDP layer wrapping and BF16-compatible NF4 storage must actually be configured. Quantization-scale metadata is retained on each rank; newer implementations that shard it can use slightly less.
Training KV cache is disabled. Inference is a replicated, unmerged-adapter model with two 16-bit KV tensors, at the same context and per-GPU batch. Its workspace is approximated by one layer; cached generation and prefill kernels vary.
The trainable-state convention retains a 32-bit master copy. BF16-only optimizers may use less; Choose 32-bit gradient storage for native FSDP unless keep_low_precision_grads=True is configured. FP32 adapters and unusual optimizer precision require an appropriate allowance. An 8-bit/paged optimizer does not make CPU offloading free.
Standard materialized HF/PyTorch cross-entropy uses a modeled 14-byte vocabulary-cell peak including backward. Custom/fused loss has no automatic memory saving: enter its measured total head-buffer bytes divided by batch × sequence × vocabulary. The coefficient must be recalibrated for another implementation, architecture or context.
Analytical activation, gather and live gradient buffers, a 15% tensor allowance, 1.5 GiB fixed framework reserve and 10% unused capacity are distinct planning assumptions. Loading, saving, evaluation and allocator spikes can exceed the steady training peak.
Peak throughput is dense BF16 with FP32 accumulation, counting a multiply-add as two FLOPs. Default RTX 4090 rate is the stock 165.2 TFLOP/s from NVIDIA Ada Appendix A. GPU power, rental rates, MFUs, token budget and activation multiplier are editable assumptions, not catalog measurements.
Workload means processed tokens after packing/padding, including tokens masked out of the loss. Attention follows PaLM's dense score-matmul convention. MFU accounts for causal/fused kernel savings, recomputation, data loading and communication.
Cost is in USD. The 2026-10-27 planning price defaults are illustrative user-editable values, not market quotes. Rental excludes host/storage charges and billing granularity; local cost excludes equipment purchase, depreciation and labor.
Deferred synchronization models FSDP no_sync behavior; framework support varies. Synced accumulation requires a loop that actually synchronizes every microbatch. Adapters are conservatively grouped by block for live-buffer sizing; root wrapping or larger buckets can need more memory. Prefetch and gradient dtype also change throughput, so recalibrate MFU.

Where it applies

Use for first-pass capacity and budget planning, then validate one training step and a representative pilot. There is no universal fine-tuning VRAM requirement independent of sequence length, microbatch, attention/loss kernels, optimizer dtype and framework.

The exact reference cases test ZeRO model-state accounting and the published GPT-3 LoRA budgets, not end-to-end framework peaks. At 7B, 2,048 tokens, microbatch 10, rank 64 all-block adapters, accumulation 1 and two full-sharded GPUs, the default allowances predict about 24.1 GiB per GPU. Answer.AI's March 2024 dual-4090 table reports 20.97 GiB and 466 s without CPU offload. The conservative sum now exceeds that measurement; this is a scale cross-check with different adapter/precision, loss-buffer lifetime and allocator details, not an exact reproduction. Do not reduce buffers merely to reproduce this benchmark. For their 1,024 × 2,048 processed-token workload, this model at QLoRA MFU 0.15 predicts about 1,362 s, showing why an assumed MFU is not a measured runtime. The same useful-work convention would imply MFU around 0.44 for that reported time; it is a diagnostic, not a supplied default.

Answer.AI's 70B dual-24GB result uses CPU offloading, which this calculator does not model. It must not be used as evidence that sharding alone fits every 70B configuration.

Unsloth's Llama-3.1 8B context benchmark specifies batch 1, rank 32, all linear targets and padded inputs: its reported 24GB limits are 78,475 tokens with Unsloth and 5,789 with Hugging Face + FA2. The optimized result uses additional checkpointing/loss-memory techniques outside this generic estimator. These measurements motivate pilot calibration of activation and custom loss coefficients; they are not universal limits. No MLPerf results are used.

Suggested maximum sequence is zero below the supported 128-token minimum and otherwise capped by the entered model context, and maximum batch is only a memory limit. Both are sensitive to buffer assumptions. Time and cost for an over-budget configuration remain hypothetical. Mixture-of-experts, distributed inference, CPU offload, multimodal models and RL training need different models.

References

Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (2020), pp.8–11 and Table 1. https://arxiv.org/abs/1910.02054
Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021), §4.1 and p.20. https://arxiv.org/abs/2106.09685
Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023), pp.4–5. https://arxiv.org/abs/2305.14314
Korthikanti et al., Reducing Activation Recomputation in Large Transformer Models (2022), pp.4–9. https://arxiv.org/abs/2205.05198
Chowdhery et al., PaLM: Scaling Language Modeling with Pathways (2022), Appendix B, p.66. https://arxiv.org/abs/2204.02311
Dettmers et al., 8-bit Optimizers via Block-wise Quantization (2022). https://arxiv.org/abs/2110.02861
Touvron et al., Llama 2: Open Foundation and Fine-Tuned Chat Models (2023). https://arxiv.org/abs/2307.09288 ; Meta architecture implementation: https://github.com/meta-llama/llama/blob/main/llama/model.py
Answer.AI, Benchmarking QLoRA+FSDP, February–March 2024; dual-4090 measured results and offloading caveats. https://github.com/AnswerDotAI/fsdp_qlora/blob/main/benchmarks_03_2024.md
Unsloth, Unsloth Benchmarks, context-length benchmark; accessed 2026-10-27. https://unsloth.ai/docs/basics/unsloth-benchmarks
Anthony, Biderman and Schoelkopf, Transformer Math 101 (EleutherAI, 2023), memory/compute/sharding cross-check, read from the official repository. https://github.com/EleutherAI/website/blob/main/content-blog/transformer-math.md ; https://blog.eleuther.ai/transformer-math/
NVIDIA Ada GPU Architecture white paper, Appendix A: dense BF16 with FP32 accumulation; stock RTX 4090 165.2 TFLOP/s. https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvidia-ada-gpu-architecture.pdf
Answer.AI, You can now train a 70b language model at home (2024-03-06), FSDP-QLoRA and replicated quantization state. https://www.answer.ai/posts/2024-03-06-fsdp-qlora
PyTorch 2.14 FSDP documentation, no_sync(), BackwardPrefetch and MixedPrecision; accessed 2026-10-27. https://docs.pytorch.org/docs/2.14/fsdp.html
Hugging Face Transformers, ForCausalLMLoss and fixed_cross_entropy, source inspected 2026-10-27. https://github.com/huggingface/transformers/blob/main/src/transformers/loss/loss_utils.py
PyTorch ATen, LossNLL.cpp, materialized cross-entropy implementation, source inspected 2026-10-27. https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/LossNLL.cpp

Inputs

Input	Symbol	Unit	Default
Selected fine-tuning method	`method`		QLoRA (NF4 base)
Base model parameters (billions) Use the actual checkpoint parameter count where known. Default: a rounded Llama-2 7B example.	`N_b`		7
Transformer layers	`L_layers`		32
Hidden width	`d_model`		4096
Feed-forward width Default matches the Llama-2 7B gated MLP.	`d_ff`		11008
Query attention heads	`H_heads`		32
Key/value heads Set equal to query heads for MHA; use the model's smaller KV-head count for GQA.	`H_kv`		32
Vocabulary size Used for unquantized embeddings/output head and logits memory.	`V_vocab`		32000
MLP architecture	`mlp`		Gated MLP (SwiGLU)
Tied embedding and output weights	`tied`		false
Model context limit Memory capacity does not extend the model's trained context window.	`S_limit`		4096
Training sequence length Include prompt, response and padding. Training assumes use_cache=false.	`S_tokens`		2048
Microbatch per GPU Sequences resident at once on each GPU; not the effective batch.	`B_micro`		1
Gradient accumulation steps Processes microbatches serially, so activation memory is unchanged. With FSDP no_sync and accumulation > 1, full unsharded gradients remain on each GPU.	`G_acc`		8
LoRA rank One rank for every selected projection. Applies to both adapter rows in the comparison, including when Full is selected.	`r_rank`		16
LoRA target projections Used by both adapter rows. Embeddings, output head and biases are not adapted.	`targets`		All block linear projections
NF4 double quantization 64-weight blocks and 256-scale secondary blocks. The model keeps embeddings/output head in 16-bit.	`double_q`		true
Adam moment precision Trainable weights use 2 bytes and master weights 4 bytes. Adam adds 8 bytes for FP32 moments or approximately 2 for 8-bit moments; gradients use the separately selected precision.	`optimizer`		32-bit moments
Gradient checkpointing (one layer per checkpoint) Stores layer boundaries and allows one layer of temporary activations. Time impact belongs in the entered MFU.	`checkpoint`		true
Memory-efficient attention (FlashAttention/SDPA) Omits the saved quadratic attention-score term. Eager attention can use much more memory.	`flash`		true
Loss/head buffer implementation Standard: BF16 logits plus FP32 cross-entropy buffers, 10 B/cell forward and 14 B/cell at backward peak. Custom/fused: enter a measured total head workspace coefficient; no automatic fused-kernel saving is assumed.	`loss_kernel`		HF + PyTorch materialized cross-entropy
Activation allowance multiplier Multiplies the analytical activation estimate. Calibrate from a pilot; GQA, fused kernels and saved tensors differ by framework.	`act_factor`		1
GPU count Homogeneous GPUs in one data-parallel group; no tensor or pipeline parallelism.	`G_gpus`		1
VRAM per GPU Enter usable device capacity from nvidia-smi, not summed VRAM.	`M_vram`	GiB	24
Training distribution Activations remain per GPU. Quantized full sharding requires a compatible FSDP-QLoRA stack. CPU offload is outside this estimate.	`shard`		Replicated / DDP
Fixed framework reserve per GPU CUDA context, kernels, communication buckets and allocator reserve; an allowance, not a measured constant.	`M_fixed`	GiB	1.5
Additional tensor-memory allowance 0.15 means add 15% to weights, training state, activations and temporary buffers.	`mem_allow`		0.15
VRAM safety headroom fraction 0.10 leaves 10% of installed VRAM unused, in addition to the explicit framework allowances.	`safety`		0.1
Dense BF16 peak per GPU (FLOP/s) Arithmetic operations per second, not clock frequency. Stock RTX 4090: 1.652e14; A100: 3.12e14. Use dense FP32-accumulating tensor rates, not sparse/INT4 headline TOPS.	`F_peak`	1/s	165200000000000
Full fine-tuning MFU Model FLOPs / (elapsed seconds × aggregate dense peak). Includes communication, recomputation and kernel inefficiency.	`mfu_full`		0.25
LoRA MFU An editable planning assumption, not a benchmark guarantee.	`mfu_lora`		0.2
QLoRA MFU Dequantization and small batches often lower MFU. Defaults are planning assumptions; measure a pilot.	`mfu_qlora`		0.15
Processed tokens per epoch (millions) Count all tokens actually processed, including padding and masked prompt tokens. Packing can reduce padding.	`T_million`		10
Epochs	`epochs`		3
Loading, compilation and evaluation allowance Added once to training wall time. Checkpoint I/O and evaluation are included only to the extent you allow here.	`t_setup`	h	0.25
Cost basis	`cost_mode`		GPU rental
Rental price per GPU-hour (USD) User-entered USD per GPU-hour, not a live quote. Planning default dated 2026-10-27; excludes tax, CPU/storage charges and minimum billing.	`r_gpu`	1/h	0.5
Average GPU draw per device Use measured training draw rather than TDP.	`P_gpu`	W	350
Other system power CPU, RAM, fans and other local equipment, excluding GPUs.	`P_host`	W	150
Facility power multiplier (PUE) Includes cooling and power conversion; 1 for an at-the-wall whole-system measurement.	`pue`		1.1
Electricity tariff (USD per kWh) User-entered tariff, dated to your bill; excludes hardware purchase and depreciation.	`r_kwh`	1/kWh	0.15
Gradient synchronization during accumulation For sharded gradients, deferred synchronization retains a full gradient copy per GPU when accumulation exceeds 1. Choose Every only if your training loop actually reduces each microbatch.	`sync_policy`		Defer sync: retain full gradients (FSDP no_sync)
Gradient storage precision FSDP normally upcasts stored gradients to FP32. The 16-bit estimate requires keep_low_precision_grads=True or a stack that keeps 16-bit gradients; otherwise select 32-bit.	`grad_precision`		16-bit gradients (ZeRO 16 B/parameter convention)
Measured head workspace per logit cell Measure peak logits/loss/autograd buffers, exclude model weights and parameter gradients, then divide by microbatch × sequence × vocabulary. Recalibrate when architecture, context or implementation changes.	`b_head_custom`	byte	14
Backward parameter prefetch PRE overlaps current gradients with current and prefetched weights. POST and None use one live weight unit here. All modes still need current unsharded gradients; extra communication buckets remain in framework reserve.	`prefetch`		BACKWARD_PRE: current + next weights

Results

With the default inputs above, the calc gives:

Result	Value
Training fit estimate	Estimated fit; validate with a pilot
Inference per GPU (replicated)	7.33 GiB
Selected training per GPU	8.43 GiB
Training headroom after safety reserve	13.17 GiB
Maximum sequence at this microbatch	4,096
Maximum microbatch per GPU at this sequence	8
Trainable parameters	39,976,960
Effective global batch (sequences)	8
Projected wall time if memory fits	10.83 h
Time with MFU at 1.5× to 0.5×	7.3 to 21.4 h
GPU-hours including setup	10.83 h
Projected cost if memory fits (USD)	5.42

Checked against sources

Each case below was calculated by this calc and agrees with the value from its source (within the tolerance where one is shown).

ZeRO 7.5B: ddp persistent state

Source: Rajbhandari et al., ZeRO, arXiv:1910.02054, Table 1, attached PDF p.11. Decimal GB. Checks persistent model state only, excluding activations, gathers and allowances; tolerance covers the published rounding. Uses one synchronized microbatch and 16-bit gradients, matching the paper's accounting.

Inputs: Selected fine-tuning method = Full fine-tuning; Base model parameters (billions) = 7.5; Gradient accumulation steps = 1; the others as above.

Result	Source	This calc
Full persistent state (decimal GB; reference table)	120 GB	120 GB

ZeRO 7.5B: z1 persistent state

Inputs: Selected fine-tuning method = Full fine-tuning; Base model parameters (billions) = 7.5; GPU count = 64; Training distribution = ZeRO-1: optimizer sharded; Gradient accumulation steps = 1; the others as above.

Result	Source	This calc
Full persistent state (decimal GB; reference table)	31.4 ± 0.05 GB	31.406 GB

ZeRO 7.5B: z2 persistent state

Inputs: Selected fine-tuning method = Full fine-tuning; Base model parameters (billions) = 7.5; GPU count = 64; Training distribution = ZeRO-2: optimizer + gradients sharded; Gradient accumulation steps = 1; the others as above.

Result	Source	This calc
Full persistent state (decimal GB; reference table)	16.6 ± 0.05 GB	16.641 GB

ZeRO 7.5B: z3 persistent state

Inputs: Selected fine-tuning method = Full fine-tuning; Base model parameters (billions) = 7.5; GPU count = 64; Training distribution = ZeRO-3 / FSDP full shard; Gradient accumulation steps = 1; the others as above.

Result	Source	This calc
Full persistent state (decimal GB; reference table)	1.88 ± 0.0051 GB	1.875 GB

LoRA GPT-3 query/value rank 1: 4.7M

Source: Hu et al., arXiv:2106.09685, p.20, GPT-3 parameter budget rq=rv=1; architecture 96 layers, width 12288 (Korthikanti et al., Table 3). Published count rounded to 0.1M.

Inputs: Selected fine-tuning method = LoRA (16-bit base); Base model parameters (billions) = 175.3; Transformer layers = 96; Hidden width = 12,288; Feed-forward width = 49,152; Query attention heads = 96; Key/value heads = 96; Vocabulary size = 50,257; MLP architecture = Two-projection MLP (GeLU); Tied embedding and output weights = yes; Model context limit = 2,048; LoRA target projections = Query and value (q, v); LoRA rank = 1; the others as above.

Result	Source	This calc
Trainable LoRA parameters	4,700,000 ± 50,000	4,718,592

LoRA GPT-3 query/value rank 8: 37.7M

Source: Hu et al., arXiv:2106.09685, p.20, GPT-3 parameter budget rq=rv=8; 96 layers, hidden width 12288. Published count rounded to 0.1M.

Result	Source	This calc
Trainable LoRA parameters	37,700,000 ± 50,000	37,748,736

EleutherAI 8-bit optimizer plus master: 6 bytes/parameter

Source: EleutherAI, Transformer Math 101, Optimizer States, official repository snapshot attached p.3. Explicitly lists 6 bytes/parameter including a 4-byte master copy and two 1-byte moments; quantization metadata and small-tensor exceptions remain in allowances.

Inputs: Adam moment precision = 8-bit moments; the others as above.

Result	Source	This calc
Master weights plus optimizer states	6 byte	6 byte

Independent PyTorch tensor-lifetime cross-check: about 14 B/cell

Source: Own independent execution of PyTorch 2.6 CPU profiler: 64×2048 BF16 logits kept alive, FP32 cast, functional.cross_entropy and backward. Measured peak 1,835,528 bytes / 131,072 vocabulary cells = 14.003967 B/cell. The small tolerance excludes label/scalar buffers outside the per-cell model. This tests tensor lifetime with a separate library, not GPU allocator behavior or a published training benchmark. Source implementation: https://github.com/huggingface/transformers/blob/main/src/transformers/loss/loss_utils.py and https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/LossNLL.cpp . Reproduction: head_workspace_probe.py in the working folder.

Inputs: the default inputs above.

Result	Source	This calc
Selected peak head buffer bytes per vocabulary cell	14.004 byte	14 byte

Related Resources

Related courses

Artificial Intelligence for Everyone – Easy to Understand

Introduction to CNC Programming

19,99 €

Fine-Tuning VRAM and Time Calculator: Full, LoRA and QLoRA