bitcalc

🧠 LLM VRAM Calculator

Simple: pick your card and context, see which models fit. Pro: weights, KV cache and overhead as separate line items.

Calculate VRAM requirements

Families sorted by popularity (Hugging Face likes, 31 Oct 2026).

8,03 Mrd · 131k Kontext

Bits per weight: Q4_K_M 4.85 · MXFP4 4.25 · BF16 16.

FP8 halves the KV cache and almost always removes the bottleneck. In vLLM: --kv-cache-dtype fp8.

Total parameters — for MoE including every expert.

Every layer keeps its own KV cache.

Not the attention heads — only KV heads count for the cache.

Dimension per KV head, usually 128 or 256.

vLLM --max-model-len. 4096, 8192, 32768, 131072 are the usual steps.

How many requests should be served at the same time (--max-num-seqs).

--tensor-parallel-size: splits weights and cache across cards.

NVIDIA, AMD and Intel Arc — the card's full VRAM each.

The vLLM default is 0.9; the main branch now uses 0.92. The rest stays with the driver and other processes.

Activations, CUDA graphs and CUDA context. vLLM logs this as the difference between reserved memory and weights — 1 to 2 GiB is normal for 8B models.

Memory budget

Model weights (total)—
Model weights per GPU—
KV cache per token—
KV cache per sequence—
KV cache for all sequences—
Overhead per GPU—
VRAM required per GPU—
Usable per GPU—
Headroom per GPU—
vLLM KV cache capacity—
Maximum concurrent sequences—
Max. context length at this concurrency (VRAM limit)—

GPU check (tensor parallel = 1)

CardVendorVRAMRequiredSequencesVerdict

🧮 Memory sorted, power not? Check the running cost next.

🧠 Why 7B in 4-bit is not 7 GB

A 24 GB card, a 32B model in Q4_K_M, and vLLM dies after loading the weights with "No available memory for the cache blocks". The weights were not the problem: 19.9 GiB is exactly what 32.76 billion parameters take at 4.85 bits per weight. What was missing is the memory the model needs once it starts serving.

Three line items, one budget

Weights are fixed once loading is done. KV cache grows with context length times concurrency, and it takes one more step for every token a request appends. Overhead is everything vLLM reserves for activations, CUDA graphs and the CUDA context before the first request arrives. Plan with item one only and you systematically come up short — the longer your contexts, the shorter you come up.

The KV cache formula

KV cache per token = 2 × layers × KV heads × head dim × bytes per element. The 2 stands for key and value, and the factor runs over all layers because every layer keeps its own cache. Llama 3.1 8B has 32 layers, 8 KV heads and head dim 128: 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes, which is 128 KiB per token. At 8,192 tokens that is 1 GiB per sequence — while the weights in Q4_K_M take only 4.5 GiB.

That is why the math flips so abruptly. Llama 3.3 70B sits at 320 KiB per token: 8k context is 2.5 GiB per sequence, 32k is 10 GiB. On 70B models with few KV heads the cache is the item that decides your hardware — not the model.

Bits per weight are not a rounding detail

BF16 to Q4_K_M is a factor of 3.3, and Q4_K_M to Q3_K_M is still 20 percent. llama.cpp's k-quants carry scales and minimums as fp16, which is why it is 4.85 bits and not 4. GPTQ and AWQ with group size 128 land at 4.25 bits — 4 bits per weight plus 2 × 16 bits of scale and zero per group. Whoever computes "4 bit = 0.5 byte" underestimates every model by six percent.

With MoE you pay for all experts

Mixtral 8×7B has 46.7 billion parameters but only routes a token to two of eight experts. That is a gift for throughput, not for VRAM: every expert has to be resident, otherwise there is nothing to route to. gpt-oss-120b comes in at 116.8 billion parameters — even in native MXFP4 that is 58 GiB of weights that will not quantize away. With MoE always compute with the total, never with the active parameter count.

What vLLM does with the budget

vLLM reserves gpu_memory_utilization of the card's memory — 0.9 by default, 0.92 on the current main branch — subtracts weights and overhead, and hands the rest to PagedAttention as KV cache blocks. Its log then reports "GPU KV cache size: N tokens" and "Maximum concurrency for 8,192 tokens per request: M×". Those are exactly the two lines this calculator produces before you download the model.

There are three knobs: --max-model-len caps context per request, --max-num-seqs caps concurrency, and --kv-cache-dtype fp8 halves the cache per token. FP8 in the KV cache costs measurable quality, but on most chat workloads less than a second server costs. If the weights alone do not fit, only --tensor-parallel-size helps: at TP=2 half the weights and half the cache live on each card.

Worked example: 8B on an RTX 4090

Llama 3.1 8B in Q4_K_M, 8k context, 8 concurrent sequences, one RTX 4090 with 24 GiB at 90 percent utilization: 4.53 GiB weights, 8 GiB KV cache and 1.5 GiB overhead add up to 14.03 GiB of the 21.6 GiB usable — 7.57 GiB remain, enough for 15 parallel 8k requests. The same card with 32k context and eight sequences needs 32 GiB of KV cache and fails. If you need the context, do not halve it — use FP8 for the KV cache: that is 16 GiB and it fits again.

AMD and Intel Arc: same math, different driver

The formula does not know the vendor. Radeon cards run on ROCm (officially supported by vLLM), Intel Arc through PyTorch's XPU backend — the KV cache needs the same bytes per token. What changes is the overhead: on ROCm, 2 to 3 GiB per card is more realistic than 1.5, and a 12 GiB Arc shares its VRAM with the desktop. Enter the value from the log and the numbers hold there too. The families in the dropdown are ordered by their popularity on Hugging Face; every model stays below 130 billion parameters, because beyond that nothing fits a single card without tensor parallelism.

Frequently asked questions

Why does this differ from what vLLM prints in its log?

Because vLLM does not estimate overhead, it measures it at startup with a dummy batch — and that number depends on the model, the batch size and the CUDA graph configuration. Enter the value the log shows as the difference between reserved memory and weights; then the numbers agree up to block rounding.

Does the calculator include the embedding layer and the LM head?

Yes, because the parameter count is the full safetensors sum. For quantization schemes that keep embeddings and LM head in BF16 — common with GPTQ and AWQ — the result is therefore slightly low. On current models the error stays below one percent.

What about Gemma 3 and gpt-oss, which use sliding window attention?

On those models the real KV cache is smaller than the linear formula suggests, because only some layers hold the full context. There the formula gives an upper bound — good for a hardware decision, poor for the exact sequence count.

Isn't CPU RAM plus swap enough?

For llama.cpp with offloading, yes; for vLLM, practically no. PagedAttention needs its blocks in GPU memory, and a model that migrates over PCIe on every token loses exactly the advantage you picked vLLM for. If the weights do not fit in VRAM, that is a hardware question, not a settings question.

How many sequences should I plan for?

A single user needs 1 to 4. Continuous batching starts paying off around 8 concurrent sequences, and beyond 32 the KV cache usually becomes the bottleneck. The calculator shows both: whether your number fits and how many actually do.

Sources & further reading

Architecture values from the models' config.json files (Llama 3.1 8B, Llama 3.3 70B, Qwen3 8B/32B, Gemma 3 27B, Mistral Nemo 12B, Mistral Small 3.2 24B, Mixtral 8×7B, gpt-oss-20b/120b) and parameter counts from the safetensors metadata on Hugging Face. Formula and defaults follow the vLLM documentation (PagedAttention, engine arguments: gpu_memory_utilization, max_model_len, max_num_seqs, kv_cache_dtype) and the llama.cpp quantization reference. More in the portfolio: capacity converter, power cost calculator, IOPS calculator.

⚡ On to the power cost calculator — or the capacity converter.