GPU Memory Calculator

A 7-billion parameter model in FP32 precision takes 28 billion bytes (26.1 GiB) just for the weights, before you add a single input token or training gradient. Estimates that stop at the weights leave out the KV cache, activations and optimizer states.

This calculator counts the largest memory consumers; the CUDA context and framework buffers come on top. Model parameters store the learned weights. Activations hold intermediate values as data flows through layers. During training, you also need gradient storage for backpropagation and optimizer states for Adam or similar algorithms. The KV cache and activations grow linearly with batch size, sequence length and the number of layers.

Precision choice matters enormously. FP32 uses 4 bytes per parameter, FP16 uses 2 bytes, and INT8 quantization uses 1 byte. LLM.int8() (Dettmers et al., 2022) ran models of up to 175B parameters in 8 bits without performance degradation. Training is usually done in mixed precision, FP16 or BF16 compute with FP32 master weights, and FP16 needs loss scaling so small gradients do not underflow (Micikevicius et al., 2018). The calculator assumes standard transformer memory patterns — CNN or RNN architectures have different scaling characteristics.

Use this calculator before buying GPU hardware for AI projects. Consumer GPUs top out at 32 GB (RTX 5090; the RTX 4090 has 24 GB), while data-center cards such as the H100 have 80 GB. Knowing your memory requirements prevents expensive hardware mistakes — you cannot upgrade GPU memory after purchase.

Calculate memory requirements when planning training runs. Large batch sizes improve training efficiency but quickly exhaust GPU memory. This calculator helps you find the largest viable batch size for your hardware, maximizing throughput without crashes.

Memory estimation becomes critical for model deployment and inference serving. Production systems need consistent memory usage predictions to auto-scale GPU resources. A model that barely fits on one GPU cannot handle multiple concurrent requests without memory optimization or larger hardware.

The biggest mistake is forgetting optimizer memory during training. Adam keeps two extra tensors the size of the model (momentum and variance), and mixed precision adds an FP32 copy of the weights: 16 bytes per parameter in all, eight times the FP16 weights. Developers often calculate inference memory then wonder why training crashes with out-of-memory errors.

Sequence length multiplies the KV cache and the activations: 4,096 tokens need twice the memory of 2,048 for both. Standard attention also builds a score matrix that grows with the square of the length; memory-efficient kernels such as FlashAttention avoid storing it in full.

Batch size miscalculation happens because activation memory scales per sample. A batch of 32 holds 32 times the activations (and, for inference, 32 times the KV cache) of a single sample, while the weights stay the same. Many tutorials show single-sample examples but production systems need larger batches for efficiency, requiring much more GPU memory than initial prototypes suggest.

Memory calculation starts with parameter count times bytes per parameter. A 7B parameter model in FP16 requires 7,000,000,000 × 2 = 14 billion bytes just for weights, which is 13.0 GiB (binary gigabytes of 1,073,741,824 bytes; in decimal gigabytes it is 14.0 GB). This calculator gives every result in GiB because VRAM is built in binary sizes: a card sold as 16 GB holds about 16 GiB, a little less after the driver's reserve. For inference, the other big consumer is the KV cache: batch_size × sequence_length × 2 × layers × hidden_size × 2 bytes (NVIDIA). The calculator does not ask for layers or hidden size, so it assumes a shape like Llama 2 7B (32 layers, hidden size 4,096): hidden size = 128 × layers, with parameters ≈ 12 × layers × hidden_size² (Kaplan et al., 2020). For 7B parameters, batch 1 and 2,048 tokens that is about 1.1 GiB on top of 13 GiB of weights. Models with grouped-query attention need less.

Training needs far more (EleutherAI, Transformer Math 101): weights and gradients at your precision, plus AdamW states of 12 bytes per parameter in mixed precision (an FP32 copy, momentum and variance) or 8 bytes in pure FP32, so about 16 bytes per parameter before activations. Activations with selective recomputation add about 34 × sequence_length × batch_size × hidden_size × layers bytes. You need the model weights, gradients of the same size for backpropagation, and the optimizer states. Total training memory ≈ model_memory + activation_memory + gradient_memory + optimizer_memory. Inference needs the weights plus the KV cache.

Precision changes the weights and gradients; the Adam states stay in FP32, and at INT8 or INT4 the KV cache here stays FP16. INT4 quantization stores weights in 12.5% of their FP32 size; methods such as GPTQ use a small calibration set to keep accuracy. Mixed precision keeps an FP32 master copy for the optimizer and runs the forward pass in FP16, balancing memory and numerical stability. Batch size scales activations and the KV cache linearly, so doubling it doubles those parts but not the weights, which makes it the easiest parameter to adjust for available hardware.

The KV cache and activation estimates assume a dense transformer shaped like Llama 2 7B, with full multi-head attention; real models vary widely in shape. Sparse attention (like in Longformer) or mixture-of-experts models have completely different memory profiles. A mixture-of-experts model must hold every expert in memory even though each token uses only some of them: Mixtral 8x7B has 46.7B parameters and uses about 12.9B per token (Mistral AI), so enter 46.7B here.