GPU workload guide
How much GPU VRAM do you actually need?
A practical memory budget for inference, fine-tuning, image generation, and video workloads—from 24 GB to 192 GB.
By AnchorGPU · Updated · 6 min readVRAM is a budget, not a model label
GPU memory is consumed by several independent categories: model weights, inference KV cache, temporary activations, framework workspaces, allocator cache, and—during training—gradients and optimizer state. A model fitting in one configuration does not prove that it will fit with a longer context, larger batch, different precision, or more concurrent requests.
Plan from the exact model revision and runtime configuration. Labels such as 7B, 13B, or 70B describe parameter scale, not total operating memory, so they should never be mapped to a GPU universally.
Estimate raw weights first
As an explicitly hypothetical example, suppose a model has exactly 13 billion parameters and stores every parameter in a 16-bit format. Raw weight storage is 13,000,000,000 × 2 bytes = 26,000,000,000 bytes, or about 24.2 GiB. That number excludes every other allocation, so it is a lower bound rather than a GPU recommendation.
Quantization can reduce weight storage, but real footprints also include scales, metadata, dequantization buffers, duplicated tensors, and runtime workspaces. Use the format produced by the actual loader rather than dividing parameter count by a nominal bit width and treating the result as final.
Inference adds KV cache and working memory
Autoregressive serving retains key and value tensors for active sequences. KV-cache demand depends on model architecture, cache precision, concurrent sequences, and the number of live tokens across those sequences. Raising maximum context or concurrency can exhaust memory even when the weights load successfully.
Temporary activations, attention workspaces, compiled kernels, communication buffers, and the framework allocator consume additional memory. vLLM reports its available GPU KV-cache capacity and an estimated concurrency for the configured sequence length; treat those startup figures as planning evidence, then replay representative prompts before committing a production workload.
Training has a different memory shape
Fine-tuning adds gradients, optimizer state, saved activations, and sometimes additional full-precision copies. Their exact size depends on the optimizer, precision policy, trainable parameter set, and sharding strategy, so a single inference multiplier is misleading.
DistributedDataParallel replicates the training state needed by each worker; it does not turn several GPUs into one contiguous memory pool. Fully Sharded Data Parallel can shard parameters, gradients, and optimizer states across workers, while activation checkpointing reduces saved-activation memory by recomputing selected work during backward. Both techniques exchange simplicity or compute for memory.
Measure the configuration you will run
Test model loading and a complete representative inference request or training step, including optimizer update, with the exact framework, precision, context, batch, and parallelism settings. Record peak allocated and reserved memory, then repeat with realistic concurrency or gradient accumulation. Leave operating headroom instead of targeting the last available byte.
Choose a larger-memory GPU when the measured peak cannot be reduced safely. Add GPUs only when the software explicitly shards the model or workload and the delivered interconnect is known. Re-run the measurement after any model, runtime, quantization, context, or batch change.
Sources & editorial method
Official documentation supports the technical explanations. Hardware recommendations are our workload-dependent interpretation, not a measured performance guarantee.
vLLM — parallelism and scaling ↗PyTorch — FullyShardedDataParallel ↗PyTorch — activation checkpointing ↗