GPU workload guide
Choosing a GPU for LLM inference in production
How model size, quantization, context, batching, and latency targets change the best GPU—and the real cost per request.
By AnchorGPU · Updated · 7 min readSpecify the serving contract
Separate interactive serving from offline batch work. Define maximum prompt length, typical and maximum output length, target concurrency, time to first token, inter-token latency and error tolerance. A single short request is not a capacity test.
Keep the model and tokenizer revisions, quantization format, chat template and decoding settings fixed when comparing hardware. Otherwise a faster result may come from a different workload or a quality tradeoff rather than the GPU.
Budget more than the model weights
Inference needs model weights, runtime buffers, activations and usually a KV cache for active sequences. More simultaneous sequences and longer contexts change the required memory. Quantization support and its effect on output quality need validation on the chosen runtime.
A 24 GB or 48 GB card is a candidate for a model that fits, not a promise that every model in a parameter-size class will run. Test the actual model and context limit. Larger-memory accelerators are useful when they reduce partitioning or provide measured concurrency headroom.
Replicas or a distributed model?
If the model fits on one GPU, independent replicas may be a straightforward way to serve more requests. Tensor parallelism distributes parts of the model and introduces communication between accelerators. Pipeline parallelism has different placement and utilization tradeoffs.
Check the current vLLM guidance and the delivered machine topology before choosing a card count. Do not assume that SXM, PCIe, four cards or an aggregate VRAM number implies a particular connected fabric.
Qualify the software stack before timing
Confirm the driver, CUDA or ROCm build, GPU architecture support and availability of the operators your model needs. Container tags should be pinned and a fresh environment should reproduce the same serving result.
Bind a first test to loopback, use an API key and do not expose an unauthenticated model endpoint. Remote access, TLS, ingress controls and service monitoring require an explicit deployment configuration; they are not created by selecting a catalog image.
Measure realistic traffic
Warm the server and separate model-loading time from steady-state results. Replay a representative mixture of prompt lengths, output lengths and concurrent requests. Report accepted request throughput, p95 latency, errors and memory use together.
Increase traffic until the latency or reliability target stops being met, then leave capacity margin. Keep cold-start and recovery tests separate: operational readiness includes restoring service after a process or node failure.
Compare the cost of useful output
For a measured interval, divide the complete allocation cost attributable to that interval by successfully delivered output. If reporting cost per million tokens, say whether input and output tokens are included and use the same convention for every candidate.
A fixed 7- or 30-day rental is paid for the whole term, including idle time. Project utilization explicitly and include options before comparing configurations. AnchorGPU catalog prices are modeled inputs; no live serving benchmark or production availability is asserted here.
Sources & editorial method
Official documentation supports the technical explanations. Hardware recommendations are our workload-dependent interpretation, not a measured performance guarantee.
vLLM — quickstart and serving ↗vLLM — distributed serving and parallelism ↗vLLM — GPU installation requirements ↗