Serve an LLM with vLLM

Define the serving contract

Record the model identifier and revision, weight format, quantization, maximum context length, prompt-length distribution, concurrency and latency objective. These settings are part of the workload, not optional benchmark notes.

Weights share GPU memory with the KV cache and temporary workspaces. A model that loads successfully with one short prompt may still run out of memory under the intended traffic. Keep operating headroom and test the longest allowed requests.

Start with one GPU when it fits

A single GPU avoids inter-GPU communication and makes a first deployment easier to diagnose. Consider tensor parallelism when the model and required runtime state do not fit on one accelerator, or when a representative measurement justifies splitting it.

Independent replicas serve separate requests; tensor parallelism splits model computation. They solve different problems. A catalog node with several cards does not, by itself, prove a particular interconnect.

Qualify the complete environment

The AnchorGPU vllm image is available for NVIDIA and AMD in the catalog. Before real execution, identify the actual driver, CUDA or ROCm runtime, PyTorch, Python and vLLM versions. Check model architecture, quantization support and any compiled attention kernels together.

This local site saves configurations but does not run a remote model server. The following commands belong on a real, provisioned Linux node with the required software and model access.

Launch a private endpoint

Set the model and a private API key in the shell used to launch the server. Replace both placeholders. The key is a vLLM serving credential, not your AnchorGPU account API key. Bind to loopback while validating; remote access needs an authenticated TLS or private-network path.

export MODEL_ID="your-organization/your-model"
export VLLM_API_KEY="replace-with-a-private-serving-key"
vllm serve "$MODEL_ID" --host 127.0.0.1 --port 8000

For a model intentionally split across four visible GPUs on the same node, add --tensor-parallel-size 4. Confirm the hardware layout and software support before making that change.

Check the API before load testing

In another shell, set the same VLLM_API_KEY and query the model list. A chat-completions request also requires a model with an appropriate chat template; a successful server launch does not guarantee every API task is supported.

curl http://127.0.0.1:8000/v1/models \
  -H "Authorization: Bearer $VLLM_API_KEY"

Use the model identifier reported by the server in client requests. Keep authentication enabled while testing.

Measure sustained capacity

Replay sanitized requests with the intended prompt lengths, output limits and concurrency. Record time to first token, end-to-end latency, throughput, error rate, GPU memory use and the KV-cache capacity reported by the server.

Measure cold model loading separately from steady-state serving. Increase concurrency gradually. If memory becomes the limit, evaluate lower context/concurrency, a suitable quantization format or more memory before assuming more compute will help.

Convert completed work into cost using the fixed reservation price. A headline tokens-per-second number is not a capacity promise, and we do not claim AnchorGPU benchmark results in this guide.

Resolve the common failure modes

Chat request fails: check the model’s chat template and supported API task. Memory failure after load increases: check KV-cache pressure, context and concurrency. Multi-GPU initialization hangs: confirm visible devices, collective libraries and the validity of the tensor-parallel split.

Save the model revision, container digest, launch configuration, validation results and client API contract. Copy operational artifacts away from the node before releasing it.