
How to Fix vLLM Out of Memory (OOM) and KV Cache Allocation on Linux
Diagnose vLLM out-of-memory failures by their actual stage
A vLLM process can fail during model loading, CUDA graph capture, KV-cache allocation, or inference under concurrent load. Those failures have different causes. A generic command that changes memory utilization, eager execution and CPU swapping all at once makes the outcome harder to interpret.
Technical correction, November 11, 2026: This guide previously contained an unmeasured GPU-specific VRAM sizing table, claimed that vLLM always preallocates 90% of the GPU to the KV cache, and recommended legacy --swap-space 8 for concurrent request exhaustion. Those descriptions were misleading for vLLM V1. It also asserted that eager execution universally reduces performance by 5–10%, without measured evidence. These claims are removed or qualified below, preserving the original article URL and publication date.
The most useful references are the vLLM serve CLI, optimization guidance, and V1 architecture differences.
1. Identify the failure before changing a setting
Use the following symptom-to-diagnosis map rather than guessing from the generic error message:
- CUDA OOM while loading weights: Model size, dtype, competing processes or runtime allocations. First record GPU memory and model precision.
max seq len ... larger than ... KV cache: Requested context exceeds cache capacity for that build. First inspect the startup cache estimate and context cap.- CUDA graph capture OOM during warmup: Graph-capture overhead on top of other allocations. Compare one bounded run with eager execution, keeping everything else unchanged.
- Runtime preemptions and long waits: Active requests competing for KV-cache capacity. Observe cache usage and try a smaller batch-concurrency limit.
- OOM in a custom attention kernel: Extension/backend allocation or compatibility. Identify the actual failing kernel and installed versions before changing unrelated options.
A model that cannot load its weights will not fit merely because you reduce --max-model-len. By contrast, a model that fits but cannot provide its requested context may benefit from a smaller context limit. Diagnose the error text rather than assuming these are the same failure.
2. Record the environment without terminating GPU workloads
The following commands inspect the current environment. They do not deliberately allocate a large GPU tensor or stop other processes:
python --version
python -m pip show vllm torch
nvidia-smi
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv
vllm serve --help | grep -E 'gpu-memory-utilization|max-model-len|max-num-seqs|enforce-eager'
Identify whether the device is a workstation display GPU or a dedicated accelerator, whether other processes own memory, and whether the specific vLLM build accepts the configuration flags you intend to use.
Do not use destructive commands to kill other GPU users in an effort to free memory. If the GPU is shared, arrange an authorized staging allocation instead.
3. Understand the GPU memory budget
--gpu-memory-utilization is the fraction of device memory budgeted to the current vLLM model executor, not a statement that the same fraction is exclusively filled by KV-cache blocks. The effective default can vary by release and cache implementation; use the pinned build’s --help and startup logs rather than repeating an outdated universal default.
Weights, CUDA and PyTorch runtime allocations, compiled kernels, graphs, attention workspaces, multimodal encoders and KV-cache capacity all contribute to memory demand. Quantized weights can reduce one component but do not reduce all overhead by a fixed factor.
Why a universal 7B/32B/70B GPU table is unreliable
The prior table declared, for example, a fixed 3.8 GB minimum KV cache for a 32B AWQ model at 8K context. That number is not portable across checkpoints, model architectures, attention variants, precision, cache data types and concurrency. No local GPU measurement was attached to those figures, so we have removed them.
A better method is to record the actual startup cache capacity and vary one parameter at a time, observing the change against the same model and test requests. If you need to estimate weight memory before downloading a checkpoint, label the figure as weight-only arithmetic and include the memory omitted by the estimate.
A reproducible, offline KV-memory budget example
Before changing a live vLLM service, you can test the arithmetic with Agentic Pulse’s inspectable offline KV capacity planner and synthetic configuration fixture. These downloadable Python/JSON files are hosted by the site, not a third-party API. Read the source before executing it. The planner uses only the Python standard library and does not download weights, inspect credentials, connect to a GPU, or contact the network itself. Downloading the files from the website, of course, uses HTTPS.
Place both downloaded files in the same directory, then reproduce the example:
python3 kv_cache_capacity_audit.py --self-test
python3 kv_cache_capacity_audit.py \
--config kv_cache_example_synthetic.json \
--gpu-vram-gib 24 --weights-gib 17.1 \
--runtime-reserve-gib 1.4 --gpu-utilization 0.9 \
--context-tokens 8192 --sequences 1 \
--block-size 16 --kv-dtype bf16
We executed the script’s self-test and a separate eight-test source regression suite on synthetic data. The current self-test reported 4 positive cases and 7 invalid-input rejections. For the fictional full-attention GQA model in the fixture (64 layers, 8 KV heads, 128-dimensional heads), the calculation gives 262,144 bytes per KV token and 2 GiB per 8,192-token sequence. Under the user-declared 24 GiB physical GPU budget, 90% utilization, 17.1 GiB resident weights and 1.4 GiB runtime reserve, one such sequence fits the declared budget but two do not. These are reproducible calculations, not GPU measurements, VRAM-fit guarantees, benchmark results or claims about a real checkpoint.
The planner rejects some recognizable unsupported configuration indicators, including a declared qwen3_5/qwen3.8 model type, kv_lora_rank for MLA, a nonzero sliding_window, nonempty per-layer attention types, and a nested text_config or vision_config. Its source cannot detect every hybrid architecture or verify how a model is distributed across GPUs. Independently confirm that the real checkpoint uses only full-attention MHA/GQA layers, the model metadata is complete, and deployment actually uses a single GPU without KV offloading before interpreting its estimate. This is important when following the Qwen3.8-27B serving guide. For unsupported architectures, use the real model’s serving documentation and startup KV-cache allocation logs. Even when the declared arithmetic fits, actual vLLM scheduling, CUDA graphs, memory fragmentation, prefix reuse and other work can still trigger OOM.
For interpreting V1 runtime preemptions after startup, see the KV-cache preemption diagnostic guide.
4. Begin with a bounded configuration, not a promised fix
Before loading any model, consult the installed release’s vllm serve --help output and evaluate each limit separately. A convenient planning checklist for single-node local serving is:
| Parameter to inspect | Hypothetical starting assumption | Validation before an actual launch |
|---|---|---|
| Model repository | Qwen/Qwen2.5-Coder-7B-Instruct (illustration) | Model card, GPU architecture, quantization and chosen vLLM release |
| HTTP listener | 127.0.0.1 for local-only access | Network boundary and authenticated endpoints separately |
| GPU memory fraction | 0.80 as a comparison assumption | Real other-process usage and model startup allocation logs |
| Max model context | 4096 tokens as an example | Required prompt/output lengths for your application |
| Max concurrent sequences | 4 as a comparison point | Live scheduling pressure, queueing and offered load |
This table is not an executable command, release-specific tested configuration or guarantee that the model fits a particular GPU. Take actual supported flags and checkpoint requirements from the installed vLLM release. Changing several values at once obscures which one affects the error.
These values are deliberately illustrative, not certified as safe on every GPU. They also do not promise that the model will fit a 12 GB card. Compare the actual model’s memory requirements and your current free GPU memory first.
--gpu-memory-utilization: change only after examining available memory and competing workloads. A lower fraction gives the vLLM executor a lower budget and can itself leave insufficient room for the model or KV cache.--max-model-len: sets the model context length. A smaller supported limit may make a previously impossible context/KV requirement feasible, but only if your application can tolerate that limit.--max-num-seqs: caps sequences processed in one iteration. Lowering it can reduce concurrent KV-cache pressure but may increase queueing.
These parameters address different constraints. Do not assume any single setting automatically resolves all out-of-memory exceptions.
5. Test CUDA graph capture as a separate hypothesis
vLLM V1 can use CUDA graphs, which introduce memory overhead during capture and execution. Only when the failure stage points to graph capture, compare the same configuration once with --enforce-eager:
Use the documented --enforce-eager option in a controlled test of an otherwise unchanged, correctly launched vLLM configuration. Do not paste a bare option as a shell command.
An eager run can help separate graph-capture memory pressure from other failures. However, eager execution changes the execution mode and may affect latency or throughput. The previous blanket claim of a 5–10% speed loss had no documented test conditions and should not be treated as a measured result.
Record startup success, GPU peak memory, request latency and throughput for both configurations. If the same model still fails when using eager execution, continue investigating model weights, other allocations and kernel-specific errors rather than claiming the issue is solved.
6. Runtime KV exhaustion is not the old V0 CPU swap mechanism
The previous --swap-space 8 example is not appropriate for V1. The V1 engine removed legacy GPU↔CPU KV-cache swapping for request preemption. Newer versions may offer separate opt-in KV-cache offloading features; these have different flags and requirements and should not be confused with the removed swap option.
If concurrent workloads trigger repeated preemptions or long queueing:
- Inspect the instance’s running/waiting request counts and KV-cache usage metrics.
- Compare the same workload with a smaller
--max-num-seqsor batch-token budget. - Check whether prompt and output length limits reflect the real application.
- Record completion and queueing; simply reducing concurrency can lower preemption while worsening throughput.
For exact preemption and metric guidance, see our KV-cache exhaustion companion guide and the official optimization guide.
7. Verify that the endpoint works, not merely that the command starts
When the model finishes startup, a read-only request to the local bound server can check its model listing. vLLM’s online serving reference documents the /v1/models endpoint:
curl --max-time 5 -fsS http://127.0.0.1:8000/v1/models \
| python -m json.tool
This is only an endpoint smoke test. It does not demonstrate that long prompts, tool calling or concurrency are correct. For a real acceptance test, run a small, agreed workload and inspect the actual answers, GPU peaks, completion counts, and timing under controlled load.
Avoid exposing an unauthenticated inference server by casually replacing 127.0.0.1 with 0.0.0.0. External access needs a deliberate authentication and network boundary.
8. Before and after: what to save
Maintain a compact measurement record with:
- Model repository and checkpoint revision, quantization/dtype and context limit.
- GPU model, compute capability, driver, installed vLLM/PyTorch versions and free memory before starting.
- Exact launch command and startup failure or cache-capacity log.
- Any graph-capture errors, engine preemption warnings or OOM messages.
- Accepted/failed requests, throughput, time-to-first-token, latency and queueing for the same workload.
Successful startup alone does not prove that a change improved the intended application. A plausible configuration needs evidence that it both fits the target GPU and meets the task requirements.
Primary references
- Current vLLM
serveCLI arguments - vLLM V1 engine differences and legacy swap removal
- Official vLLM optimization guidance
- vLLM V1 metrics and legacy CPU cache notes
- PyTorch CUDA memory management reference
- vLLM online serving endpoints
Revised November 12, 2026. The offline synthetic KV calculations were executed and regression-tested; no model was run on a GPU and no universal performance improvement was measured.