How to Fix CUDA Out of Memory (CUDA OOM) in PyTorch on Linux
PyTorch & CUDA(Updated )

How to Fix CUDA Out of Memory (CUDA OOM) in PyTorch on Linux

A CUDA out-of-memory error tells you that an allocation failed; it does not by itself identify why. The useful response is to locate the failing process and phase, establish what changed, and test one plausible cause at a time. An inference request with unusually long input, an optimizer step, and a distributed initialization can require different investigations.

The guidance below distinguishes the supplied PyTorch 2.15 documentation from current main-branch documentation. Check the documentation for your installed release and allocator backend before applying an API or configuration option. The supplied excerpts do not establish when those behaviors first appeared or whether they apply unchanged to another release.

1. Establish the failing context

Record the complete exception and the operation or training phase in which it occurred. Alongside it, capture:

  • PyTorch version and build, CUDA runtime and driver versions, GPU model, and available device capacity.
  • Model and optimizer configuration, training or inference mode, precision, input shapes, micro-batch size, and sequence length.
  • Process and distributed rank, device mapping, allocator backend, and relevant environment settings.
  • Whether the failure occurs on the first iteration, only after particular inputs, or after many iterations.

Inspect the device used by the failing work. PyTorch 2.15 documents that CUDA tensors are allocated on the currently selected GPU by default, while operations on an already allocated tensor place results on that tensor’s device. A later device selection therefore does not move existing tensors. This matters when a process uses more than one GPU. See PyTorch 2.15 CUDA semantics.

The PyTorch 2.15 device_count reference describes a count of available GPUs. A count alone does not establish which device a failing allocation used or how much capacity is free. Check the actual process, rank, and tensor placement.

2. Follow the evidence

Use this decision tree to select the next investigation, not to declare a diagnosis:

  1. Does the same bounded input fail at the same phase? If so, compare the workload configuration and memory observations at that phase with a smaller representative input. If only particular inputs fail, record their shapes and length distribution.
  2. Does the failure appear only after repeated iterations? Compare observations at equivalent points over time. Inspect retained outputs, metric histories, and references to tensors or computation graphs; establish whether they are intentionally kept before changing them.
  3. Does it occur only with multiple processes or ranks? Record each rank’s device and failure phase. Check whether multiple ranks share a device as intended and investigate each process separately.
  4. Is other work using the device? Identify ownership and coordinate before making changes. Do not terminate another user’s process or reset a shared GPU as a diagnostic shortcut.
  5. Is the cause still unclear? Preserve the exception, workload inputs, version details, and observations. Form one hypothesis for the next controlled comparison.

For PyTorch 2.15, the CUDA semantics documentation describes memory_reserved() and max_memory_reserved() as measures of total memory managed by the caching allocator, and describes memory_stats() and memory_snapshot() as tools for more detailed investigation. Collect observations in the process and device executing the workload, at comparable points. A separate process is not a substitute for examining the failing process.

The supplied excerpts do not establish that these allocator tools capture every device allocation. Treat their output as one part of the investigation, and verify the scope of each tool in the documentation for the pinned release before drawing conclusions about an unexplained difference.

3. Test workload changes before assuming fragmentation

If a representative smaller workload completes where the original fails, compare the two configurations and their outputs. Candidate changes include reducing the micro-batch or limiting input length, provided those changes preserve the intended use. For training, a smaller micro-batch can change optimization behavior unless the training procedure accounts for it; validate the resulting behavior rather than assuming equivalence.

If memory grows across iterations, inspect whether the application intentionally retains tensors or graph-connected results. Change retention only after confirming that the values are not needed for a later backward pass or other computation. Compare equivalent iterations before and after the change.

Activation checkpointing, mixed precision, and distributed sharding may be appropriate design options, but their APIs, constraints, numerical effects, and tradeoffs require release-specific verification. The supplied excerpts do not substantiate a configuration recipe or a savings estimate for them. Consult the pinned release’s official documentation and validate model behavior and performance on the actual workload before adoption.

4. Treat allocator tuning as a conditional experiment

Do not infer fragmentation solely from an OOM or from a difference between memory observations. First establish the allocator backend, gather the diagnostics supported by the installed release, and check the release’s definitions and limitations.

The PyTorch 2.15 CUDA semantics documentation describes max_split_size_mb as a last-resort option for an OOM workload showing a large amount of inactive split blocks. It says the option is meaningful for the native backend, is ignored with cudaMallocAsync, and can have a performance cost ranging from negligible to substantial depending on allocation patterns.

Before considering it:

  • Verify that the installed release documents the option, its complete syntax, and the active backend.
  • Confirm that the release’s diagnostic definitions support the inactive-split observation.
  • Plan a separate comparison with the same representative inputs and one configuration change.
  • Record completion, memory observations, output behavior, and performance; revert if the result is unsuitable.

The supplied excerpt does not provide complete configuration syntax, so it is not enough to construct a runnable setting here. It also does not substantiate an expandable_segments recipe. Do not copy an allocator setting from another release without checking its official documentation.

The current main-branch environment-variable reference calls PYTORCH_CUDA_ALLOC_CONF a backward-compatible alias for PYTORCH_ALLOC_CONF. That is a current main-branch statement, not proof that either spelling or a particular option is supported by your installed release.

5. Understand cache clearing

PyTorch 2.15 documents that empty_cache() releases unused cached memory for use by other GPU applications; memory occupied by tensors is not freed. The documentation also says the call cannot increase the GPU memory available for PyTorch itself. It is therefore not a substitute for identifying live workload pressure, and it is not a guaranteed fix. Verify the equivalent reference for your pinned release before relying on this behavior.

6. Validate safely

Avoid deliberately exhausting a shared GPU. A synthetic multi-gigabyte allocation may interfere with other work, but its actual effect depends on available capacity and the environment; eviction is not established by the supplied evidence.

Instead, agree on a bounded window and run a representative portion of the existing workload where authorized. Keep the input, device, versions, and measurement points consistent between comparisons. A candidate change is acceptable only after the intended workload completes and its outputs, numerical behavior, and performance have been checked against the requirements. If it still fails, retain the evidence and test the next hypothesis.

Primary sources and validation boundary

This page provides a diagnostic framework, not an execution receipt. No command, workload, configuration, or performance result has been independently run or validated here. The supplied source excerpts were reviewed for this correction on November 13, 2026; they do not establish what the documentation or behavior was on the original publication date.

BC

Written by B. Cho

Systems & AI Infrastructure Engineer

Specializing in GPU inference engines (vLLM, SGLang, Ollama), CUDA memory allocation, and Model Context Protocol (MCP) infrastructure on Linux workstations.