How to Fix 'RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasCreate(handle)' in PyTorch on Linux
PyTorch & CUDA(Updated )

How to Fix 'RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasCreate(handle)' in PyTorch on Linux

CUBLAS_STATUS_ALLOC_FAILED is not permission to kill every process attached to the GPU. A shared accelerator can have several healthy users; indiscriminate termination destroys their work and may not address the cause of this failure.

Updated November 12, 2026: An older version suggested broad fuser -k on all NVIDIA devices, universal allocator flags, host IPC sharing, 16 GiB shared memory, system-wide memlock changes and a stress test that supposedly certified production readiness. These have been replaced with source-scoped diagnostics and explicit limits.

Understand what NVIDIA actually documents

The official NVIDIA cuBLAS status reference defines CUBLAS_STATUS_ALLOC_FAILED as a resource allocation failure inside cuBLAS, usually caused by a cudaMalloc failure. The status does not establish an exact 16–64 MB workspace requirement, a specific driver defect, an orphaned process, virtual address fragmentation, or Docker shared memory exhaustion.

The PyTorch CUDA memory notes distinguish memory occupied by tensors from memory reserved by the caching allocator. torch.cuda.empty_cache() releases only some unoccupied cached blocks; it does not free live tensor allocations. Memory fragmentation is a hypothesis to measure, not an automatic diagnosis.

Step 1: Inspect the failing workload and memory

Run these read-only commands in the authorized training environment:

nvidia-smi
python3 -c 'import torch; print("PyTorch:", torch.__version__); print("Build CUDA:", torch.version.cuda)'
python3 -m pip show torch

Record the GPU identity, application PID, current allocation, model revision, batch shape, operation that first failed and full CUDA stack trace. A Python import can initialize process resources; do not run diagnostics on another user’s environment without permission. A display of high GPU utilization is not proof that a process is abandoned.

For memory pressure inside your own Python process, PyTorch offers allocated/reserved memory counters and summaries:

import torch

if not torch.cuda.is_available():
    raise RuntimeError("CUDA is unavailable in this Python environment")
print(torch.cuda.memory_summary(device=0, abbreviated=True))

The above snippet queries a CUDA device and can initialize runtime state. Execute it only as part of an authorized diagnosis, not inside a production process that must remain untouched.

Step 2: Identify a genuine contention problem

Observation Safer next action Do not automatically do this
Another active user owns GPU memory Ask the user or resource scheduler; choose another device or schedule Force-kill everyone holding /dev/nvidia*
Your own batch grows and fails Reduce batch size or concurrent jobs, document memory use Set arbitrary allocator flags without comparison
Many inactive split blocks Inspect memory_summary() and supported allocator settings Assume all errors are fragmentation
A worker reports SIGBUS or shared memory exhaustion Investigate that separate error and container memory budget Assume it caused cuBLAS handle allocation failure
Driver/runtime initialization fails Compare supported build and driver compatibility Add a GPU driver PPA or rewrite system-wide library paths

If a scheduler-managed job genuinely failed, stop only the identified job that you own through its normal graceful shutdown or cancellation mechanism. No broad fuser -k, pkill, or device-wide kill command is an acceptable default diagnostic.

Step 3: Change allocator behavior only with evidence

PyTorch’s CUDA memory-management guidance explains allocator configuration, including max_split_size_mb. That option is a last-resort candidate when a workload aborts due to OOM and statistics show many inactive split blocks; it applies to the native allocator backend and is ignored with cudaMallocAsync. It can have a substantial performance cost depending on allocation patterns.

Do not combine multiple allocator flags, an enlarged model context, increased tensor-parallel count and a bigger batch into one experiment. Change a single measured variable at a time, keep the original launch configuration for rollback, and compare failure recurrence and memory statistics.

Step 4: Keep container IPC and system limits isolated

Docker’s run reference says --ipc=host uses the host system IPC namespace. It reduces isolation and is not a general cuBLAS repair. --shm-size sizes container shared memory and must fit the real resource budget. A 16 GiB value is not universally required.

Do not edit /etc/security/limits.conf or set memlock unlimited without a demonstrated, separately identified lock-limit failure and administrator approval. The existence of a cuBLAS allocation error does not prove locked-memory limits are the cause.

Step 5: Validate a small, authorized operation

After inspecting the runtime and freeing your own unused allocations, use a small operation only where resource limits permit. This example allocates GPU resources and is not read-only:

import torch

if not torch.cuda.is_available():
    raise RuntimeError("CUDA build/device not ready for this test")

a = torch.randn((16, 16), device="cuda", dtype=torch.float32)
b = a @ a
torch.cuda.synchronize()
print("matmul output shape:", tuple(b.shape))

Passing one small matrix multiplication only shows that this particular operation completed in that environment. It does not demonstrate that the original model, long-running training, memory budget or driver configuration is production-ready. The example was not executed as part of preparing this article.

Primary sources

Reviewed November 12, 2026. This is a source-based diagnostic workflow, not a reproducible performance benchmark.

BC

Written by B. Cho

Systems & AI Infrastructure Engineer

Specializing in GPU inference engines (vLLM, SGLang, Ollama), CUDA memory allocation, and Model Context Protocol (MCP) infrastructure on Linux workstations.