
How to Fix 'RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED' in PyTorch on Linux
The NVIDIA cuDNN API documentation describes CUDNN_STATUS_NOT_INITIALIZED as a possible cudnnCreate() return when no compatible GPU is available, a CUDA driver is disabled or missing, or CUDA runtime initialization fails. The status alone cannot identify which situation occurred in a PyTorch application. The actual cause depends on the selected PyTorch build, device, runtime environment and surrounding error messages.
Updated November 12, 2026: This guide prioritizes read-only inspection and isolated environment testing. It does not prescribe changes to host-wide dynamic linker configuration, automatic cache deletion, or an arbitrary GPU memory allocation ceiling.
First record the software environment
Run these commands within the same Python environment where the failure occurs:
python3 -c 'import torch; print("torch:", torch.__version__); print("torch CUDA build:", torch.version.cuda); print("CUDA available:", torch.cuda.is_available()); print("cuDNN:", torch.backends.cudnn.version())'
python3 -m torch.utils.collect_env
nvidia-smi
Use these outputs for environment comparison; Python imports or device queries may initialize runtime resources, but the shown commands do not request package installation or host settings changes. torch.cuda.is_available() can be false on non-NVIDIA or CPU-only builds, so first establish whether the installed PyTorch build is expected to use CUDA at all. Use the official PyTorch CUDA semantics and local installation selector to understand the requested compute platform.
If the failure is inside a container, record the runtime and GPU device grants. Do not infer that a host CUDA toolkit should be installed or replaced merely because a Python package reported an initialization error.
Separate likely failure families
| Observation | Next diagnostic | Avoid as an automatic fix |
|---|---|---|
| CPU-only torch build | Check the installed build and official install selector | Blindly uninstalling packages or replacing the NVIDIA driver |
| Driver/build mismatch suspected | Compare actual driver, CUDA build and environment | Editing global /etc/ld.so.conf.d based on an assumed path |
| Memory pressure near failing operation | Inspect live GPU utilization and model/batch sizes | Claiming exactly 512 MB or 1 GB of cuDNN workspace is required |
| Failure only with multiprocessing | Check when accelerator runtime initializes, process start method | Globally forcing Python’s spawn method in a running service |
| Only specific convolution shapes fail | Reduce to a small authorized test with captured exception | Deleting other users’ CUDA caches |
PyTorch’s CUDA notes distinguish live tensor allocations from caching allocator reservations. torch.cuda.empty_cache() does not free memory still occupied by live tensors. A CUDA OOM, a cuDNN error, and a driver runtime error should not be collapsed into one cause.
Check an isolated minimal CUDA operation
If CUDA is expected and the machine has sufficient resources, a resource-consuming GPU test under your authorized quota using your existing environment can help determine whether initialization fails without the original large model. This code allocates GPU tensors and executes convolution; it is not read-only and must not be run on a production or shared machine without permission. It was not executed to prepare this article.
import torch
print("torch:", torch.__version__)
print("cuDNN:", torch.backends.cudnn.version())
if not torch.cuda.is_available():
raise RuntimeError("This environment does not expose a CUDA-capable PyTorch device")
device = torch.device("cuda:0")
layer = torch.nn.Conv2d(3, 4, kernel_size=3).to(device)
x = torch.randn(1, 3, 16, 16, device=device)
with torch.no_grad():
y = layer(x)
torch.cuda.synchronize()
print("output shape:", tuple(y.shape))
If this small test works but the application fails, investigate the specific workload’s batch size, inputs, model allocations and start order. If the test also fails, collect the full trace and version/build information before reinstalling anything.
Multiprocessing: use a supported context only when relevant
PyTorch’s multiprocessing guidance describes why CUDA runtime initialization before a fork can cause poison-fork issues, and documents spawn or forkserver as alternatives for CUDA work in child processes. That is a specific failure hypothesis, not proof that all DataLoader errors originate in cuDNN.
For an application that actually uses accelerators in worker processes, select the context in its own process lifecycle and make sure worker functions and objects are picklable. Avoid mutating the multiprocessing start method globally in a library or notebook simply to silence one exception. For normal CPU DataLoader workers, changing this context may be unnecessary and may introduce new overhead.
Do not change system-wide loader paths to fix one virtual environment
A conflicting shared-library load is possible, but it must be demonstrated with environment-specific diagnostics rather than assumed from a guessed file name such as _C.cpython-310-x86_64-linux-gnu.so or a single ldd output.
Do not add a Python virtual environment’s NVIDIA library directory to /etc/ld.so.conf.d/ or run host-wide ldconfig as a general troubleshooting step. Those affect programs outside the virtual environment. If the cause is verified as a package mismatch, reproduce in a new isolated virtual environment with the supported PyTorch installation instructions, preserving the original for rollback.
Likewise, do not delete CUDA or Torch extension caches recursively without identifying what they contain and which jobs depend on them.
Verification and limitations
- Preserve the original traceback and note whether it happens at import, first allocation or specific model operation.
- Determine expected accelerator backend and compare actual package/driver versions.
- Check a minimal local operation only when resources and access permit.
- Change one application-scoped setting at a time after a log-supported hypothesis, then rerun the exact failed workload.
- Confirm the original environment and other users’ processes remain unaffected; retain a rollback path.
This article documents a diagnostic procedure, not measured GPU memory requirements or a guaranteed repair.
Official references
- PyTorch CUDA semantics
- PyTorch multiprocessing best practices
- PyTorch supported installation choices
- NVIDIA cuDNN status for cudnnCreate
- NVIDIA CUDA documentation
Reviewed November 12, 2026. No drivers, host linker configuration, CUDA caches or production workloads were modified during preparation.