
How to Fix 'RuntimeError: Watchdog caught collective operation timeout' in PyTorch NCCL on Linux
The PyTorch distributed API defines a process-group timeout for collective operations; NCCL collectives that exceed it can be aborted asynchronously. A WorkNCCL(OpType=ALLREDUCE) timeout report is therefore an observed symptom, not proof that the network interface is broken or that increasing the timeout is a repair. Collective order, an earlier CUDA failure, different code paths among ranks, resource pressure and network problems all deserve investigation.
Updated November 12, 2026. This guide prioritizes finding the first failing rank and operation, respects the distinction between global RANK and LOCAL_RANK, and removes universal network-override, GPU P2P-disable and timeout-increase recipes. Nothing was executed on a live NCCL cluster during this editorial update.
Start with the first failure, not the last watchdog line
Collect timestamped logs from every rank of the same failed job, with the model version, PyTorch/NCCL/CUDA stack, WORLD_SIZE, global RANK, node identity and local device assignment. Protect any secrets, infrastructure addresses and training data before sharing.
| Observed evidence | Investigate next | What it does not prove |
|---|---|---|
| One rank hit CUDA OOM/illegal access before timeout | That rank’s original CUDA operation and allocation | The communication network caused the failure |
| Only some ranks entered the expected collective | Divergent input count, conditional model branch, exceptions, earlier collective order | Increasing timeout synchronizes incompatible code paths |
| A rank is CPU/I/O blocked before collective | Dataset loading, worker exception, process state | Setting more DataLoader workers universally fixes it |
| NCCL logs show an unreachable NIC/interface | Routes and actual NIC in the process/container namespace | All jobs should force an interface copied from another machine |
| Logs show shared-memory allocation failure | Container /dev/shm budget and permissions |
Host IPC sharing is the default required solution |
| P2P/IB errors only on certain topology | Device links, firmware/driver, verified transport details | Disabling P2P/InfiniBand is a free performance fix |
The official NCCL troubleshooting guidance addresses transport, shared memory, system configuration and debugging as separate issues. Prioritize what the logs actually report.
Step 1: Read-only inventory on the affected worker
The following commands inspect local state; they do not run distributed collectives. Some may be unavailable on a system without NVIDIA utilities or the appropriate permissions.
# Run only on the affected host with authorization.
nvidia-smi -L
nvidia-smi topo -m
ip -brief addr
ip route
df -h /dev/shm
Record kernel/driver and container image versions separately. In Docker or Kubernetes, NIC names and routes in the container namespace may differ from those on the host. A one-time nvidia-smi observation does not reconstruct memory pressure from the failure’s timestamp.
Step 2: Inspect PyTorch’s ProcessGroupNCCL debug options
The PyTorch 2.15 NCCL environment reference documents TORCH_NCCL_TRACE_BUFFER_SIZE, TORCH_NCCL_DUMP_ON_TIMEOUT and TORCH_NCCL_ASYNC_ERROR_HANDLING. A timeout dump requires a nonzero trace-buffer size. Capturing a trace is for authorized diagnostic sessions and may record internal metadata.
For an isolated, reproducible job you own, review the actual accepted values in your installed version before modifying its launcher. The official references identify these diagnostic knobs:
| Diagnostic option | Officially documented purpose | Safety boundary |
|---|---|---|
TORCH_NCCL_TRACE_BUFFER_SIZE |
Size of the ProcessGroupNCCL tracing buffer; a timeout dump needs a value larger than zero | Choose an appropriate bounded size for an authorized reproduction |
TORCH_NCCL_DUMP_ON_TIMEOUT |
Request diagnostic dump behavior on timeout when tracing is enabled | Check accepted values in the installed PyTorch release; not a universal failure explanation |
NCCL_DEBUG |
Controls NVIDIA NCCL diagnostics; INFO prints debug information |
Temporary testing only, not a system-wide default |
These are configuration concepts, not a tested launch script or a claim that the settings can reproduce a problem on every NCCL version.
These flags do not diagnose the culprit automatically, and the dump may not identify a Python source line or reveal what a CPU-blocked rank was doing. Review the recorded collective sequence alongside the original exception and application logs.
Do not set TORCH_NCCL_ASYNC_ERROR_HANDLING=1 just to suppress warnings: the documented modes have different process and communicator tear-down behavior. In a shared environment, use the approved error-handling policy. Remove temporary verbose debug overrides after investigating.
Step 3: Use LOCAL_RANK for a local GPU, not global RANK
The PyTorch torchrun reference differentiates LOCAL_RANK and global RANK. On a multi-node run, setting the CUDA device to dist.get_rank() can request a nonexistent local GPU when the global rank exceeds the node’s visible device count.
A fragment for an existing torchrun-launched program is:
# Illustrative setup; do not run outside a valid torchrun job.
import os
import torch
import torch.distributed as dist
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)
dist.init_process_group(backend="nccl") # torchrun supplies global rank/world size
try:
print("global rank:", dist.get_rank(), "local device:", local_rank)
# Run only your own known-good, rank-matched training workload.
finally:
dist.destroy_process_group()
This example is not a reproduction of a watchdog failure and has not been executed. In particular, merely initializing a process group and printing ranks cannot prove that a later all-reduce or long training job will succeed.
Step 4: Change transport settings only after identifying the cause
NVIDIA’s official NCCL environment variable reference states that some debugging overrides can reduce performance or even break functionality when left enabled in production.
- Interface:
NCCL_SOCKET_IFNAMEselects prefixes for network interfaces. Only if logs identify a wrong choice, select the real reachable interface in the worker’s namespace. The NVIDIA reference describes a leading equals sign to match an exact name; do not assume every host useseth0,ens3, or a specific Mellanox HCA. - P2P:
NCCL_P2P_DISABLE=1disables GPU peer-to-peer transport. It is a narrowly scoped diagnostic comparison when topology/logs support that hypothesis, not the default fix for consumer GPUs. - InfiniBand/RoCE:
NCCL_IB_DISABLE=1disables IB transport and can remove a working high-performance link. Do not set it on production machines without a documented reason. - Timeout: increasing the process-group timeout may postpone a failure without repairing mismatched collective order, missing ranks or a prior GPU exception.
If a verified transport experiment is authorized, compare the same workload with and without one change and remove any diagnostic override afterward. Avoid editing global shell profiles, host networking, firewall or system-wide NCCL settings.
Step 5: Recheck the reduction order before calling the fault fixed
Inspect which ranks actually reached each collective, and whether their input data, exceptions and control flow diverged before the watchdog message. The PyTorch distributed reference documents operation ordering and process-group behavior for the installed release. Historical PyTorch DDP design notes describe bucket scheduling for an earlier implementation; do not treat a v1.4 architecture note as proof of all current process-group internals. A mismatch is a testable hypothesis, not a diagnosis of every NCCL timeout.
A safe investigation report records:
- Rank and first error timestamp, not just the final timeout line.
- The collective name/sequence as supported by the installed diagnostic tools.
- Device identity and local/global rank mapping.
- Whether the same workload fails after one isolated change, and whether new errors appear.
- Which temporary flags were removed after the test.
No cluster workload or GPU benchmark was run to prepare this article. The method distinguishes plausible causes and verifies source-defined knobs; it cannot certify a production NCCL repair without an authorized before/after reproduction.
Official references
- PyTorch ProcessGroupNCCL environment controls
- PyTorch torchrun launcher and rank environment
- PyTorch distributed collectives
- PyTorch DDP reduction ordering
- NVIDIA NCCL environment configuration
- NVIDIA NCCL troubleshooting
Reviewed November 12, 2026. The correct transport configuration depends on the specific NCCL release, device topology and job.