
How to Fix 'RuntimeError: Address already in use' in PyTorch Distributed on Linux
RuntimeError: Address already in use during PyTorch distributed startup can indicate that a rendezvous listener could not bind the requested address and port. Do not kill every process using port 29500. On a shared GPU server the listener may belong to an unrelated, healthy training job.
Updated November 12, 2026: This version focuses on authorized listener inspection, appropriate PyTorch rendezvous modes and the Linux kernel distinction between FIN_WAIT_2 and TIME_WAIT. No broad process-termination or shared-memory deletion instructions are provided.
Best first option for independent single-node training
The PyTorch 2.15 elastic launch documentation describes standalone rendezvous. For a single node, two local workers and a compatible existing training script:
torchrun --standalone --nnodes=1 --nproc-per-node=2 train.py
Choose worker counts appropriate to your own GPUs and script. This is a documented single-node launch mode; whether it avoids a particular port conflict still depends on the actual launch options and environment. The official documentation also explains c10d rendezvous with a loopback endpoint assigned an available port for separate single-node jobs; such loopback settings are not multi-node defaults.
Step 1: Inspect the actual port, without stopping its owner
Port 29500 is an example; always check the address and port in the error.
ss -ltnp '( sport = :29500 )'
The Ubuntu ss manual documents listener and process inspection with the -l and -p flags. If the owner is not shown, the process may have exited or permissions may be insufficient; check scheduler inventory or ask an administrator. Before intervening, identify a live listener and its owner; do not infer ownership from an abandoned job name or an unrelated process.
Step 2: Choose the correct rendezvous layout
| Job | Appropriate approach | Avoid |
|---|---|---|
| Single-node independent job | torchrun --standalone --nnodes=1 |
Reusing a shared fixed port |
| Multiple single-node jobs | Launcher-supported c10d rendezvous on localhost:0 as documented for single-node independent jobs | Assuming a random number is collision-free |
| Multi-node training | Shared reachable --rdzv-endpoint, unique --rdzv-id, consistent backend and nodes |
Each host connecting to its own loopback address |
| Scheduler-managed job | Scheduler-owned stop/restart by job ID | Terminating processes by name |
PyTorch documents these flags. For multi-node jobs, the intended participants must reach the same rendezvous host and use the correct job identity. Selecting a port by opening and immediately closing a temporary socket creates a time-of-check/time-of-use race; use the launcher’s supported mechanism.
Step 3: Diagnose before making changes
Record the exact error, version, launch options, requested address/port and job owner.
python3 -c 'import torch; print("torch:", torch.__version__)'
ss -ltnp '( sport = :29500 )'
A TCP bind conflict differs from DNS resolution failure, firewall refusal, IPv6 configuration error, NCCL initialization failure and GPU OOM. Seeing an IPv6 address in a log alone does not prove a dual-stack self-collision.
If the listener belongs to another job, leave it alone. For a failed job you own, verify its scheduler ID, then use the scheduler’s ordinary graceful stop/restart workflow; escalate only with authorization. Do not run broad pkill or fuser -k by port, and do not delete /dev/shm/torch* as a generic socket fix. These can disrupt active work and do not diagnose a TCP bind failure.
TIME_WAIT is not configured by tcp_fin_timeout
The Linux kernel TCP sysctl reference defines tcp_fin_timeout for orphaned FIN_WAIT_2 connections. Its documented purpose is the orphaned FIN_WAIT_2 state rather than a general TIME_WAIT remedy. Observing TIME_WAIT by itself does not prove that it caused a listener bind error, and changing global TCP settings is not a safe default response to one job’s failure.
Verification and limits
- Identify the conflicting listener and its owner without changing it.
- For a single-node compatible test job, compare the launcher-managed standalone mode with your old rendezvous setup.
- Verify unrelated jobs were untouched and that the original error no longer appears.
- Successful rendezvous proves startup only. It does not prove gradient synchronization, NCCL health or correctness of training.
No distributed workload, GPU benchmark or process termination was performed to prepare this guide.
Official references
Reviewed November 12, 2026. Commands are for inspection or controlled, owner-authorized test jobs.