PyTorch DataLoader Bus Error: Diagnose /dev/shm Without Host IPC
PyTorch & CUDA(Updated )

PyTorch DataLoader Bus Error: Diagnose /dev/shm Without Host IPC

Important correction, November 12, 2026: An earlier version claimed every DataLoader worker bus error was caused by shared-memory exhaustion, recommended a universal 16 GiB allocation or host IPC sharing, changed the host’s shared-memory mount and boot configuration, and said file_system IPC would avoid /dev/shm by writing normal files to disk. Those claims are inaccurate or unsafe as blanket advice. This guide separates evidence collection from environment-specific changes.

PyTorch multiprocessing can produce a message such as:

RuntimeError: DataLoader worker (pid ...) is killed by signal: Bus error.
It is possible that dataloader's workers are out of shared memory.

The word possible matters. Shared-memory exhaustion is common, particularly inside small Docker mounts, but a SIGBUS signal alone does not prove which memory mapping or resource failed. Other issues involving memory-mapped input data or storage must be diagnosed from the worker’s earlier errors and the container/host environment.

The official PyTorch data-loading tutorial describes shared memory when num_workers is greater than zero and recommends reducing memory demand as a first response.

1. Collect read-only evidence first

Before changing host mounts, Docker configuration or the training script, inspect what actually happened:

# Check shared-memory mount size and current occupancy.
df -h /dev/shm

# Check overall available memory without changing limits.
free -h

# Look for recent memory-kill evidence, where permitted.
journalctl -k --since "30 minutes ago" --no-pager | grep -Ei 'out of memory|oom|killed process'

On a host that runs Docker, locate the confirmed training container name and inspect its configured limits and OOM state:

docker ps --format 'table {{.Names}}\t{{.Status}}'
# Replace the example name with your actual, authorized container.
docker inspect --format '{{.Name}} shm={{.HostConfig.ShmSize}} ipc={{.HostConfig.IpcMode}} oom={{.State.OOMKilled}}' TRAINING_CONTAINER

A 0 in a single OOM flag or a missing kernel log does not prove that the /dev/shm limit was the root cause. Match the actual worker failure time with memory and application logs. On managed Kubernetes installations, examine the pod’s memory limit and mounted volume configuration through authorized read-only interfaces.

2. Reduce the batch queue before changing system limits

The DataLoader reference documents that prefetch_factor is a per-worker batch count when multiprocessing is enabled. With four workers and a prefetch factor of two, up to eight batches can be queued, in addition to other memory held by the training job.

Start by testing the same dataset and model with fewer workers or a smaller batch size. A small example, with dataset representing your own configured dataset:

from torch.utils.data import DataLoader

loader = DataLoader(
    dataset,
    batch_size=8,        # Example only; measure for your actual tensor shape.
    num_workers=2,
    prefetch_factor=1,
    persistent_workers=False,
)

For diagnosis, num_workers=0 can help identify whether the problem is specifically connected to worker multiprocessing. It is not necessarily the throughput-optimal production setting.

Be cautious about treating persistent_workers=True as a memory-leak cure: PyTorch documents that it keeps worker processes and Dataset instances alive between iterations. That can reduce respawn overhead but can also retain memory longer.

Do not blindly increase the system-wide /dev/shm mount based on a fixed number of images. Tensor dimensions, data types, number of workers, prefetch queue depth, pinned buffers, transforms and container/cgroup limits determine real demand.

3. Docker: prefer a bounded private shared-memory mount

Docker’s container resource documentation specifies a 64 MiB default for a Docker container’s shared-memory mount when no explicit size is configured. This is a configuration limit, not evidence every multi-worker job fails.

When logs and measured queue usage support the diagnosis, a container owner can choose an appropriate private shared-memory budget using the Docker Compose service shm_size setting, for example:

services:
  trainer:
    image: YOUR_PINNED_PYTORCH_IMAGE
    shm_size: "1gb"  # Illustrative ceiling, not a universal recommendation.
    volumes:
      - .:/workspace
    working_dir: /workspace
    command: python train.py

This Compose excerpt is a configuration illustration, not a complete ready-to-run GPU training stack. The image is intentionally a placeholder. The chosen allocation must fit real host resources and the container’s security and memory controls. Document why that particular size was selected in your environment.

An older revision presented host IPC sharing as a default remedy. Docker’s IPC mode reference documents that host mode uses the host’s IPC namespace, rather than a private container IPC namespace. This reduces isolation and must not be a default workaround. Never rewrite /etc/fstab or remount the host’s shared-memory filesystem as a first-line fix.

For Kubernetes, the official emptyDir documentation describes memory-backed volumes. A pod-level shared-memory mount can use emptyDir.medium: Memory with an appropriate sizeLimit and container memory limits. The Kubernetes volume reference explains that memory-backed volume files count toward the memory limit of the container writing them. Raising the mount ceiling does not create more node memory. Apply such changes through your cluster’s approved configuration process.

4. The actual meaning of file_system sharing

This is the most important technical correction.

PyTorch’s multiprocessing documentation explains that the two common CPU tensor sharing strategies both use POSIX shared-memory mechanisms:

Strategy How objects are identified Trade-off
file_descriptor Passes descriptors obtained with shm_open between processes Keeps descriptors open; may exhaust low file-descriptor limits
file_system Uses names returned by shm_open to identify shared-memory objects Fewer cached descriptors but can leave shared-memory objects after crashes

The file_system strategy is not a disk-backed substitute for /dev/shm. It does not automatically bypass a small shared-memory mount or guarantee that SIGBUS cannot occur. PyTorch documents a background torch_shm_manager to help clean up orphaned objects, but shutdown and crash behavior still require attention.

Switching sharing strategies can be sensible when the actual issue is file-descriptor exhaustion, not when a Docker container has too little shared memory. For a verified descriptor-limit problem, PyTorch supports this controlled option before workers are created:

import torch.multiprocessing as mp

if "file_system" in mp.get_all_sharing_strategies():
    mp.set_sharing_strategy("file_system")
    print("sharing strategy:", mp.get_sharing_strategy())

This does not delete or migrate existing workers. Re-run the relevant workload and observe both memory occupancy and open-descriptor use.

5. Verify the exact fix and define rollback

To claim that a DataLoader failure is resolved, repeat the same training loop and data shape while recording worker count, queue depth, batch size, /dev/shm use and the process exit state. Compare one change at a time.

  • If reducing num_workers or prefetch_factor removes the error and shared-memory occupancy tracks it, shared-memory pressure is a plausible cause.
  • If the failure persists with a single-process loader, investigate dataset files, memory mappings, filesystem integrity and native-extension errors.
  • If a larger private container mount is required, document its actual peak occupancy and its effect on memory limits.
  • If the change worsens other jobs or consumes too much memory, roll back the resource setting instead of enlarging host limits again.

The article provides read-only inspections and bounded options. No actual DataLoader workload, GPU training run or container mutation was executed to validate a specific hardware fix for this correction.

Primary references

Technical correction, November 12, 2026: Original article URL, hero image and publication date preserved. The corrected guide distinguishes shared-memory evidence from a verified incident and removes unsafe blanket host-resource changes.

BC

Written by B. Cho

Systems & AI Infrastructure Engineer

Specializing in GPU inference engines (vLLM, SGLang, Ollama), CUDA memory allocation, and Model Context Protocol (MCP) infrastructure on Linux workstations.