How to Fix NVIDIA Docker GPU Passthrough for Local LLMs on Linux
PyTorch & CUDA(Updated )

How to Fix NVIDIA Docker GPU Passthrough for Local LLMs on Linux

When a Linux host sees an NVIDIA GPU but a Docker workload reports could not select device driver "" with capabilities: [[gpu]], first identify which Docker daemon received the request. Host driver health, daemon configuration, container GPU access and model startup are separate boundaries. Success at one boundary does not establish success at the next.

This article provides a diagnostic framework, not a verified runtime installation or model deployment procedure. Use it to narrow the failing layer, then consult the documentation for the exact releases installed on the machine before making a change.

1. Locate the failing layer

Record the host, user account, Docker context and daemon endpoint used by the failing invocation. A client connected to a remote engine needs checks on the daemon host. Rootless and system daemons can also be separate engines on the same machine.

Follow this decision tree:

  1. Does the daemon host’s NVIDIA diagnostic report the expected GPU? If not, investigate the host driver, kernel and device first.
  2. Is the selected daemon configured for NVIDIA GPU access? If uncertain, inventory the daemon, toolkit and driver versions. Compare the actual configuration with vendor documentation for those releases.
  3. Can a minimal, compatible GPU container see the intended device? If not, investigate runtime registration, device selection and daemon logs. Choose the diagnostic image and invocation from documentation for the pinned releases, after checking image provenance and driver compatibility.
  4. Does the minimal container work while the model fails? Inspect model logs, framework compatibility, device assignment and memory demand. A model-specific failure does not by itself establish a daemon configuration fault.

Capture the exact error and timestamp, daemon identity, GPU model and driver version. Record Docker Engine, toolkit, image digest and framework versions as they become relevant. Review logs for sensitive information before sharing them.

2. Check the selected daemon and Compose reservation

Determine how the selected daemon was installed and managed, who owns its configuration, and whether another tool manages that configuration. A packaged engine, a system service and a user service may have different management arrangements. Do not apply a rootful or rootless recipe until you know which engine will receive the change.

For the specific toolkit and Docker releases, verify prerequisites, configuration paths, commands and restart steps in the corresponding versioned vendor documentation. The supplied evidence does not establish a complete setup procedure.

Before an authorized change:

  • Save the existing configuration through the normal change process and establish how to restore it.
  • Identify affected containers and dependent services; arrange a maintenance window if a daemon restart is required.
  • Record the initial daemon and runtime state, then verify the same daemon after the change.
  • Avoid uninstalling Docker or unloading GPU kernel modules as a shortcut. Establish the cause and impact on existing workloads first.

For Compose, inspect the service’s device reservation as well as daemon readiness. Docker’s Compose GPU guidance requires capabilities and says count and device_ids are mutually exclusive. It also documents errors for an invalid device ID or a count exceeding the available devices. If neither a count nor device IDs are specified, review the documented default: it may select all available GPUs. These checks do not replace host and daemon diagnosis.

3. Review inference API exposure

Before starting an Ollama, vLLM or other inference service, identify its listening address, published ports, network mode, authentication and intended clients. Pin a documented image and configuration for the selected release, and verify model and GPU compatibility in that release’s documentation.

Docker’s port publishing documentation says that publishing without a host address maps the port on any host address. A loopback binding is a useful starting point for host-local development, but inspect the effective network configuration as well. Docker warns that, on releases older than 28.0.0, hosts on the same Layer 2 segment can reach localhost-published ports. Direct routing can also affect reachability.

After deployment, verify access from the host and from an appropriately authorized external vantage point, including IPv6 where applicable. For remote access, use an authenticated access layer and review firewall and routing rules. A port binding does not itself provide authentication.

4. Diagnose shared memory and IPC separately

If workload logs point to shared-memory trouble, inspect the effective IPC mode, /dev/shm capacity and usage, process layout and available host memory. Do not infer an IPC requirement solely from GPU use.

Docker’s container run reference distinguishes private, shareable, container-joined and host IPC modes. When no mode is specified, the daemon default can be private or shareable depending on its version and configuration. Host IPC joins the host’s IPC namespace and changes the isolation boundary. The reference describes --shm-size as the size of /dev/shm; assess its effect in the actual IPC and mount configuration.

The supplied vLLM GPU documentation is labeled latest developer preview. It notes that PyTorch uses shared memory between processes, particularly for tensor parallel inference. That guidance does not establish that every release or single-GPU workload requires host IPC. Consult the pinned stable release, select an appropriate isolation mode and measure capacity before changing it.

5. Investigate termination and NVML symptoms

For a terminated container, preserve its reported status, container state and logs around the timestamp. Correlate those with relevant host and cgroup memory events and service-manager activity. Distinguish evidence of host or cgroup memory pressure from a reported GPU allocation failure before choosing a remedy; a status alone is not a diagnosis.

For an NVML driver/library mismatch message, record the loaded driver and installed package versions and investigate their consistency. Avoid unloading modules used by live workloads.

A different symptom, Failed to initialize NVML: Unknown Error after a container update, has a documented possible mechanism. NVIDIA’s Container Toolkit troubleshooting page explains that updates can remove GPU access injected by its runtime hook, and says the affected container must be deleted and restarted to regain access in that case. Confirm that the mechanism fits the observed sequence and assess workload impact before recreating anything.

If another process occupies GPU memory, establish its owner and purpose and coordinate with its operator before taking action.

6. Validate each boundary

For the maintenance record, capture the before-and-after daemon identity and versions, the approved change, the minimal container’s GPU visibility and the model startup result. Separately check intended network reachability and effective IPC configuration.

A GPU visible in a diagnostic container establishes visibility for that test at that time. It does not establish model fit, inference correctness or multi-GPU reliability. If a check fails, preserve the output and return to the corresponding layer in the decision tree. Stop when the evidence no longer supports the next change.

Primary sources

Validation boundary: This revision relies on the supplied documentation excerpts; no command, container, GPU workload, network test or model was executed for it. Source-review notice: November 13, 2026. Live documentation and the supplied excerpts do not establish what those pages contained on an earlier date.

BC

Written by B. Cho

Systems & AI Infrastructure Engineer

Specializing in GPU inference engines (vLLM, SGLang, Ollama), CUDA memory allocation, and Model Context Protocol (MCP) infrastructure on Linux workstations.