Docker Exit Status 137 and OOMKilled: Evidence-Based Diagnosis on Linux
PyTorch & CUDA(Updated )

Docker Exit Status 137 and OOMKilled: Evidence-Based Diagnosis on Linux

An exit status of 137 is a reason to collect evidence, not a diagnosis of an out-of-memory event. The Docker container-list reference describes 137 in relation to SIGKILL and lists multiple causes; it does not establish that a particular container was killed by the kernel OOM killer. Preserve records from the affected container and its host before restarting or changing limits. The goal is to identify what happened to that execution, which resource boundary was involved, and whether a proposed change addresses it.

Docker documents the interpretation of 137 in its container-list reference, but that does not independently establish the OOMKilled field semantics for every Engine version, nor identify the cause of an individual incident. Treat both values as observations to investigate. Before interpreting them, consult authoritative documentation for the deployed shell or runtime and Docker Engine release, then corroborate the interpretation with incident-time evidence.

1. Preserve the recorded state

Docker’s inspect reference describes low-level object information, JSON output, formatting, and a type filter for disambiguating objects with the same name. Use your established read-only procedure to capture the affected container’s state by ID. The excerpt does not define the state fields, so verify their meanings against documentation or source code for the installed Engine release.

Record the container ID, image digest, deployment revision, node, relevant timestamps, exit status and available state fields. Also retain application logs, runtime events, orchestration history and host monitoring for a bounded window around the incident. Note clock offsets, collection gaps and retention limits.

Observation Investigation
Exit status is 137 Verify the status convention for the component reporting it; seek evidence of the termination mechanism.
OOMKilled is present Preserve its value and verify the field’s semantics for the deployed Engine release before using it as evidence.
State or field is unavailable Record the gap and look for retained runtime events and host records for the same instance.
A later state shows a different status Determine whether it belongs to the incident execution or a subsequent restart.

A current inspection is a snapshot, not necessarily a complete history. Correlate it with instance-specific events and timestamps before attributing a later state to the failure.

2. Decide which evidence to pursue

Use this sequence:

  1. Can you identify the execution? Match the alert to the container ID, node and restart history. If the records do not establish the match, keep the attribution tentative.
  2. Is there evidence of memory pressure or an OOM event? Correlate host and runtime records with the incident time and affected process. Docker explains that, under Linux system memory exhaustion, the kernel can kill processes to recover memory, including important host services. See Docker’s resource constraints guidance.
  3. Which boundary was involved? Compare evidence for an individual container constraint with evidence of pressure affecting the host or other workloads. Do not select a scope from an exit status alone.
  4. Is another termination mechanism supported? Check deployment, timeout, scheduler and operator records alongside application logs. Verify the reporting component’s status convention before deciding what the number implies.
  5. Are records missing or contradictory? Preserve them and report the cause as unconfirmed. A missing kernel message may reflect inaccessible or expired logs.

This distinction matters operationally: raising one container’s limit may be inappropriate if the host itself lacks capacity, and a later clean run cannot reconstruct missing incident evidence.

3. Check cgroup evidence in context

The kernel cgroup v2 documentation describes a hierarchy of cgroups, process migration and selectively enabled controllers. Its supplied excerpt is from development documentation and does not define the memory files needed to interpret an incident. Consult the memory-controller documentation for the running kernel release before relying on file names, values or counter behavior.

Use this non-executable checklist:

  • Identify the process and confirm that its identity and start time match the intended execution. A stopped container may have no live process or surviving cgroup to inspect.
  • Determine whether the target uses cgroup v1, v2 or a hybrid arrangement. Do not apply a v2 procedure unchanged to v1.
  • Identify the cgroup mount’s filesystem type, mount point and mount root, along with the mount and cgroup namespaces visible to the inspector. Resolve membership against the actual mount root rather than blindly appending a path to /sys/fs/cgroup.
  • Verify that the resolved directory belongs to the intended process and hierarchy. Namespace isolation, a bind mount rooted below the hierarchy, a different host or insufficient permissions can make a plausible path misleading or inaccessible.
  • Check the pinned kernel documentation and mount options for each available memory file, its scope and its reset behavior. A missing file is a visibility or capability question, not a zero.
  • Compare snapshots only when they refer to the same cgroup instance and documented scope. Recreation, migration or descendant activity can complicate comparisons.

Current cgroup data describes the observed environment; it is not automatically evidence of an earlier exit. If mapping cannot be verified, rely on retained host snapshots and logs, and state the limitation. Starting a stopped container to inspect it creates a new execution to investigate.

4. Match the remedy to the finding

A container memory constraint is supported by evidence. Compare the effective limit with incident-time demand and available host capacity. Docker documents --memory as a maximum memory setting and notes that resource controls depend on kernel support. Verify the installed Engine and kernel behavior and the deployment’s effective configuration before proposing a change. Account for sibling workloads when evaluating capacity.

Swap is relevant. Docker documents --memory-swap as meaningful with --memory; when set to a positive value, it represents the combined memory-and-swap allowance. Frequent swapping carries a performance penalty. Verify the pinned release and host capability, then evaluate latency and host pressure as well as whether the workload survives.

Host-wide pressure is supported. Examine aggregate demand, placement and concurrent peaks. Reducing concurrency or moving work may address the constraint more directly than increasing one limit. Docker cautions against circumventing OOM safeguards with extreme priority settings or indiscriminately disabling OOM killing.

The evidence points to an application, GPU or shared-memory error. Investigate that error and resource independently. Such an error alone does not establish an OOM kill. Record the runtime version and workload phase before selecting a remedy; the supplied sources do not establish a PyTorch-specific fix.

Make one scoped change through the normal deployment process. Record the expected effect, capacity assumptions and rollback criteria.

5. Validate the change

For an approved test, retain the image and workload characteristics where feasible and document differences from the incident. Compare exit state, effective limits, documented counters, application errors, latency, host pressure and effects on sibling services. One successful run is evidence about that run; it does not establish that the original cause is resolved under every load.

Primary sources

Validation boundary: No commands, scripts, container tests, GPU runs or benchmarks were executed for this correction; no versioned execution receipt was supplied. The provided excerpts do not establish exit-status encoding, explicit exit 137, Docker’s OOMKilled semantics, a cgroup lookup command or memory-counter definitions. Verify those against authoritative sources and the pinned environment before using them operationally. Source review notice: November 13, 2026. The supplied current excerpts do not establish what the sources said on a historical date.

BC

Written by B. Cho

Systems & AI Infrastructure Engineer

Specializing in GPU inference engines (vLLM, SGLang, Ollama), CUDA memory allocation, and Model Context Protocol (MCP) infrastructure on Linux workstations.