How to Fix Subprocess Zombie Leaks and File Descriptor Exhaustion in AI Agents on Linux
MCP & Agents(Updated )

How to Fix Subprocess Zombie Leaks and File Descriptor Exhaustion in AI Agents on Linux

An AI coding agent that invokes subprocesses repeatedly may eventually encounter defunct child processes, too many open files, or commands that stop producing output. Those symptoms do not share one universal cause. An exited process that has not been waited for is a zombie. A still-running process whose parent exits becomes an orphan, which may be adopted by a process responsible for reaping it. An open pipe descriptor is yet another kind of resource.

Corrected November 12, 2026: An earlier version offered a drop-in global SIGCHLD handler calling waitpid(-1, WNOHANG), an arbitrary-shell runner claiming guaranteed cleanup of all descendants, a fixed pipe capacity, and a made-up normal descriptor count. Those recommendations are unsafe or unsubstantiated. This guide replaces them with a process-ownership diagnosis, a bounded test using only a fixed trusted child, and conditional container/init guidance.

1. Identify the failure before changing a supervisor

Symptom What it indicates What it does not prove
A child appears with process state Z or <defunct> It exited but its termination status has not yet been collected All orphan processes are zombies, or a PID limit was reached
A child or grandchild continues running after a tool timeout Lifecycle and supervision need inspection A global SIGCHLD handler can safely kill or reap all descendants
OSError: [Errno 24] Too many open files The affected process encountered a file-descriptor limit Every open descriptor is a leaked pipe
A child is blocked after writing output to a pipe The parent may not be consuming a pipe promptly Every Linux system has one fixed 64 KB pipe capacity
A container fails to stop promptly PID 1 behavior, forwarded signals and child cleanup may matter All Python processes acting as PID 1 are automatically broken

The official Linux wait(2) manual defines how a parent obtains a child process’s termination status, and why an uncollected terminated child can remain a zombie. The same page describes WNOHANG; it changes whether waiting blocks, not who owns the child or whether other libraries are also waiting for it.

2. Read-only diagnostics with a verified process ID

Inspect the actual process tree and whether zombie entries recur without killing anything:

# Show parent IDs and process states; Z indicates an exited zombie.
ps -eo pid,ppid,stat,comm | awk 'NR == 1 || $3 ~ /^Z/'

Next, identify the intended agent PID from your own process supervisor or service manager. Do not automatically run broad pgrep -f and assume it returned exactly one safe target. For a PID you have verified to belong to the application:

# Replace this example with the ONE PID confirmed by your supervisor.
TARGET_PID=12345
case "$TARGET_PID" in
  ''|*[!0-9]*) echo 'Invalid PID' >&2; exit 2 ;;
esac
ps -p "$TARGET_PID" -o pid,ppid,stat,comm
# Count names in this PID's /proc/fd directory (Linux only).
# Races are possible if the process exits or descriptors change.
python3 - "$TARGET_PID" <<'PYFD'
import os
import sys
pid = sys.argv[1]
try:
    count = len(os.listdir(f"/proc/{pid}/fd"))
except OSError as exc:
    sys.exit(f"FD inspection failed ({type(exc).__name__}); do not report zero")
print("Open FD snapshot:", count)
PYFD

These commands inspect metadata, not the data held in open files. If /proc/PID/fd is inaccessible, report a permissions or process-lifetime limitation rather than claiming zero open descriptors. Observe repeated comparable snapshots at the same job state and compare after the workload completes.

There is no universal normal range such as 25–45 file descriptors for ten tasks. An LLM agent with open sockets, databases, log files and tool streams may legitimately hold more. Only a reproducible growth trend, correlated with incomplete child cleanup or pipes, supports a leak hypothesis.

3. Let one owner collect each child; avoid global waitpid(-1)

The Python asyncio subprocess API provides create_subprocess_exec(), Process.wait() and Process.communicate(). When a framework already owns a child process through one of these interfaces, do not install a second global SIGCHLD handler that calls waitpid(-1): it can collect termination statuses that the framework expects to receive itself.

close_fds=True controls inherited file descriptors at child creation; it is not a guarantee that parent-side stdout and stderr pipes will be drained or closed. The official asyncio subprocess documentation warns about a deadlock when using Process.wait() while the child has piped output and its pipe fills. A communicate() call is one documented remedy for a bounded known output, not a license to buffer unlimited output in memory. Prefer communicate() for known small bounded output, or a carefully designed streaming/logging policy with explicit output limits for arbitrary tools.

A small, fixed-command smoke test, not a generic shell runner or production supervisor:

# Runs only a fixed, trusted Python child printing one short line.
# Do not replace this with untrusted model-generated shell commands.
import asyncio
import sys

async def smoke():
    proc = await asyncio.create_subprocess_exec(
        sys.executable, "-c", "print('child-ok')",
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )
    try:
        output, errors = await asyncio.wait_for(proc.communicate(), timeout=3)
    except (asyncio.TimeoutError, asyncio.CancelledError):
        # Cleanup is scoped to this direct child; NOT its descendants.
        if proc.returncode is None:
            try:
                proc.kill()
            except ProcessLookupError:
                pass
        await proc.wait()
        raise
    assert proc.returncode == 0, (proc.returncode, errors)
    assert output == b"child-ok\n"
    print("fixed direct-child smoke PASS")

asyncio.run(smoke())

This test’s child produces a known tiny response. It is not a secure wrapper for unbounded output, user-supplied commands, LLM-generated scripts, background grandchildren, repeated task cancellation, or a hostile process. If a production agent invokes arbitrary commands, its command approval, process-group ownership, output size, cancellation and timeout behavior need a separate reviewed implementation and multi-process tests.

4. Why a subreaper is not a one-line repair

On Linux, PR_SET_CHILD_SUBREAPER can mark a supervisor so that some orphaned descendants are reparented to it. The Linux manual for PR_SET_CHILD_SUBREAPER explains that the subreaper then has the responsibility to use waiting APIs on adopted, terminated children.

That is not permission to reap every PID returned by waitpid(-1) in a Python application whose asyncio, subprocess, worker pools, or third-party agents already own child processes. A blanket SIGCHLD handler may race and steal statuses from their normal owners. Adding a subreaper to an existing service is a supervisor architecture change requiring child ownership design and testing, not a startup snippet.

For a known, owned direct subprocess, wait or communicate through the same API that launched it. If the tool starts unrelated grandchildren, do not assume that killing one process group always reaches children that have changed sessions, groups or namespaces. A robust multi-process supervisor requires its own documented containment strategy.

5. Containers: use an init where warranted, not blanket limits

Docker’s multiple-processes guidance describes --init as an option to run a small init process that forwards signals and reaps child processes. The docker run reference documents the flag’s CLI semantics. The Docker Compose service specification separately documents the init: true setting for a Compose service.

For a deployment already approved and configured for Compose, consider the following configuration fragment:

services:
  agent-runner:
    init: true

This is not a runnable standalone Compose file, a guarantee that every child is cleaned up, or an instruction to restart a shared container. Investigate the actual PID 1 process, existing entrypoint, image initialization, orchestration and already-running jobs before changing it.

Do not arbitrarily set extremely high nofile or nproc values to make leaking children less visible. Do not install a second init on top of one already handling that duty without checking how signals and process ownership work.

6. A reproducible verification plan

Test the actual agent on a disposable, permissioned environment. Use a small fixed number of known subprocess requests, then compare process state and descriptor snapshots at equivalent points:

  1. Record the agent PID, process ancestry, deployment version and current subprocess ownership mechanism.
  2. Run a small fixed set of trusted commands and verify each direct child’s exit status is collected.
  3. Observe processes and file descriptors before, during and after requests. Do not infer a leak from one mid-request peak.
  4. Separately exercise authorized timeout, cancellation and bounded-output scenarios. A passing trivial child does not prove descendant cleanup.
  5. Only when a specific failure reproduces should you change the child’s lifecycle design, init policy, or output handling; repeat the exact failed scenario afterward.

Verification boundary: This article was source-reviewed and does not claim that the site reproduced a production PID leak, tested all forms of cancellation, or proved a full descendant-reaping implementation. A safe sample handling one direct child does not establish a safe general-purpose AI agent executor.

Primary references

Independent source correction on November 12, 2026; original publication date and hero asset preserved.

BC

Written by B. Cho

Systems & AI Infrastructure Engineer

Specializing in GPU inference engines (vLLM, SGLang, Ollama), CUDA memory allocation, and Model Context Protocol (MCP) infrastructure on Linux workstations.