AI Coding Agent Says Tests Passed? What MCP Logs Actually Prove (2026)
The tests passed. But which inputs did they test?
An agent’s statement that tests passed is a report, not a complete validation record. A tool response can provide stronger evidence, but only if it identifies what ran, what the underlying process returned, and which code and configuration it evaluated. Even a genuine passing run can become stale before submission.
An earlier version of this article described a Codex session, passing tests, post-test changes, checker results, and an independent rerun. Those details were an author attestation; no independently verifiable, versioned execution receipt was supplied to substantiate them. They should not be treated as established experimental results. The scenarios below are hypothetical failure modes, offered to help teams design and review their own evidence.
Read the tool result at the right layer
The MCP Tools specification for protocol revision 2026-07-28 describes tools/call requests and shows a tool result with an isError field. That field belongs to the tool response. It is not, by itself, the operating-system exit code of a subprocess that the tool may have launched.
A test tool therefore needs an explicit result contract. Review its implementation and versioned documentation to determine whether it records the process exit code, how it represents interrupted or timed-out runs, and whether its summary comes from the actual runner output. A completed MCP response alone does not answer those questions.
Use this decision path:
- Is there an inspectable tool result tied to the claimed invocation? If not, the claim is unverified. A missing record can also mean the observer did not cover that execution path; it does not prove the command never ran.
- Does the result identify the underlying process outcome? If the exit code, termination state, or output is missing or ambiguous, investigate or rerun under an instrumented runner.
- Does the run satisfy the validation policy? Check the selected tests, collection outcome, failures, and any required coverage or integration checks. A successful process status alone does not establish that the intended tests ran.
- Does the evidence apply to the state being accepted? Compare the tested inputs and environment with the submitted state. If a relevant input changed, validate the new state.
Treat a zero-test collection as a failure to meet a policy that requires tests to run. Interpret the runner’s exact exit codes using documentation for the pinned release; the supplied evidence does not establish a particular pytest exit-code meaning.
Bind a result to the tested state
A useful receipt should let a reviewer reconstruct both the invocation and its scope. Record, at minimum:
- The exact command and arguments, working directory, runner and tool versions, timestamps, termination state, exit code, and retained output.
- The test selection and collection summary, including how many tests were collected, passed, failed, or skipped where the runner reports those values.
- The repository revision and whether the working tree contained uncommitted or untracked inputs. A commit identifier alone does not describe a dirty workspace.
- The relevant configuration and dependencies: test files, discovery settings, scripts, lockfiles, build inputs, generated artifacts, and environment settings that can affect the run.
- The identity and provenance of the runner and receipt, plus the final revision or artifact the result is intended to validate.
Avoid recording secrets in receipts. Record the identity or version of a secret-backed dependency where needed, without exposing its value.
A content manifest can help detect changes that timestamps miss. Define its boundary before running: include relevant paths and contents, represent additions and deletions, and specify exclusions. Capture the manifest before and after the run, then compare it with the state to be accepted. If inputs changed during the run, or the final manifest differs, treat the result as stale until the effect is resolved and validation is repeated.
For example, suppose a manifest covers only Python source files. A later edit to a test-discovery configuration file could alter which tests run while leaving that manifest unchanged. Expanding the boundary to include that configuration addresses this particular blind spot; it does not automatically capture remote services, generated inputs, runtime state, or every environmental dependency.
Make the acceptance decision explicit
Use a policy that distinguishes pass, stale, fail, and unknown:
- Pass: Required checks ran successfully against the identified final inputs, and the receipt satisfies the team’s evidence requirements.
- Stale: A relevant input changed after the run, or changed during it without an adequate explanation. Revalidate the final state.
- Fail: The required validation ran and reported a failure, or did not meet an explicit requirement such as a minimum intended test selection.
- Unknown: Evidence is absent, incomplete, contradictory, or outside the observer’s coverage. Obtain sufficient evidence before relying on the claim.
An independent CI run can add a distinct trust boundary if it checks out the intended revision, uses controlled dependencies, retains its logs, and reports the actual required checks. Independence is a property to inspect, not a label: identify who can alter the workflow, runner, artifacts, and acceptance policy. A manifest or separate runner does not make evidence tamper-proof or prove that tests cover the requirements.
Check GitHub’s actual merge gate
For GitHub repositories, inspect the check associated with the commit GitHub requires, not merely an earlier green run. GitHub’s required-status-check troubleshooting guide says required checks must pass on the latest commit SHA; where a test merge commit has a status, its status is the one to inspect.
The same guide documents several operational traps. A workflow skipped by path or branch filtering can leave an associated required check pending, while a job skipped by a conditional can report success. A green skipped job is therefore not evidence that its test steps executed. Check the triggering event too: GitHub lists eligible events for workflow-job checks evaluated against a pull request, and a merge queue requires the separate merge_group event for required GitHub Actions checks.
Before changing a gate, inspect the repository’s branch protection or ruleset, the expected check source, workflow triggers, and job conditions together. If a required check is absent or unexpectedly green, examine the corresponding run and job steps rather than inferring execution from the status label.
Version and review boundaries
The supplied official MCP Python SDK documentation identifies itself as documentation for the v2 line and states a Python 3.11+ requirement. It also directs readers migrating from v1 to the migration guide. Do not assume an example or API from one SDK generation applies to another: pin the actual package and Python versions, then verify signatures, transport behavior, and result handling against documentation and source for that release.
The protocol excerpt is specifically for revision 2026-07-28. Confirm the revision and capabilities negotiated by the client and server in your own deployment before applying its requirements. For pytest or any other runner, verify the pinned release’s collection rules, exit codes, and configuration precedence from its official versioned documentation before implementing an automated decision.
Primary sources
- MCP Tools specification, revision 2026-07-28 — tool discovery, invocation, and response examples.
- GitHub required-status-check troubleshooting — required SHA, skipped checks, event eligibility, and merge queues.
- Official MCP Python SDK v2 documentation — SDK generation and Python requirement.
Validation boundary: This article provides a review framework, not evidence that any particular command, test, agent session, or CI configuration ran successfully. The supplied source excerpts support the cited claims as observed when retrieved; they do not establish what those live pages contained on the original publication date. Source review reflected in this correction: November 13, 2026.