Overview

shipmoor test-evidence checks a claim your coding agent makes constantly: that it ran the tests and they passed. It harvests the actual test run, checks whether the test surface quietly shrank, and checks whether a kept test was quietly gutted, then reports three signals: passed, reduced, and weakened.

It is a standalone command, a peer to scan, review, and claim-check in the same shipmoor binary. It’s also the evidence layer Claim Check composes in automatically for any entitled caller, so the two are meant to be used together even though neither strictly requires the other. Test Evidence needs its own test_evidence entitlement, in the Shipmoor IC plan alongside claim_check; see Plans & tiers. Its lifecycle hook has no such gate: shipmoor test-evidence hook always runs, for everyone.

The problem it solves

“I ran the tests and they pass” is one of the most common sentences a coding agent produces, and a careless human isn’t above it either. It’s also one of the easiest claims to fake. An agent under pressure to turn a red suite green has several quiet options: delete the test that was failing, hollow out its assertion until it can never fail again (an assert True standing in for a real check), or route the run through something like pytest || true so a red exit code never surfaces. None of that necessarily shows up in a diff review, especially if no one remembers exactly what the test used to check.

Test Evidence exists because a claim like “tests passed” deserves the same scrutiny as any other claim about a change. Instead of trusting a bare result, it asks three narrower questions:

• Did the tests actually run and pass, grounded in real, harvested output rather than a bare exit code?
• Was the test surface quietly shrunk?
• Was a kept test gutted so it can never fail again?

The first two have deterministic, provable answers. The third does not, and it’s built differently because of that.

Three signals

Each question above maps to one signal, and none of them collapse into a single score.

passed: grounded, not assumed

Deterministic, with real counts. Rather than re-running your suite from scratch by default, Test Evidence harvests a result that already exists, preferring, in order: a hook-recorded spool entry bound to the exact working-tree state, a CI report bound to the head commit, and then an on-disk report. If nothing fresh matches any of those, it says so plainly: not checked, never a guess dressed up as a pass.

reduced: did the test surface shrink

Deterministic. Flags a test that silently disappeared between base and head, the single highest-value cheat there is, since deleting the failing test is the fastest route to a green suite. A disclosed, reasoned removal, recorded in .shipmoor/test-removals, is the honest way to clear this. A removal with no stated reason clears nothing.

weakened: judgment, and it can only subtract

A test can keep its name, stay in the suite, and still be gutted: an assertion changed to something that can never fail, an assertion deleted outright, an early return added before the real checks, an expected failure quietly loosened. No mechanical rule catches every way to do this, so weakened rides the same judge machinery Claim Check uses.

weakened has no concept of a passing test. It can report weakened, citing what changed and where, or it can stay silent. There is no third state where it certifies a test as fine. Silence means nothing was found to flag, not that the test was checked and approved.

This is a deliberate design choice, not a gap. A judge that could also certify a “pass” would be making a claim about code correctness that no amount of careful prompting actually earns. Withholding that claim entirely, and only ever being able to subtract trust, is what keeps the signal honest.

See Signals & anti-gaming for the full depth on all three: exactly what each one catches, what clears it honestly, and where each still has blind spots.

Binder, not oracle

Your test suite is the oracle for whether your tests passed. Your pytest, your go test, your cargo test, decided that already, correctly or not. Test Evidence doesn’t re-decide it and doesn’t run a second opinion on your code’s correctness. Its job is narrower and mechanical: harvest what actually ran, check it against the diff, and disclose honestly what it found. It tells you whether the tests actually ran and passed, whether the test surface shrank, and whether a kept test was quietly gutted. It reports as a binder disclosing evidence, never as an oracle pronouncing your code correct.

This is why weakened is built the way it is. A judgment call like “was this assertion gutted” has no mechanical answer, so Shipmoor routes it to the same bring-your-own-agent machinery Claim Check uses, and holds it to the same discipline: it can flag a divergence, or it can decline to. It cannot hand back a verdict of correct. That restraint, not a fancier model, is what makes this corner of Shipmoor’s verification surface trustworthy.

The command and the hook

Two mechanisms produce the evidence.

The lifecycle hook, shipmoor test-evidence hook [--dir PATH] [--config PATH], is wired as a Claude Code PostToolUse hook on the Bash tool. Your coding harness pipes each tool-result event to its stdin, and it recognizes a real test run and records it to a local evidence spool (.shipmoor/test-evidence/spool/). It is never entitlement-gated; it always runs, for everyone. A non-test command, or running outside a git repository, is a silent no-op, and the hook always exits 0: a lifecycle hook must never interrupt an agent’s turn.

The standalone command, shipmoor test-evidence [target] [flags], is a peer to scan, review, and claim-check, not a flag on any of them. It defaults to your working tree, the same as the rest of the CLI, and narrows with --staged, --diff <spec>, --all, or a committed --from/--to range. --agent <preset-or-cmd> is the one flag that matters for weakened; passed and reduced are fully deterministic and never touch a model. --attest-out <path> writes the in-toto TestEvidence attestation, the exact artifact Claim Check consumes. --json, --terse, and --no-color cover machine output and busy CI logs. See the CLI reference for the complete flag set.

--high-assurance discloses honestly; it doesn’t verify yet. The flag is meant to spot-verify a harvested result by re-running a subset of it. The subset re-runner isn’t built yet, so today --high-assurance reports an unverified harvest as not checked rather than trusting it blindly. That’s a deliberate choice: disclose the gap, don’t fake the check.

Relationship to Claim Check

Test Evidence is a fully separate, standalone command; it isn’t a mode of shipmoor claim-check. You don’t need to run it yourself first. For any caller entitled to both, Claim Check composes its signals in automatically. You can still run it on its own, for its own report, and either way the strongest passed signal depends on the same thing: the PostToolUse hook wired into your coding agent. See Using Test Evidence with Claim Check for how the composition works and what it changes in the verdict.

Measured in our own evaluation harness

These numbers come from Shipmoor’s own evaluation harness, not a customer deployment. Read them as a measurement of the mechanism, not a field result.

Across five scenarios built to probe the three signals, we measured zero false positives and zero false negatives:

Harvesting also measured faster than re-running: a 100% harvest hit rate on the in-loop scenario, and roughly 3.6x faster than a naive re-run on a representative suite (about 180ms harvested versus about 670ms re-run). Harvesting has its own fixed cost, roughly 150 to 200ms from a couple of git subprocess calls, so this advantage widens as a suite grows and is marginal on a very fast one.

What these numbers do not tell you. The weakened results above measure the mechanism correctly routing a supplied verdict into the right signal state, not a live model’s own judgment quality against real, novel code. That’s a meaningfully different claim from “the AI reliably catches gutted tests,” which this evaluation doesn’t support. A live-agent accuracy study, testing the judge’s actual semantic judgment rather than the plumbing around it, is a stated follow-up, not yet done.

Next

• Quickstart: wire the hook and read your first report in about five minutes.
• CLI reference: every flag, selector, and exit code.
• Signals & anti-gaming: the full depth on passed, reduced, and weakened, and how each resists specific ways of gaming a suite green.
• Using Test Evidence with Claim Check: how the signals compose into a claim-check run.
• Claim Check: the verdict layer this evidence feeds.