Signals & anti-gaming

shipmoor test-evidence reports three signals on a change: passed, reduced, and weakened. The first two are deterministic. The third is a judgment call, with a schema built so it can never certify a test as fine, only ever flag it as weakened or say nothing at all. The design behind all three is the same: your own test suite is the oracle for whether your tests passed. Shipmoor harvests what already ran, checks it against the change, and discloses honestly what it found and what it couldn’t check.

passed: harvested, not re-run

By default, passed is harvested, never re-run from scratch. Harvesting looks in this order and stops at the first hit:

1. A harness-hook spool entry bound to the exact working-tree hash. The strongest signal: because it’s tied to the literal tree state rather than a commit, it catches uncommitted edits too. See the hook subcommand.
2. A CI report bound to the head commit.
3. An on-disk report found at a conventional path (test_evidence.report_globs in config).
4. Honestly disclosed as not_checked if none of the above exist.

Three assurance levels

A harvested result carries one of three assurance levels, so a shaky result is never silently trusted the same as a strong one:

The anti-gaming design

Three things stop a passed result from being trivially gamed:

• Freshness binds to the working-tree hash, not the commit. A single uncommitted edit makes a report instantly stale, which is what catches an agent that runs the tests and then keeps editing.
• The exit code alone is never trusted. The classic pytest || true trick, forcing a zero exit regardless of what actually happened, is caught because a passed result also requires the recorded output to parse into a real report with clean failure information. A zero exit code with no parseable report behind it is not evidence.
• The run has to have genuinely exercised the relevant tests. The collected test set must be non-empty, and if the change itself touched test files, at least one of the collected tests must come from one of the changed test files. A suite that quietly skips the new or changed tests doesn’t count as evidence that they passed.

reduced: a test-id set difference

reduced is deterministic: a set difference between the test ids collected at base and the test ids collected at head. A test id that existed at base and is gone at head is a reduction, and it’s flagged:

reduced   ✗ 1 test removed since base (no removal on record)

This catches the single highest-value cheat available to an agent under pressure to turn a red suite green: deleting the failing test instead of fixing the code.

The escape hatch: disclosed removal

Tests get removed for real reasons too: consolidation, a flaky test replaced by a better one, a feature that no longer exists. reduced has a narrow, deliberate escape hatch for that. An entry in .shipmoor/test-removals naming the removed test id and stating a reason clears it, as disclosed and intentional:

tests/test_payments.py::test_refund_retries_on_timeout    flaky against the sandbox gateway; replaced by test_refund_retry_policy

A bare id with no stated reason clears nothing. It still fires:

reduced   ✓ no tests removed

The bar is disclosure, not just awareness that the test is gone.

weakened: judgment, subtractive only

Deleting a test is one cheat. Keeping the test but gutting it is another, quieter one: an assertion changed to something that can never fail (assert True standing in for a real check), an assertion deleted outright, an early return inserted before the real checks run, the test body emptied out, or an expected-failure or exception check loosened until it no longer constrains anything.

No count-based check catches any of this. The test id is still there, the collection is still non-empty, the exit code is still zero. It’s a semantic judgment about what the test’s content actually does now, compared to what it did before, and there’s no deterministic oracle for that question.

The same judge machinery as Claim Check

weakened doesn’t invent new machinery for this. It reuses the same judge machinery as Claim Check’s intent fidelity judge: a BYO agent transport, automatic secret-masking of anything sent to that agent, a tolerant response parser, per-sample retry, and the same K-sampling and agreement calibration approach. Like the intent fidelity judge, each check is sampled K times (default 5), and only agreement at or above the same floor (4-of-5 by default) is enough to flag weakened.

”Pass” is not in the schema

The judge’s own output schema makes a pass literally unrepresentable: there is no pass field, no ok field, nowhere in the shape. The signal can only ever be in one of three states:

There is no fourth state that certifies a pass. Behind that last line sits the fuller disposition the judge produced:

  rationale: the assertion on the refunded amount was replaced with `assert True`; the
             test can no longer fail on a wrong refund total.
  resolving_pointer: tests/test_refund.py:42 (resolved)
  judge: claude · samples: 4/5 agreed

Notice what never appears next to weakened, in any of the three states: a checkmark. Silence (not judged or nothing flagged) means the judge didn’t find weakening to report, or wasn’t asked to look. It never means “this test is fine,” and it is never a certification of anything.

Biased toward silence, on purpose

The judge’s prompt carries a deliberate, hard bias against flagging. An honest refactor, renaming a test or restructuring its body while keeping the real assertions intact, is explicitly, deliberately not treated as weakening. The judge is tuned to avoid crying wolf on legitimate cleanup, because a signal that fires on ordinary refactoring just trains people to ignore it.

The pointer-must-resolve rule

A static gate sits over the whole signal: any claimed weakening must resolve to a real, specific file:line location in the diff, or it’s discarded rather than trusted. A judge that names a location that doesn’t exist doesn’t get to flag anything.

samples, honestly disclosed

The samples field, how many of the K samples agreed, is absent or null when no --agent was configured, or when nothing under a recognized test path changed. It is never silently defaulted to a fabricated placeholder. No judge run means the field says so by being empty, not by pretending a run happened.

Measured findings

Measured in our own evaluation harness, not a production or customer result.

Zero false positives and zero false negatives across those five scenarios.

Harvesting isn’t free: that fixed cost means the speed advantage is a curve, not a constant. It widens as the suite gets bigger and is marginal for a suite that was already fast.

Honest limitation. The weakened numbers above measure the mechanism: that a supplied verdict is routed to the right place, correctly, every time. They do not measure a live model’s own semantic judgment quality on real, never-before-seen code. A live-agent accuracy study, how often a real agent looking at a real unseen diff correctly calls weakening, is a stated follow-up we have not done yet. What we measured is that the plumbing works, not that the model’s judgment is good.

Next

• CLI reference: the --agent, --high-assurance, and config surface these signals sit behind.
• Using Test Evidence with Claim Check: how passed, reduced, and weakened each land in the gate.
• Test Evidence overview: the shape of the problem this solves.
• Quickstart: wire the hook before you need any of this.
• BYO-Judge: the K-sampling, agreement floor, and masking boundary weakened shares with Claim Check’s own judge.