Green Tests Prove Nothing

We could disable any of eight guards and every test stayed green. How to catch decorative tests, the kind that multiply once agents write your tests.

We had an agent add eight fail-loud guards, with tests. Everything was green. Then we commented the guards out one at a time. Whichever guard we disabled, every test stayed green.

The same habit for completion reports: don’t trust “done”.

Rule: if breaking it doesn’t fail, it isn’t a test

Break the target on purpose, confirm the test fails, then restore it. In one session we found five patterns:

1. A self-referential threshold: “test at half the constant” tests nothing once the constant is 0.
2. A traversal order that wasn’t guaranteed and only happened to be stable.
3. Isolation of one module accidentally hiding a bug in another.
4. Coarse fingerprints: mtime in seconds plus size misses same-second, same-size overwrites.
5. test || exit 1 exiting early, so later contamination checks never ran.

Mutation testing can lie too

A same-length constant mutation (16384 → 24576) that fits inside one second reuses stale .pyc bytecode. Set PYTHONDONTWRITEBYTECODE=1 and shift the mtime.

Doubting the measuring side first is covered more broadly in Doubting the AI’s Root Cause.

Mocks hide deleted code

A refactor deleted a method but left a call to it. Production crashed with an AttributeError on every shutdown, and nobody noticed for six months, because a bare AsyncMock() simply invented the missing method. Use AsyncMock(spec=RealService). For an order that keeps the deletion itself from breaking things, see Deleting Thousands of Lines in a Refactor.

753/753 passing, 33 wrong deletions

An audit removed 102 “unused” translation keys with all 753 tests passing. An adversarial review found that 33 of them were used through dynamically built key names. Check imports, CSS @import, build config and package scripts, then actually run the tool.

How we run those reviews: AI review rounds.

Tests that lock in bugs

• Seeds with neatly ordered timestamps never reproduced production, where 117 rows shared an identical timestamp.
• A LIKE '%PUBLIC%' permission check was always false. Add a GRANT and confirm the test flips.
• Mocked auth passes with any helper. Write a separate test with a real JWT.
• A fixture with a fixed date is a time bomb.
• When a suite slows from 2.4 s to 62 s, a test may be replacing the very thing it is testing.

Constraints we give test-writing agents

Additions only, never deletions (check wc -l went up). Run numbers before asserting them. Show each new test fails against the old code. Report skipped items as unverified. A small model once decided “one test is enough” and deleted 195 lines of existing tests.

For running this in CI, with its costs and runaway jobs, see Running Claude Code in CI: Costs and Traps.