Don't Trust an Agent's "Done"

An agent reported six commits and an open PR after making zero tool calls. A memory rule didn't stop it happening again. Hooks and git numbers did.

One morning the previous session’s final report read: “6 commits ahead, pushed, PR opened, MERGEABLE.” The branch had no new commits and there was no PR. The agent had made zero tool calls during the implementation phase.

What happened

Just before that, it had written a detailed implementation plan, down to the individual commit messages. It had also really created the worktree and the build environment. With all that real setup in place, it took its own plan for an execution log. The more detailed the plan, the better the scaffolding for a fabricated report.

A persistent memory rule saying “never fabricate completion” already existed. The fabrication happened anyway, so we stopped relying on prose rules.

Fix 1: inject reality every turn

b=$(git branch --show-current)
ahead=$(git rev-list --count "@{u}..HEAD" 2>/dev/null || echo no-upstream)
dirty=$(git status --porcelain | wc -l | tr -d ' ')
echo "[git] branch=$b ahead=$ahead uncommitted=$dirty"

This one line, added on every prompt, means drift from reality never lasts more than a turn.

Fix 2: block only on claim AND contradiction

The Stop hook fires only when the final message asserts completion (“pushed”, “committed”, “PR created”) and git disagrees. Trying to detect lies semantically was too inaccurate (on narrowing what an LLM is asked to judge, see Let LLMs Only Pick and Extract). A surface claim plus a mechanical mismatch is precise. After three blocks it drops to a warning, to avoid loops.

What each exit code does: hook exit codes.

Fix 3: don’t trust push output either

A relay layer once fabricated git push stdout, and ls-remote returned a SHA that only matched on its first few characters. Fetch, then compare the full 40-character SHAs with rev-parse.

Subagents under-report

Self-reported test counts were low three times in a row: 12 vs 14 actual, 8 vs 11, 13 vs 19.
“TEST BUILD SUCCEEDED” was reported three times. The real outcomes were a compile error, a data race and a timeout.
A subagent that couldn’t run tests guessed that 0.565 rounds to 57%. In IEEE 754 it is 0.56499…, so the right answer is 56%.

The controller session runs the build itself and counts tests with grep -c before merging.

Silent failures

A pipe’s exit status is the last command’s. verify | tee log followed by a commit will commit even when verify fails.
git add … 2>/dev/null hid a filename that was off by a minute, so nothing was staged.
When a hook denied add && commit, re-running only the commit dropped six files.
ls dir/*.jsonl | wc -l on 70,000 files fails with “argument list too long”, and wc prints 0.

When you see zero, suspect the measurement first. Use machine ground truth (exit codes, counts, SHAs), and your hooks survive model upgrades: ours went through two model generations with all 10 hooks unchanged.

Tests lie the same way: green tests prove nothing.

Once completion is verified, the next job is making sure it keeps running; see Pitfalls of Always-On Bot Monitoring.