Issue 00 · Autumn 2026日本語 · RSS
AgentKoji

Doubting the AI's Root Cause

Claude Code's stated root cause is sometimes wrong. Check controls, environment, artifact freshness and the measurement setup, in that order, before you dig.

Agents state causes confidently. When “the cause is X” sounds plausible, you want to go fix X. In our test environment that diagnosis was wrong many times, and the misses follow patterns. Here is the order we doubt things in. For when to stop repeating a fix, see Don’t Make It Fix the Same Thing Thrice; this one is about checking the cause before you touch anything.

Get a control first

For a web app rendering bug we were told, “It reproduces in three environments, so it’s an app bug.” All three were the same person’s same browser: zero controls. A clean browser rendered fine. Ask “did you see one environment where it does not reproduce?” before “how many times did it reproduce?”

Then suspect the environment

Before reading app code, rule out the environment. Examples, product names removed:

Each time the agent suspected app code first, because code is visible and the environment isn’t.

For environment differences when something passes locally but fails in CI, see Passes Locally, Fails in CI.

Suspect artifact freshness

On “it behaves differently than a few days ago,” the agent started reading diffs. There were no code changes; an old build artifact was still in use. When retired logs reappear, check when the running artifact was built. We prefix build/run commands with:

echo "CWD=$(pwd) BRANCH=$(git branch --show-current) HEAD=$(git rev-parse --short HEAD)"

Suspect the measurement

A URL check run 16-way parallel marked many URLs “invalid.” Serially, all were fine. The checker’s DB connection pool was too small and pool timeouts were counted as “invalid.” We reproduced it locally with dummy URLs and dummy work (pool of 4, 0.2 s acquire timeout, all assumed values):

parallel 1:  ok=30 invalid=0
parallel 16: ok=12 invalid=18

Nothing about the targets changed. This is one run and counts depend on timing; look at the shape, not the numbers.

Measurement mismatches get worse as hypotheses pile up. In another case, the agent kept missing on a measurement discrepancy and nearly proposed an irreversible technology choice on the grounds that the result was “theoretically impossible.” Once you see the contradiction, stop adding hypotheses and write a ~30-line probe to measure again.

Suspect clocks and responses

Right after waking from sleep, a machine’s date was seven hours behind and curl returned stale responses: 200 OK, consistent JSON, nothing to notice. The agent used that to assert “no new notifications.” When time or “latest state” matters, re-fetch before trusting it.

Doubt “all”

We once tallied a log export as “17,041 lines, the full set.” It was a window holding only the last 48 minutes; the real population was 250 times larger. An API limit means “the latest N,” not “the unread N.” State files are past declarations, not current observations: trusting one that said “not posted,” we sent the same notification repeatedly to something already posted.

How to ask the AI

The best practices suggest pasting the actual error and asking Claude to address the root cause rather than suppress the error, and giving Claude a way to verify its work: tests, build exit codes, scripts (https://code.claude.com/docs/en/best-practices, checked 2026-10-16). We add the order:

Before naming a cause, check and report in order:
- Did you see at least one environment where it does not reproduce?
- Did you rule out the environment (resident apps, devices, tool updates)?
- When, and from which commit, was the running artifact built?
- Is the measuring side (parallelism, connections, timeouts) the same?
- Is "all" really all, or a window or limit?

If the only check is “tests are green,” the tests may verify nothing: see Green Tests Prove Nothing.

Summary