When to Stop AI Code Review

A design plan needed four review passes before it got a GO. How to tell an endless loop from one you must not cut short, plus review CLI traps.

When a second model reviews your code, the hard question is when to stop. Stop too early and holes remain; too late and you get endless “could be stricter” findings. The basic stopping rule is in Choosing a Coding Agent. Here are cases and CLI traps we didn’t cover there. For having another AI rebut a design before any code exists, see Have Another AI Rebut Your Design. CLI behavior was checked with Codex CLI (@openai/codex, codex-cli 0.163.0) on 2026-10-16.

Things that never pass in one round

Plans. One plan went through four passes, v1 to v4.1, absorbing 9 Critical and 12 Important findings in total. Every place that claimed “fully backward compatible” had a slightly different response shape. Check compatibility by diffing real response bodies.

Fail-open code. An implementation meant to keep working when notifications fail piled up P2s for five rounds: per-step outcomes, silent exits under set -e, URLs leaking through argv, a missing --fail-with-body. Walking the reviewer through “secret unset, payload failure, HTTP failure, transport failure” one layer at a time made it converge.

Review the fix too

Fix commits produce their own P2s. “Retry every failed push” queued permanent 4xx errors forever. Run one more round on the fix diff alone.

Don’t rush to fix every finding

A reviewer without conversation history re-raises points you already settled, sometimes as Critical. Check the agreement document and reject with the section number. Before calling existing code a defect, read the decision log and commit bodies. In one consultation, all three findings re-opened settled decisions; we had paid another model to confirm them. Ask only whether a premise changed.

Isolated reviewers for heavy conclusions

For irreversible conclusions, assign several reviewers per claim, each with a different lens and no view of the others. In one analysis, one of nine claims survived.

An adversarial review also caught 33 wrong deletions.

CLI traps

• stdin hang: codex exec in the background waits on stdin. Pass - < file or add < /dev/null. codex exec --help (v0.163.0) says it reads stdin when no prompt (or -) is given.
• Exit 0 doesn’t mean the body arrived: output sometimes landed minutes to tens of minutes later. In three of four rounds the file was empty at first check; waiting produced 11 P1/P2 findings. Detect completion with a line-anchored marker and wait up to 15–20 minutes.
• Markers match the prompt echo: codex exec - < PROMPT.md echoes the whole prompt first, so “print DONE at the end” matched itself. Use ^DONE$.
• --base and a prompt are exclusive, despite the usage line:

$ codex review --base main "check"
error: the argument '--base <BRANCH>' cannot be used with '[PROMPT]'

We pipe the prompt to codex exec and start it with “review git diff main...HEAD”.

• base resolves to local refs: fetching without advancing the local branch compares against a stale ref.
• Conflict markers in the worktree produce a fake “doesn’t compile” P1.
• Keep output short: six reviews piled up in /tmp reached several MB and slowed the shell. Ask for “under 1000 characters, conclusion first,” check wc -c, read the tail.

How review output differs by tool: Claude Code vs Cursor vs Codex in Practice.

Summary

• Plans and fail-open code need multiple rounds; schedule for it.
• Re-review the fix diff.
• Reject re-raised settled points by section number.
• Close the three CLI traps first: stdin, late bodies, echoed markers.