AI Code Review Is Statistical: Why One Run Is Not Enough

Run an AI code review twice on the same code, and you can get different findings. Even after multiple reviews, the next run can uncover a real issue.

The code hadn't changed. The reviewer found something the earlier runs had missed.

A finished AI code review is one sample

Linters and type checkers have taught us a deterministic mental model: run the check, get the result. Run it again, get the same result.

An AI reviewer doesn't work like that. Finding bugs with it is statistical. A completed review tells us what that run found. There may still be problems another run would catch.

So "the code has been reviewed" means less than it sounds. It means one reviewer looked once.

My own review data shows the same thing from the other side. Across six review runs on one project, three defects got past every reviewer and were found later by GitHub Copilot's pull request review. Two of the files were in a reviewer's work queue and never examined. The third was examined, too shallowly. On one change of 31 files, the two instances of the same reviewer looked at 13 and 11 of them.

I'd accept that variability and build it into the workflow.

Make multiple runs the default

My recommendation is to make multiple runs part of running a code-review agent. You can run them sequentially, or launch separate instances in parallel against the same change.

Parallel runs cost more tokens, but they add review runs without making you wait for each one in sequence.

Give every reviewer a fresh context

Give each reviewer a fresh context, with access to the code and the relevant project instructions. Let it inspect the change without inheriting the coding agent's reasoning or another reviewer's conclusions.

A reviewer that starts from someone else's explanation tends to check that explanation. A reviewer that starts from the code has to build its own picture of what the change does. I wrote more about why the validator should get evidence and not reasoning in If AI Writes and Reviews the Code, What Is the Human Approving?

Separate contexts don't make the reviewers statistically independent. They often run on the same model, with the same training behind it. What the fresh context gives each run is room to investigate without starting from someone else's answer.

Vary the review angle, and repeat the ones that matter

This is what one of my reviews looks like when it starts, using the aw-review plugin from my personal agent marketplace:

Five review angles, two of them run twice, all on the same change at the same time.

Different review angles are useful: correctness, security, tests, the project's own guidelines. Each reviewer is asked to look for something different, so each looks in different places.

But even two instances of the same reviewer can be worth running. They have the same assignment, yet one may notice a problem the other misses. Different specialists buy different angles. A repeated specialist buys another sample of the same angle.

Model choice adds another variable. Haiku, Sonnet and Opus can produce different findings on the same change. I'd compare what each adds: does switching models uncover valid issues the other runs missed?

More reviewers need deduplication and validation

Several reviewers on the same change produce overlapping findings, and some of those findings are wrong. Raw findings are not the result.

In my setup this happens in two separate steps. A deduper agent groups findings that describe the same defect. Then every surviving finding gets its own validator, running in parallel with the others. Each validator sees one finding and the code, not the other findings or the reviewers' conclusions. The same fresh-context rule applies here as to the reviewers.

This is one real run, on a change of 31 files with seven reviewer instances:

Seven findings became two, and I accepted both. Without those steps, I would have read seven, and five of them were repeated or wrong.

One duplicate is worth a closer look. Both instances of the general reviewer reported the same issue, finding 0001, and the validator rejected it: the evidence they cited did not match the code. Two runs agreeing is not proof. Every finding goes through validation, including the ones more than one reviewer found.

More reviewers without this step just creates more noise.

Watch what each extra run costs

With enough review runs, validating the code can consume as many tokens as writing it. That's a budget worth paying attention to, especially when choosing which models to use. Token counts alone don't tell you the cost when the models are priced differently.

I've published the token table for one whole feature, agent by agent, in If AI Writes and Reviews the Code, What Is the Human Approving? (section 7, What one feature costs). There, the final review of the feature diff, six reviewers plus a second bug reviewer that was stopped, produced 107,771 output tokens. The implementation produced 332,058. In output tokens, one review cost about a third of writing the code. Every extra run adds to that third.

I wouldn't prescribe a fixed number of runs for every change. I'd track the new validated issues each extra run or model adds, their severity, and the cost of finding them. That gives me a basis for deciding where another review is worth the spend.

My review ledger does that now. Over those six runs, the reviewers produced 18 raw findings, 11 of which survived deduplication and validation. The six runs billed 43.6 million tokens, about 4 million per surviving finding, and 88% of them were cache reads.

The split of the tokens is worth a look. The reviewers themselves account for 28% of the billed tokens. The orchestrator takes 40% and the support agents, the deduper and the validators, take 32%. The reviewers are less than a third of the cost of a review.

A valid finding still isn't the same as a useful one. Validation tells me whether a finding is true. Whether it was worth fixing is my call, so Claude Code shows me every finding that survived, one at a time, and I pick the ones to fix:

The interesting number isn't how many reviewers you run. It's how many findings worth fixing each additional one buys.

One run is not enough

What I would change is the default assumption that running the review once is enough.

If another run can find a real bug in unchanged code, that's something worth designing for.