How Much Should You Trust an AI Code Review?
Every AI code review ends the same way. A list of findings, and a developer deciding what to do with it.
Approve. Fix. Or go and check something yourself.
For a long time I made that decision on the list alone. Each finding had a file, a line and a confident explanation. Nothing in the list told me how much of the change the reviewer had actually looked at, or how much weight its findings deserved.
The AI never says "this is my opinion, and here is what I based it on." But that is what a review is. An opinion.
I don't think the point of AI code review is more findings. The point is that the developer makes a better decision, faster. In my experience, that only works when the opinion comes with evidence: what the agents actually did, what they read, and what held up when someone tried to disprove it.
Most AI reviews give you almost no evidence
Some review tools add a summary of what was reviewed. But a summary written by the agent is still the agent's opinion. It is not the same as a record of what actually entered its context.
What you usually don't get:
- Which files the reviewer opened.
- How much of the change it actually saw.
- Whether its findings survived any independent checking.
- What it has missed before.
A short list feels like good news. It might just mean the reviewer saw a third of the change.
The first step was not improving the review. It was building a way to see what the reviewers had done, recorded by the system and not reported by the agent.
I have written before about storing everything an AI dev flow produces as project artifacts, so the work can be steered by data instead of by what an agent says. Captured tool output outweighs a model's summary. Review needed the same treatment.
In my review skill, AW Review (the aw-review plugin in my personal agent marketplace), every run now writes to a ledger: what each reviewer's tool calls returned, every finding with its evidence, the validator's verdict, and the tokens, time and cost. Each record carries the version of the reviewer's prompt, so every number traces back to the prompt that produced it. A dashboard turns it into numbers.
One thing it does not record yet is why a reviewer stopped. That is the next field I want.
The first time I looked, the review had come back with one finding. The change was 95 files and 2,589 changed lines of code. 1,789 of those lines had not been read by any reviewer.
The review did not say that. It never does.
The setup and what I measured
AW Review runs five reviewers in parallel: correctness, general, security, tests and guidelines. General and tests ran as two separate instances, marked ×2 in the tables. A validator agent checks every finding before I see it. Reviewers run on Sonnet 5.5, guidelines on Haiku, the validator on Opus. Guidelines only reads the files its rules apply to, so I leave it out of the coverage numbers.
In October 2026 I took one real diff and replayed it: same 95 files, same commit range, same tools. Between replays I changed one area of the prompts or settings. It was iterative engineering, not a lab experiment, and some steps changed more than one thing.
The replays below are examples from a larger set of runs. Each step ran once, so a difference of a few points means nothing. The pattern I lean on, the effect of prompt wording, has repeated across runs.
A changed line counts as read when one of the reviewer's own tool calls returned it. Nothing else counts. If the content never came back to the agent through a tool, the agent never saw it.
Every number has limits, so I wrote down what each one can and cannot tell me:
| Metric | What it tells me | What it does not tell me |
|---|---|---|
| Code read | How much of the change reached the reviewer | Whether it understood what it saw |
| Unread by all | Code that no reviewer looked at | Whether that code has bugs |
| Raw / live findings | What reviewers claimed / what survived dedup and validation | Whether the survivors matter |
| Rejected by validator | How many claims failed an independent check | Whether the validator is right |
| Confirmed misses | Real defects the review did not report | The misses nobody has noticed yet |
| Tool calls, time, cost | How much effort was spent | Whether the effort was worth it |
None of these is "review quality". A review that reads 30% of a change and finds the critical race condition is worth more than one that reads 100% and reports three naming issues. Together, though, they say how much a run deserves to be trusted.
This was the starting point:
| Step | Change | correctness | general ×2 | security | tests ×2 | Unread by all | Raw / live | Cost |
|---|---|---|---|---|---|---|---|---|
| Baseline | Original prompts, effort medium | 30% | 29% · 27% | 25% | 13% · 13% | 1,789 lines | 2 / 1 | $1.66 |
Share of changed code lines each reviewer read. A ×2 reviewer gets one figure per instance.
A cautious prompt, a reviewer that stopped early
The first thing I looked at was wording.
My prompts had sentences meant to keep the reviewers focused. "Do not flag speculative concerns." "Only as far as … stop." Some of them had a punishing tone.
They were written to prevent noise. The agents seemed to read them as permission to stop.
I removed them, cleaned up the rest and moved the output instructions to the end. Effort stayed at medium.
| Step | Change | correctness | general ×2 | security | tests ×2 | Unread by all | Raw / live | Cost |
|---|---|---|---|---|---|---|---|---|
| Baseline | Original prompts, effort medium | 30% | 29% · 27% | 25% | 13% · 13% | 1,789 lines | 2 / 1 | $1.66 |
| Prompt cleanup | Stop-inviting and punishing wording removed | 74% | 62% · 48% | 33% | 16% · 0% | 637 lines | 9 / 7 | $4.95 |
Correctness more than doubled. Unread code dropped by two thirds. Live findings went from one to seven, and a race condition in removeList showed up for the first time.
Same model. Same effort. Different sentences.
This is the result I have seen again and again since. How a prompt is worded changes how far an agent goes, often more than any setting.
Tests stayed at 16% and 0%. Both tests instances read 98% of the changed test lines and almost none of the code those tests were supposed to cover. A test reviewer that never reads the code cannot tell you what the tests miss.
A change aimed at one reviewer moved another
The next step changed three things at once. Shorter instructions per agent for Sonnet 5.5. The review queue sorted by path instead of importance, because sorting by importance pushed frontend files to the end. And a frontend example for general.
| Step | Change | correctness | general ×2 | security | tests ×2 | Unread by all | Raw / live | Cost |
|---|---|---|---|---|---|---|---|---|
| Prompt cleanup | Stop-inviting and punishing wording removed | 74% | 62% · 48% | 33% | 16% · 0% | 637 lines | 9 / 7 | $4.95 |
| Queue and shorter prompts | Shorter per-agent prompts, queue by path, frontend example | 54% | 77% · 88% | 32% | 13% · 1% | 285 lines | 10 / 4 | $4.16 |
General improved, as intended. Correctness dropped from 74% to 54%.
Was that the shorter prompt? Or just a different path through the same code? With one run and three changes, I can't tell. What I can tell is that a single run would have let me believe either story.
One removed sentence changed how an agent worked
During the shortening I had removed one sentence: "Read the patch first, open the full file only to prove a finding."
After that, the tests reviewer started reading whole files with cat. More reading, less of the change.
I put the sentence back.
| Step | Change | correctness | general ×2 | security | tests ×2 | Unread by all | Raw / live | Cost |
|---|---|---|---|---|---|---|---|---|
| Queue and shorter prompts | Shorter per-agent prompts, queue by path, frontend example | 54% | 77% · 88% | 32% | 13% · 1% | 285 lines | 10 / 4 | $4.16 |
| Patch-first | Patch-first sentence restored | 36% | 79% · 62% | 30% | 14% · 14% | 509 lines | 6 / 5 | $3.73 |
Correctness fell again, to 36%. In this run it used five tool calls on a 95-file change.
The cat episode taught me something about the metric itself. An agent can read a lot and still read the wrong thing. If I made coverage the target, I could get reviewers that fetch more code without thinking harder about it. So coverage is a guardrail for me, not a goal. It tells me when a run is not worth trusting. It does not tell me when it is.
Effort high: the biggest jump and the biggest bill
Then I changed two things together again. The tests reviewer got a two-phase workflow: first read every patch, then try mutations on the candidates. Correctness, security and tests moved to effort high. General stayed at medium as a reference point.
| Step | Change | correctness | general ×2 | security | tests ×2 | Unread by all | Raw / live | Cost |
|---|---|---|---|---|---|---|---|---|
| Patch-first | Patch-first sentence restored | 36% | 79% · 62% | 30% | 14% · 14% | 509 lines | 6 / 5 | $3.73 |
| High effort | Effort high, two-phase tests | 89% | 50% · 89% | 50% | 90% · 90% | 253 lines | 18 / 14 | $11 |
Tests went from 14% to 90%. Test gap findings went from zero in the baseline to eleven, and the validator confirmed all eleven.
The work changed with it:
| Reviewer | Baseline | Patch-first | High effort |
|---|---|---|---|
| correctness | 9 calls, 22 s | 5 calls, 24 s | 24 calls, 132 s (high) |
| security | 4 calls, 11 s | 5 calls, 20 s | 14 calls, 51 s (high) |
| tests ×2 | 6–7 calls, 15–22 s | 8 calls, 30 s | 24–27 calls, 157–159 s (high) |
| general ×2 | 6–7 calls, 13–24 s | 8–17 calls, 44–76 s | 10 calls, 50 s (medium) |
The cost tripled. I can't say how much of the jump came from effort and how much from the tests workflow, only that the combination worked.
Effort is one dial. The model is another, and it did not change in these runs. Which model is the most cost-effective for each reviewer role is not something to guess. It is something to test the same way: same diff, one change, compare.
Put together, the five replays looked like this, next to the real runs:
Same prompt, same diff, same run
I have written before that AI code review is statistical: run the same review twice on unchanged code and you get different findings. The replays here changed something between every step, so they can't show that cleanly. The cleanest evidence is somewhere else.
General runs as two instances. Same prompt, same diff, same run, started at the same moment. Nothing differs between them.
Sometimes the two agree. Sometimes one reads half the change and the other reads almost all of it.
This is what "one run is one sample" looks like in numbers. If I had run general once, I would have got either 46% or 84%, and no way to know which.
Read is not found
This is the part that changed how I think about coverage.
| Finding | Where it was found | Severity |
|---|---|---|
| BulkWriter: unhandled write promises | Every replay where the reviewers completed | high in Prompt cleanup, medium or low elsewhere |
removeList race | Prompt cleanup onwards, nearly every replay | |
| AppLayout returns 401 (frontend) | Only Patch-first, by general | high |
| Error text for NOT_FOUND | Only Patch-first | |
| Name uniqueness checked outside the transaction | First in High effort, by general on medium |
The AppLayout bug was a high severity finding. It was found once.
In the High effort run, three reviewers read that file. None of them reported it.
That was not a coverage problem. They saw the code. They judged it differently.
The BulkWriter bug was found every time, but its severity moved between high and low. Same bug, same code, different opinion.
Then there is the validator. In earlier runs it rejected one to three findings per run. In High effort it rejected zero out of eighteen. That can mean the findings got better. It can also mean the validator got easier to convince.
A validator verdict is independent evidence, not ground truth. It is a different model from the reviewers, but from the same family, and it can share their blind spots. CONFIRMED means someone tried to disprove the finding and could not. That is worth a lot. It is not proof.
This is why "the AI found no problems" is such a weak statement. It mixes three separate questions: what did it read, what did it notice, and how sure should I be.
The data made the decisions
None of the decisions after these runs came from a feeling about the review. Each one came from a number.
| What the data showed | What I decided |
|---|---|
| Wording alone doubled coverage at medium effort | Fix wording before raising effort |
| Without patch-first, tests read whole files instead of the change | Keep patch-first |
| Correctness stopped early on medium in several runs | Correctness and security on high |
| High effort made two tests instances cost about 300k tokens together | Tests ×2 → ×1 |
| General varies the most, but produced the best new production finding | Keep general ×2, give it the two-phase workflow next |
Without the ledger, I would have changed the prompt, seen a different finding list and called it better or worse based on how the list felt.
What it looks like on real branches
Replays are the controlled part: one diff, compared against itself. Real branches show whether the result holds in normal work. They are different diffs, so the comparison is not clean. It is the test that matters, though.
Since the High effort step, the reviewers have run seven more real reviews on six branches, with the same agent versions, except that tests now runs once.
| Run | Files | Code lines | correctness | general ×2 | security | tests | Unread by all | Raw / live | Cost |
|---|---|---|---|---|---|---|---|---|---|
| A | 98 | 2,607 | 89% | 49% · 89% | 50% | 100% | 0 | 5 / 5 | $7.52 |
| B | 102 | 2,593 | 89% | 46% · 84% | 53% | 91% | 226 | 9 / 6 | $9.73 |
| C | 109 | 2,604 | 90% | 57% · 84% | 47% | 92% | 212 | 9 / 4 | $9.81 |
| D | 53 | 1,012 | 91% | 90% · 94% | 86% | 92% | 41 | 2 / 2 | $5.41 |
| E | 98 | 1,255 | 93% | 85% · 85% | 58% | 92% | 40 | 1 / 0 | $8.71 |
| F | 208 | 3,532 | 93% | 76% · 63% | 52% | 96% | 73 | 9 / 7 | $18 |
| G | 151 | 2,780 | 97% | 70% · 74% | 60% | 98% | 0 | 8 / 7 | $16 |
Together with the High effort run, compared with the 30 real runs on the old agent versions:
Correctness and tests now hold their level from run to run. That is the change I wanted most. Not a higher peak, a higher floor.
General still swings from 46% to 94%. It is the one reviewer I have not given the two-phase workflow yet.
In the latest run, every changed code file was read by at least one reviewer. Seven of seven live findings were confirmed. The validator rejected one: a guidelines finding where the arithmetic was right, but the change sat in a field the rule itself excludes.
That is the kind of evidence I want next to a finding. Not just "here is a problem", but "here is what someone checked, and what held".
Then security. It reads about half of the changed code, and across the whole ledger it has produced two findings, about 692,000 tokens each. Tests produced 85 at about 45,000 each. That comparison is tempting and wrong. Security findings should be rare, and one critical finding can justify a lot of scanning.
Security does not need to read every line either. It should read the authentication, the input handling and the data boundaries, not the i18n files. So the real question is not whether security is worth running, but whether it reads the right half. Answering that needs coverage of security-relevant changes, not of all changes.
What nobody found
Coverage tells me what was read. Validation tells me what held. Neither of them tells me what every reviewer missed.
That needs input from outside the review.
When a pull request is reviewed by people, or by another model, their comments point at things AW Review did not report. The review skill pulls those comments back in. A person confirms each one, and it goes into the ledger as a miss, with a cause.
So far the ledger has two confirmed misses. Both were general's, and both had the same cause: the defect was in a file the reviewer never looked at.
That closes the loop. Every review a PR gets, human or AI, becomes data about how good my own review was.
From opinion to evidence
So how do you know an AI review was any good?
From one run, you can't. Not with certainty.
What you can know is how much to trust it. I ask three questions now, about any review:
- What did it read? If the reviewer saw a third of the change, a short finding list means nothing.
- Did the findings hold? A finding that an independent validator tried to disprove, and could not, is worth more than a confident explanation.
- How has this reviewer done before? One run is one sample. The ledger tells me whether this reviewer is usually thorough or usually lucky, and what it has missed.
Since this change, I decide on practically every review this way. Not because the reviews got perfect. Because the data shows me which findings and recommendations I can trust, and where I still need to look myself.
The variance will not go away with a better prompt. What can change is what the developer decides on. Not the AI's opinion, but data about what the AI did, stored with the project and visible when the decision is made.
In my experience, that is how a developer makes better decisions faster. Not by reading more AI comments. By knowing which ones to trust. It is not the only way to get there, but it is the model that has worked for me.
When your AI review says "no problems found", do you know what it read?