Presentation · AI code review · October 2026
How much should you trust an AI code review?
I replayed one real diff through my review agents, changed the prompts and settings between replays and measured what the reviewers actually read. What changed, and what I now look at before I approve a pull request.
P presents · ↓ or a remote moves one screen · F fullscreen · Esc exits
The decision
One finding.Is the rest fine?
If you trusted the review completely, this is easy. Fix the one finding, approve, move on.
If you don't, you are back to reading the whole change yourself.
● Review finished. 1 finding.
MEDIUM · data · found by general
backend/src/services/bulletin-publishing.service.ts:305-314
Edit can change the recipient lists of an emergency message, but its receiving units stay fixed.
How it fails: units on the new list can read the message but cannot acknowledge it, and the publisher status never completes.
What the review did not say
One finding.Most of the change unread.
- 95
- 2,589
- 1,789
800 of 2,589 changed lines read, 1,789 unread.
A short list feels like good news. It might just mean the reviewer saw a third of the change.
Change 00 · Measure first
Recorded by the system.Not reported by the agent.
What I changed
Every review run writes a ledger of what each reviewer's tool calls actually returned.
Read = one of the reviewer's own tool calls returned the line. Nothing else counts.
Baseline · effort medium · share of changed code read
- Unread by all
- 1,789
- Findings · reported / kept
- 2 / 1
- Cost per run
- $1.66
What I learned
Nothing in the review changed yet. But now I could see it.
Change 01 · Prompt wording
Two sentences removed.Correctness 30% → 74%.
What I removed
Do not flag speculative concerns.
Only as far as … stop.
Written to prevent noise. The agents seemed to read them as permission to stop.
Same model, same effort (medium), same diff. Output instructions moved to the end.
Prompt cleanup · dashed line = previous step
- Unread by all
- 637
- Findings · reported / kept
- 9 / 7
- Cost per run
- $4.95
What I learned
Wording changes how far an agent goes.
Change 02 · Three things at once
Aimed at general.Correctness fell 74% → 54%.
What I changed
- Shorter prompts per agent
- Queue sorted by path
- Frontend example for general
Queue and shorter prompts
- Unread by all
- 285
- Findings · reported / kept
- 10 / 4
- Cost per run
- $4.16
What I learned
Aim at one reviewer. Measure all of them.
Change 03 · One sentence back
Reading more.Seeing less.
What I put back
Read the patch first, open the full file only to prove a finding.
Without it, the tests reviewer read whole files with cat.
An agent can read a lot and still read the wrong thing. Make coverage the target and you get reviewers that fetch more code without thinking harder about it.
Patch-first
- Unread by all
- 509
- Findings · reported / kept
- 6 / 5
- Cost per run
- $3.73
What I learned
Coverage is a guardrail, not the goal.
Change 04 · Effort and workflow
Effort medium → high.
What I changed
- Effort high: correctness, security, tests (general stays medium)
- Tests in two phases: read, then mutate
- correctness
- 5 calls →24 calls
- tests, each
- 8 calls →24–27 calls
Test gap findings 0 → 11, all confirmed.
High effort
- Unread by all
- 253
- Findings · reported / kept
- 18 / 14
- Cost per run
- $11
What I learned
A better review was possible. It was not free.
The whole journey
From 30% to 91%.Not in a straight line.
Five replays of the same diff, then the current version on real branches. Wording lifted correctness, the next change took most of it back, and only effort and the two-phase workflow moved tests. On real branches the floor held.
Share of changed code read by correctness over five replays: 30%, 74%, 54%, 36%, 89%. Tests: 13%, 8%, 7%, 14%, 90%. Lines no reviewer read: 1789, 637, 285, 509, 253. On 38 real runs: correctness 91%, tests 92%, unread <3%.
Read ≠ found
Three reviewers read it.None reported it.
The AppLayout file was read in two runs. In Patch-first, general read it and reported a high severity bug: AppLayout returns 401. In High effort, three reviewers read it and none of them reported anything.
The AppLayout bug was high severity, and it was found once. In the High effort run, three reviewers read that file. None of them reported it.
The BulkWriter bug was found every time. Its severity moved between high and low. Same bug, same code, different opinion.
BulkWriter: unhandled write promises
Every replay where the reviewers completedSeverity: High once, medium or low elsewhere
removeList race
Prompt cleanup onwards, nearly every replay
AppLayout returns 401 (frontend)
Only Patch-first, by generalSeverity: High
Name uniqueness checked outside the transaction
First in High effort, by general on medium
What I learned
Coverage tells me what reached the reviewer. Not what it noticed.
Validation · Try to disprove it
Same finding.Now with evidence.
The finding from the start, with its evidence and what the validator checked when it tried to prove the finding wrong.
Earlier: 1–3 rejected per run.
High effort: 0 of 18.
Better findings, or an easier validator?
The validator runs on Opus, the reviewers on Sonnet. A different model, but the same family, and it can share their blind spots.
● Validating…Validation done. 1 confirmed.
MEDIUM · data · PENDINGCONFIRMED · found by general
Edit can change the recipient lists of an emergency message, but its receiving units stay fixed.
Evidence:
- bulletin-publishing.service.ts:305(defect)
- bulletin-publishing.rules.ts:343(context)
- emergency-acknowledgement.service.ts:112(call_site)
Validator: save() resolves receiving units only when mode === 'create'. On update the list is empty, so the units are never rewritten.
What I learned
CONFIRMED is evidence. Not proof.
Same reviewer, twice
Same prompt.Same diff.Same run.
General runs as two instances, started at the same moment. Nothing differs between them.
Sometimes they agree. Sometimes one reads half the change and the other almost all of it.
Which one is the review?
What I learned
Run it once and I get 46% or 84%, with no way to know which.
On real branches
Not a higher peak.A higher floor.
What nobody found
2 confirmed misses
Both general's. Both in a file it never read.
Found later on the PR. Written back to the ledger.
Latest run: every changed code file read by at least one reviewer. Seven of seven live findings confirmed.
General still swings from 46% to 94%. It is the one reviewer without the two-phase workflow yet.
What I learned
Every review a PR gets is data about how good mine was.
Take this with you
Three questions for any AI review.
What did it read?
Mine0 unread
If the reviewer saw a third of the change, a short finding list means nothing.
Did the findings hold?
Mine7 of 7 confirmed
A finding a validator tried to disprove, and could not, is worth more than a confident explanation.
How has it done before?
Mine238 runs · 14 misses
One run is one sample. The ledger says whether a reviewer is usually thorough or usually lucky.
When your AI review says “no problems found”,do you know what it read?
Every number here, the replays it comes from and what each metric cannot tell you are in the post.
How much should you trust an AI code review?