Part 4 of a series. My chief of staff has a reviewer, Elissa, who labels every finding must, should or nit, and only a must blocks. She has no edit tool and no way to run a command, so a quiet fix is not part of her job. Her best catches are not bugs. They are claims stated as fact that nobody had checked, and options that were skipped in silence.
On Thursday 29 October, at 13:13, a research page of mine was told it was not good enough.
The page was a comparison for a side project I work on at night. The question was how to let someone drag items around on a floor plan, and which library to build it on. Yara had gathered the tools. Dan had read the code and picked one. Nadim had put it together into a page with a clear answer on line 7. It read well. Every claim had a number or a file next to it.
Elissa's review opened like this: "Not approved: three musts."
The first one was the worst. The page said that one library had no keyboard support and that another was the only one with a built-in keyboard sensor. The runner-up's weakness rested on exactly those claims. But those claims had come from table cells that Yara had marked from memory, unverified, and Dan had never opened the source to check. On the page they had become plain facts.
Nobody lied. Each step handed the next one something slightly more certain than it received. That is how a doubt turns into a fact without anyone deciding to do it, and that is exactly the kind of problem a review exists to catch.
This part is about the reviewer: what she is, what she is not allowed to do, and why the second list matters more than the first.
What she is
In part 2 I introduced the four experts. Elissa is the one who reviews. Her definition file is short, and the first line of it is the job: she looks for "claims without sources, summaries that outrun their evidence, and the thing that was skipped in silence."
Then come the rules she keeps even if she reads nothing else:
- Lead with the worst thing. No context first, no credit first, no "strong overall."
- Be specific or say nothing. If she cannot point at the line, she does not have a finding.
- Never soften it for the relationship. Nadim wrote most of what she reads, and will be standing there when she says it.
- Never manufacture a problem. "I read it, I found nothing" is a complete finding. Invented criticism gets her ignored on the day she is right.
- Never fix it, never decide. She names the problem and hands it back.
That last pair is mostly her tools and partly her brief. Her list is Read, Glob, Grep, Write, WebSearch, WebFetch. She has no edit tool and no way to run a command, so she cannot patch a sentence in place. She does have Write, which can in theory overwrite a file, and nothing in the tool list stops that. What stops it is the brief, and a rule that her output is a findings file and nothing else. "Never decide" has no tool behind it at all. It is only a rule. I would rather tell you that than pretend the lock is stronger than it is.
🧾 The auditor does not keep the books
An auditor who also fixes the ledger is no longer an auditor. They are a second bookkeeper with an opinion, and nobody is left to check them. The value is in the gap between finding and fixing. Someone else has to look at the problem, decide it is real, and change the thing. That is two chances to be wrong out loud, instead of one chance to be wrong in silence.
Must, should, nit
Every finding gets one of three labels, and the labels decide what happens next.
A must blocks. Nothing moves until she has read the fix and approved it. If she does not, the loop goes again.
A should does not block. It is a real observation, and I look at it, but a should never stops a merge.
A nit does not block either. A wrong line number, a clumsy word.
The rule that holds the whole thing together is a plain one: only a must stops the work. Without it, a reviewer becomes a gate that every small opinion can close, and people learn to route around her.
There is a cap on rounds. The default is two. If she still says no after the last round, the job does not push through. It stops as Blocked and the decision comes to me. Every round is a fresh dispatch with its own input, the change and the open findings. She does not remember the last round and does not need to. A reviewer who remembers her own earlier verdict has a reason to defend it.
And there is one more rule, which she did not write and I did. It came from watching a tidy report do nothing. On 1 October I told Nadim: when Elissa finds a documentation discrepancy, fix it, do not just document it. A list of known problems that nobody fixes is a more polished way of ignoring them. The fix goes in during the same job, and she re-checks it once.
One real review
Back to the 13:13 review. Here is the whole of it, filtered.
Must: claims stated as fact, sourced from "from memory, unverified." The line for the library comparison said the numbers had been measured. They had, for version, date, license and size. Nothing covered the column that decided the winner. Fix: that column now names where each claim comes from. Two were re-read from their sources that afternoon. The rest carry the label unverified on the page itself.
Must: a whole category skipped in silence. I had told the team to look at "plugins or 3rd parties," and the brief asked for any hosted floor-plan service. Yara's own note said they were not searched. The page never said so. Fix: a paragraph now names two hosted services found in one search, marked "not evaluated in depth," so a reader knows the door exists and was not walked through.
Must: a question in the brief with no answer. The brief asked what a screen-reader user gets when moving an item. The page described focus and arrow keys, and the status line was visual. Fix: the page now says what is announced. It is marked as Nadim's design and in no source, because it is.
Should: a reference that looked wrong. She thought a decision number the page cited pointed at something else. Nadim checked. The number was right, but the wording around it had drifted from what the document said. Nadim pushed back and reworded the sentence instead of the number. The should did not block, and it did not need to.
Nit: a wrong line citation. Fixed in a minute.
She also listed what she had checked and found sound: the tool claims traced to dated sources, and the code claims matched the repository. That matters. A review that only lists faults cannot tell you what she looked at. This one told me which parts of the page I could stop worrying about.
At 13:20 she read the changed parts again. Her verdict, quoted on the page: "All five fixes hold."
The first review came at 13:13. Nadim's page records her approval at 13:20. At 13:12 I would have signed that page.
None of the three was a bug in the usual sense. There was no broken code. The page overreached in a way that looks exactly like being right: confident, sourced in form, and resting on a cell that said I am not sure. Yara's cell said so. Dan inherited the claim. Nadim wrote the sentence. The doubt was in the file the whole time and got lost in the handoffs.
Here is the part I did not expect. All three musts were about the distance between what was said and what was shown. Not one of them was about the library choice itself. She never told me the pick was wrong. She told me the page had not earned the right to say it yet.
A check against yourself is not a check
The failure I like least in this whole setup is the circular one.
On 23 September a job produced a page that counted completed jobs. Its check passed. Then Elissa looked and found that the page was missing four jobs. The check had compared the page against the output of the same command that built it, so if the command skipped four jobs, the page and the check skipped the same four together and agreed.
We wrote that down as a rule: a check against the pipeline's own output is circular. Verify against an independent source, a second command that does not share the first one's assumption. A count taken from the same query is not a count. It is the same answer twice.
It is also why she is a separate expert and not a step Dan runs at the end of his own work. Dan checking Dan is the oil seller from part 2, tasting his own oil.
Not every remark is a ticket
A reviewer who is never ignored is a reviewer who is going to drown you.
On 10 October, one batch of follow-up tickets from her findings held roughly twenty-seven items. Three were worth doing. I said so out loud to Nadim: we do not have to do everything Elissa says. Her shoulds and nits are input, not instructions. Nadim files the few worth the time, in one ticket, and drops the rest. No ticket at all if none is.
That sounds like I am letting the reviewer off. It is the opposite. If every remark must be acted on, she will learn to say less, and the one that mattered gets buried among the ones that did not. A review's value is in its signal, not in its volume.
What she cannot do
I want to be fair about the limits.
She is only as good as the files she is shown. She caught the unverified cells because they were written down. If Yara had been rounding in her head and writing only the confident version, Elissa would have had nothing to read. The reviewer cannot see what nobody wrote.
She does not know the product. Her best catches are about evidence and silence, not taste. A page can pass every one of her checks and still be a dull, wrong-headed idea. That part is still mine.
She is also Nadim's reviewer, and that cuts both ways. When she finds a problem with Nadim's own judgment and Nadim disagrees, the argument does not stay between the two of them. It comes to me, with both positions written out. Nobody marks their own homework, and that includes the one who wrote the homework policy.
The record is thinner than I would like. There are 94 committed review files in the repo today, counted by file name (git ls-files, round-numbered files only): 53 first-round files and 41 later ones. A file is not a round, and some rounds left no file, so treat it as approximate. It tells me she has been busy. It does not tell me how many of those rounds found something that mattered, or how often a first round has a must. I tried to count the musts and could not do it honestly, because the reviews are written in different formats. A number I cannot reproduce is not going in this post.
The takeaway
I used to think a reviewer's job was to find mistakes. After a few weeks of watching Elissa, I think it is smaller and stranger: she checks whether the page has the right to be as confident as it sounds.
Most of the damage in AI work is not a wrong answer. It is a right-sounding answer built on something nobody confirmed. The only defense I have found is a second reader who has no stake in the answer, no mandate to change it, and permission to say no in a labeled sentence. Me, the puny human in the cycle.
If you are putting AI in front of your own work, give your reviewer a pen that is meant only to mark, and write the rule down. Make her say which findings block and which do not. And cap the rounds, so a no ends in a decision and never in a stalemate.
The series
- The One Where We Meet Nadim
- The One Where the Avengers Assemble
- The One Where It Works the Night Shift
- The One Where Elissa Says No (you are here)
- The One Where Every Rule Has a Scar, next Sunday
Next week: the rulebook as a changelog of incidents, and why "merged" is not the same as "running."
A good review does not make the work better. It makes the work stop pretending.
Subscribe to Life in Production
New essays on engineering, career, and life. No spam, unsubscribe anytime.