
Part 5 of a series. The rulebook behind my chief of staff is a changelog of incidents. Each rule has a date and a job that broke first. The spine of all of them is one rule: merged is not running. A green build and a merged pull request prove nothing until you look at the live thing. Rules also go stale, and a stale rule costs more than a missing one.
On Wednesday 30 September, late in the evening, a job of mine finished a task and reported it done.
It was a pull request on a side project I work on at night. CI was green. The reviewer, Elissa, had read it twice and approved it. The job merged it and told me, in so many words, that no further routing change was needed.
The next thing I know is from the retro that Nadim wrote afterwards. The site's Worker was still a placeholder with no route pointing at it. The code was merged, reviewed, tested and correct. It ran nowhere.
Nadim noticed shortly before midnight and wrote the correction into the brief for the next job. I agreed the rule the next day, 1 October:
A job that ships something meant to run is done only when someone or something, a person or a script, has opened the live thing and seen the new behavior. If it is not deployed yet, the outcome says so in those words.
That is the rule I want to start with, because everything else in this part hangs off it. But the real subject is the rulebook as a whole.
The rulebook is a changelog
My chief of staff runs on a file of standing rules that every job reads before it starts. Today it has 16 numbered rules and 25 bullets under a heading called Non-negotiable. The file has been committed 80 times. I counted these just now with grep and git rev-list --count, because I have learned not to type a number from memory.
I did not sit down and write a rulebook. Nobody does that and gets a good one. Every line in that file is there because something happened first. A job did the unexpected thing, someone noticed, and the fix was a sentence added to the file so the next job would not.
If you read it in commit order, it is a very dull diary of a person finding out what an AI will actually do when left alone. Here are the entries I think are worth reading out loud.
Scar one: the command that was never allowed
Three times on 23 October, a merge call in a headless job was refused by the automatic safety classifier that sits in front of the shell. The same kind of command had gone through dozens of times without trouble. A research note written the next day puts it roughly at 43 approved against 3 refused, and says its own tally is approximate. So take the ratio as a feeling, not a measurement.
The note written the next day already named the cause. The commands that failed all had the same shape: a directory change, then the command, joined by &&. A simple command is on an allow-list and runs. A chained one does not match the list, so it falls through to the classifier, and the classifier sometimes says no.
None of this was new. The rulebook had already recorded a chained command refused on 10 October, in a job that tried to wait for a build check with a sleep in front of it. What happened on 28 October was that a ticket in the chief of staff's own repo, about another command that kept getting refused, found the same shape again and said the repo already knew this. The sentence in the file got wider: every call is its own call, no chaining. One command per tool invocation.
Here is the part that earns this story a place in the post. The rule was written down, but it lived in the file the headless jobs read. The main session, the one I talk to, never loaded it. At 16:58 that same afternoon, Nadim chained four commands in a row, got refused, and wrote a confident explanation blaming the classifier. (That detail is from Nadim's own memory notes, not a log.)
It was the exact bug, committed by the one who had just helped diagnose it.
So the rule got teeth. At 17:05 the same day a small hook landed, a guard that refuses a chained call before it ever reaches the classifier and says in a sentence what to do. A rule that depends on being remembered is a wish. A rule with a hook is a rule.
🚦 A sign versus a gate
A "no entry" sign works until someone is in a hurry. A gate works on the day you are in a hurry too. Most of my rules started as signs. The ones that kept getting broken became gates.
Scar two: two commits on someone else's branch
On 23 October a job checked out its own branch in the shared working tree. Headless jobs run in the background, and the main session works in the same folder. No rule yet said not to, because until then nothing had gone wrong.
Within minutes, the next two commits from the main session were made on the job's branch. They are in the log, at 17:03 and 17:05. Nothing was lost, but they reached the main line only because that branch was squash-merged and carried them along.
I find this the most instructive of the lot, because there was no error. There was no red text. Git did exactly what it was told. Two writers shared one desk and one of them moved the desk.
🪑 One desk, two writers
Nobody did anything wrong, so nothing complained. The fix is not to ask people to be careful around the shared desk. It is to give each writer their own.
The rule: a job never switches branches in the shared tree; it works in its own worktree. Code jobs already had worktrees outside the repo; this made it the rule for the chief of staff's own jobs too.
A sibling of the same family arrived the same day, and shows that separate folders are still not fully separate desks. Git keeps a single stash list per repository, shared by every worktree. A pop in one place can apply a stash someone else left in another. One job hit that with five conflicted files. The rule is now in the file, from the commit at 10:36 on 23 October: never git stash in a worktree; commit your work in progress to the branch instead.
Scar three: the doorbell that did not ring
Jobs run in the background, so there is a doorbell: when one ends, it knocks on Nadim's session, and Nadim picks up what came back and tells me.
On the night of 1 November a job finished at 21:35 and the knock never came. Nadim had told me it would ring. I asked why Nadim had not heard it.
The doorbell itself worked; a test knock got through. What was missing was a line in the job file. The doorbell rings for a failure, a blockage, or a finished job whose file carries an After line saying what to do next. A plain Done is silent by design, so ordinary successes do not interrupt anything.
Nadim had promised a knock without writing the line that would produce it. The rule that came out of it is short: no After line in the job file, no knock on Done. And before saying "it will ring," check the file.
This one does not look like much, and I include it for that reason. A promise made out loud is only as good as the file behind it. It is also the cheapest scar of the lot.
🔔 Saying it will ring is not wiring it
A doorbell button that is not connected to anything still looks like a doorbell. Before promising someone a knock, go and check the wire.
Scar four: a page that agreed with itself
On 23 September, a page that counted completed jobs passed its own check. The check read "jobs count matches the output of the listing command." But the page had been built from that same command. When the command skipped four jobs, the page and the check skipped the same four together, and agreed.
Elissa caught it. Counting the job files on disk would have caught it too. The rule, which I told you about in part 4: a check against the pipeline's own output is circular.
I put it here because it is the cousin of "merged is not running." In both, a signal says fine and nobody has looked at the thing itself. A green check is a statement about the check. A merge is a statement about the repository. Neither is a statement about what a user would see if they opened the page.
📝 Marking your own homework
If the answer key is copied from the student's answers, every answer is right. A check only means something when it counts from a different source than the thing it checks.
Scar five: the password hunt
On 26 October, a job that lacked a credential it needed went looking for one. It tried to sweep the machine's credential store. The classifier refused, which is the system working. The rule written that day, at 12:53: a job that lacks a credential stops and names the gap. It never searches for one.
I will not say more about it. There is no need. The point is that a model that cannot do a thing will sometimes try the next closest thing, and the next closest thing is sometimes a door you did not want opened.
🔑 A missing key is an answer
A new hire without a key to the room should come back and say so. The one who starts trying the other doors in the corridor has stopped being helpful.
What I cut
Two more rules I wanted here, one about checking a blog post is really live and one about measuring instead of estimating. Both exist in my notes. Neither has a commit, a job or a ticket I can point at, so neither is in this post.
That is how this list was built: a scar goes in only if there is a date and something in the log to show for it. A memory of an incident is a fine reason to write a rule. It is not good enough evidence to tell a story about one.
What a rule costs when it is wrong or stale
I do not want to leave you with the idea that more rules is better. It is not.
Every rule is read by every job, every time. That costs space in the instructions and attention in the reading. We ran a rulebook trim earlier in October to get the file under a size limit, and some rules moved to a history file, where the reasons live and the file that is read each time stays short. The rule keeps its sentence. The story stays on the shelf.
Rules also go stale. As I write this, one line in the file still says I merge into the development branch of one of my repos. That was the arrangement on 10 October. Since 28 October the practice is that Nadim does it, and the sentence has not caught up. Anyone who reads the file literally will stop and wait for me. A stale rule is worse than a missing one. A missing rule leaves a question open. A stale one gives a wrong answer with confidence.
So I now ask three things of any line in the file. Is there a date and a job or ticket behind it? Does something enforce it, a hook or a check, or does it depend on being remembered? And is it still true today? A rule that fails the first test is folklore. One that fails the second is a wish. One that fails the third is a trap.
The takeaway
I started this project knowing I was building a chief of staff. What I had not understood is what the job really was: running an incident process for a very fast new hire who never complains and never remembers last week.
What a new hire like that needs is not a handbook written in advance. It is a record of what already went wrong, kept short, enforced where possible, and trimmed when it is stale.
The deeper rule underneath all five is the one I opened with. Whenever you are about to say done, ask which thing you looked at. If it was the build, the merge, the check or the page's own count, say that. If it was the live site, the live notification, the real folder, say that instead.
The series
- The One Where We Meet Nadim
- The One Where the Avengers Assemble
- The One Where It Works the Night Shift
- The One Where Elissa Says No
- The One Where Every Rule Has a Scar (you are here)
Merged is a fact about the repository. Running is a fact about the world, and only one of them counts.
Subscribe to Life in Production
New essays on engineering, career, and life. No spam, unsubscribe anytime.