I Gave My Terminal a Chief of Staff: The One Where It Works the Night Shift

Share X LinkedIn Facebook WhatsApp
Back to Life in Production
I Gave My Terminal a Chief of Staff: The One Where It Works the Night Shift
TL;DR

Part 3 of a series. My chief of staff runs jobs at night, each one its own headless Claude Code session with a step cap, a time cap and a row it updates as it goes. It took a silent four-day outage, a misread exit message and a closed-lid test to get there. The night shift is not the jobs. It is the log they leave.

On 30 September I opened a file and found four days of nothing.

The file is called OVERNIGHT.md. Every job Nadim schedules gets a row in it: when it fires, what it should produce, and an outcome. Four of those rows still said pending. They were supposed to run on the evening of the 25th. Nothing had run since. A weekly retro was scheduled for the 27th, and it fired into nothing too.

Nothing crashed. Nothing sent an alert. The account hit a usage limit, the laptop went to sleep, and the night shift simply stopped showing up for work. I found it by reading the file.

It was not even the first time. On 21 September two jobs never fired and sat pending for two days before anyone looked. When Nadim wrote its retro on 2 October, the line was blunt: "Runs that never fired: at least seven."

This part is about the night shift: the work that runs while I sleep, and everything I had to build before I could trust it.

Why nights

In part 2 I described the team: Dan builds, Yara researches, Elissa reviews, Fady keeps the dates, and Nadim decides who does what. During the day, that team is mostly a conversation. I ask, Nadim staffs it, an expert does it, I read the result.

But my laptop time is for deciding, not for watching a progress bar. The real value shows up when the work happens without me in the room: a ticket I approved in the evening is built, reviewed, merged into development and checked by the time I open the laptop again.

That only works if three things are true. The work has to start on time. It has to stop on its own when it should. And it has to leave behind a record I can trust in the morning, without asking anyone what happened.

The September outage failed the first and the third at once.

A cron is a nudge, not a promise

The jobs are started by crons inside Claude Code. A cron fires a prompt at a time. It is also session-only: if the session ends, the cron is gone. If the laptop sleeps, it does not fire. A Claude Code cron expires after seven days anyway.

So on 22 September we made a decision that sounds backwards: the schedule does not live in the crons. It lives in a plain file, dates.md. That file is the commitment. A cron is only a nudge to act on it. A cron dying overnight, which they all eventually do, is a missed nudge, not a missed deadline. Every morning, the check compares the crons that are alive with the ones that should be, and reports the difference in numbers, never as "crons checked".

⏰ The alarm is not the meeting

If your phone dies, the meeting still happens. You just did not get reminded. Nobody puts the meeting in the alarm app and deletes the calendar. Crons are the alarm. The file is the calendar. When the alarm fails, the file still says what was supposed to happen, and someone can see that it did not.

That is also why the outage got caught at all. The rows still said pending. Today, /nad jobs marks any job past its time with no report as MISSED or not yet reported, so I do not have to read the whole file to notice.

One job, one session

Until October, a night job was a prompt fired into the session I had been talking to all day. That sounds harmless. It is not.

Claude Code re-sends the whole conversation with every request. Anthropic's own documentation says a scheduled prompt inside a session "fires on its interval even while the session is idle, sending your full context each time." So a job at 01:00 started by paying for my entire day: every question, every file read, every tool output since the morning. Then it added its own on top, and the next job paid for both. By morning one terminal held three jobs' worth of output and nobody had been there to /clear it.

There was a second problem, cheaper and worse. On 7 October the main session ran a git commit -am while a job was halfway through its own edits in the same folder. The commit swept the job's unfinished work into a commit that was not the job's. Two sessions, one working tree, and the convenient command took both. Since then every session stages files by name, and a guard script refuses git add -A and git commit -a.

So on 7 October we ran a trial. Job87 had run a few days earlier the old way, inside the main session. Job88 ran as its own headless claude -p. From the cost log, where "priced" weighs each kind of token by what it costs (a cached re-read is a tenth of fresh input):

  • job87, in the main session: every reply re-read 53k tokens of context, and cost 8.8k priced tokens.
  • job88, its own session: every reply re-read 30k, and cost 7.9k.

They were different jobs, so only the per-reply figures compare. Context fell by more than 40%. Cost per reply fell by about 10%, because re-reading history is billed at the cheap cached rate. Still, it was the direction we wanted, and that afternoon headless became the default, and a standing rule every job reads first.

Then Nadim re-measured it for this post, and the number did not hold. Across the 137 runs logged since 8 October, the median reply re-reads 52k tokens of context and costs 9.7k. The 24 runs in the six days before re-read 49k and cost 9.1k. The jobs also got longer: the median run went from 49 replies to 104, and a longer job carries more of its own context. So I cannot honestly tell you headless saved X tokens per job. The trial was one job against one job, and the next 137 did not repeat it.

What it did change is harder to put in a number. A job's cost no longer depends on how long my day was. A job at 01:00 starts from its job file, not from fourteen hours of me. My own session stays small enough to clear. And one job dying no longer takes the next one down with it. Headless did not make a reply cheaper. It made the cost of a job stop depending on everything else.

Now each job is its own claude -p run: a fresh session that gets one prompt, "run the job described in this file", and nothing else. The job file is the whole brief. If it is not in the file, the job does not know it. It is the same discipline as part 2's briefs, applied to Nadim itself.

The rule is one job, one session, and it binds Nadim too. On 23 October, to save itself a few launches, Nadim wrapped two jobs in one shell script, with a second script waiting to run the next two. I stopped it. After every job, the main session reads the row, sends the notification and resumes the job if a limit stopped it. A script in the middle swallows all three. One job, one session, one ending you can read.

Every headless job has hard limits:

  • A step cap. A small job gets 50 tool calls, a big one 120. At the cap it stops, commits what is done, and writes down why.
  • A clock. 100 minutes, and a kill at 120.
  • Three lines at the end. Done, Blocked or Failed, one link, one note. That is all the session that launched it gets back.

And since 8 October, a fourth: phases. After every meaningful step (an expert returns, a pull request opens or merges, a deploy goes green) the job appends a timestamp to its own row and pushes it. The row is the signal, not the last three lines.

That rule came from a night when the last three lines lied.

The night the exit message lied

In the early hours of 8 October, a job ran from 01:13. Elissa approved it on round two. Nadim merged it into development at 01:36. Two minutes later, the login expired.

A headless run prints only its last message. Its last message was an authentication error. So Nadim read that output and concluded the job had died at the start. It had not. It had done almost everything, and the proof was sitting in the git history and the board the whole time. The one-line summary was simply the wrong place to look.

A process that only reports at the end is a process you cannot see. So now it reports as it goes, and anyone, including me on my phone, can read the row and see exactly how far it got.

One real night, lid open

Every night row now carries two words in its fire time: lid open. That is because of something I got wrong.

The laptop runs a small utility that keeps it awake. I was sure it held through a closed lid. Nadim had written down that it only blocked idle sleep, then changed it on my word. So on 22 October, at 11:31, with the utility on and the cable in, I closed the lid. The power log showed "Clamshell Sleep" at 11:32:34. A logger writing a timestamp every 30 seconds went silent. The utility blocks idle sleep and nothing else. A closed lid is a closed shift. Never overrule a measurement with a memory, even when the memory is mine.

That night, 22 to 23 October, the lid stayed open and the screen off. Three jobs in a chain, each one only allowed to start if the one before it finished Done. Every time below is from the rows.

01:13, job221 starts: a change to how a side project at night records who did what. Built, approved by Elissa on the first round, merged into development, deploy green and checked live. Done at 01:40.

03:07, job218 starts: a set of admin screens made usable on a phone. Elissa finds one must, a confirm dialog that opened off-screen on a phone. Fixed and merged at 03:26. Then the phone-width screenshots show four screens still overflowing. Fixed and merged at 03:50. At 04:00 the session limit stops it mid-flight. That is not a failure: the rule says a limit is not Blocked. The main session waits for the reset and resumes it with bin/resume-job.

05:13, the resume, and the tooling broke. bin/resume-job finds a job's session by searching the transcripts for its prompt. The main session's own cron quotes that same prompt, so it found the parent, not the child. Nadim caught it, resumed the right session by hand, and fixed the script at 05:14. Done at 05:30.

05:43, job219 starts. At 05:55 it stops on a refusal: a git stash had pulled in an old stash from another job, because every worktree in a repo shares one stash list. Restoring those files was my decision, so the job stopped and put the question on the ticket. It waited for breakfast. There is now a rule: never git stash in a worktree.

One job done straight through, one stopped by a limit and resumed, one waiting on a question for me. One script fixed, one rule added. Nothing that needed me was decided at night. It waited, written down, until I was awake.

What broke, and what is still broken

Nadim counted on Friday. Of the 187 job rows in the current OVERNIGHT.md, 15 mention a resume and 9 mention a session limit. Some of those resumes follow a limit, others follow a decision of mine. Either way, a stopped job got picked back up instead of being started again, which is what it was built for.

What is still fragile:

It is still one laptop. The whole night shift depends on a machine with its lid open and its power cable in. The decision on 22 September was to accept that for now, and revisit it when Nadim no longer lives on my laptop. That trigger has not fired yet.

Resuming is still a judgement call. The rule says how to resume. The night of the 23rd showed that the tool for it can pick the wrong session, and that the catch came from Nadim looking, not from a check. One script fix later, the next mistake is still waiting somewhere.

The guardrails stop jobs at night. Job219 stopped at 05:55 and waited for me until morning. That is the right call, a refusal is a stop and nobody routes around it, but it means some nights end with work half done and a question waiting for breakfast.

The takeaway

I thought the night shift would be about scheduling. It turned out to be about evidence.

The work at night is the same work as during the day. What changes is that nobody is watching. So everything a person would have noticed in the room has to be written down: when it started, what each expert did, where it stopped, and why. A cron that silently dies, a summary that shows only the last line, a lid that closes: all three fail the same way, quietly.

If you are building something that runs while you sleep, start with the log, not the scheduler. Make every job write where it is after every step. Then, in the morning, read the file, not the summary.

The series

  1. The One Where We Meet Nadim
  2. The One Where the Avengers Assemble
  3. The One Where It Works the Night Shift (you are here)
  4. The One Where Elissa Says No, next Sunday

Next week: the reviewer who only ever names problems, and what must, should and nit actually mean.

The night shift is not the jobs. It is the log they leave behind.

Charbel Wakim — Staff Software Engineer at Emma Sleep, writing about production systems, career, and life across the Levant, Europe, and the GCC.

Back to Life in Production