2026-09-08  ·  devops · macos · automation · programming

A Scheduled Job That Works and Never Runs

I have a job that publishes one article a day. Over four days I rebuilt its scheduling three times, and each rebuild fixed a real failure that the previous version could not have shown me.

Every version passed the test I gave it. The test was always "does it work now", never "does it work where it is going to live".

Version 1: a loop in /tmp

nohup bash /tmp/daily-loop.sh &

It ran. It published. Four scheduled items were riding on it.

Then the machine restarted and all four were gone. /tmp is not a place you put a program, and nohup does not survive a reboot. I knew both of those facts and still did this, because the thing I was testing was the publishing logic, and the publishing logic was fine.

Nothing reported the loss. The next morning there was simply no new article, and no error anywhere, because the process that would have logged the error did not exist.

Version 2: launchd with KeepAlive

Moved the script into the repository, wrote a launch agent, KeepAlive: true so it comes back if it dies.

The next morning:

scripts/daily-publish.sh: line 59: node: command not found

A job started by launchd gets almost no PATH. Mine was /usr/bin:/bin:/usr/sbin:/sbin. Node is in /opt/homebrew/bin. When I ran the same script from my own shell it worked, because my shell had the PATH.

The part that made this expensive: the same loop also runs a reconciliation step written in Python, and python3 is in /usr/bin. So that step kept working. The log filled up with successful runs every fifteen minutes.

A partly working automation is harder to notice than a dead one. The dead one has no log. This one had a log that looked healthy.

Fix is boring:

export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"

I put it in every script the scheduler touches, not only the one that broke. One entry point was not the whole story: another script could invoke the same thing from a different parent.

Version 3: the loop was frozen, not running

PATH fixed, publishing worked, and then the log went quiet for three hours. The process was alive:

$ ps -o pid,etime,command -p 72127
  PID     ELAPSED COMMAND
72127    02:55:52 sleep 900

sleep 900 had been alive for two hours and fifty-five minutes.

That is a laptop. It sleeps. A sleep that is supposed to last fifteen minutes does not necessarily advance while the system is suspended, so every suspend adds an unbounded gap to a while true; do work; sleep N; done loop.

The loop was not broken. It was not stuck on a network call. It was doing exactly what I wrote, and what I wrote assumed wall-clock time keeps flowing.

Version 4: let the scheduler schedule

<key>StartInterval</key>
<integer>900</integer>
<key>RunAtLoad</key>
<true/>

The script now does one pass and exits. No loop, no sleep. StartInterval is the mechanism built for this: launchd fires the job on wake if the interval elapsed while the machine was asleep, and it will not start a second instance of the same label while one is running, so the lock I would otherwise need is gone.

I also deleted the old looping script instead of leaving it beside the new one. Two entry points for the same job means never being sure which one is running.

One more trap on the way out

I checked whether the interval had registered:

$ launchctl list com.example.daily
{
    "Label" = "com.example.daily";
    "LastExitStatus" = 0;
    ...
};

No StartInterval. I was one command away from concluding the schedule had silently failed to load and rewriting it again.

$ launchctl print gui/$(id -u)/com.example.daily
    state = not running
    runs = 4
    last exit code = 0
    run interval = 900 seconds

launchctl list does not print every key it holds. launchctl print does. The absence of a key in one tool's output is not evidence about the job.

What the four versions have in common

Each fix was correct. Each was verified. Each verification was performed in an environment the job would never actually run in.

• I tested the publishing logic in my shell, so the reboot never came up
• I tested the launch agent by triggering it once, while my shell's PATH still hung around in my head as "the" PATH
• I tested the loop by watching one iteration, which is exactly the case where a sleep that does not advance looks identical to one that does

The rule I ended up with is narrow and I have not managed to argue myself out of it yet:

When you change where something runs, run it once in the new place before believing anything you knew about it.

Not a fresh test of the logic. A test in the new environment, of the same logic, looking for the environment's own failure modes: what the PATH is, what happens across a reboot, what happens across a suspend, and what the tool you use to inspect it declines to tell you.

The fourth version has now survived several suspends and one reboot. That is not proof, but it is the first version whose evidence came from the place it lives.

---

I publish the configuration for splitting Claude Code into separate personas — Architect, Coder, Reviewer, Conflict Resolver — under MIT. Copy it, run ./setup.sh, and it works. It does not depend on your tech stack.

https://github.com/quintetkit/quartet

I built one real tool using nothing but this workflow. Every Issue, PR, review and merge is still there. The parts that went wrong were not deleted.

https://github.com/quintetkit/mdlinkcheck

The version that adds a UI Designer persona, review criteria, a per-Issue parallel execution script and a 10-chapter guide is on the product page.

The full kit — five personas, the scripts and the complete guide — is available here.

https://quintetkit.gumroad.com/l/quintet