Field notes · 2026-09-16 · RunVouch

Our deploy job reported success for three days. The site stood still.

A brake counter, set -e and grep -c that exits 1 on zero matches killed our deploy three lines before it deployed. No error, no failed run, three days of a stale site, and one duration alert we ignored.

Our own deploy job reported success every morning for three days while the site it deploys stood still. This is the write-up, including the part where the watchdog did its job and we ignored it.

What happened

Every morning at 06:20 a job called refresh-formd rebuilds three paid reports, puts them on the site and deploys. It takes about nine minutes. On 4, 5 and 6 September it took 239, 224 and 197 seconds, exited 0, and reported a clean run. Visitors kept getting the edition of 3 September.

The difference between 550 seconds and 200 is almost exactly the Netlify build, which takes five and a half minutes. The job was not faster. It was stopping before the part that matters.

The line that did it

The deploy script has a brake on how many deploys it may do per day, because build minutes cost money. The brake counts today's deploys in a ledger file:

TODAY_COUNT=$(grep -c "^$TODAY_UTC" "$LEDGER")

When grep -c counts zero lines it prints 0 and exits 1. The script runs under set -e. The first deploy of any day has, by definition, zero lines for today. So every morning the script died on the counter, three lines before the deploy step, with no output and no error message anywhere.

The fix is three characters plus two guards: || true so a count of zero is not fatal, a hard error when the ledger exists but cannot be read (otherwise the brake silently reads zero and is off), and an ERR trap that writes the line number and exit code into the log so the next silent death names itself.

Why nothing failed loudly

This is the shape of failure that costs the most. There was no exception, no non-zero exit at the level anyone watches, no error in the deploy log, and no missing file: the previous edition was still there and still served. Every check that asks "did the job run" answered yes. The job ran. It just stopped doing its work partway through and said nothing.

A heartbeat monitor would have stayed green for three days. An uptime check would have stayed green, because the site was up. A log search would have found nothing to search for.

The alert we ignored

One thing did fire. RunVouch compares each run against the job's own trailing baseline, and on all three mornings it raised DRIFT: duration 239 against a median of 561, then 224, then 197. The alerts were delivered. Nobody acted on them, and they sat unacknowledged for twelve days until somebody went through the open list.

That is worth saying plainly, because it is the more common failure. The detector was right the first morning. The gap was not detection, it was that a DRIFT alert on a deploy job looked like the same noise as a DRIFT alert on a scraper whose runtime swings with the network. We have since changed the detector: a value inside the range a job has actually produced before is no longer reported, which removed six false alerts over 30 days on our fleet and kept the real ones, including this one.

Three things we took from it

Duration is evidence. A job that suddenly finishes in 40 percent of its usual time did not get faster. Something stopped happening. Of the eight things we watch for, duration drift is the only one that would have caught this on the first morning.

Exit 0 is a claim, not a result. The wrapper reported success because the command it wrapped returned success. If the job had been asked for evidence, in this case that the deployed edition carried today's date, it would have failed on day one instead of on day four.

An alert nobody acknowledges did not happen. Delivery is not the finish line. If a kind of alert is routinely ignored, either the detector is too loose or the message does not say what to do, and both are our problem to fix, not the reader's.

The numbers behind this

We publish the failure rates of our own fleet, rebuilt weekly from the production database: how many runs finished, how many failed, and every alert broken out by kind. The counts are free to quote and the runs behind them are hashed into a public daily chain, so the history cannot be edited after the fact. See how often does an unattended job actually fail and the machine-readable version at /api/failure-rates.json.

If you run jobs nobody watches at night, the cheapest version of this lesson is somebody else's: wrap one job, give it a cadence and an evidence file, and let the first surprise be a message instead of a customer.

Related field notes

Try it: free for 3 agents · Docs: Claude Code · cron