Field notes · 2026-10-19 · RunVouch

Your agent didn't fail, it drifted. Detecting output and duration drift with a 7-run baseline

Agents that exit 0 and write the wrong thing. How a 7-run median plus MAD baseline catches output and duration drift, and why mean and stdev miss it.

The worst run my nightly report agent ever had exited with code 0. It ran for 11 seconds instead of its usual six minutes, wrote 40 bytes instead of 40 KB, and cron was perfectly happy. The 40 bytes said: "I was unable to access the data source." Nobody read them for four days.

That is drift. Not a crash, not a timeout, not an exception. The job still runs on schedule, still returns success, and quietly does something else than it did last week. After two years of running agents and a crypto trading bot unattended, I think drift is the failure that costs the most, because every other check says green.

RunVouch is a watchdog for unattended AI agents: a dead man's switch, a cost cap and an outcome check, with a drift detector that compares every run against the last seven. This article is about how that detector works and why the obvious approach, a mean and a standard deviation, is the wrong tool.

What AI agent output drift looks like in practice

Drift is a run that completes but no longer resembles the runs before it. Three shapes I have seen on my own machines:

Collapse. A summariser that normally prints 40 KB prints 40 bytes. The model got a 403 from an upstream API, apologised in one sentence, and the wrapper script wrote that sentence to the output file. Exit 0.
Short circuit. A six minute job finishes in 11 seconds. A changed prompt file made the agent decide there was nothing to do. Exit 0.
Bloat. A 90 second job takes 25 minutes and the output triples. The agent started re-reading every file in the repository on each call because a tool description changed. Not a loop, so no retry storm, just a slow, expensive job doing far more than it should.

In all three, the content is wrong or useless, but the only two signals a scheduler sees, the exit code and the fact that it ran, look fine. The two signals that do move are the duration and the size of what the job produced. Those are cheap to measure and hard to fake, which is why they are what RunVouch watches.

Why mean and standard deviation miss agent drift

The first version of my own drift check, years before RunVouch, took the mean of the last runs and alerted when a run fell outside three standard deviations. It never fired on the 40 byte run. Here is why.

Suppose the last seven output sizes were 41, 39, 42, 40, 38, 41 and 40 KB. Mean 40.1, standard deviation about 1.3. A 0.04 KB run is thirty deviations out. That one is caught. Now suppose one of those seven was an earlier bad run of 2 KB that nobody noticed. The mean drops to 34.7 and the standard deviation jumps to about 14. The new 0.04 KB run is now 2.5 deviations out, inside a three sigma band. One bad run in the history has hidden the next one.

That is the known weakness of the mean and standard deviation: both are pulled by the very outlier you are trying to detect. Leys and colleagues made exactly this point in a 2013 paper whose title is its conclusion, Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. The NIST Engineering Statistics Handbook says the same in drier words: extreme values do not distort the median absolute deviation because it is based on ranks, so it gives a more stable estimate of spread when the tails are heavy. Agent run histories always have heavy tails, because the history is where the previous incidents live.

How a 7-run MAD baseline works, explained simply

MAD stands for median absolute deviation. Three steps, no squares, no square roots:

runs  = [41, 39, 42, 40, 2, 41, 40]      # last 7 successful runs, KB
median = 40                               # sort, take the middle
deviations = [1, 1, 2, 0, 38, 1, 0]      # |run - median|
MAD = 1                                   # median of the deviations

new run: 0.04 KB
|0.04 - 40| = 39.96  >  4 x MAD = 4     # DRIFT

The bad 2 KB run in the history contributes one deviation of 38 to a list of seven, and the median of that list ignores it. The baseline stays at 40 plus or minus 1, which is what the job actually does. With the mean and standard deviation, that same history produced a band from about 7 to 62 KB. The Wikipedia article on MAD puts the difference in one sentence: in the standard deviation the distances are squared, so outliers weigh heavily; in the MAD the deviations of a small number of outliers are irrelevant.

RunVouch runs this calculation twice after every successful run, once for duration and once for output size, against the median of the previous seven successful runs. A run is flagged DRIFT when it sits more than four MADs from that median. Four is deliberately wide. Leys suggests 2.5 for research data; a watchdog that pages you at three in the morning needs to be more conservative than a journal.

Making the detector quiet enough to keep on

A pure MAD check is still too noisy for real jobs, and a noisy detector gets muted, which is worse than no detector. These are the rules on top, each one added after a false alarm on my own agents:

A floor on the spread. Seven runs of exactly 40 KB give a MAD of zero, and any change becomes infinite drift. The MAD is never allowed below 10 percent of the median, 5 seconds for duration or 512 bytes for size. A deviation under 25 percent of the median is also ignored outright.
A floor on the baseline. Jobs with a median under 30 seconds or under 2 KB are not checked. A one second job that takes three seconds is noise.
At least four runs first. Nothing fires until there is a history to compare against.
Known modes are not drift. One of my own jobs alternates between an incremental pass of about 350 seconds and a full pass of about 700. A median plus MAD assumes one normal level, so it flagged every full pass. Now a value the job has reached at least twice in its last 25 runs is treated as a known mode. A value the job has never produced still alerts.
The observed band is not drift. A scraper of mine takes anywhere from 8 to 72 seconds with identical output. Any value between the second lowest and second highest of the last 25 runs is accepted. A collapse to almost nothing falls below that band and still alerts, which is the case the detector exists for.

The alert itself names the numbers, so you can judge it from your phone:

DRIFT nightly-report: output size 40 vs trailing median 40960 (MAD 4096).
Task may be silently doing something else.

Setting up drift detection for a scheduled agent

Drift detection is on for every agent; it needs nothing but a history. Wrap the command:

rv agent nightly-report --cadence 24h
rv run nightly-report --evidence-file out/report.html -- claude -p "$(cat prompt.md)" --output-format json > out/report.json

The client records the start, the end and the size of what the command printed. With the Claude Code plugin or a claude -p run with JSON output, cost and tokens come along as well, so BUDGET_RUN sits next to DRIFT. The evidence file adds a third, independent check: if the file was not modified during the run, or is empty, you get NO_EVIDENCE regardless of what the drift maths say. If a job cannot run a client at all, the ping URL per agent accepts a plain HTTP call at the start and the end, and duration drift still works from the timestamps.

None of this kills a running process. DRIFT is an alert to Telegram, Slack or a webhook, nothing more, because a job that is slow is not necessarily a job that is wrong. Only a crossed cost cap pauses the agent, and even then the brake is applied to the next rv run, never the current one. The client fails open: if the watchdog is unreachable, the job runs as if it were not there.

The honest limit: a drift detector looks at shape, not meaning. An agent that writes 40 KB of confident nonsense at the usual speed passes. For that you need an outcome check of your own, a schema, a test, a diff against yesterday. What the 7-run baseline buys you is the four days I lost: the morning after the first bad run, instead of the morning you finally open the file.

FAQ

Why seven runs and not thirty? Seven is enough for a median to be stable and short enough to follow a deliberate change. If you rewrite the prompt and the job legitimately produces half the output, the baseline has moved within a week. Thirty runs would alert for a month. The wider 25-run history is used only to recognise known modes.

Does drift detection work for an agent that produces different output every day? Different content, yes; different size, within limits. News digests, reports and summaries vary in content but stay within a band of sizes. The band rules above learn that band from your own runs. A job whose output size genuinely ranges over two orders of magnitude will never trip the detector, and should rely on the evidence file and a cost cap instead.

Can I tune the threshold? On self-host, the multiplier is an environment variable, default four. The floors and the known-mode rules are fixed, because every one of them exists to prevent a false alarm I have already had. On the hosted plans the defaults apply; three agents are free, Solo is 9 dollars and Team 29 dollars a month, hosted in the EU, and the code is MIT.

Related field notes

Try it: free for 3 agents · Docs: Claude Code · cron