RunVouchGet a free key

Field notes · 2026-10-26 · RunVouch

Scheduled LLM jobs in GitHub Actions: exit 0 is not done

Cron in Actions, artifacts as evidence, an rv run step and cost reporting: how to know a scheduled LLM job ran, produced output and stayed on budget.

For two years I ran agents and a crypto trading bot unattended. The lesson that cost me the most was simple: a process that exits 0 has only told you it stopped. In GitHub Actions the same lesson wears a green checkmark. A scheduled workflow that runs claude -p or a Python script against an LLM, finishes, and shows green has proved one thing: the last step returned exit code 0. It has not proved that the job ran on time, that it produced the report it was supposed to produce, or what it cost.

RunVouch is a dead man's switch, cost cap and outcome check for unattended AI agents: each run reports start, end, cost and evidence, and you get an alert when a run is missed, fails, produces nothing, or crosses a budget.

This article is about the Actions case specifically: cron in a workflow, artifacts as evidence, an rv run step, and cost reporting.

Why a green scheduled GitHub Actions run is not proof the job did its work

Start with what the platform itself tells you. The events reference says the schedule event "can be delayed during periods of high loads", that scheduled workflows only run on the default branch, and that in a public repository they are "automatically disabled when no repository activity has occurred in 60 days". None of those three produce a failed run. A delayed run is still green. A disabled schedule produces no run at all, so there is nothing to be red.

Then look at who hears about failures. Per the notifications page, notifications for scheduled workflows go to the user who initially created the workflow, or to whoever last changed the cron syntax. On a team that is usually one person, often the one who left the project six months ago.

And finally the LLM part. When a model is the thing doing the work, the exit code describes the wrapper, not the work. The script that calls the model returns 0 when the HTTP call succeeded. The model may have answered "I was unable to find the input file" and the script wrote that sentence into report.md and exited cleanly. I have had a nightly summary job do exactly this for four days. Every run green.

Monitoring a scheduled AI job in GitHub Actions: the five questions

Each of these is a separate failure mode, and a green run answers none of them on its own.

Actions gives you the second one for free. The other four you have to build or bring.

Artifacts as evidence for a scheduled LLM job

Uploading the output as an artifact is a good habit and I do it in every scheduled workflow. The upload-artifact action keeps files for 90 days by default, and it ignores hidden files unless you tell it otherwise, which matters if your agent writes to a dotfile.

But an artifact is storage, not a check. Nobody gets an alert because an artifact is 0 bytes. Nobody notices that the upload step was skipped because the model never wrote the file and the step had if-no-files-found: warn. The workflow syntax reference even gives you continue-on-error, which "prevents a job from failing when a step fails". Useful for flaky steps, fatal for evidence.

So keep the artifact for the human who wants to read last Tuesday's report, and add a check that looks at the file the moment the run ends. That is what the evidence flag in rv run does: it records the mtime and size of the named file after the command exits, and compares them with the run start. A file that was not touched during the run, or is empty, is NO_EVIDENCE.

The rv run step in a GitHub Actions workflow

Register the agent once with the cadence and the limits you want enforced, from any machine with the key:

rv agent nightly-digest --cadence 24h --cap-run-cost 2 --evidence

Then wrap the command in the workflow. The client is a single stdlib Python file, so there is nothing to install beyond pip:

on:
  schedule:
    - cron: "15 3 * * *"

jobs:
  digest:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - run: pip install runvouch
      - env:
          RUNVOUCH_KEY: ${{ secrets.RUNVOUCH_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          rv run nightly-digest --evidence-file out/report.md -- \\
            python digest.py --out out/report.md
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: report
          path: out/report.md

Three things about this step. rv run posts the start, runs the command, posts the end with the exit code and the evidence, and returns the command's own exit code, so the Actions status stays truthful. It fails open: if api.runvouch.com is unreachable, the command still runs and the step still reflects its result. Monitoring never breaks the job. And the timeout-minutes is yours to set. Without it, the Actions limits let a job run for up to 6 hours, which for a looping agent is a 6-hour bill.

The cadence is what catches the schedule that was disabled after 60 days of inactivity, or the one that silently stopped when someone changed the default branch. If no run starts within the window, MISSED fires to Telegram, Slack or a webhook. No run, no green, but still an alert.

Cost reporting for an LLM job in GitHub Actions

Actions bills you for minutes. The expensive part of an LLM job is tokens, and Actions knows nothing about them. You have to report them.

If the job is Claude Code, the RunVouch plugin does this through hooks: it reads tokens and cost from the session transcript and attaches them to the run, no changes to the prompt. For a plain API script, report the cost from your own code with the Python or Node client, or call the per-agent ping URL with the amount. Either way the number lands on the run, and BUDGET_RUN fires when it crosses the cap.

What happens then matters, and I want to be precise because this is where I was burned before. A crossed cap pauses the agent. The run that crossed it finishes. RunVouch never kills a running process, because a process killed halfway through writing a file is a worse outcome than one that overspent by a dollar. The next rv run for that agent is refused, with a clear exit and a clear message in the Actions log. The schedule keeps firing, and every refused run is one you can see in the log, until you look at what happened and unpause.

Reporting without a client: the ping URL

Some workflows do not have Python, or the job is a composite action you would rather not touch. Every agent has a ping URL. Three curl steps cover start, end and failure:

- run: curl -fsS https://api.runvouch.com/ping/$TOKEN/start
- run: ./job.sh
- if: failure()
  run: curl -fsS https://api.runvouch.com/ping/$TOKEN/fail
- if: success()
  run: curl -fsS https://api.runvouch.com/ping/$TOKEN

You lose the evidence check and the cost in this mode, but you keep MISSED, FAILED and STALLED, which is already more than the green checkmark gives you.

The free tier covers 3 agents, Solo is $9 a month and Team is $29. The server is MIT and self-hosts in one file with SQLite. The hosted version runs in the EU.

FAQ

Does GitHub Actions alert me when a scheduled workflow does not run? No. Notifications are sent for runs that happen. A schedule that is delayed, disabled after 60 days of inactivity, or removed from the default branch produces no run and no notification. An external cadence check is the only way to notice absence.

Can I use artifacts instead of an evidence check? Use both. Artifacts store the output for 90 days so a person can read it. The evidence check looks at the file at the end of the run and alerts when it is missing, stale or empty. Storage does not alert.

Will the cost cap cancel my Actions job? No. The run that crosses the cap completes. The agent is paused and the next rv run is refused with a message in the log. The workflow itself keeps running on schedule until you unpause.

Related field notes

Try it: free for 3 agents · Docs: Claude Code · cron