Docs › Ollama nightly batch job monitoring with RunVouch
Ollama nightly batch job monitoring with RunVouch
Ollama nightly batch job monitoring where nothing costs money: with a local model the failure modes change. The process hangs while the model loads, the GPU is taken by another job, or a model update changes output length by half. None of them cost money, all of them cost the result.
How it runs on Ollama and local LLM batch jobs
Wrap the batch script with rv run and set --max-runtime on the agent so a hang becomes STALLED.
Store the key
An environment variable in the cron environment.
Wrap the job
rv agent nightly-classify --cadence 24h --max-runtime 2h --evidence 0 1 * * * rv run nightly-classify --evidence-file /data/labels.jsonl -- python classify.py --model llama3.1
rv fails open: if RunVouch is unreachable the job still runs and you get one warning line.
Register the cadence and caps
rv agent nightly-report --cadence 24h --grace 30m --max-runtime 1h --evidence --cap-run-cost 2
Register the agent once, from anywhere with the key. Cadence is what turns a schedule that stopped into an alert; --evidence makes a run without evidence a failure; the caps pause the agent when it overspends.
What goes silent on Ollama and local LLM batch jobs
- A hang has no exit code; STALLED fires when max runtime passes without an end.
- Output size drift (DRIFT) is the first sign a model update or a prompt change broke the pipeline.
- Zero cost is fine; leave the cost caps unset and keep evidence and drift.
What Ollama and local LLM batch jobs does not tell you
Ollama logs requests to its server log and returns per-request timings in the API response; there is no job history, no notion of a schedule and no alerting. A request that blocks while a model is loaded onto a busy GPU has no timeout unless the client sets one, and a model pulled to a new version answers with a different length without saying so.
What RunVouch detects
| Alert | What it means here |
|---|---|
| MISSED | cadence plus grace passed and no run started |
| FAILED | non-zero exit or a reported failure, with the stderr excerpt |
| NO_EVIDENCE | the run said ok but the file, URL or assertion you required is missing |
| STALLED | a run started and never ended within max runtime |
| RETRY_STORM | the same tool called with identical input many times in one run |
| BUDGET_RUN / BUDGET_DAY | cost cap crossed; the agent is paused until you resume it |
| DRIFT | duration or output size far off its 7-run baseline |
Set up in two minutes
- Get a free key (3 agents, no card) and store it where this page says.
- wrap the job: copy the snippet above into the scheduled job.
- Register the cadence once:
rv agent nightly-report --cadence 24h --grace 30m --evidence, or let the first run create the agent and set the cadence on the dashboard. - Send one test alert:
curl -X POST https://api.runvouch.com/v1/settings/test-alert -H "X-API-Key: $RUNVOUCH_KEY". The next missed, failed or empty run reaches the same channels.
Also running agent frameworks?
OpenClaw (verified-run skill). LangGraph. CrewAI. AutoGen. The full list of runtimes is on integrations, and the same agent can report from several of them: one job, one cadence, wherever it happens to run.
Need a key? Get a free key · Stuck? contact