Skip to main content

Cron system, two schedulers

entity · created 2026-07-09 · actor claude-cowork · confidence high · status active tags: cron scheduling registry

Now​

Two independent schedulers, one registry doc (~/.hermes/CRON_REGISTRY.md, last full audit 2026-06-13):

  • Hermes cron: ~/.hermes/cron/jobs.json (217 jobs as of 2026-09-27, 215 enabled; disabled are heating-profile and air-fryer-done-watch; jobs is a top-level array, file wrapped as {jobs, updated_at}). --no-agent --script pattern = silent-when-clean watchdog. CLI quirk (v0.18.0): hermes cron create rejects positional prompts, append agent jobs directly to jobs.json (backup first, copy an enabled job as template). Gateway rewrites fire_claim/state live, so never hand-edit the file while the scheduler is running.
  • Execution ledger: ~/.hermes/cron/executions.db, tables executions (id, job_id, status, started_at, finished_at, error, delivery_outcome) and cron_incidents (job_id, error_sig, state, failure_type, first_seen_at, last_seen_at, closed_at). A run recorded completed is not proof the task succeeded; check the job's own output. All 1,002 execution rows from 2026-09-06 to 2026-09-13 read completed except two in-flight rows. Re-read 2026-09-20: 1,000 rows since 2026-09-13 read completed and one (f24afd3d344e, this job) was in flight, so the ledger held no failures across the week either. delivery_outcome is a second field to read: suppressed means the job ran and produced silence by design.
  • OpenClaw cron: gateway-managed, store ~/.openclaw/cron/jobs.json + sqlite state (34 jobs). Agent jobs carry per-job --fallbacks (local ollama/gemma4:31b since 2026-07-09) and --timeout-seconds.

Rules that keep it sane (registry "Principles"): one job one purpose; no duplication (cross-system overlap only as producer→reviewer chain); no same-minute clusters on OpenClaw (simultaneous isolated spawns stall at runtime-plugins phase); explicit delivery targets (never last:); house format Verdict/Latest/Active/Status/Next.

Producer→consumer chains (do not reorder): tech-leaders 06:30→07:05→07:30→07:35; arXiv 12:17→12:55; engagement scout → OC review (08:40/45, 20:40/45, Sun 15:30/35); daily debrief 20:40→20:45→21:00; cameras collect→analyze (06/12/18:00 → 03/06/12/18:10).

Fleet-layer jobs (2026-07-09): bot-coordination-sync 06:50, knowledgelayer-sync 00/06/12/18:25, fleet-layer-guard 07:12, weekly-brain-distill Sun 05:05, knowledgelayer-weekly-digest Sun 07:45 (human-voiced brain summary to home room), knowledgelayer-hourly-digest :07 hourly (change-triggered note, silent when brain unchanged), weekly KG rebuild Sun 04:16.

Weekly synthesis job (knowledge-synthesis-weekly, id f24afd3d344e, Sun 06:00, deliver telegram:[redacted-id]): failed 2026-08-23 with RuntimeError: HTTP 502: Post "https://ollama.com:443/v1/chat/completions": dial tcp: lookup ollama.com: no such host, a transient DNS failure on the ollama.com upstream, not a code fault. Same job also hit a Telegram delivery timeout on that run. It is agent-backed (model deepseek-v4-flash:0731-cloud), so it depends on the ollama.com cloud endpoint being resolvable.

  • Pausing a recurring job has no auto-resume. hermes cron <cmd> pause <job_id> sets enabled: false and state: paused (cron/jobs.py::pause_job, verified 2026-09-20 against v0.21.3). There is no paused_until field and no scheduler support for one, resume_job only recomputes next_run_at from now, and the CLI's hermes cron resume --at <ISO> calls rearm_oneshot, which re-arms a ONE-SHOT and does not apply to a recurring job. Pattern for any temporary pause (holiday, maintenance): pause the job, then create a no_agent one-shot that runs a tiny hermes cron resume <job_id> script at the intended resume time. Arm it a little BEFORE the job's first real slot so that slot is not missed. Verified live 2026-09-20 on heating-profile (db57480d29dd, */15 10-14 * * *) paused for a week with one-shot b9575bdc478e at 2026-09-27T09:45 to catch the 10:00 slot; the resume command was exercised against the real CLI and the job re-paused to leave the window intact. Also note hermes cron list shows paused jobs only with --all.

Log​

  • 2026-10-04 (weekly synthesis, Talent Watch halt burst has cleared): The 2026-09-27 entry above read Australian AI Talent Watch (fe0814a7945e, every 130m) as producing the tool-loop halt constant on 27 of 50 runs and called it "the fleet's most visible cron fault." That is no longer true. The 50 retained output files now span 2026-09-29 to 2026-10-04; 41 bodies are exactly [SILENT] and the other 9 are full talent reports. Zero halt-constant outputs remain, so the earlier halt period has aged out of the retained set entirely rather than merely going unobserved. executions.db shows every grouped job completed since 2026-09-27. Two residual quality faults show in the surviving reports: two runs on 2026-10-02 and one on 2026-10-04 append a ⚠️ File-mutation verifier block complaining it could not overwrite a stale hermes-verify-*.py in $TMPDIR, and one run states outright "this runtime has no shell or code-execution tool." Those are verifier-hygiene noise, not job failure. Sources: ~/.hermes/cron/output/fe0814a7945e/ (50 files, 2026-09-29 to 2026-10-04), ~/.hermes/cron/executions.db, read 2026-10-04 07:00-07:15 AEDT., hermes.

  • 2026-09-27 (weekly synthesis): ~/.hermes/cron/jobs.json holds 217 jobs, 215 enabled, 2 paused (heating-profile and air-fryer-done-watch). ~/.hermes/state/cron-rigour-audit-alert-state.json reported 245 jobs across both schedulers at 2026-09-26T21:06 (215 Hermes, 28 OpenClaw; 55 agent-backed, 190 scripted; 0 critical). Two cron-quality items from the week's own output: weekly-memory-reindex (3f8b2d29e66f) reports the manifest covering 217 jobs, 161 memory files, 260 scripts and 207 wiki pages, wiki lint finding 1 issue in 207 pages and 11 domain pages over 30 days stale; wiki-compiler (3fef5cddd7f2) reports 13 pages from a 200-entry KB query in every run since 2026-09-18, so that source may be capped. The week's most visible cron fault is not a scheduler failure: Australian AI Talent Watch (fe0814a7945e, every 130m) delivered the hardcoded tool-loop halt constant on 27 of its 50 runs between 2026-09-22 and 2026-09-27, so the fleet's most frequent user-facing job produces noise instead of a report about half the time. See the lesson file named below, updated 2026-09-27. Sources: live cron/jobs.json, the rigour-audit state file, the week's cron output directories, read 2026-09-27 05:15-06:07 AEST., hermes.

  • 2026-09-27 (publish-site crontab line still points at the stale clone; the job has never once succeeded): The 2026-09-20 lesson lessons/2026-09-20-the-kl-command-on-path-reads-a-stale-clone-so-duplicate-chec.md fixed the kl PATH shim and explicitly left the crontab line open. It is still open. crontab -l on 2026-09-27 05:15 AEST carries 40 */6 * * * $HOME/Projects/knowledgelayer/tools/publish-site.sh >> $HOME/.hermes/logs/publish-site.log 2>&1 unchanged, which runs the stale clone (/Users/dwayne/Projects/knowledgelayer, HEAD 5fb5499, 2026-08-24) instead of the canonical brain. Over the log's full span (307 lines, 2026-07-10 to 2026-09-27) the tally is 233 fatal: could not read Username for 'https://github.com' and 49 BLOCKED: ai-hub working tree dirty, against zero published knowledgelayer snapshot and zero no change; nothing to publish markers: the job has never produced a success outcome. The root cause is auth, not the dirty tree: the stale script relies on the machine's ambient git credential helper (its header says "whatever git auth the machine already has, gh credential helper on Pro"), which in a cron context is an osxkeychain helper holding no GitHub credential, while the canonical script sets credential.helper '!gh auth git-credential' and publishes from a dedicated ~/.cache/publish-site/ai-hub clone hard-reset to origin/main. The two same-named scripts (stale 2650 bytes mtime 2026-07-10; canonical 3743 bytes mtime 2026-07-17) share no publish strategy, per diff. Fixed in this pass: nothing. Note the 2026-09-19 entry lessons/2026-09-19-publish-site-old-script-reverts-humanizer.md already showed a manual run of the stale script republished an Aug-24 snapshot and reverted the humanizer pass - a hand run in a clean tree stays destructive. Verification required when the fix lands: run the canonical script once by hand, then require a dated published knowledgelayer snapshot <date> or no change; nothing to publish line in ~/.hermes/logs/publish-site.log. Source: crontab -l on macbook-pro-3 2026-09-27 05:15 AEST, ~/.hermes/logs/publish-site.log, diff of both tools/publish-site.sh, git log -1 in the stale clone. Lesson: lessons/2026-09-27-a-stale-clone-kept-running-from-crontab-after-the-path-shim-.md. - hermes.

  • 2026-09-21 (ai-token-economics-weekly silent-source repair): ai-token-economics-weekly (99ddaf4ac09e, 58 7 * * 1, deliver telegram:[redacted-id]) had reported "No structured local pricing memory found" on every run since at least 2026-08-24, because both copies of ai_token_economics_report.py (~/.hermes/scripts/, the copy the scheduler actually executes, and /Users/dwayne/clawd/scripts/) read a dead directory, /Users/dwayne/.openclaw.pre-migration/memory/. That directory does not exist; the four files (ai-token-economics.json, ai-token-economics-summary.txt, gpu-cost-trends-summary.txt, gpu-cost-trends.json) only survived inside /Users/dwayne/.openclaw.pre-merge-[redacted-id].tar.gz (archived 2026-04-13). Restored them to the live /Users/dwayne/clawd/memory/ and made MEMORY_DIR resolve the first existing candidate from clawd/memory then the legacy path. Also fixed two rendering faults in the .hermes copy: selected_lines(GPU_SUMMARY, 6) printed the same boxed note six times instead of GPU rows, and selected_lines matched only provider-name/model keywords so price rows were dropped, now it also matches $ and %. Verified by running both copies: exit 0, 153-entry pricing_history read, latest local pricing memory 2026-04-13T08:19:31.585358, GPU rows for AWS p4d.24xlarge A100 ($4.1 on-demand, $1.23 spot). Source: live filesystem reads, ~/.hermes/cron/output/99ddaf4ac09e/, ~/.hermes/cron/executions.db, 2026-09-21 AEST. The underlying staleness is unresolved: the restored pricing snapshot is dated 2026-04-13, so the weekly report will keep quoting April prices until a refresh job repopulates it., hermes.

  • 2026-09-20 (weekly retune, 15:30-15:48 AEST): 7-day cron health from executions.db: 1001 runs, 993 completed, 7 failed, 1-2 running at read time, 213 of 215 jobs enabled. All 7 failures are the same job, air-fryer-done-watch (0aeb740a318c, */1, no_agent, deliver local), between 14:59 and 15:05 AEST, each exiting 1 with {"ok": false, "status": "observation_unavailable", "error": "sensor.apple_macbook_pro_power state is unavailable"} (the 14:59 run got HTTP 404 while the entity was still absent). Root cause is not the script: Home Assistant restarted at 14:59 AEST, and HA history shows sensor.apple_macbook_pro_power unavailable from 05:00:00 UTC to 05:06:30 UTC (15:00-15:06 AEST), then 0.0. The job recovered on its own and has run completed since 15:34. Design note: a one-minute observation job that returns exit 1 when its source entity is unreadable makes an upstream restart look like a job fault; observation_unavailable is a better third status than a failure. Two duplicate definitions of the same job exist (daac573675c7 disabled, 0aeb740a318c enabled) - harmless but worth removing the disabled twin. Source: ~/.hermes/cron/executions.db, ~/.hermes/cron/jobs.json, /Users/dwayne/.homeassistant/home-assistant.log (+.1), live HA /api/history/period, 2026-09-20. - hermes.

  • 2026-09-20 (holiday pause): heating-profile (db57480d29dd) paused for Dwayne's week on the Gold Coast, at his request. Cron has no auto-resume for a recurring job, so a no_agent one-shot (heating-profile-resume-after-holiday, b9575bdc478e, script ~/.hermes/scripts/resume_heating_profile.py) is armed at 2026-09-27T09:45 to re-enable it just before the 10:00 protected-morning-hold slot. Both the pause and the resume path were exercised against the live CLI and the job was re-paused to leave the window intact. The underlying limit is now recorded in the Now block., hermes.

  • 2026-09-20 (07:31 morning digest, 2 error jobs triaged): the digest reported 2 erroring jobs out of 213 enabled. Both were triaged and one was repaired. (1) daily-tech-leaders-status (2737efb76f44): TypeError: record_event() takes 0 positional arguments but 4 were given at ~/.openclaw/workspace/bin/generate-daily-status:234; record_event is keyword-only (scripts/knowledge_graph.py:242). Fixed to keyword args and verified end-to-end (exit 0, latest_brief_date=2026-09-20). (2) globird-ac-optimizer (b9d72391d9cf): the 07:00 run failed on the dead automation.smart_ac_cool_down entity. The 404 guard was added to ~/.hermes/scripts/winter_ac_baseline.py at 07:11, after the 07:00 fire, so the job's last_status still reads error; a rerun of the same 07:00 phase exits 0 with a valid receipt. Its next_run_at is 11:00. Both incidents are cron_incidents rows in state detected, which clear on the next successful tick. Source: live jobs.json, executions.db and cron_incidents reads plus reruns, 2026-09-20 07:31-07:40 AEST., hermes.

  • 2026-09-20 (weekly synthesis): Job count corrected again, jobs.json holds 214 jobs, 213 enabled, with only air-fryer-done-watch disabled (source: live read of ~/.hermes/cron/jobs.json, updated_at 2026-09-20T06:01:04+10:00). The executions ledger held 1,000 completed rows since 2026-09-13 and one in-flight row for this job, so no scheduler-level failure landed in the week. Added delivery_outcome to the Now block: it records suppressed for jobs that ran and returned silence by design, which is how [SILENT] watchdogs appear in the ledger.

  • 2026-09-26 (weekly-project-janitor cannot push oversize repos): cb63565fd572 weekly-project-janitor (Mondays 10:15) runs ~/clawd/scripts/ProjectJanitor.py, which staged with git add . and force-pushed the WIP branch. It had no size guard and no credential guard, and run_cmd capped every subprocess at 30s. Effect: any repo holding a single file over GitHub's 100 MB limit failed on EVERY run, because the oversized blob stays in history once committed, so the push can never self-heal. First casualty was the claw-proj-removed-mindroom-backup-20260919 mirror (the decommissioned MindRoom tree), whose single commit 848349a4 carried a 424 MB log, a 191 MB log, the 186 MB zig binary and a 120 MB team db; its remote had zero branches, so nothing ever landed. The same run also timed out pushing dodge-wip-20260921 because of the 30s cap. Fixed in c12e4bc (claw-core): stage only paths passing a guard (50 MB ceiling, credential names, credential directories whose names contain "credential"), log what was skipped, give the push 600s. Guarding at STAGE time matters, not after - git add . writes the blob into .git before any unstage. Verified end-to-end with isolated-HOME runs on scratch repos; falsified by running the pre-fix script on an identical fixture, which committed the 60 MB blob. Repo repaired by rebuilding the single commit without oversized/credential/audio paths and force-pushing (remote was empty; pack 36 MiB, 1186 blobs, largest 3.57 MB). Backup scripts/ProjectJanitor.py.bak-pre-largefile-guard-20260926 (sha256 44d12457f656a27f2341f0fd760cfefa074c6074e754b6d19d9c41a515ba84b2)., hermes.

  • 2026-09-15 (fire-fence recurrence + voice stale source): the 2026-09-14 fence burst recurred in the morning window. Six runs failed between 07:56:01 and 08:14:02 AEST (98dad8a650cc script timeout, 7eb6264fdd4e fire claim ownership lost, and ac50b5c52e5b/a219e899392e/871469f27c34/ bc5db7e029e6/d2dcdd1402ef interrupted by shutdown) with nine local fire fence timeouts between 08:03:37 and 08:13:35 across nine job ids. gateway_state.json stayed running with restart_requested: false and no SIGTERM in gateway.log, so the interruption source is still unverified; host load averaged 38-62 on 12 cores. Downstream: bc5db7e029e6 wrote no output, so the xAI voice companion 6d7c8f85e925 exited 1 at 08:15 with stale source, waiting for today's report. Both were recovered with hermes cron run (08:40) plus xai_voice_research_summary.py --force. 98dad8a650cc daily-arxiv-site-publish remains error with a 300s script timeout. See lessons/2026-09-15-fire-fence-burst-recurrence-and-voice-stale-source.md.

  • 2026-09-14 (fire-fence lock timeouts): ten ERROR cron.jobs: Timed out waiting for local fire fence /Users/dwayne/.hermes/cron::\<job_id>; failing closed lines between 13:13:38 and 13:21:47 AEST across nine job ids; eight runs recorded failed in cron/executions.db. The "local" wording means the in-process per-job threading.RLock in _fire_job_lock() (30s cap, jobs.py:94), not the cross-process .fire-<uuid5>.lock flock. Delivery and recovery were unaffected: affected jobs kept delivering and every one passed on its next tick. cron-rigour-audit (6a4fbe99815e) still shows last_status=error until its 21:06 tick. See lessons/2026-09-14-fire-fence-lock-timeout-under-concurrent-cron-burst.md. , hermes.

  • 2026-09-14 (workspace rehatch breakage): the OpenClaw workspace rehatch (pre-rehatch backup at ~/.openclaw/workspace-pre-rehatch-20260831/) did not carry every helper script into the live ~/.openclaw/workspace/. weekly-fsd-tracker (Hermes job 212ad6760d34, 0 14 * * 1, script fsd_tracker_report.py) failed with Failed (code 2) ... can't open file '/Users/dwayne/.openclaw/workspace/scripts/fsd-tracker.py': [Errno 2] No such file or directory. Restored scripts/fsd-tracker.py and memory/fsd-tracker.json from the backup; verified by running the Hermes wrapper end-to-end (exit 0, full report rendered). 13 other backup-only scripts are absent from the live workspace (ha-health-audit.py, people_moves_report.py, publish_post_model_smoke.py, telegram_to_nexus_bridge.py and others); no enabled Hermes cron job references any of them as of this read, so only the FSD job was affected. Check comm -23 of the two scripts/*.py listings after any workspace rehatch., hermes.

  • 2026-09-14 (lesson): hermes_reflex.py wrapped hermes cron status in a 20s cap (measured 13.5-22s on Zappo), and a cap failure was read as "scheduler down", triggering gateway restarts on a healthy stack. Separately, gateway_state=draining with restart_requested: true (gateway deferring stop() for in-flight turns, #77184) was misread as a fault and re-restarted every tick. Fixed: 90s budgets, draining-with-pending-restart is healthy and silent, caller budgets raised (attention 60→150 catches TimeoutExpired, sentinel 90→150). See lessons/2026-09-14-reflex-cron-status-timeout-and-draining-restart-loop.md., hermes.

  • 2026-09-13 (weekly synthesis): Corrected the stale job count (was 56 jobs, ~34 enabled) to 157 jobs, all enabled, and recorded the jobs.json wrapper shape. Added the executions.db location, both table schemas, and the rule that a completed ledger row is not task success. Sources: live ~/.hermes/cron/jobs.json and ~/.hermes/cron/executions.db reads 2026-09-13., hermes.

  • 2026-08-24: Recorded knowledge-synthesis-weekly 2026-08-23 failure (ollama.com DNS lookup failure, HTTP 502) + Telegram delivery timeout. Source: ~/.hermes/cron/jobs.json., hermes.

  • 2026-07-09: created from CRON_REGISTRY.md + live job stores, claude-cowork.

Sources: ~/.hermes/CRON_REGISTRY.md · ~/.openclaw/cron/jobs.json · ~/.hermes/cron/jobs.json