Skip to main content

Stale cron error records outlive the fix that resolved them

entry

Date: 2026-09-20 Lane: hermes Confidence: high

The symptom​

The Hermes Brainstem ops status reported degraded, naming three faults:

  • jobs: 213 enabled, 2 errors, 1 delivery error
  • cron errors: globird-ac-optimizer, daily-tech-leaders-status
  • delivery errors: zappo-daily-combined

All three were stale records, not live faults. Each was contradicted by the deployed code, proved by running the exact script the job runs.

Evidence per fault​

daily-tech-leaders-status (2737efb76f44). Recorded error: TypeError: record_event() takes 0 positional arguments but 4 were given. The deployed ~/.openclaw/workspace/bin/generate-daily-status passes keyword arguments, and knowledge_graph.record_event is keyword-only (scripts/knowledge_graph.py:242, def record_event(*, source, job, summary, ...)), so the call is correct. The file's mtime is 07:33, after the 07:30 failure. Running it directly and through the job's wrapper both exit 0.

globird-ac-optimizer (b9d72391d9cf). Recorded error: HTTP 404 on automation.smart_ac_cool_down. That entity was removed when the cool-down automation was retired on 2026-09-15. winter_ac_baseline.py now guards both the GET (line 194) and the POST (line 172) against a missing automation, treating an absent entity as disarmed. The script exits 0 and reports ok: true.

zappo-daily-combined (c972d50650ac). Recorded as delivery_failed: Telegram send failed: Timed out. This is transport, not config. The response was 1,178 characters, well under Telegram's 4,096 limit, so size was not the cause. A live send to telegram:[redacted-id] succeeded. The job fires daily at 11:17, so the next successful run clears the field.

How the records cleared​

hermes cron run <id> re-executes the job and rewrites its status. Running 2737efb76f44 and b9d72391d9cf returned Ran now: succeeded, both flipped to last_status: ok, and hermes cron doctor dropped from 3 issues to 1.

hermes cron incidents is a separate ledger from the job's last_error: the two script incidents auto-resolved to state resolved once the jobs succeeded. The delivery error is NOT tracked as an incident, so it only clears when the job next delivers.

Lessons​

  • A cron last_error / last_delivery_error is a historical field, not a live signal. Before acting on one, run the exact script the job runs and compare.
  • Check file mtime against the failure timestamp. A fix applied minutes after a failed run leaves a stale record that repeats in every downstream report until the job is re-run.
  • Two distinct ledgers hold cron failures: the job's own last_error fields (read by cron doctor) and cron incidents. Acknowledging or resolving one does not touch the other.
  • A delivery timeout on a small payload is transport weather. Prove reachability with a live send before changing any delivery config.
  • The Brainstem ops status is a derived report over these fields, so it inherits every stale record. Clearing the record is what makes the report healthy, not editing the report.