Two cron ledgers, two clearing paths: ack for incidents, a real run for job status

_entry

Date: 2026-09-20 Lane: hermes Confidence: high

The distinction

Hermes keeps cron failure state in two separate places, and they do not clear each other.

store	read by	clears when
job fields last_error / last_delivery_error	hermes cron doctor, Brainstem ops status	the job next runs and the outcome is recorded
hermes cron incidents ledger	hermes cron incidents	the job succeeds (auto-resolved) or hermes cron incidents ack <id>

hermes cron incidents ack closes an incident row. It does NOT touch the job's last_status. hermes cron run rewrites the job fields. It does NOT close a stale incident row. Clearing one and reporting the other as fixed is the trap.

Worked example, 2026-09-20

The Brainstem ops status read degraded with three faults. All three were stale records contradicted by the deployed code:

• daily-tech-leaders-status: recorded TypeError: record_event() takes 0 positional arguments but 4 were given; the deployed file passes keyword arguments to a keyword-only function and its mtime (07:33) was after the 07:30 failure.
• globird-ac-optimizer: recorded HTTP 404 on automation.smart_ac_cool_down, an entity retired on 2026-09-15. The script now guards both the GET and the POST.
• zappo-daily-combined: delivery_failed on a Telegram timeout, with a 1,178-character payload, well under the 4,096 limit. Transport, not size or config.

Resolution:

• hermes cron run <id> for the two script jobs -> both last_status: ok, cron doctor dropped 3 issues to 1, and both incidents auto-resolved.
• hermes cron run c972d50650ac -> the job actually delivered, and last_status moved from delivery_failed to ok.
• Two further alerted incidents from 2026-09-06/07 sat on jobs that had run clean since (13th and 14th September). Those were closed with hermes cron incidents ack <id>.

End state: cron doctor "no issues" over 213 jobs, zero unresolved incidents, Brainstem sentinel emits [SILENT].

Rules

• A cron error field is historical, not live. Before acting, run the exact script the job runs and check its mtime against the failure timestamp. A fix applied minutes after a failed run leaves a record that repeats in every derived report.
• A delivery timeout on a small payload is transport weather. Prove reachability with a live send before touching delivery config.
• The Brainstem ops status is derived from these fields, so it inherits every stale record. Clear the record; never edit the report.
• An incident sitting on a job that has since run clean is stale by definition. Ack it with evidence (later successful runs), not by assumption.
• [SILENT] from the sentinel is the healthy state, not a missing report.