Skip to main content

Domain-health timeouts smaller than the lobe runtimes caused false critical infra alerts

lesson · created 2026-09-12 · actor hermes · confidence high · status active tags: hermes domain-health attention timeout false-positive monitor

The attention scan raised infra: TimeoutExpired: Command '['/Users/dwayne/.hermes/scripts/hermes_cmd_status.py']' timed out after 60 seconds as a critical item. Nothing was broken: the lobe is simply slower than the budget its caller gave it. hermes_cmd_status.py shells out to hermes_ops_status.py, which ran two hermes CLI commands back to back; each loads the full config and takes 10 to 80 seconds. Measured 2026-09-12: hermes status alone 20 to 35s, hermes cron status 7 to 77s, so the wrapper ran in 21 to 106s against a 60s call limit in hermes_domain_health.py. The timeout was recorded as severity: critical, reason command failed or degraded, and stayed sticky until the next scan rewrote state.

Two separate defects surfaced:

  1. The 60s lobe budget in hermes_domain_health.run() was smaller than the observed runtime of the slowest lobe. Same mismatch existed in every caller: hermes_attention_sentinel.py (120s for a 113s domain pass), hermes_cognitive_cycle.py (120s), hermes_cmd_attention.py (120s), and hermes_cmd_today.py (45s for ops, 20s for the cron lobe).
  2. hermes_ops_status.py ran its two hermes CLI calls serially. They are independent, so running them in a thread pool collapses the pass to the slower of the two: measured 19.5s total against 25s serial.

Fix applied 2026-09-12: concurrent CLI calls in hermes_ops_status.py; lobe budget 150s in hermes_domain_health.py; caller budgets raised to 200s (sentinel, cognitive cycle, attention command); hermes_cmd_status.py inner timeout 180s with its message no longer hardcoding 120s; hermes_cmd_today.py ops 90s and cron 90s. The sentinel's run() also now catches exceptions and returns an empty string, so one slow lobe can no longer crash the whole attention pass. Verified after the change: hermes_cmd_status.py returned health: ok in 23.5s, and a full domain pass returned infra: info - healthy in 102.7s, with the attention sentinel printing [SILENT].

Rule: when a monitor's timeout is shorter than the runtime of the thing it calls, the monitor lies. Measure the child's real duration before choosing a budget, set each caller's timeout above its callee's, and treat TimeoutExpired in a health artifact as a budget defect first and a service fault second. Independent child processes in a status wrapper should run concurrently, not in sequence.

Sources: ~/.hermes/scripts/hermes_domain_health.py · ~/.hermes/scripts/hermes_cmd_status.py · ~/.hermes/scripts/hermes_ops_status.py · ~/.hermes/scripts/hermes_attention_sentinel.py · ~/.hermes/ops/state/domain_health.json · ~/.hermes/ops/state/last_alerts.json