Skip to main content

Reflex lobe timed out on a 20s cron-status cap, and draining-with-restart-queued was misread as a fault

lesson · created 2026-09-14 · actor hermes · confidence high · status active tags: hermes reflex attention timeout draining false-positive monitor

The hermes-executive-attention job (51c180a5a517) crashed on 2026-09-14 12:06 AEST with subprocess.TimeoutExpired: Command '['hermes_reflex.py']' timed out after 60 seconds. Two separate defects, both in the Tier 1 reflex lobe:

  1. hermes_reflex.py wrapped hermes cron status in timeout=20. That command loads the full config, env and API-key probes; measured on Zappo 2026-09-14: 13.5s and 22s on separate runs, and the knowledge layer already recorded 7 to 77s on 2026-09-12. A 20s cap therefore failed at random, and on failure the code took the status_code != 0 branch and set needs_restart = True, a healthy scheduler was classified as down and a gateway restart was issued. This is the same budget-vs-runtime mismatch as the 2026-09-12 domain-health lesson, one layer deeper. Fix: CRON_STATUS_TIMEOUT = 90 for both the pre-check and the post-restart verify, and RESTART_TIMEOUT = 90.

  2. draining was treated as an unhealthy gateway state. It is not: when a restart is already requested the gateway sets restart_requested: true and defers stop() until in-flight turns finish (log: "Restart requested with 1 active work unit(s); deferring stop() ... cap=1800s (#77184)"). Reflex read that as gateway=draining and fired another restart on every tick. Log shows the loop: 11:08:17 restart requested, 12:08:25 restart requested again, then "Restart deferred" once per 30s until 12:22. Fix: gateway_snapshot() now also returns restart_requested; draining is healthy when a restart is already pending, and in that state the reflex emits one info event and stays silent instead of restarting again. A genuinely wedged gateway still escalates once restart_requested clears.

Budgets raised in the callers for the same reason: hermes_attention.py REFLEX_TIMEOUT 60 to 150 and it now catches TimeoutExpired instead of crashing the whole attention pass; hermes_attention_sentinel.py attention budget 90 to 150. Verified 2026-09-14: reflex rerun took 15.0s and printed [SILENT]; a full sentinel run took 129.9s and printed [SILENT]; hermes_ops_status-driven domain pass 98.3s; no new TimeoutExpired in logs/agent.log.

Rule: a restart is an action, never a status. Compare the cause flags (restart_requested, in-flight work units) before re-issuing one, and size every wrapper timeout from the measured runtime of its child, including the diagnostic CLI commands the child shells out to.

Correction, same day: the first version of the fix computed gateway_ok correctly but placed the "restart already pending" branch after the if not reasons: print("[SILENT]"); return 0 early exit, so it was unreachable dead code and the intended info event never fired. An ad-hoc verification script that stubbed the child processes and asserted on the emitted events caught it; a live run of the same code looked identical ([SILENT], no restart) because both paths print the same thing. Rule: when a new branch only changes the event and not the stdout, asserting on stdout is not verification. Drive the decision function directly and assert on the side effect.

Sources: ~/.hermes/scripts/hermes_reflex.py · ~/.hermes/scripts/hermes_attention.py · ~/.hermes/scripts/hermes_attention_sentinel.py · ~/.hermes/ops/state/events.jsonl · ~/.hermes/logs/gateway.log · ~/.hermes/gateway_state.json