Fire-fence burst recurred 2026-09-15; a failed upstream run makes the voice companion report a stale source

_{lesson · created 2026-09-15 · actor hermes · confidence medium · status active tags: hermes cron locks scheduler false-positive concurrency voice}

On 2026-09-15 the fire-fence burst pattern first recorded on 2026-09-14 (see lessons/2026-09-14-fire-fence-lock-timeout-under-concurrent-cron-burst.md) recurred in the morning tick window. Between 07:56:01 and 08:14:02 AEST the ledger recorded six failed runs:

run (job)	error
98dad8a650cc daily-arxiv-site-publish	Script timed out after 300s
7eb6264fdd4e zappo-fim-check-hourly	Fire claim ownership lost before terminal completion
ac50b5c52e5b ai-hub-main-guard	Interrupted by shutdown before terminal completion
a219e899392e pro-nexus-zombie-cleanup	Interrupted by shutdown before terminal completion
871469f27c34 zappo-home-job	Interrupted by shutdown before terminal completion
bc5db7e029e6 nightly-research-summary-delivery	Interrupted by shutdown before terminal completion
d2dcdd1402ef go-live-readiness-dashboard-refresh	Interrupted by shutdown before terminal completion

errors.log logged nine Timed out waiting for local fire fence ... ; failing closed lines between 08:03:37 and 08:13:35 across nine distinct job ids, and agent.log logged five matching fire claim ownership lost; interrupting stale run lines. Hourly counts confirm the 07:00 (47) and 08:00 (89) hours of 2026-09-15 carry almost all of the day's fence errors; the host was under heavy load (1-minute load average 38-62 on 12 cores, swap used ~10.7 GB of 12 GB) from the Android emulator, a Vagrant/Colima VM and a node process at 100%+ CPU.

Not established: which component requests the shutdown that produces Interrupted by shutdown before terminal completion. gateway_state.json read gateway_state: running, restart_requested: false, active_agents: 2, Telegram connected, and no SIGTERM or gateway.run: Received SIGTERM line appears in gateway.log for 2026-09-15. Treat the shutdown cause as unverified, as on 2026-09-14.

The downstream effect worth remembering

bc5db7e029e6 (nightly-research-summary-delivery, 6 8 * * *) failed at 08:08:27 and wrote no output file, so at 08:15 the xAI voice companion 6d7c8f85e925 (xai-voice-research-summary, 15 8 * * *) found only cron/output/bc5db7e029e6/2026-09-14_08-06-28.md and exited 1 with xAI voice delivery blocked: Research summary: stale source, waiting for today's report (2026-09-14_08-06-28.md). The source_rejection() guard in ~/.hermes/scripts/xai_voice_delivery_report.py requires the source's filename date to match today in Australia/Sydney, so one failed upstream run takes the companion down with it and the error text points at the companion, not the upstream job.

Recovery (verified 2026-09-15)

hermes cron run bc5db7e029e6 completed in 31s (08:40:32, delivery_outcome=suppressed because deliver: local), writing cron/output/bc5db7e029e6/2026-09-15_08-40-32.md. Running xai_voice_research_summary.py --force against that fresh source produced ~/.hermes/voice-deliveries/xai_voice_research_summary/20260915_084144_2026-09-15_08-40-32.ogg (Ogg Opus, 867 KB, verified with file). hermes cron run 871469f27c34 also completed and delivered. hermes cron doctor then showed only 98dad8a650cc still red.

Rules:

• When a voice-companion job reports stale source, waiting for today's report, check the upstream job's ledger row before touching the companion. Fix the upstream run, then re-run the companion with --force; the companion is rarely the fault.
• A local fire fence burst plus Interrupted by shutdown rows is a scheduler contention event, not a per-job defect. Confirm delivery and next-tick success before repairing job state, and never clear the .fire-*.lock files.
• hermes cron run <job_id> is the sanctioned manual re-run. Calling it while the scheduler already has the run in flight answers Job is already being fired by the scheduler; not run again., that is a guard, not a failure, and it also means the current agent session IS that run.

2026-09-16 recurrence: a slow upstream, and the companion now waits

The same class of failure recurred at 20:08 AEST on 2026-09-16, but the upstream did not fail: the 20:05 daily-state-of-the-union (4fa833abcfab) self-repaired its report generator mid-run and ran until 20:39:57, so the 20:08 voice companion found only 2026-09-15_20-05-23.md and blocked.

Fix, verified 2026-09-16: xai_voice_delivery_report.py now has wait_for_today_source(job_id, wait_seconds=WAIT_FOR_TODAY_SECONDS) with WAIT_FOR_TODAY_SECONDS = 150. On a stale source rejection in run(), the script polls the producer's output directory for a same-day file before reporting the block. The 150s budget stays under the global cron.script_timeout_seconds (300) so synthesis still has room. Ad-hoc verification (temp script, six cases) covered: stale-only source waits the full bounded time rather than returning instantly, today's file present returns immediately, a file landing mid-wait is picked up, a missing output directory returns None instead of raising, the budget stays under the script timeout, and the stale source, waiting for today's report wording is unchanged for the never-arrives case.

• A companion script that requires a same-day source must wait for it, not just reject. A scheduled gap of minutes is not a guarantee the producer has finished, especially when the producer is self-repairing.
• Keep such a wait bounded and under cron.script_timeout_seconds; an unbounded poll turns a stale report into a killed script.

---

_{Sources: ~/.hermes/logs/errors.log · ~/.hermes/logs/agent.log · ~/.hermes/cron/executions.db · ~/.hermes/cron/jobs.json · ~/.hermes/cron/output/bc5db7e029e6/}