Local fire-fence lock timeouts during a concurrent cron burst marked healthy jobs failed
hermes · confidence medium · status active
tags: hermes cron locks scheduler false-positive concurrency
On 2026-09-14 between 13:13:38 and 13:21:47 AEST the Hermes scheduler logged ten
ERROR cron.jobs: Timed out waiting for local fire fence /Users/dwayne/.hermes/cron::<job_id>; failing closed lines, across nine distinct jobs. Eight of those runs were then recorded failed in
cron/executions.db with either Interrupted by shutdown before terminal completion. or
Fire claim ownership lost; stale result was discarded.. This is the first occurrence recorded in
the ledger: select count(*) from executions where error like '%fire fence%' or error like '%Interrupted by shutdown%' or error like '%ownership%' returns 8, all dated 2026-09-14.
The message is about the local fence, not the file lock. _fire_job_lock() in hermes-agent/cron/jobs.py
keys a threading.RLock on f"{cron_dir.resolve()}::{job_id}" and acquires it with
_JOBS_LOCK_TIMEOUT_SECONDS (30.0s, jobs.py:94). Logging "local fire fence" means that in-process
acquire expired; the cross-process flock on .fire-<uuid5>.lock logs "Timed out waiting for fire
fence" without the word local. Failing closed is the designed behaviour: the tick yields False and the
run does not proceed.
Observed context, all in the same window: a large concurrent burst starting at 13:08 to 13:13
(51c180a5a517, 6a4fbe99815e, b8263a1030af, 4ceb83f2ee96, eae30e4b5008 all dispatched inside
five minutes), a pro-NEXUS Finance Monitor run that occupied the scheduler from 13:05:38 to 13:33:12,
and ha-exposed-camera-stills-refresh (835ebfa7498e) whose terminal call hit the 420s sequential tool
budget (agent.deadline: 'terminal.wait:LocalEnvironment' timed out after 402.0s at 13:16:02).
Recovery was automatic and no output was lost. Jobs whose runs were marked failed still delivered their
messages: b8263a1030af delivered Telegram message 66719 at 13:13:46, 51c180a5a517 delivered 66719's
neighbour at 13:13:46 and ran clean again at 13:23:13, and every affected job's next tick succeeded
(b8263a1030af ok 15:08:14, 4ceb83f2ee96 ok 15:07:22, 51c180a5a517 ok 15:06:48, d2dcdd1402ef ok
15:10:01, 33d94fde6946 ok 15:09:09, eae30e4b5008 ok 15:08:05). One job, cron-rigour-audit
(6a4fbe99815e, 6 7,13,21 * * *), still reads last_status=error with
Interrupted by shutdown before terminal completion. because its next tick is 21:06; its 13:06 run's
report did deliver (message 66720 at 13:17:17).
Not established: why the per-job RLock was contended for ten different job ids in a ten-minute window. The per-job keying means cross-job contention should not occur, so the plausible causes are scheduler thread starvation under the burst (a 30s acquire expiring late) or an ownership/delivery path taking the fence on a run that had already finished. Treat the root cause as unverified.
Rule: local fire fence in the log means in-process lock contention, not a dead file lock and not a job
logic fault. Before treating a burst of those as broken jobs, check cron/executions.db for the same
job ids on their next tick and confirm the Telegram delivery rows, the fail-closed path records failed
even when the user already received the report. Do not clear or churn the .fire-*.lock files: a local
fence timeout leaves no held lock behind.
Update 2026-09-14 17:06 AEST (source: ~/.hermes/scripts/hermes_ops_status.py, ~/.hermes/cron/jobs.json):
the stale last_status=error on cron-rigour-audit was cleared to ok with last_error=null,
failure_streak=0 and an explicit last_verification_note, backing up jobs.json to
.bak-rigour-stale-error-repair-20260914T170616 first. Evidence for clearing: the 13:06 run's report
delivered at 13:17:17 (delivered to telegram:[redacted-id] via live adapter message_id=66720), all 157
enabled jobs then showed errors=0, and the scheduler logged no fire-fence timeout after 13:21:47. A
failed run whose output already reached the user leaves a permanent red flag in jobs.json; clearing it
by hand with a verification note is the repair, because the field only refreshes on the next tick and
this job's next tick is 21:06.
Sources:
~/.hermes/logs/agent.log · ~/.hermes/cron/executions.db · ~/.hermes/cron/jobs.json · ~/.hermes/hermes-agent/cron/scheduler.py · ~/.hermes/hermes-agent/cron/jobs.py