A tool-loop cap turns a research job into a chat reply

_entry

Date: 2026-09-20 Lane: hermes Confidence: high

The symptom

The cron job Australian AI Talent Watch (fe0814a7945e, every 130m, delivers to Telegram) sent Dwayne this text six times between 2026-09-19 19:07 and 2026-09-20 07:43:

I stopped retrying because I kept running web_search 50 times without making progress. The last result above shows what blocked it. Tell me how you'd like to proceed, or send continue and I'll try a different approach.

A user answered continue, which is meaningless: the job is unattended.

Root cause 1: the text is a hardcoded agent constant, not model output

run_agent.py:1224 _toolguard_controlled_halt_response() returns that sentence verbatim when a turn guardrail halts. It is emitted by agent/turn_tool_round.py:165, appended to the transcript, and returned as final_response, so the cron delivery path sends it like a report.

Consequence: no cron prompt edit can suppress it. The prompt is not consulted on this path. Telling the model "never ask a question" does nothing, because the model never produces this reply.

Root cause 2: the job genuinely exceeded the cap

Per-session web_search counts, read from messages.tool_calls in ~/.hermes/state.db:

searches	outcome
37, 38, 44	brief produced
27, 40 (after the fix)	brief produced
50, 51, 51, 51, 52, 53, 53	halted at the cap

The cap is tool_loop_guardrails.loop_caps.max_web_searches, default 50, per turn, with counters reset per turn (they do not accumulate across runs). The job's real appetite is 37-53 searches, so it straddled the limit: completion depended on how many queries it happened to spend. That is flaky by construction.

Root cause 3: wasted budget on a dead search backend

13 of 52 searches in one run returned the same 679-byte keyless-Firecrawl error (403 Forbidden for api.firecrawl.dev/v2/search). web.search_backend and web.backend are both '' in config.yaml and FIRECRAWL_API_KEY is absent, so the search chain falls to the keyless Firecrawl default and burns budget on a guaranteed failure. One repeating 403 occupied 25 percent of the run.

What was changed

1. hermes cron edit fe0814a7945e --no-continuity - context_from: ["self"] was removed. It re-embedded the previous run's output inside the next run's output, compounding seven levels deep (blocks per run: 0,1,2,3,4,5,6,7,7,7), and it echoed the halt text forward as if it were a prior report. Only one job in the fleet used context_from; blast radius was that job alone.
2. Output contract appended to the prompt: exactly one of brief / [SILENT] / coverage-failure notice, never a question.
3. Search budget added: at most 30 web_search and 6 web_extract per run, plan queries first, stop and write when the budget is reached.
4. Presentation rule added: the brief starts with its heading on line one and carries no tool, verification, ledger or process notes. The 07:49 run had led with ~2,000 chars of tooling commentary that reached Telegram, which AGENTS.md "Generated content hygiene" forbids.

Verification

Three forced runs after the fix (hermes cron run fe0814a7945e), reading the delivered response and the session's real search count:

• 27 searches, brief, no halt
• 40 searches, brief, no halt
• final run: response starts AUSTRALIAN AI TALENT WATCH, no tooling commentary, delivery target telegram:[redacted-id], last_delivery_error: None

Lessons

• A cron job that ends in a question is broken. The halt reply is a constant, so prompt hardening cannot fix it; budget the job below the cap instead.
• Measure a research job's tool appetite before trusting an interval schedule. A job that needs 37-53 calls against a cap of 50 will fail intermittently, and the failure looks like a chat message rather than a cron error.
• context_from: ["self"] nests output inside output. It grows the prompt every run and can echo a failure forward as if it were real prior work. Prefer an explicit on-disk ledger for deduplication.
• When a job has a search-heavy profile, check the search backend is live. A repeating keyless-Firecrawl 403 silently consumed a quarter of the budget.
• last_status: ok on a halted run is misleading; the guardrail path returns a normal final response. Judge these jobs by the delivered text.

Addendum 2026-09-26 06:26 AEST: the fix did not hold, the job is halting on every run## 2026-09-27 addendum: the ledger half improved, the halt half did not

Re-measured 2026-09-27 06:00-06:07 AEST, one week after the previous addendum.

• The halt constant is still delivered. ~/.hermes/cron/output/fe0814a7945e/ holds it on 1 run on 2026-09-22, 3 on 2026-09-23, then a steady 7 per day on 2026-09-24, 2026-09-25 and 2026-09-26, and 2 by 05:17 on 2026-09-27: 27 of 50 runs, or 54%. state.db holds 74 rows carrying the constant. Latest is the 2026-09-27 05:17:04 run.
• The dedup ledger now advances on the runs that finish. The previous addendum read last_checked = forty-sixth run, 2026-09-25 with mtime 2026-09-25 21:15:15. It now reads last_checked = fifty-second run, 2026-09-27 with mtime 2026-09-27 02:58, and that record is a substantive brief (six verified open roles), not a halt. So the second-order claim needs narrowing rather than reversing: a halted run still writes nothing (the 05:17 halt left the ledger at its 02:58 state), but completed runs do advance it, so the coverage loss is proportional to the halt rate, not total.
• Backend burn is unchanged in scale. gateway.error.log DDGSException lines by day: 83 on 2026-09-22, then 343, 342, 359, 270 and 92 (partial) on the following days, with the paired provider line meaning roughly half those lines are real failures. TimeoutException on the underlying engines stayed live at 18, 10, 22 and 2 lines. Search-cap halts held at 7 per day on 2026-09-24 to 2026-09-26.
• Nothing was changed; cadence, budget and the cap remain Dwayne's call. Source: the cron output directory, state.db content scan, gateway.error.log date histograms, the ledger file mtime, read 2026-09-27 06:00-06:07 AEST., hermes.

Supersedes this lesson's "what was changed / verification" reading, which treated the halt as resolved. The same symptom is live and now deterministic.

Measured, not inferred. Per-run Web search via ddgs counts read from ~/.hermes/logs/agent.log are 50, 50, 50, 50, 50, 50 across the six consecutive runs starting 2026-09-25 16:34, 18:52, 21:09, 23:26, 2026-09-26 01:42 and 03:59. The cap is 50 (hermes_cli/config_defaults.py:521), so every run ends by hitting the wall instead of finishing its research. The run starting 06:16 also spent 50 and ended 06:24:13.

The halt text is delivered, not suppressed. At 2026-09-26 04:05:38 the turn ends reason=guardrail_halt; at 04:05:40 the scheduler logs Job 'fe0814a7945e': delivered to telegram:[redacted-id] ... message_id=70677, and state.db holds the constant as that turn's content. Deliveries of the halt text in the last 24 hours: message_id 70524 (21:15), 70529 (23:32), 70531 (01:49), 70677 (04:05). Code path re-verified: run_agent.py:1224 _toolguard_controlled_halt_response builds the string, agent/turn_tool_round.py:165 returns it as final_response.

Prevalence. Halt text appears in 29 distinct Talent Watch sessions. Per day: 4 on 2026-09-19, 3 on 2026-09-20, 3 on 2026-09-22, 3 on 2026-09-23, 7 on 2026-09-24, 7 on 2026-09-25, 2 so far on 2026-09-26. At a 130-minute interval the fleet's most frequent user-facing job is now a noise source rather than a report.

Why the 2026-09-20 fix does not bind. The job prompt still carries the 30-search budget added then, and the job spends 50 inside a single turn. A prompt budget is not enforcement; the per-turn cap is. The lesson "budget the job below the cap" was correct and was not implemented as a constraint. The remaining choices are to raise the cap to the job's real appetite, to cut the appetite by rotating coverage across runs, or to accept the halts knowingly. Job cadence and budget are Dwayne's call, so nothing was changed.

Second-order, and it compounds the failure. The run that halted at 21:15 never wrote the dedup ledger: /Users/dwayne/.hermes/state/australian_ai_talent_watch.json mtime is 2026-09-25 21:15:15 and it still reads last_checked = forty-sixth run, 2026-09-25. A halted run cannot record what it saw, so the next run re-searches the same ground, burns the same 50 searches, and halts again. Reducing the budget without repairing the ledger trades one failure for another.

New ddgs failure class. Alongside DDGSException: No results found (171 on 2026-09-24, 180 on 2026-09-25, 41 in the first six hours of 2026-09-26), TimeoutException now appears on the underlying engines: 0 on 2026-09-23, then 9, 5, 6 on the three following days, across grokipedia (16), mojeek (10), startpage (6) and yahoo (6). Each failure costs a retry and a rescue, so a degraded backend converts directly into budget burn. Capacity and backend health are one problem here, not two.

Source: agent.log per-run search counts and the 04:05:39 turn-end line, gateway.error.log cap and ddgs date-histograms, state.db halt-text rows and delivery message_ids, cron/jobs.json, the two code call sites, and the ledger mtime. Measured 2026-09-26 06:22-06:26 AEST., hermes.

Addendum 2026-09-27: a second failure mode, and it is not the cap

Measured 2026-09-27 06:22-06:26 AEST. The job now fails two different ways, and the newer one has nothing to do with search capacity.

The verify-on-stop gate replaces a finished brief with a tooling complaint. agent.verify_on_stop is true in config.yaml:60. After any non-prose file edit, agent/verification_stop.py:158 build_verify_on_stop_nudge fires via agent/turn_stop_gates.py:129 and forces another turn. The job's only edit is state/australian_ai_talent_watch.json, and .json is not in _NON_CODE_VERIFY_EXTENSIONS (verification_stop.py:19, prose and log types only). With no runnable recipe the nudge asks for a temporary script under /private/var/folders/... (verification_stop.py:207-215). The job has enabled_toolsets: ["web","file"] in cron/jobs.json, so it cannot run a script. It answers with a blocker, and the blocker is delivered as the report.

Evidence, run starting 2026-09-27 02:54:32: the brief was written at message index 98 (starting AUSTRALIAN AI TALENT WATCH 27 September 2026), then a 1136-character blocker at index 109 beginning "Cannot produce the requested verification script" became the final text. Turn ended 02:59:23 reason=text_response response_len=1136; delivered 02:59:24 as message_id 71599. The brief never went out. The run at 2026-09-26 17:51 kept the brief but appended verification noise to it (response_len=2963, message_id 71312), which the job's own prompt forbids. The run at 2026-09-26 15:35 wrote its ledger entries and was then suppressed entirely ([SILENT] — skipping delivery).

Rate over the window. Ten runs, ten deliveries' worth of work. Seven delivered the halt question (message_ids 70706, 70720, 70874, 71587, 71594, 71596, 71605), one delivered a tooling complaint, one delivered the brief, one was silenced. One clean brief in ten runs.

The ledger counter now moves without recording. After the 02:54 patches the file reads last_checked = fifty-second run, 2026-09-27 and seen_jobs carries first_seen: fifty-first run 2026-09-26 entries, but material_facts jumps from Forty-sixth to Fiftieth: no Fifty-first or Fifty-second entry exists among 213 facts. A run that advances the counter without recording facts misleads the next run more than one that never writes at all.

Rule. Do not attach a runtime-verification gate to a job that has no runtime. verify_on_stop: true is a host-wide setting; its legacy auto mode exists precisely to skip messaging and cron surfaces (verification_stop.py:50). The live bare boolean forces the gate everywhere, including jobs whose only "code" edit is a state file.

Source: config.yaml:60, agent/verification_stop.py, agent/turn_stop_gates.py, cron/jobs.json, state.db transcripts and delivery message_ids, agent.log turn-end lines, gateway.error.log cap lines, state/australian_ai_talent_watch.json., hermes

Addendum 2026-09-27 20:45-20:55 AEST: halt rate measured per run, and the scout double-counted it

Measured from ~/.hermes/cron/output/fe0814a7945e/*.md (body after the echoed prompt, not the file as a whole) across the 50 retained runs:

window	runs	halts	briefs	other
all retained	50	31	-	-
last 24h to 20:45	10	8	1	1 verification-blocker
2026-09-27 calendar	9	7	1	1 verification-blocker

The 18:52 run hit the cap at search 50 with 52 DDGS No results found lines in its own five-minute window; the turn ended reason=guardrail_halt, and Telegram message_id 71915 carried the halt constant. The 16:36 run produced a real brief and advanced the ledger (last_checked = fifty-third run, mtime 16:35), so the coverage loss stays proportional to the halt rate.

Reporting defect found and fixed. engagement_scout.py:guardrail_halt_lines() counted log LINES, and one halt writes two within milliseconds: the runtime status line Tool guardrail halted web_search: loop_web_search_cap and the delivered reply I stopped retrying .... The scout therefore reported halts_24h=16 for eight real halts, and the 20:40 evening-review input carried that doubled figure. Fixed to collapse marker lines sharing a timestamp into one event, preferring the delivered reply text. Verified: python3 -m py_compile clean, and a live scout run at 20:55 returned halts_24h=8 with one entry per halt timestamp. Backup at ~/.hermes/backups/engagement_scout.py.bak-halt-dedupe-20260927-205505.

Still unfixed, and not fixable from the job. The 30-search prompt budget does not bind; the 50-call per-turn cap (tool_loop_guardrails.loop_caps.max_web_searches) is the only enforcement, and loop_caps is global config with no per-job override in the cron schema. The halt sentence is a runtime constant, so no prompt edit can replace it with the job's contracted coverage-failure notice. Real options remain cadence, budget and the search backend, all Dwayne's call.

Source: cron output bodies and md5-deduplicated sizes, cron/executions.db, agent.log turn-end line for session cron_fe0814a7945e_20260927_184719, gateway.error.log marker timestamps, the repaired scout's live run. Measured 2026-09-27 20:45-20:55 AEST., hermes