Skip to main content

Hosts & services, Pro

entity · created 2026-07-09 · actor claude-cowork · confidence high · status active tags: infra pm2 launchd services

Now​

Host layout changed 2026-08-14. The Nexus swarm and all live PM2 services now run on macbook-pro-3 (Zappo, Tailscale 100.80.168.13, this Mac) where /Volumes/FATTY is a local APFS volume. Pro (Tailscale 100.87.212.111) is a stale/retired mirror: its /Volumes/FATTY volume is unmounted, so every PM2 app under Pro runs from a source path that does not exist there. Pro's Nexus/eq/carry PM2 group was a pure zombie, zero live PIDs, junk pidfiles, and reported "unhealthy" to monitors even though the real swarm was healthy on macbook-pro-3.

Single primary host note (historical): Pro (Dwayne's MacBook Pro, arm64; Tailscale 100.87.212.111). External volume /Volumes/FATTY holds source repos, anything PM2 runs from there dies if the volume unmounts (this was the assumed: risk, now observed 2026-08-14: Pro lost its FATTY mount and its whole Nexus PM2 group became zombie).

Retired host: Pro-2 (dwaynes-macbook-pro-2, Tailscale 100.97.49.102) is offline. HA and MindRoom now run on macbook-pro-3 (ComputerName MacBook Pro (2), Tailscale 100.80.168.13). All cron scripts referencing the old 100.97.49.102 HA URL were updated to 100.80.168.13 on 2026-08-02. Source: live script audit and HA reachability check, 2026-08-02.

Two supervisor stacks (observed 2026-07-09, source ARCHITECTURE.md 2026-06-08):

  • launchd (~17 ai.openclaw.* + ai.hermes.* plists): both gateways, openclaw node, ha-tunnel, watchdogs, backups, doc-audit family, auto-update, healthchecks.
  • PM2 daemon fleet from /Volumes/FATTY/src/openclaw-*: openclaw-evolution (self-improvement orchestrator), sentinel (:18794 status API, polls Nexus), swarm-ws (:11470 ws / :18790 http deliberation), swarm-sentinel, oc-voice-bridge (STT/TTS), oc-status-page (:18792), oc-location, openclaw-workflows. max_restarts: 10; dump at ~/.pm2/dump.pm2; always pm2 save after changes.

Ports: OpenClaw gateway 127.0.0.1:18789 (+:3334), Hermes gateway API 127.0.0.1:8642, Ollama 127.0.0.1:11434, sentinel 18794, swarm 11470/18790, status page 18792.

Foundation libs every daemon uses: /Volumes/FATTY/src/claw-core/ (HA token store, cron helpers), /Volumes/FATTY/src/claw-architecture/, both git-tracked, had uncommitted files at 2026-06-08 audit.

Watchdog: ~/.openclaw/workspace/agent-state/agent-watchdog.sh every 60s; patched 2026-04-21 (read PM2 PIDs from pm2 jlist, not stale JSON) and 2026-04-28 (gateway PID from launchd label, kickstart restart). This watchdog auto-spawns an empty ~/.pm2 daemon on Pro every ~60s (it runs pm2 jlist), so a bare pm2 kill on Pro never sticks, the daemon respawns holding only pm2-logrotate, no Nexus apps.

PM2 zombie cleanup (2026-08-14)​

Pro's two PM2 homes (~/.pm2, ~/.pm2-nexus-strategies) held 27 + 7 dead Nexus/eq/carry apps under unmounted /Volumes/FATTY. Fixed:

  • Renamed keepalive launchd plists to .disabled: ~/.Library/LaunchAgents/pm2.dwayne.plist → .plist.disabled (label com.PM2, system domain, KeepAlive+RunAtLoad resurrects default daemon), and com.PM2.nexus-strategies.plist → .plist.disabled.
  • Backed up and emptied both dump.pm2 files (now [], so pm2 resurrect restores nothing). Backups: dump.pm2.bak-zombie-cleanup-20260814.
  • Killed both daemons. Live Nexus apps were NOT touched (they run on macbook-pro-3).
  • Recurrence watchdog: Hermes cron pro-nexus-zombie-cleanup (job a219e899392e, */30 * * * *, no_agent, deliver local) runs ~/.hermes/scripts/pro_nexus_cleanup_watchdog.py, kills the zombie daemons only if Pro is up, FATTY unmounted, and no app has a live PID; [SILENT] otherwise.
  • To re-enable Pro's PM2 if ever needed: rename .disabled plists back to .plist, launchctl load/bootstrap, restore dump.pm2 from the .bak files.

ARCHITECTURE.md names a migration target: OpenClaw → Hermes Agent via hermes claw migrate, not yet executed as of 2026-06-08 review.

Log​

  • 2026-09-25 06:50 AEST (ddgs search backend: the 2026-09-22 pin is live but not quiet): The 2026-09-24 entry above reads the web layer as quiet. That is true only of the exa/firecrawl class, and it stays true, a precise scan for exa/firecrawl in gateway.error.log (exact match, excluding the substring in "example.com") finds 7 lines on 2026-09-23, all before the 06:40 restart, and 0 on 2026-09-24 and 2026-09-25. The noise did not stop; it relocated to the pinned backend. ddgs fails on roughly one query in five with DDGSException: No results found. Direct measurement in the Hermes venv (ddgs 9.16.0, the current latest, installed 2026-09-22 14:10): 2/12 failed paced at 3s (17%), 3/15 at 0.6s (20%), 4/14 in an unpaced burst (29%). The same queries succeed on retry and a 25s cooldown clears it, so this is rate-shaping at the DuckDuckGo backend, not a broken pin and not a bad query. Retrying the specific failures (victorian planning reform, tafe victoria courses, zip pay news, seek jobs melbourne) succeeded afterward; an explicit backend='duckduckgo' kwarg still returns rows, so the failure is transient per-call rather than a dead engine. Failure events per day: 40 on 2026-09-22, 170 on 2026-09-23, 180 on 2026-09-24, 44 in the first 6h22m of 2026-09-25; each event carries a paired provider-level line, so the log holds about two lines per real failure. The rescue path absorbs it and works: search_with_failover returned usable rows on 6/6 then 8/8 consecutive calls across a mix of the exact queries that had just failed natively (ddgs ring order ['parallel','keenable']). Cost is a wasted round-trip on ~20% of searches, roughly 180/day at current volume. Visible knock-on, guardrail pressure: loop_web_search_cap halts (agent/tool_guardrails.py:80, 50 searches per turn) ran 9 on 2026-09-23 and 17 on 2026-09-24, against 5-6/day on 2026-09-18 to 2026-09-22. A backend that throws makes search-heavy turns retry and hit the cap more often; job cron_fe0814a7945e (Talent Watch) at 2026-09-25 00:37-00:42 stacked 4-5 web_search calls per step and tripped the cap at 00:42:12. Telegram is not a standing fault: sticky-path failures ran 133 on 2026-09-24 but 1 on 2026-09-25 (03:30, one IP, recovered). Gateway healthy and long-lived: pids 55088/55089, started 2026-09-23 06:40:00, elapsed 1d23h42m, no restart. Residual, and the real exposure: every search on this host is keyless, ddgs pinned, and parallel/keenable are the only failover vendors and both are themselves unkeyed. No EXA_API_KEY, FIRECRAWL_API_KEY, PARALLEL_API_KEY, TAVILY_API_KEY, KEENABLE_API_KEY or BRAVE_SEARCH_API_KEY is set in the Hermes dotenv. The structural fix is a key on any search-capable provider, which needs Dwayne's decision (credentials are a hard brake). Source: direct ddgs and search_with_failover calls in the Hermes venv 2026-09-25 06:30-06:50 AEST, gateway log date-histograms, config + env key scan, state.db session read., hermes.

  • 2026-09-24 06:35 AEST (exa + firecrawl removal confirmed in the live process): The 2026-09-23 entry left one thing unproven: it said the gateway restart was deferred, so the running process might still hold exa and firecrawl in its startup plugin registry. It completed. The old gateway pair (pids 88209/88210, started 2026-09-20 02:29) is gone; the live gateway is pids 55088/55089, started 2026-09-23 06:40:00, elapsed 23h44m at read time. The restart was requested 06:38:10 and the drain finished inside two minutes, far short of the 1800s cap. Runtime proof the removal holds: gateway.error.log keyless-failure lines attributed to Firecrawl or Exa ran 17-30 per day from 2026-09-13 to 2026-09-17, then climbed to 109 / 133 / 140 / 115 per day on 2026-09-19 through 2026-09-22, then 6 on 2026-09-23 (all before 05:38, i.e. before the change) and 0 after the 06:40 restart. That is the defect gone from the live path rather than merely masked. The two tier pins and both backend names were read at call time and were already live before the restart; the restart only dropped the two providers from per-process plugin discovery. Keenable is not a regression. Its keyless-extract failure lines are steady and predate the pin: 1-2 per day from 2026-09-13, 2 on 2026-09-22, 6 on 2026-09-23, 1 on 2026-09-24 04:20. Direct call against 7 real hosts (seek.com.au, macquarie.com.au, zip.co, nearmap.com, tal.com.au, nrma.com.au, example.com) returned content on 7/7, 119-6516 chars. The 04:20 line is Upstream forbidden, i.e. the vendor rate-limiting an individual host, not the backend being unusable. Failover is confirmed working: with keenable forced to return an empty-content result, _walk_ring order was ['keenable', 'parallel'] and parallel served the request (165 chars). Residual unchanged: parallel is the only keyless failover vendor and is itself unkeyed, so a parallel outage leaves no second keyless extract vendor. Source: live ps/launchctl, gateway log date-histogram over gateway.error.log, direct keenable_extract_keyless calls, _walk_ring failover simulation, config read, 2026-09-24 06:25-06:35 AEST., hermes.

  • 2026-09-23 06:55 AEST (web backends: exa + firecrawl removed): Removed both dead vendors from every web path, on Dwayne's instruction "remove exa and firecrawl". Why they were dead: neither has a key on this host (EXA_API_KEY and FIRECRAWL_API_KEY/FIRECRAWL_API_URL all unset), so each was reachable only through the anonymous keyless tier, where they fail structurally rather than intermittently. Direct proof: POST https://api.firecrawl.dev/v2/scrape with no key returns HTTP 403 with the body "your IP address looks suspicious, so Firecrawl can't be used without an API key from here"; exa's keyless MCP endpoint returns a malformed response the client rejects as "Unrecognized MCP response shape". A 2026-09-19 Talent Watch run logged 3 Firecrawl 403s + 1 Exa shape error in a single turn, which is what surfaced this. Four changes, at two independent layers, because one layer was not enough:

    1. plugins.disabled: ['web-exa', 'web-firecrawl'] (via hermes config set), stops both providers registering, so they are absent from the registry snapshot and unreachable by name.
    2. web.provider_tier.exa: paid and web.provider_tier.firecrawl: paid, required because the keyless failover ring is a hardcoded tuple and ignores plugins.disabled. plugins/web/keyless_mcp.py:299 _KEYLESS_RING = ("exa", "parallel", "firecrawl", "keenable") is not filtered by plugin registration, so with only change 1 in place both vendors stayed reachable whenever a search or extract was throttled. provider_tier == "paid" is the switch that excludes a vendor from the walk (_ring_order, line 336). Verified before/after: ring order pinned keenable went from ['keenable','exa','parallel','firecrawl'] to ['keenable','parallel'].
    3. web.extract_backend: keenable, required because _get_backend() hardcodes firecrawl as the no-selection fallback. tools/web_tools.py:110: when web.backend is empty but a selection exists it returns "firecrawl" with no probe. Disabling firecrawl therefore broke extraction outright ("web is configured to use 'firecrawl', but no registered web extract provider has that name"). keenable was chosen because it is the remaining keyless vendor that supports extract (ddgs is search-only and fails outright; verified).
    4. web.search_backend: ddgs, unchanged from the 2026-09-22 fix, re-verified. Supersedes the 2026-09-22 19:45 entry's recommendation to leave the extract pin alone. That entry concluded the right fix was "a credential-bearing extract backend ... or leaving the pin as-is and treating the noise as known", and declined to set extract_backend because empty resolved to firecrawl and ddgs cannot extract. Both were correct at the time; keenable was not considered. Pinning keenable removes the noise without a credential, so the deferral is no longer needed. That entry's claim that extract_backend was parallel is superseded: the live value at the time of this change was '', and every config backup on disk agrees, a scan of all 13 config.yaml.bak-* files (2026-07-09 through 2026-09-23) found extract_backend empty in all of them, so no parallel value is recoverable from any point in the file's history. The cited backup ~/.hermes/config.yaml.bak-extract-backend-20260922-193119 does not exist. Verified (9/9, $TMPDIR/hermes-verify-exa-firecrawl.py): neither vendor registered; both listed disabled; both tier-pinned paid; the ring excludes both from every start vendor; ddgs search returns 3 hits on a real query; keenable extract returns content on a real URL; a throttled keenable result fails over to parallel (165 chars). Falsified: with the two tier pins stripped in a temp HERMES_HOME, the script reports 7/9 and exits 1, so it genuinely detects the broken state rather than passing vacuously. Also confirmed through the real live tools (web_search, web_extract) rather than only direct provider calls. Gateway restart: hermes gateway restart was accepted but deferred (drain-first, 1 active chat work unit); the restart only drops the two providers from the live process's startup plugin registry, the tier pins and backend names are read at call time and were already live before it. Config backed up at ~/.hermes/state/config.yaml.bak-20260923-exa-firecrawl (sha256 2e9c947e56219730f5bf253a5e8b3b78f1f2e35a464f47f58c6d61b699020be1). Residual: parallel remains as the sole keyless failover vendor, itself unkeyed; if it dies too there is no second keyless extract vendor. Source: live config reads, plugins/web/keyless_mcp.py ring inspection, direct curl of the Firecrawl endpoint, real tool calls, temp-home falsification., hermes.

  • 2026-09-22 19:45 AEST (search backend correction, follow-up to the 09:45 entry): The search half of that entry holds. The extract half does not, and the entry's own claim that web.extract_backend was '' cannot be reproduced. Live read of ~/.hermes/config.yaml (mtime 2026-09-17 09:15:56, unchanged until the edit below) shows extract_backend: parallel was already set; only backend and search_backend were empty. The claim of a config change is falsified: no config-write journal entry, no backup, and no mtime movement on 2026-09-22, and the cited receipt $TMPDIR/hermes-verify-search-backend.py no longer exists. The ddgs install claim is also falsified by date: ddgs-9.14.4.dist-info is dated 2026-07-01 08:00, not today, and is version 9.14.4 rather than the claimed 9.16.0, so no package was installed on 2026-09-22. ddgs being importable and _get_search_backend() returning ddgs are both true. Real defect found in the extract path instead. extract_backend: parallel names a provider that cannot work: the parallel SDK is absent from the venv, PARALLEL_API_KEY is unset, and the resolved provider reports is_available(): False, with an empty search-backend fallback ring (_KEYLESS_PREFERENCE = ("exa","parallel","firecrawl","keenable"), no keys for any). The runtime consequence is visible in the logs: 31 Keyless Parallel extract failed lines in gateway.error.log from 2026-09-09 to 2026-09-19 against host='openrouter.ai', i.e. the parallel endpoint 404s and the keyless rescue path silently performs every extraction. Extraction does currently succeed because failover covers it: example.org and rfc-editor.org/rfc/rfc2606.txt both returned full content. This is the same class of bug as the search half, one layer over: a backend that is configured, named in resolution, and incapable. Not changed, deliberately: setting extract_backend: ddgs makes the tool fail outright rather than attempt rescue (verified: DuckDuckGo (ddgs) is a search-only backend and cannot extract URL content); setting it empty resolves to firecrawl, which is also unavailable and would log the same noise class. Both were tested and reverted, and config.yaml is byte-identical to its pre-change state (d6fb009a7c5ecdc9). The correct fix is a credential-bearing extract backend (firecrawl/tavily/exa/parallel key) or leaving the pin as-is and treating the noise as known: see hermes-operations skill references/web-extract-noise-triage.md. Verified: live _get_extract_backend and get_active_extract_provider() from a fresh process reading config off disk, importlib.util.find_spec for each SDK, env-key presence scan, plugins/web/keyless_mcp.py ring inspection, two live web_extract calls, package and config mtimes, diff against the backup at ~/.hermes/config.yaml.bak-extract-backend-20260922-193119. Source: live probes 2026-09-22 19:20-19:45 AEST., hermes.

  • 2026-09-22 14:12 AEST (search backend, Talent Watch coverage note): Fixed the dead search backend that was wasting roughly a third of every search-heavy cron job's queries. Root cause: web.search_backend, web.backend and web.extract_backend were all '', so _get_backend() ran its never-configured autodetect ladder and fell through every keyed candidate (no Tavily/Perplexity/Exa/Parallel/Keenable/Firecrawl key exists on this host) to the keyless free tier, whose _KEYLESS_PREFERENCE = ("exa","parallel","firecrawl","keenable") ends at Firecrawl with no FIRECRAWL_API_KEY. Direct proof of the failure: POST https://api.firecrawl.dev/v1/search without a key returns HTTP 403. That is exactly the "keyless Firecrawl search backend returned HTTP 403 on about a third of query attempts" line in the 2026-09-22 Australian AI Talent Watch brief: the job's queries were being served by a backend that cannot work, and the runner was silently rerouting the rest. Fix: installed the optional ddgs package (9.16.0, brings click/lxml/primp) into the Hermes venv (132 -> 135 packages) and pinned web.search_backend: ddgs via the sanctioned hermes config set (the direct file write was correctly refused by the config guard: "Refusing to write to Hermes config file"). ddgs = DuckDuckGo, genuinely keyless; Hermes ships a first-class provider at plugins/web/ddgs/provider.py that gates is_available() on package importability and runs each search in a disposable child process with a hard 30s cap (_SEARCH_TIMEOUT_SECS), so a rate-limited response cannot hang the agent loop. Why explicit rather than autodetect: a bare keyless walk would also pick ddgs, but pinning makes it strict and keeps the dead Firecrawl path unreachable; the config comment records how to revert. Verified: _ddgs_package_importable() True; _get_search_backend() now returns ddgs from a fresh process reading config off disk; the six real Talent Watch query shapes (Zip Co, Macquarie 22831, Nearmap, Arinco, TAL, NRMA) all returned success with 3 hits each, 6/6, 18 hits, 0 failures, where previously about a third 403'd; the old keyless Firecrawl endpoint still returns 403, so the defect is genuinely gone from the path rather than masked. 5/5 ad-hoc checks at $TMPDIR/hermes-verify-search-backend.py. Note ddgs is an optional package, not a declared dependency, so a future hermes update that rebuilds the venv may remove it; if searches start failing again, reinstall ddgs in /Users/dwayne/.hermes/hermes-agent/venv. This affects every search-heavy job, not just Talent Watch. Source: live _get_search_backend/_configured_backend, plugins/web/ddgs/provider.py, tools/web_tools.py resolution ladder, direct Firecrawl curl, config.yaml before/after, pip freeze counts., hermes. [PARTIALLY SUPERSEDED 2026-09-22 19:45: the search finding stands and is re-verified; the extract_backend and ddgs-install claims in this entry are falsified. See the correction entry above.]

  • 2026-09-20 (cross-status field cleanup, 17:29 AEST): Removed the last hard-coded retired hosts from the shared status scripts. ~/.hermes/bin/cross-status probed gw-pro2 at pro-2 (100.97.49.102) and doge at 100.122.127.91; neither is a tailnet peer (live tailscale status shows only macbook-pro-3, iphone182 and macbook-pro), so both fields read offline on every run and carried no information. ~/.hermes/bin/ha-tag-status kept pro-2 as a last-resort HA candidate and SSHed it to grep a token, a path that could never succeed. Both probes are gone. Also corrected a mislabel: the field named gw-pro was never measuring Pro, since it checks this host's ai.hermes.gateway launchd label and local hermes_cli.main gateway run process, so it is renamed gw-zappo, the host it actually measures. doge is not a host: Doge is the local home-energy persona (~/.hermes/scripts/doge_home_report.py, live 17:18 read Doge Home, 441W whole-house). Live output is now nexus=4/5 | ha=202/248 | gw-zappo=ok in 1.1s, down from five fields. Three now-dead imports were also dropped (time from cross-status; subprocess and sys from ha-tag-status, the first made dead by this edit). Consumers updated: cross-status-refresh diffs ("nexus","ha","gw-zappo") and engagement_scout.py:441 watches gw-zappo=down. Ad-hoc verification 34/34, script at $TMPDIR/hermes-verify-g5y4_xq5.py, hashes verified after the import cleanup. Separately found: cross-status-refresh claimed a "2 min via launchd" cadence in its own header, but no launchd agent or cron job runs it. The cache refreshes only on demand from engagement_scout.py (08:40, 20:40, Sun 15:30, when over 15 min stale) and fleet_bridge.py, which is why the cache sat 104 min stale at read time. Comment corrected; the missing scheduler is not yet fixed., hermes.

  • 2026-09-20 (weekly retune, 15:30-15:48 AEST): The shared cross-status headline carries two permanently dead fields. ~/.hermes/bin/cross-status probes gw-pro2 at 100.97.49.102 and doge at 100.122.127.91; both hosts are retired and neither appears in tailscale status (only macbook-pro-3, iphone182, and MacBook Pro 100.87.212.111 are tailnet peers), so gw-pro2=offline and doge=offline are constant and convey no information. Worse, gw_pro() checks the LOCAL launchd label ai.hermes.gateway and the local hermes_cli.main gateway run process, so gw-pro=ok reports this host's Hermes gateway, not Pro - the field name misleads. Also confirmed: the host running this retune is macbook-pro-3 (MacBook-Pro-3.local, Tailscale 100.80.168.13), which is the same machine as Nexus, Home Assistant, Ollama and the OpenClaw gateway; the task's host list treats macbook-pro-3 as a separate remote. Pro (100.87.212.111) was Tailscale-offline at read time, last seen ~30 min earlier, SSH port 22 timing out; nothing depends on it - its PM2 group is the documented zombie set. Source: live tailscale status --json, tailscale ping, ssh, launchctl list, ~/.hermes/bin/cross-status, 2026-09-20. - hermes.

  • 2026-08-14: Nexus swarm actually runs on macbook-pro-3, not Pro. Pro's FATTY volume unmounted (assumed risk became observed) → its 27+7 Nexus/eq/carry PM2 apps were zombies with zero live PIDs. Cleaned via .disabled plists + emptied dumps + daemon kill; recurrence watchdog cron pro-nexus-zombie-cleanup installed. Source: live PM2/ps/launchd inspection over SSH 2026-08-14., hermes.

  • 2026-08-03: Superseded Pro-2 as active host, now retired. HA and MindRoom moved to macbook-pro-3 (100.80.168.13). All cron scripts updated. Source: live script audit and HA reachability check, 2026-08-02., hermes.

  • 2026-07-09: created from ARCHITECTURE.md + RUNBOOK.md, claude-cowork.

  • 2026-09-27 (control-room brief, 19:03 AEST): Two live defects found on this host. (1) Hermes checkpoint store was wedged by a stale git index lock: /Users/dwayne/.hermes/checkpoints/store/indexes/51bf226495121bd8.lock (created 17:57, 0 bytes) caused seven tools.checkpoint_manager: git add -A rc=128 failures between 17:58 and 18:17. Quarantined to 51bf226495121bd8.lock.stale-20260927-190318 via mv (same fix pattern as the 2026-09-20 .lock.stale-20260920-153638 pair). No live git process held it. (2) Pro's launchd agent ai.openclaw.sync-inbound runs /Users/dwayne/.openclaw/bin/sync-inbound.sh every 15 s with SRC='dwayne@100.122.127.91:.openclaw/media/inbound/'; 100.122.127.91 is the retired doge host, not a tailnet peer (live tailscale status on both hosts shows only macbook-pro-3, macbook-pro and iphone182). Every run exits 255 with ssh: connect to host 100.122.127.91 port 22: Operation timed out. The log holds 311,088 exit=255 lines over 46 MB, still growing ~180 lines/hour, first entry 2026-06-30. This is the same retired-host class of bug already removed from cross-status on 2026-09-20; sync-inbound was missed. Source: live launchctl print, grep/wc over ~/.openclaw/logs/sync-inbound.log, tailscale status on both hosts, 2026-09-27., hermes.

Sources: ~/.openclaw/ARCHITECTURE.md (reviewed 2026-06-08) · ~/.openclaw/RUNBOOK.md