Skip to main content

Gateways & endpoints

entry

Now​

All loopback-bound, single machine (Dwayne's MacBook Pro). Observed 2026-07-09 from configs/logs.

  • OpenClaw gateway, bind: loopback, token auth from env OPENCLAW_GATEWAY_TOKEN (name only; value in env). Version 2026.6.11 (update to 2026.7.x pending, adds on-exit cron, openclaw attach). Source: ~/.openclaw/openclaw.json.
  • Hermes gateway API, OpenAI-compatible, http://127.0.0.1:8642/v1/chat/completions, model name hermes-agent. Source: ~/.hermes/logs/gateway.log (api_server listening lines).
  • MindRoom, REMOVED 2026-09-19. No process, no container, no port 8765 listener, no chat.mindroom.* launchd agent. Source and runtime moved to /Users/dwayne/Projects/removed-mindroom-backup-20260919/. The paragraphs below describing it as a live service at v2026.9.11 are historical. Source: REMOVAL-REPORT.md in the backup dir, re-read live 2026-09-20.
  • MindRoom (historical, superseded 2026-09-19), observed 2026-09-10: editable source /Users/dwayne/mindroom, installed distribution 2026.9.11.post1.dev94+gd787f7abf.d20260905, live PID 97237 uses its .venv/bin/mindroom run. Matrix and MindRoom health recovered without restart. Telegram bridge login owner is @dwayne:localhost, but the human ghost alias still targets the internal service user; correction awaits authenticated owner control. P8 initiative ops-autopilot-ba0760ba remains owned by the originating OpenClaw parent. Repairs are isolated, not live; release .37 conflicts are resolved, 423 isolated target tests pass, and a private journal-copy migration preserves 3,742 event identities and 2,390 visible-history rows. Full customization overlay, durable P8 owner wiring, coordinated rollback and deployment approval remain incomplete; no production upgrade occurred. Source: /Users/dwayne/.openclaw/workspace/artifacts/mindroom-plan-execution-2026-09-10/continuation-report.md. <!-- project: path:/Users/dwayne/.openclaw/workspace -->
  • MindRoom web UI (historical, superseded 2026-09-19), tailnet-only HTTPS at https://macbook-pro-2.tail9e4b21.ts.net, served by Tailscale Serve as / → http://localhost:8765. Verified 2026-07-20: valid TLS, redirect to /login?next=%2F, final HTTP 200. Source: live tailscale serve status and curl.
  • Ollama, http://127.0.0.1:11434/v1; cloud models via ollama-launch provider (kimi-k2.7-code, glm-5.1/5.2, deepseek-v4-flash, gemini-3-flash-preview, kimi-k2.5, minimax-m2.7 + local llama3.2:3b, llama3.1:8b, llava, embedders). Single upstream - fleet-wide single point of failure; mitigated by OpenClaw fallbacks (2026-07-09). Bind scope corrected 2026-09-20: the listener is NOT loopback-only. Live lsof -nP -iTCP:11434 -sTCP:LISTEN shows ollama pid 3539 on TCP *:11434 (LISTEN), and curl http://192.168.68.79:11434/api/version from the host's own LAN address returns {"version":"0.34.2"}. The macOS application firewall is enabled with stealth mode on, and /Applications/Ollama.app/Contents/Resources/ollama carries an explicit "Allow incoming connections" rule, so the port is reachable from the LAN with the firewall on. Ollama has no authentication. The ollama_listener string in ~/clawd/state/local-security-audit.json is the same evidence and is the source of the standing "Ollama is not clearly bound to localhost only" finding. Not a new finding this week; recorded here with the concrete probe because the entity previously claimed loopback binding. Mitigation when wanted: set OLLAMA_HOST=127.0.0.1 on the launchd job that starts the app.
  • Home Assistant, http://192.168.68.114:8123, token at ~/.openclaw/.ha-token (symlinked from ~/.hermes/.ha-token). Allowlisted curl: curl -m 5 http://192.168.68.114:8123.*.
  • Exec approvals (OpenClaw), socket ~/.openclaw/exec-approvals.sock; file exec-approvals.json is gateway-rewritten live, never hand-edit.
  • Hermes cron, ~/.hermes/cron/jobs.json, ~150 jobs, gateway-managed (fire_claim/state rewrites), never hand-edit; add jobs via Zappo's cronjob toolset. Staged additions: ~/.hermes/cron/proposed-jobs-20260709.md.
  • Codex Telegram sidecar, ~/.hermes/scripts/codex_channel_loop.py reads Hermes Telegram session exports and replies through the bot. Observed 2026-07-19: One.ai delivery has a Telegram migration loop: stale session id [redacted-id] returns chat not found, while the old basic-group id [redacted-id] returns migrate_to_chat_id=[redacted-id]. The sidecar now parses migration targets and avoids retry loops; actual channel delivery still needs the bot/Mindroom bridge made present in the migrated supergroup. Source: ~/.hermes/codex-lab/mindroom-channel-response-migration-loop-2026-07-19-0131/2026-07-19_01-32-06-mindroom-channel-response-migration-loop-2026-07-19-0131.md.

MindRoom host: the logical host previously called Pro-2 is currently MacBook-Pro-3.local (ComputerName MacBook Pro (2), Tailscale 100.80.168.13). The stale Tailscale device dwaynes-macbook-pro-2 (100.97.49.102) is not a separate current MindRoom host. Source: Dwayne correction plus live local hostname, process, listener, and container checks, 2026-07-25.

  • MindRoom REMOVED 2026-09-19, source /Users/dwayne/mindroom (2.5 GB) and runtime /Users/dwayne/.mindroom (2.6 GB) were moved to /Users/dwayne/Projects/removed-mindroom-backup-20260919/ (mindroom-repo/, dot-mindroom/), not deleted. All seven chat.mindroom.* launchd agents booted out and plists deleted; the five mindroom containers, their images and four volumes removed; matrix_default network gone. Port 8765 no longer listens. com.dwayne.pacman-swarm was repointed from the MindRoom venv to /Users/dwayne/miniforge3/envs/ha314/bin/python3 and verified HTTP 200 locally and over Tailscale. Hermes cron had zero functional dependency (215 jobs checked). The Hermes and OpenClaw gateways are independent and untouched. Source: REMOVAL-REPORT.md in the backup dir, read live 2026-09-20. Do NOT treat MindRoom as a running service; any script referencing /Users/dwayne/mindroom is stale and dormant.

Machine hostname on LAN: dwaynes-macbook-pro (allowlisted ollama health curl).

Current incidents (observed 2026-07-08 … 2026-08-29, source: kg query, sessions):

  • fleet-health recovered, was warnings on 2026-08-03 07:30 UTC; has reported all green on every daily run since 2026-08-25 (observed 08-25, 08-26, 08-27, 08-28). The 2026-08-03 warning is superseded; no active fleet-health incident.
  • 2026-09-09 transient network outage, ollama.com DNS/timeout, 15:05 to 16:38 AEST a connectivity window made ollama.com unresolvable/unreachable. Local Ollama 0.20.5 proxies :cloud models to ollama.com, so Hermes calls to 127.0.0.1:11434 returned HTTP 502 ... lookup ollama.com: no such host / i/o timeout. Affected: weather-combined, conflict-rss-scan, hourly-fim-audit (agent inference) and p8-portfolio-pulse (Telegram media send timeout at 16:04). All scripts verified working manually after the window; inference healthy; last_error/last_delivery_error in jobs.json clear on each job's next fire (fim 20:02 today, weather/conflict 07:00/07:30 next morning, p8 16:00 next day). Readiness dashboard HOLD at 80% was a symptom of these stale error rows, not a config fault. Do not chase a custom_providers ollama_cloud misconfiguration for this signature: the 2026-07-08 entry is gone; the modern path is local Ollama proxying cloud models.
  • cron-health-observer recovered, was 3 erroring on 2026-08-03; has reported 0 erroring, 3 stale, 10 transitions on every run since 2026-08-25. The erroring count dropped to 0; the 3 stale jobs are the known low-cadence set (ai-home-stack-upgrade-maintenance, autonomous-improvement-loops, improvement-loop-closer, incident-observer, notify:cron per kg stale). No active erroring incident.
  • Approval-mode parser alias, Hermes cron-approval-mode-alias-triage patched the config loader so legacy approvals.mode=allow maps to off (the old "allow everything" value is no longer in the enum). Tests passed; gateway restart was deferred. This is a backwards-compatibility shim, not a policy change, live exec-approvals.json remains gateway-managed and must never be hand-edited.
  • OpenClaw exec allowlist misses, agents in the OpenClaw runtime hit exec denied: allowlist miss for ha_entity_guard.py, birdclaw scout, and some version/upgrade commands. Root config: ~/.openclaw/openclaw.json tools.exec and agents.defaults.toolPolicy. Hermes cannot change OpenClaw policy; route to OpenClaw agent or Dwayne for policy update.
  • OpenRouter key missing, OpenClaw gateway configured with openrouter/openai/gpt-5 as primary model, but OPENROUTER_API_KEY is not set in ~/.openclaw/.env or ~/.hermes/.env. Gateway crash-loop breaker tripped after 23 failed starts. Resolution requires either adding the key or switching primary to a provider with existing auth (Anthropic, xAI, local). Source: live config audit, 2026-08-01.

Log​

  • 2026-10-04 (21:57 AEST): Fixed the recurring git add -A (rc=128) failures in the filesystem checkpoint store for the /private/tmp project. Root cause was NOT the 2026-09-20 lock-file issue: the store's persistent index (~/.hermes/checkpoints/store/indexes/11fe14a563f7aed6) had recorded two nested git repositories in /private/tmp (ai-hub-diag, whose .git has no HEAD and is not a real repo, and ai-hub-verify) as gitlink entries. git add -A then tried to descend into them, hit fatal: 'ai-hub-diag/.git' not recognized as a git repository plus could not open directory 'msu-target-*/': Permission denied (four root-owned 0700 temp dirs), and aborted with rc=128. The store's info/exclude carried neither name, so every fresh stage re-hit the submodule path. Counted 177 such errors in ~/.hermes/logs/gateway.error.log since 2026-09-13, last 2026-10-04 21:27:35. Fix (Tier 2, reversible): appended ai-hub-diag/ and msu-target-*/ to info/exclude, and purged the two gitlink entries from the live index with git rm --cached -f. Index backed up as 11fe14a563f7aed6.bak-20261004 first; index shrank 477 K -> 113 K. Verified: git add -A against the live index returns rc=0 with zero fatal lines and zero remaining ai-hub/msu-target entries. Secondary finding, unfixed: hermes checkpoints prune reclaims 0 B and the store sits at 1.6 GB against a 500 MB cap, so _enforce_size_cap is not converging. Sources: live git add -A reproductions, ~/.hermes/logs/gateway.error.log, hermes checkpoints status/prune, 2026-10-04., hermes.

  • 2026-09-27 (weekly synthesis): The Ollama bind finding was re-verified independently by two of the week's own jobs, and the mechanism is now named. autoresearch-network-security (338a91091ac9, 2026-09-27 04:06 AEST) and security-audit-weekly (ab43c1050da9, 2026-09-27 02:11 AEST) both reported the same single finding, and the second probed the host directly rather than trusting the audit JSON: ollama pid 3539 holds 0.0.0.0:11434, the macOS launch environment for the app sets OLLAMA_HOST=0.0.0.0, and the API answered HTTP 200 from the LAN address 192.168.68.79:11434 and the tailnet address 100.80.168.13:11434 on 2026-09-27. The application firewall is enabled but block-all is disabled, so it is not closing the port, and Ollama has no authentication by default, so anything that reaches it can list and pull models and run inference as this user. New fact: the override is the app's own launch environment, which makes the previously recorded mitigation (set OLLAMA_HOST=127.0.0.1 on the launchd job that starts the app) the correct one rather than a guess. Left unapplied: it is a live service configuration change and needs Dwayne's approval. Sources: ~/.hermes/cron/output/338a91091ac9/2026-09-27_04-06-32.md, ~/.hermes/cron/output/ab43c1050da9/2026-09-27_02-11-07.md, read 2026-09-27., hermes.

  • 2026-09-23 (20:44 AEST): Fixed a silent blind spot in ~/.hermes/scripts/tailscale_mesh_health.py, the cron watchdog that alerts when a critical tailnet node drops. It matched critical hosts with crit in hostname while scanning [self, *peers], so macbook-pro matched this host's own name macbook-pro-3. Pro could sit offline and the probe reported [SILENT]. Replaced with exact first-DNS-label matching (_host_keys()), which also normalises the MacBook Pro HostName nickname. Regression-verified against all three real peer records: self now yields {macbook-pro-3}, Pro yields {macbook-pro}, and the phone's localhost nickname still resolves to iphone182. The watchdog now reports macbook-pro: offline. Pro itself is documented normal-offline; the fault was the monitor, not the host.

  • 2026-09-23 (20:44 AEST): Pro (100.87.212.111) LastSeen is 2026-09-23T10:20Z (20:20 AEST today), superseding the 2026-09-22 entry that recorded last-seen 2026-09-20 05:02 UTC. Pro was therefore reachable at some point in the last 24h and dropped again. SSH port 22 and ICMP both time out at 20:42 AEST; still not a fleet fault. Source: live tailscale status --json, 2026-09-23.

  • 2026-09-23 (20:45 AEST): Note that Hermes-Pro digest entries in ~/.hermes/state/digest-today-*.jsonl are emitted by locally-run Zappo scripts (disk_space_guard.py job 860f58246dd7, backup_freshness_guard.py, credential_expiry_guard.py, repo_hygiene_sweep.py). The tag is a historical label and is not evidence that Pro is up. Source: cron/jobs.json job-to-script mapping plus the emitting scripts, 2026-09-23.

  • 2026-09-20 (weekly retune, 15:30-15:48 AEST): Two live faults found and fixed on the local Hermes gateway. (1) providers.ollama-launch.api_key was the literal no-key-required, so agent.auxiliary_client.resolve_provider_client logged "named custom provider 'Ollama' has no resolvable api_key ... will 401" on every auxiliary call: 641 occurrences in ~/.hermes/logs/errors.log since 2026-09-15 and 104 on 2026-09-20 alone, last at 15:39:42. The Ollama endpoint ignores the key, so aux work still succeeded; the cost was log noise and a misleading 401 warning. Fixed with hermes config set providers.ollama-launch.api_key ollama (the value the model: block already carried), backup config.yaml.bak-auxkey-20260920-1540; semantic diff shows the single changed key. Falsified both directions: a temp HERMES_HOME copy with the old placeholder still emits the warning, the live config emits none, and zero further warnings appeared by 15:46. (2) Three zero-byte stale git index locks (11fe14a563f7aed6.lock 2026-09-13, 51bf226495121bd8.lock 2026-09-15, 1add2d7a96fb0f18.lock 2026-07-09) in ~/.hermes/checkpoints/store/indexes/ were failing every git add -A with rc=128 Unable to create ... File exists: 59 checkpoint_manager errors in the last 24h and 154 over 7 days, against 65 M of checkpoint store. Renamed to *.lock.stale-20260920-153638 (no deletion); git add -A then succeeded against the store with plain GIT_DIR/GIT_WORK_TREE, and zero checkpoint errors were logged after 15:36. tools/checkpoint_manager.py never removes these locks when a snapshot dies mid-run, so the recurrence path is still open. Source: ~/.hermes/logs/{gateway.error.log,errors.log}, ~/.hermes/config.yaml + backup, ~/.hermes/checkpoints/store/indexes/, live git add -A, 2026-09-20. - hermes.

  • 2026-09-20 (weekly synthesis): Corrected the Ollama bind scope in Now, the live listener is *:11434, not loopback, and answers on the host LAN address 192.168.68.79. Marked the two MindRoom service entries historical now that the runtime is removed and port 8765 is dead. Sources: live lsof, curl http://192.168.68.79:11434/api/version, /usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate and --getappblocked /Applications/Ollama.app, read 2026-09-20 06:00-06:05 AEST.

  • 2026-09-10: Superseded the earlier two-unresolved-conflicts release status. Continuation resolved both conflicts, applied 21 tracked customization patches, added durable edge uncertainty fencing and P8 approval-record/cancellation checks, and passed 423 isolated tests plus a private journal-only migration rehearsal. Live comparison: 7,742 preserved files unchanged. Prior repair patch was corrupt at a newline boundary; a hash-verified rebuilt patch is available. No authentication bypass, live alias edit, timezone change or deployment. Source: /Users/dwayne/.openclaw/workspace/artifacts/mindroom-plan-execution-2026-09-10/continuation-report.md. <!-- project: path:/Users/dwayne/.openclaw/workspace -->

  • 2026-09-10: Superseded the Now entry reporting MindRoom v2026.7.309 and parked local-work-20260719 as current state. Historical July 30 upgrade from v2026.7.251 and Intel onnxruntime 1.23.2 workaround remain historical facts, not current deployment proof. Live package/process, health, bridge identity, protected backups and staged repair evidence recorded under P8 ops-autopilot-ba0760ba; no live config/security/restart/message or release deployment performed. Source: /Users/dwayne/.openclaw/workspace/artifacts/mindroom-plan-execution-2026-09-10/repair-matrix.md. <!-- project: path:/Users/dwayne/.openclaw/workspace -->

  • 2026-08-30 (weekly synthesis): Superseded the 2026-08-03 fleet-health and cron-health-observer blockers. Both recovered: fleet-health all green daily since 2026-08-25; cron-health-observer 0 erroring, 3 stale since 2026-08-25. Sources: kg query --limit 50, kg stale., hermes.

  • 2026-08-03 (weekly synthesis): Added fleet-health blocked and cron-health-observer 3 erroring to current incidents. Sources: kg query --limit 50, 2026-08-03., hermes.

  • 2026-08-03: Added MindRoom v2026.7.309 upgrade record and OpenRouter key gap to Now section., hermes.

  • 2026-07-25: Corrected the MindRoom host identity: Pro-2 refers to the live local MacBook-Pro-3.local, not the stale offline Tailscale device dwaynes-macbook-pro-2. MindRoom was listening on port 8765 and the mindroom-telegram-bridge container was running. The direct-message portal !CzLMONAFcsUmQXAVZw:localhost is healthy. A Telegram group message created portal !MYzyNgMAoChKZjfENA:localhost; assistant and router were invited, joined, and persisted, and relay login [redacted-id] was enabled for that portal. Telegram still rejects outbound sends with PEER_ID_INVALID for the portal key chat:[redacted-id]/[redacted-id], including after portal sync and a bridge-only restart. Inbound Telegram-to-Matrix delivery remains healthy. Source: Dwayne correction and live process/container/bridge/MindRoom logs, 2026-07-25.

  • 2026-07-20: Verified the MindRoom web UI is reachable through tailnet-only Tailscale HTTPS at https://macbook-pro-2.tail9e4b21.ts.net; proxy target is http://localhost:8765, TLS verification succeeded, and the login page returned HTTP 200. Source: live tailscale serve status and curl, 2026-07-20.

  • 2026-07-20: Ownership of @MindRoom_V_bot was transferred from OpenClaw to the MindRoom mautrix-Telegram bridge on Pro-2. OpenClaw account mindroom is retained but disabled (running=false, lastError=disabled); the bridge bot login is stored under @mindroom_user:localhost and reported state_event=CONNECTED, with Telegram updates running. No portal existed immediately after login, so a fresh Telegram message is still required for end-to-end portal and agent-response verification. No token value is stored here. Source: live OpenClaw probe, mautrix provisioning result, bridge DB and logs, 2026-07-20.

  • 2026-07-19: OpenClaw Telegram account mindroom connected successfully as @MindRoom_V_bot on Pro-2. The credential is referenced through a mode-0600 token file; openclaw channels status --probe --json reported configured, running, connected, polling, no last error, and no restart pending. No token value is stored here. Source: live OpenClaw channel probe, 2026-07-19.

  • 2026-07-09: File created from fleet-upgrade session; Hermes API endpoint verified in gateway.log (last "listening" 2026-06-25, no stop line after).

  • 2026-07-12 (weekly synthesis): Added approval-mode alias and OpenClaw exec-allowlist gaps.

  • 2026-07-14: All 16 OpenClaw cron delivery and failure-alert destinations were normalised to Dwayne's Telegram DM; old supergroup/session bindings were cleared. daily-stock-tracker subsequently ran ok and delivered to the DM. birdclaw-local-stats-smoke was converted from an agent turn to a direct command payload, eliminating cloud-model dependency; forced run was ok with zero consecutive errors. Source: OpenClaw cron get/runs, 2026-07-14.

  • 2026-07-14: OpenClaw memory reindex remains blocked because the main and X agent stores lack OpenAI provider auth; changing auth requires Dwayne's explicit approval. Source: openclaw memory index --force, exit 1, 2026-07-14.

  • 2026-07-19: Codex sidecar direct bot delivery gained alias-based Telegram chat not found fallback, then follow-up probing found the alias target is only an old migrated basic group. Bot API sendChatAction to [redacted-id] reports migrate_to_chat_id=[redacted-id], and getChat/send attempts to [redacted-id] report chat not found. Direct-bot channel delivery remains blocked until the bot is present in the migrated supergroup, or the mautrix Telegram bridge login/reconnect is performed/approved. Source: ~/.hermes/codex-lab/mindroom-channel-response-migration-loop-2026-07-19-0131/2026-07-19_01-32-06-mindroom-channel-response-migration-loop-2026-07-19-0131.md.

  • 2026-07-19: Reverified direct-bot delivery blocker. getMe identifies @Sportbal_herme_bot; getChat [redacted-id] and getChatMember [redacted-id] return chat not found; getChat [redacted-id] returns basic group One.ai with default can_send_messages=false; getChatMember [redacted-id] reports bot status left. Matrix and Mindroom are healthy, but the mautrix bridge has user_login=0, portal=0, message=0. Source: ~/.hermes/codex-lab/mindroom-channel-response-bot-still-left-2026-07-19-current/.

  • 2026-09-22 (19:15 AEST): Go-live readiness dashboard reported a permanent two-warn 90% plateau from two checks that could never pass. Both were fixed in ~/clawd/scripts/go_live_readiness_dashboard.py; live rerun now reads GO 100%, 10/10 pass, 0 warn.

    1. parse_ops_status() expected the retired key=value layout of ~/.hermes/scripts/hermes_ops_status.py. The script now prints a card (health: ok · <ts> then • jobs: enabled=213 errors=0 ...), so the parser returned health=None / cron_counts=None and the "Ops status" gate warned health=unknown every refresh. The parser now accepts both layouts, strips the • bullet, splits the timestamp off with ·, and falls back to whichever result value carries delivery_errors=. Verified against current, legacy and synthetic degraded inputs.
    2. The "Polling-conflict patch" gate required ~/.hermes/backups/local-patches/telegram-conflict-keep-gateway.patch to exist and to show up in git status of hermes-agent. That directory does not exist and the patch was upstreamed, so the gate could never pass. It now also passes when the live adapter (~/.hermes/hermes-agent/plugins/platforms/telegram/adapter.py) contains the gateway stays alive and will retry marker. The "Cron fleet" gate's metric also printed the unparsed ops['cron'] string; it now prints the parsed enabled/errors/delivery_errors/never_run counts. Source: live parser runs, full dashboard regeneration and go_live_readiness_dashboard_refresh_report.py rerun, 2026-09-22.

  • 2026-09-22 (19:15 AEST): Pro (100.87.212.111, MacBook Pro M5 Max) is Tailscale-offline, last seen 2026-09-20 05:02 UTC (~2d 14h). SSH port 22 times out. This is the documented normal state, not a fault: the Pro's Nexus PM2 group is the known zombie set and nothing in the live fleet depends on it. The Pro-facing cron wrappers already absorb it by design (pro_fim_check_hourly.sh treats ssh 255 as a silent skip; pro_nexus_finance_monitor_report.py now points PRO_HOST at 127.0.0.1; pro_nexus_cleanup_watchdog.py prints [SILENT]). Do not report an offline Pro as degraded. Source: live tailscale status --json, ssh probe, script source reads, 2026-09-22.