Skip to main content

Leader project suite

entity · created 2026-09-12 · actor hermes · confidence high · status active tags: leader-briefings projects evidence-collection cron

Now​

Six tracked projects derived from tech-leader briefing proposals. The suite turns proposal names into repeatable local artifacts under ~/clawd/memory/leader-projects/<slug>.json plus an index.json dashboard.

Projects: agent-eval-harness, openclaw-hardening-pass, openclaw-namespace-audit, enterprise-agent-adoption-monitor, ai-capex-dashboard, autonomy-reliability-auditor.

Owner chain: Hermes cron runs ~/.hermes/scripts/leader_project_suite_report.py (a thin wrapper), which shells out to ~/clawd/scripts/leader_project_suite.py --all --no-live. Evidence collection is local-only; there is no web fetch.

Corpus for evidence matching (candidate_files()): the 20 newest *.md under ~/.hermes/cron/output/4259a850b4c4 (the tech-leaders brief output), plus a fixed list of files under ~/clawd/memory/ and ~/clawd/wiki/pages/domains/. At 2026-09-12 the corpus was 31 files / ~2.07 MB.

Design constraints worth preserving​

  • Source terms must exist in the corpus. A term that appears zero times silently yields zero evidence and the project looks merely "not yet started" in the report. When a project returns zero, check the term frequency in the corpus before assuming there is no material.
  • Deduplicate on full normalized text. The briefing corpus repeats standing bullets across consecutive daily issues. Without dedupe, the reported count measures how many days a line persisted, not how much evidence exists. At 2026-09-12 the corpus held 10,998 non-empty lines but only 1,070 unique ones.
  • Do not cap line length low. The substantive analytical bullets are the long ones. A 500-char cap discarded them; the 593-char standing item on ambiguous model/runtime directives was the only corpus match for the namespace audit project.
  • Test running: ~/clawd has no pytest of its own; use /Users/dwayne/.hermes/hermes-agent/venv/bin/pytest from ~/clawd. The old /Users/dwayne/mindroom/.venv/bin/pytest path is DEAD (MindRoom removed 2026-09-19).
  • Every declared metric key must appear in the produced record. A declared key that nothing computes leaves the report looking complete while carrying no data. Derive what the evidence supports and write an explicit null for the rest, so a reader sees a known gap instead of an implied omission.
  • next_action is evidence-aware for corpus-sourced actions only. The adoption monitor's action reads "extract adoption numbers from leader briefings"; the corpus has zero mentions of Gemini Enterprise, Meta Business AI and Claude connectors (only Copilot appears). signal_state reports coverage as full/partial/none. The other five actions read live systems (the session DB, OpenClaw config, wiki/memory), so absent corpus terms do not block them; flagging those produced noise.
  • Match the format the data is actually in. Evidence is stored as raw fragments: dates appear as "Apr 11", vendors as | Microsoft | table cells. ISO-only patterns reported dated_items: 0 for visibly dated material.

Log​

  • 2026-09-12: Fixed evidence collection, commit 1adad65. Two projects (openclaw-hardening-pass, openclaw-namespace-audit) reported zero evidence because their source_terms used vocabulary absent from the corpus: NemoClaw, OpenShell, secrets isolation, signed-update, name-collision, Peter Steinberger - all zero occurrences across 31 files. Replaced with terms verified present (secrets, credential, allowlist, sandbox, provenance; ambiguous, upstream, vendor). Also raised the line-length cap from 500 to 2000 and added dedupe on normalized line text. Before: agent-eval 10 items / 3 distinct, capex 4/3, hardening 0, namespace 0, adoption 1, autonomy 2. After: agent-eval 10/10, hardening 10/10, namespace 9/9, adoption 5/5, capex 4/4, autonomy 2/2 - all distinct, all six non-empty. Three regression tests added, each verified to fail against the previous code (3 failed / 3 passed on old code, 6 passed on new). Known limitation: the 10-item cap fills from the first corpus file, so openclaw-hardening-pass draws all 10 items from one brief; interleaving sources is not implemented. Source: live corpus analysis and pytest runs 10:20 to 10:32 AEST this session.
    • hermes.
  • 2026-09-20: Fixed silent metric loss and a frozen next action, commit a0aec45 in ~/clawd. Before: 6 of 6 projects declared metrics the code never produced (samples, pass_rate, controls_seen, mentions, signals, vendors, dated_items, failures, modes, lessons). Now every declared key is computed or explicitly null. next_action was checklist[0] with no progress tracking; it now flags corpus coverage as full/partial/none for the one action that sources the briefing corpus. dated_items counted ISO only and reported 0 for evidence dated "Apr 11"; it now counts natural dates and table vendors. Tests 8 to 9, all passing; both defects falsified against the old code before the fix. Source: ~/clawd run and pytest 10:10 to 10:20 AEST.
    • hermes.
  • 2026-09-20: Executed all six next actions, commits 12d6d8e and bbada38 in ~/clawd. New manual tool scripts/leader_project_actions.py; deliberately NOT wired into the cron job, because a reporting cron stays a reporting cron. Results in memory/leader-projects/actions/<slug>-action.json: agent-eval-harness collected 25 sessions from state.db (scores null, grading needs a model in the loop); openclaw-hardening-pass inventoried 5 surfaces; openclaw-namespace-audit classified 998 mentions with 1 genuine ambiguity; enterprise-agent-adoption-monitor blocked, 3 of 4 products unsourced; ai-capex-dashboard extracted 78 dollar and 4 GW figures; and autonomy-reliability-auditor returned no hits. The autonomy result was checked and is truthful: all 104 autonomy lines concern fleet directives, not vehicle autonomy, and the corpus holds no autonomy failure story. Tests 9 to 20. Source: pytest and ad-hoc checks, 10:25 to 10:45 AEST.
    • hermes.
  • 2026-09-20: Fixed the corpus source list, commit e710a57 in ~/clawd. candidate_files() omitted wiki/pages/entities/ (117 pages, 350 KB), which is where the enterprise-adoption numbers live. Corpus 31 to 148 files. Adoption monitor went from 3 of 4 tracked products unsourced to 1: google.md carries Gemini Enterprise paid MAU +40% QoQ, meta.md carries Meta Business AI at ~10M weekly conversations. The remaining phrase, "Claude connectors", is a naming gap: anthropic.md covers connectors under other names, so the gap message says so. Autonomy auditor split into vehicle and agent branches: no vehicle failure exists (6 lines, all deployment news) but the agent branch found one real failure, the Claude AI outage of 23 June. hallucinat was excluded because it matched an improvement claim, and generic terms now require a named AI subject because outage alone matched 126 unrelated lines. Tests 20 to 23. Source: pytest and 22 ad-hoc checks, 10:50 to 11:10 AEST.
    • hermes.
Sources: ~/clawd/scripts/leader_project_suite.py · ~/clawd/tests/test_leader_project_suite.py · ~/clawd/memory/leader-projects/ · ~/.hermes/scripts/leader_project_suite_report.py