Skip to main content

LLM benchmark weekly

entity · created 2026-09-13 · actor hermes · confidence high · status active tags: llm benchmark cron openrouter model-selection

Now​

Cron llm-benchmark-weekly (job 559bb003fc16, 2 6 * * 0, Sunday 06:02, deliver telegram:[redacted-id]) runs ~/.hermes/scripts/llm_benchmark_weekly_report.py.

As of 2026-09-13 the script fetches live benchmark data from the OpenRouter catalogue (https://openrouter.ai/api/v1/models, no authentication required). That endpoint carries Artificial Analysis indices per model: intelligence_index, coding_index, agentic_index, plus per-token pricing. At 2026-09-13 it listed 445 models, 250 with a benchmarks block, 187 with an artificial_analysis block, and 90 rating-eligible after removing alias (~-prefixed), :batch and :free duplicates and entries with a null intelligence index.

The report shows the top 8 by intelligence index, week-on-week movers against a stored snapshot, and the standing of the four models the fleet actually runs. State lives at ~/clawd/state/llm-benchmark-weekly.json:

Watched modelRoleIntelligenceRank (2026-09-27)
openai/gpt-6-astraOpenClaw fleet52.73 of 91
z-ai/glm-5.3-flashHermes agents41.815 of 91
deepseek/deepseek-v4.1-flashfallback39.520 of 91
minimax/minimax-m3vision29.238 of 91

Leader at 2026-09-13 was anthropic/claude-fable-5.1 at 53.4, just ahead of GPT-6 Astra at 52.8. At 2026-09-20 the field widened to 94 rating-eligible models and the top five were flat; the only material mover was qwen/qwen3.8-max (displayed as Qwen3.8 Max (0902)), up 5.1 to 45.4.

At 2026-09-27 the leader changed for the first time since the script went live: anthropic/claude-opus-5.5 entered the top at 57.6, ahead of Claude Fable 5.1 (53.4, flat) and GPT-6 Astra (52.7, down 0.1). Rating-eligible models fell from 94 to 91, and 68 of them moved, so a lower score can still mean a higher rank. All four watched models lost 0.1 or 0.4 on score while their ranks slid two to five places, which is the rank-drift trap this entity already warns about.

Note on rank drift: the watched models' absolute scores did not change between runs, but their ranks moved by one position (glm-5.3-flash 11 to 12, minimax-m3 32 to 33) because the eligible field grew from 90 to 94. Compare scores, not ranks, across runs.

Source selection notes​

Probed 2026-09-13 for a no-auth live benchmark source. These did not work:

  • datasets-server.huggingface.co rows/splits for lmarena-ai/* and opencompass/* - HTTP 401 (gated datasets).
  • open-llm-leaderboard/contents - HTTP 200 but the archive is frozen at 2025-03-12 submissions; it is the retired v1 leaderboard, not current.
  • artificialanalysis.ai/api/v2/data/llms/models - HTTP 401 (needs a key).
  • lmarena.ai/api/leaderboard - redirects to arena.ai, which returns HTTP 403 {"error":"Route not allowed"}.
  • The Artificial Analysis HTML leaderboard page (HTTP 200, 2.7 MB) does embed current model names, but only 9 "name" keys, so extracting scores means scraping a Next.js payload. Not chosen.

OpenRouter won because it is a single stable JSON document that already carries the Artificial Analysis indices plus pricing, so no page parsing is needed.

Design constraints worth preserving​

  • Never report a failed fetch as health. On any fetch error the script prints the failure and says no values are available. It must not fall back to the previous snapshot as though it were current, and it must not overwrite the baseline. Test test_fetch_failure_reports_unknown_and_keeps_baseline covers this.
  • Variant entries mirror their base model. ~-prefixed aliases, :batch and :free variants carry identical indices. Deduplicate by base ID or the top-8 list fills with near-identical rows.
  • A null index is not a zero. Models with intelligence_index: null are excluded from ranking rather than sorted to the bottom.
  • Price is per token in the API. pricing.prompt is a per-token string (for example 0.00001); multiply by 1,000,000 for a per-million figure.
  • Test running: /Users/dwayne/.hermes/hermes-agent/venv/bin/pytest from ~/.hermes. The old /Users/dwayne/mindroom/.venv/bin/pytest path is DEAD (MindRoom removed 2026-09-19).

Log​

  • 2026-09-27 (weekly synthesis): Recorded the 2026-09-27 06:02 AEST run (source ~/.hermes/cron/output/559bb003fc16/2026-09-27_06-02-52.md). The top of the index changed for the first time since the script went live: anthropic/claude-opus-5.5 at 57.6 took first place, ahead of Claude Fable 5.1 (53.4, flat) and GPT-6 Astra (52.7, down 0.1). The eligible field shrank from 94 to 91 models and 68 models moved, so the watched fleet models all slipped in rank (Astra 2 to 3, GLM 5.3 Flash 12 to 15, DeepSeek V4.1 Flash 16 to 20, MiniMax M3 33 to 38) while scoring 0.1 to 0.4 lower. This is the third consecutive week the rank figure alone would mislead, and the entity's "compare scores, not ranks" rule held: no fleet model lost meaningful capability this week. Source: live run output and ~/clawd/state/llm-benchmark-weekly.json, read 2026-09-27., hermes.

  • 2026-09-20 (weekly synthesis): First run after the 2026-09-13 rewrite replaced the placeholder, and it worked: the cron output at ~/.hermes/cron/output/559bb003fc16/2026-09-20_06-02-07.md (run 06:02:04-06:02:09 AEST, status completed, delivery_outcome: delivered) printed a real top-8, one mover, and fleet standing. State file was rewritten at 06:02 with fetched_at 2026-09-19T20:02:04+00:00 and 94 models, up from 90 on 2026-09-13. Scores for the four watched models were unchanged; their ranks moved one position because the eligible field grew. Source: live cron output and ~/clawd/state/llm-benchmark-weekly.json. - hermes.

  • 2026-09-13: Replaced the placeholder report with a live fetch. The old script was a documented stub: it searched local markdown for filenames containing gpt|claude|gemini|qwen|llm|benchmark|model and printed the same two lines once per matched file, so every run repeated "Live benchmark fetcher is not configured" and "Next build: add official LMSYS/Artificial Analysis/Hugging Face fetch" five times (confirmed in ~/.hermes/cron/output/559bb003fc16/2026-09-13_06-03-01.md, identical on the five prior runs). New script reports live top-8, movers, and fleet-model standing. 12 regression tests added at ~/.hermes/tests/test_llm_benchmark_weekly_report.py; all 12 fail against the old placeholder (git-less, so verified by swapping in llm_benchmark_weekly_report.py.bak-pre-polish) and all 12 pass on the fix. Live run verified twice: first run set the baseline, second run exercised the "Movers since last run: none" branch. Fetch-failure path verified by running a copy pointed at an unreachable host: it reported Benchmark fetch failed: URLError and left the 90-model snapshot intact. Source: live API responses, cron output, pytest runs 06:20 to 06:28 AEST this session. - hermes.

Sources: ~/.hermes/scripts/llm_benchmark_weekly_report.py · ~/.hermes/tests/test_llm_benchmark_weekly_report.py · https://openrouter.ai/api/v1/models · ~/.hermes/cron/output/559bb003fc16/