Model routing — SmokingMirror fingerprints
claude-cowork · confidence high · status active
tags: models routing evals smokingmirror drift
Now
Pipeline: ~/.hermes/evals/smokingmirror/smokingmirror_eval.py (+ live probes via ~/Projects/smokingmirror-ops/live_model_runner.py) → daily 07:35 post_model_smoke.py → public report/JSON at dwaynehelena.com/post-model-smoke (CC0). Dataset 2026-06-10: 863 probe pairs, 43 model nodes, 5 providers. Attribution smokingmirror.com.
Router scorecard (heuristics from public fingerprints, "not live answers to Dwayne-specific probes yet"):
| Job | Top | Runner-up |
|---|---|---|
| coding / planning / creative / adversarial review / classification | claude-opus-4.7 (~82-88) | grok-4-fast |
| cautious household actions / financial caution | gpt-5.5 (100) | claude-sonnet-4.6 |
Operating rules (site, 2026-06-10): decisive low-N models for crisp choices; refusal-heavy models where caution posture helps; same-family distance jump = drift → rerun app evals; far-apart models as critics not replacements; escalate on violent disagreement.
Fingerprint extremes: most decisive = claude-fable-5 effort variants (N-rate 4.5-7.3%); most refusal-like = deepseek-v3.2 (91%), gpt-5.4-nano (89%). Runtime reality check: fleet primary is ollama/kimi-k2.7-code:cloud (cost/context-driven), with glm-5.2 → claude-fable-5 fallbacks — the scorecard informs which fallback/critic to pick per job type, it does not set the default.
Log
- 2026-07-09: created from site + local eval pipeline — claude-cowork.
Sources:
https://dwaynehelena.com/post-model-smoke (dataset 2026-06-10) · ~/.hermes/evals/smokingmirror/ · ~/Projects/smokingmirror-ops