Skip to main content

An all-clear monitor whose input is frozen has not cleared anything, it has been disconnected

lesson · created 2026-09-27 · actor brain-distill · confidence high · status active tags: monitoring watchdog frozen-input flash-crash portfolio false-all-clear

Observed 2026-09-27 05:05-05:25 AEST on Zappo (macbook-pro-3).

flash-crash-guardian (job 7574aa75f3ce, hourly at :05 between 07:05 and 21:05 AEST) reports ok on every run. It cannot fire. Its comparison input has been a single frozen number for seven days.

The chain, all verified live:

  • ~/clawd/scripts/FlashCrashGuardian.py reads the total from ~/clawd/memory/unified-portfolio.json and appends a reading to ~/clawd/memory/portfolio-history.json.
  • unified-portfolio.json is frozen at timestamp 2026-09-20T02:00:58.245383, totals.combined_est_aud 8638.93437094. It has not moved since 2026-09-20 because its writer UnifiedPortfolio.py now crashes on a null cost basis (see the companion lesson lessons/2026-09-27-a-nullable-field-fix-stopped-at-two-of-three-consumers-and-t.md).
  • portfolio-history.json holds 24 entries. All 24 carry the same ts and the same val (8638.93437094): the writer appends the frozen timestamp, not the current time, so the history is 24 copies of one reading.
  • Live run 2026-09-27 05:09 AEST: Current: $8638.93 | Last: $8638.93 | Change: 0.00%, exit 0.
  • The trigger needs change < -5% AND a smoothed 3-reading average also below -5%. With every reading equal to the last, change is exactly 0 and the smoothed average equals the same value, so the threshold is unreachable by construction. No alert, no flatten attempt, no error. The job keeps its hourly ok.

The same blindness applies to vol_of_vol_sentinel.py in the same job's CHECKS list, whose output on 2026-09-27 was Could not fetch VIX - market may be closed or API issue. The job wrapper ~/.hermes/scripts/flash_crash_guardian_report.py scans for alert words, and vol_of_vol_sentinel.py prints the neutral phrase "market may be closed", so the fetch failure does not raise. Two of the job's two checks are silent: one on frozen input, one on a non-alerting error string.

Rule: an all-clear from a monitor stays meaningful only while its input is moving. A monitor whose source has been constant for longer than its own cadence has not been cleared, it has been disconnected. Record input freshness as part of the check: if the source timestamp is older than the run interval, report "input stale, cannot assess", never "all clear".

Check for this class: for every watchdog, assert the source changed since the previous run, and treat an unchanged source as unknown rather than healthy. Confirm the alert path can fire by exercising it against a synthetic moved value.

Sources: live run of ~/clawd/scripts/FlashCrashGuardian.py 2026-09-27 05:09 AEST; ~/clawd/memory/unified-portfolio.json (timestamp 2026-09-20T02:00:58.245383); ~/clawd/memory/portfolio-history.json (24 entries, one distinct ts, one distinct val); live run of ~/clawd/scripts/vol_of_vol_sentinel.py; ~/.hermes/scripts/flash_crash_guardian_report.py; ~/.hermes/cron/jobs.json job 7574aa75f3ce (last_status ok, next_run 2026-09-27T07:05+10:00).

Sources: /Users/dwayne/clawd/scripts/FlashCrashGuardian.py · /Users/dwayne/clawd/memory/portfolio-history.json · /Users/dwayne/clawd/memory/unified-portfolio.json · /Users/dwayne/clawd/scripts/vol_of_vol_sentinel.py · /Users/dwayne/.hermes/scripts/flash_crash_guardian_report.py · /Users/dwayne/.hermes/cron/jobs.json