Skip to main content

System disk hit 99% full and broke the cron ledger; disk monitor watched the wrong volume

lesson · created 2026-09-11 · actor hermes · confidence high · status active tags: disk storage cron sqlite monitor false-negative

On 2026-09-11 the Zappo Data volume reached 99% used (204 GiB of 233 GiB, 3.5 GiB free). Between 15:43 and 16:05 the Hermes gateway logged 141 sqlite3.OperationalError: disk I/O error lines, all from cron/ledger.py prepare_ledger() running PRAGMA synchronous=FULL. SQLite returns I/O error rather than "disk full" when it cannot write, so the symptom looked like database corruption. The hourly-fim-audit job (fc4cd071b735) failed at 16:02 with disk I/O error; other jobs logged Cannot check completed occurrence for job ... warnings. Writes recovered once free space climbed back above roughly 4 GiB (verified 17:17: PRAGMA synchronous=FULL and a ledger read succeeded).

Root cause of the missing warning: disk_space_monitor.py watched /, which on macOS is the sealed read-only system volume (12 GiB of 233 GiB, 73%). It never crossed its 85% threshold, so the daily monitor stayed silent while the Data volume filled. Fixed 2026-09-11 by watching /System/Volumes/Data instead; a dry check now reports 99% used and raises the alert. FATTY read 87% against an 88% threshold.

Largest reclaimable items measured the same day: ~/.hermes/backups 8.7 GiB (2.4 GiB pre-update-2026-08-27 zip, 3.3 GiB pre-update-2026-09-10 zip, 1.0 GiB orphan tmpk6ql9m1g.db), ~/Library/Application Support/Claude 10 GiB, ~/Library/Caches 7.0 GiB, ~/.homeassistant/backups 9.5 GiB (3.1 to 3.2 GiB per daily tar), ~/.hermes/cache 2.1 GiB, ~/.hermes/artifacts 772 MiB. Deletion is not automatic; it needs Dwayne's approval.

Rule: on macOS, watch /System/Volumes/Data, not /. A monitor whose threshold can never be reached is worse than no monitor. Treat sqlite3.OperationalError: disk I/O error as a low-space signal first and a corruption signal second: check df before touching the database.

Log​

  • 2026-09-11 17:20-17:40: reclaimed 6.0 GiB with no user data lost, taking Data volume free space from 3.4 GiB (99%) to 9.4 GiB (96%). Removed: two abandoned partial zips in ~/.hermes/backups (.pre-update-2026-08-18-183638...partial 1.6 GiB, .pre-update-2026-08-18-184659...partial 325 MiB), the orphan tmpk6ql9m1g.db 1.0 GiB (SQLite PRAGMA integrity_check on the live cron/executions.db returned ok), and a truncated 2,147,483,647-byte ~/.homeassistant/www/camera-stills/.pool.84891...candidate.jpg from 2026-08-04 (exactly INT32_MAX, an interrupted write; the current pool.jpg is 73 KiB and refreshed 2026-09-11 17:31, and the hourly ha-exposed-camera-stills-refresh job (835ebfa7498e) has completed normally at 17:00 and 17:30). Not touched: the 3.3 GiB and 2.4 GiB Hermes pre-update zips (real rollback artifacts) and the 9.5 GiB HA daily backups. Sources: df -h /System/Volumes/Data, ls -la on both directories, sqlite3 pragma integrity_check.

  • 2026-09-15: recurrence. The Data volume fell to 695 MiB free (100% used, 233 GiB) and four cron errors fired between 11:16:48 and 11:17:56: disk I/O error on the dashboard-data-refresh job, [Errno 28] No space left on device on .jobs_9kb5z9dp.tmp and on cron/output/fa09c9c8c683/.output_uxbi6xc_.tmp, and ha-health-check stopped with "session storage could not be written". Both affected jobs completed normally on their next runs (ha-health-check at 12:12 and 13:12), so the fault was the write failure, not the jobs. Largest live contributor: OpenCore Legacy Patcher staged the macOS 15.8 (24H23) install from its com.dortania.opencore-legacy-patcher.auto-patch LaunchAgent, which started 09:31 and downloaded a 15.7 GB InstallAssistant.pkg plus a 15.6 GB payloads_overlay shadow file into /private/var/folders/wj/rdb6slrj6rj29szyxtpwwps00000gn/T/tmp2l1vilns (29 GiB), and installed the 15 GiB /Applications/Install macOS Sequoia.app. The OCLP process (--update_installed, PID 3770) logged "InstallAssistant installed" at 09:50:35 and then sat idle and sleeping for four hours holding the mounted payloads.dmg (disk9) open. Other measured candidates: ~/.cache/uv 21 GiB, ~/.ollama 12 GiB, ~/Library/Application Support/Claude 9.7 GiB, ~/Library/Android 9.3 GiB, ~/.homeassistant/backups 4.0 GiB, ~/.hermes/cache 2.2 GiB. disk_space_monitor (job 5d2bac7c0ec6) did alert at 10:03 reporting the Data volume at 99% with 2 GB free, so the 2026-09-11 monitor fix worked; the ENOSPC followed two hours later. Sources: df -h /System/Volumes/Data, ~/.hermes/logs/errors.log, ~/.hermes/cron/jobs.json, ~/Library/Logs/Dortania/OpenCore-Patcher_2.5.0_2026-09-15_09-31-29-239338.log, hdiutil info, du -sh per candidate, read 2026-09-15 13:15-13:55 AEST.

  • 2026-09-16: third recurrence of the same pressure, plus two report defects found and fixed. At the 10:00 AEST health run the Data volume read 208.3 MB free (100% used, 233 GiB, /dev/disk2s1); by 10:12 it had recovered to 19 GiB free / 91% used. The observed sequence: /Applications/Install macOS Sequoia.app measured 15 GiB at 10:11 and was gone by 10:12, its tmp payload tree in /private/var/folders/wj/.../T/ was down to 138 M, and com.apple.cache_delete ran Purge OneShot on /System/Volumes/Data at 10:06 to 10:10 with StorageManagementService active. No No space left on device line appears in any Hermes log dated 2026-09-16, so the squeeze lasted under an hour and broke no scheduled job. Largest measured candidates remain ~/.cache/uv 21 GiB, ~/.ollama 12 GiB, ~/Library/Application Support/Claude 11 GiB, ~/Library/Android 9.3 GiB, ~/.colima 8.9 GiB, ~/Library/Caches 8.1 GiB. Two defects fixed in ~/.hermes/scripts/platform_health_report.py (the 10:00 daily job 0bbf18f4eb0e): (1) hermes gateway status was called with timeout=15 while it measures 7.5-13.5s on Zappo, so a healthy gateway reported Gateway: CHECK at 10:00, timeout raised to 45s and re-verified OK; (2) the Disk line called df -h /, the sealed system volume, so it could not see the Data volume fill, same false-negative class as the 2026-09-11 monitor bug above. It now reports /System/Volumes/Data. Sources: df -h /System/Volumes/Data, os.statvfs, du -sh per candidate, /Applications listings 10:11 and 10:12, log show for com.apple.cache_delete and StorageManagement, ~/.hermes/logs/gateway.error.log, ~/.hermes/cron/executions.db, read 2026-09-16 10:04-10:30 AEST., hermes.

  • Rule added 2026-09-15: an unattended OCLP --update_installed run holds two copies of the macOS installer (payload in tmp, plus its APFS/HFS shadow overlay) and a third in /Applications. On a volume this tight, staging a macOS update consumes roughly 45 GiB before the update itself is applied. Check tmutil/OCLP staging paths and any sleeping OpenCore-Patcher --update_installed process before blaming application caches.

  • 2026-09-11: incident recorded; disk_space_monitor.py moved to /System/Volumes/Data., hermes.

Sources: df -h /System/Volumes/Data · ~/.hermes/logs/gateway.error.log · ~/.hermes/cron/executions.db · ~/.hermes/scripts/disk_space_monitor.py · job fc4cd071b735 hourly-fim-audit