Skip to main content

Arr stack and Colima Docker

entry

Now​

The media stack (Radarr, Sonarr, Lidarr, SABnzbd, Prowlarr, Overseerr, Plex, MindRoom stack) runs in Docker under Colima (macOS Virtualization.Framework, profile default, x86_64, 4 CPU / 6 GiB RAM / 100 GiB disk since 2026-09-12) on macbook-pro-3. Docker socket: /Users/dwayne/.colima/default/docker.sock. Radarr API at http://127.0.0.1:17878 (key in /Volumes/FATTY/dockerdata/radarr/config.xml).

RESOLVED 2026-09-20 02:18: Colima healthy, shares mounted, Plex serving​

Verified live: colima status running (Virtualization.Framework, virtiofs); all three guest shares readdir OK (/Volumes/FATTY, /Volumes/FATTY2, /Volumes/FATTY3); guest uptime 19:48, so the VM restarted after the 2026-09-18 host reboot. Plex container Up 20 hours, restarts 0, zero create_directories crashes in the last 3 minutes, /identity HTTP 200. The six *arr containers Up 2 hours after the 00:00 schedule run (arr-schedule.log shows all START, plex "already running", done 00:00:05). FATTY 102 GiB free (92%), FATTY2 38 GiB (95%), FATTY3 247 GiB (75%).

The 2026-09-17 20:40 guest-share fault is closed: shares recovered with the VM restart, not with a targeted remount. The two-observation "~24h into VM lifetime" offset remains a hypothesis only, not a confirmed pattern; there is no evidence of a recurring schedule.

The colima stop + colima start approval requested by the 2026-09-17 alert was never needed.

SUPERSEDED 2026-09-20 02:18: host rebooted 2026-09-18 23:38; Colima will not start, FATTY and FATTY2 no longer attached​

This block is retained as history. The recovery is the RESOLVED entry above: Colima started again, all three shares mounted, and the *arr window resumed. Do not treat this as an open fault.

Confirmed at 02:25 AEST 2026-09-19: still down, 2h47m in; /var/log/install.log shows the restart was the macOS Sequoia 15.8 (24H23) software-update install (SU:macOS Sequoia 15.8 24H23, installer boot.efi at 23:38:09), so this was a scheduled update, not a crash. FATTY and FATTY2 are still absent (ls /Volumes has only EMTEC C410, FATTY3, Macintosh HD; diskutil apfs list shows no other volume), Colima is still down, and the *arr 00:00-05:00 window is already lost. Two new knock-ons: /Users/dwayne/ARR/spillover_guard.log (mtime 02:10) is crash-looping because it calls df -k /Volumes/FATTY unconditionally, and colima_monitor/collector.py (log mtime 02:21) fails on colima ssh --profile default -- cat /proc/meminfo. See the 2026-09-19 02:12-02:25 Log entry.

The host restarted at 2026-09-18 23:38:01 AEST. log show records the shutdown initiated at 23:35:15 AEST by halt[85500]<-suhelperd[288] (the software-update helper), previous shutdown cause 3, and a shutdown_stall_2026-09-18-233535 report. Colima did not come back.

launchctl print gui/501/homebrew.mxcl.colima at 00:06 AEST: state = not running, runs = 1, last exit code = 1. /usr/local/var/log/colima.log records the single attempt 2026-09-18 23:39:12 starting colima and then 2026-09-18 23:39:22 level=fatal msg="error starting vm: error at 'starting': exit status 1". The cause is in ~/.colima/_lima/colima/ha.stderr.log: level=fatal msg="mkdir /Volumes/FATTY: permission denied" at 23:39:23 AEST. colima.yaml lists four mounts (/Users/dwayne, /Volumes/FATTY, /Volumes/FATTY2, /Volumes/FATTY3); Colima creates a missing mount source, and /Volumes is root:wheel 755, not writable by dwayne, so the missing paths are fatal rather than skipped.

Both drives are simply absent. diskutil list external shows two external disks only: disk2 EMTEC C410 62.9 GB FAT32 and disk3 Seagate BACKUP+ 1.02 TB (2HC015KJ), whose single APFS volume is FATTY3 (759.0 GB used of 1024 GB, 264.8 GB free, 74.1%). No volume named FATTY or FATTY2 exists anywhere, so neither is attached or mounted. ls /Volumes has only EMTEC C410, FATTY3 and the Macintosh HD symlink. FATTY3 holds Archive, JeepDash-2.2.0, movies (257 GB), openclaw-backups, roms (380 GB) but no dockerdata.

Knock-on state at 00:04-00:06 AEST, all verified live:

  • Colima down, so nothing in the media or MindRoom stack runs. docker ps fails on the missing socket /Users/dwayne/.colima/default/docker.sock; port 32400 (Plex) and 8008 (Synapse) both return HTTP 000.
  • arr-schedule.log records 2026-09-19 00:00:00 AEST apply hour=0 arr=up plex=up followed by 2026-09-19 00:02:28 AEST ERROR docker not ready after 120s (is Colima up?), so the whole 00:00-05:00 *arr window is lost.
  • Radarr's config and every *arr movie/download path are on FATTY, so repairing Colima alone does not bring Radarr up. FATTY must be reattached first.

radarr_shrek5_monitor.py behaved correctly for a new failure mode: at 23:53:32 AEST it classified the probe failure as a changed signature and emitted blocked. Local Radarr probe failed: ApiKey not found via docker exec (rc=1, stderr=failed to connect to the docker API at unix:///Users/dwayne/.colima/default/docker.sock; ...) instead of the off-schedule silence it uses for a clean 05:00-24:00 stop. Its container_running() correctly returned unknown rather than healthy.

Nothing was restarted and no recovery was attempted: reattaching the drives is physical, and starting Colima is consequential (it restarts every container). Both need Dwayne.

SUPERSEDED 2026-09-18 23:38: guest virtiofs EBADF fault of 2026-09-17 20:40​

Superseded by the host reboot above, which replaced that fault state rather than resolving it. The 2026-09-17 virtiofs EBADF condition (three shares dead inside the guest, Plex crash-looping 11,671 times by 2026-09-18 00:03 AEST) is no longer observable, because the VM is not running and the host state it depended on is gone. Last recorded evidence for it is retained in the 2026-09-18 00:03 update and the Log entries below. Whether the reboot cleared it is unknown, not confirmed.

Original 2026-09-17 entry follows.

All three virtiofs shares (/Volumes/FATTY, /Volumes/FATTY2, /Volumes/FATTY3) stopped working inside the guest at about 20:40 AEST while still listed as mounted. Any open or readdir under them returns Bad file descriptor [system:9]. Host side is healthy: FATTY is mounted, writable, 113 GiB free of 1.2 TiB (91%), and a host-side touch into /Volumes/FATTY/dockerdata/plex succeeds. Guest root disk is healthy too (touch /tmp/x works), so this is not the 2026-09-12 read-only remount.

Impact: Plex is fully down. Its /config bind is on FATTY, so it crash-loops on create_directories: Bad file descriptor for /config/Library/Application Support/Plex Media Server/Cache. docker logs --timestamps plex shows 2,177 crash pairs from 2026-09-17T10:40:54Z (20:40:54 AEST) through 21:19 AEST and counting. docker ps still says Up 24 hours and docker inspect says RestartCount=0, both accurate and useless here because s6 keeps the container alive while the application restarts (see lessons/2026-09-17-colima-guest-virtiofs-share-fails-ebadf.md). TCP 32400 is held by the Colima ssh mux forwarder, so the port accepts connections and returns an empty reply instead of refusing.

Home Assistant: plex integration not ready yet: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')), retry ladder 5/10/20/40/80/160/320s then 600s from 20:54:02 and 21:09:23 AEST. [plexwebsocket] lines at 20:41:00 (15s), 20:41:15 (30s), 20:41:45 (60s), 20:42:45 (120s), 20:44:45 (240s).

Still healthy: the MindRoom stack (synapse, postgres, redis, cinny, telegram bridge) and mosquitto, all up 24 hours, because their data is on the guest root disk rather than a virtiofs share. The six *arr containers were already stopped by the 05:00-24:00 schedule (arr-schedule.log STOP at 17:00) and will fail to start at 00:00 if the shares are still dead.

Recovery not attempted: the hourly ha-health-check job is a read-only probe by contract and has no authority to restart the VM. Candidate fix is the same as 2026-09-12, colima stop then colima start, which remounts the shares and restarts all 13 containers including Plex and MindRoom, so it needs Dwayne's approval.

Timing note: guest booted 2026-09-16 20:46 AEST, shares died 2026-09-17 20:40 AEST, about 24 hours into VM lifetime. Single observation, not yet a pattern.

Update 2026-09-18 00:03 AEST: fault unchanged, and now two failures compounding.

  • The 00:00 *arr start failed. arr-schedule.log records 2026-09-18 00:00:00 AEST START radarr then Error response from daemon: error while creating mount source path '/Volumes/FATTY2/movies': mkdir /Volumes/FATTY2: file exists and failed to start containers: radarr. Because arr-schedule.sh runs set -euo pipefail, that aborted the run, so sonarr, lidarr, sabnzbd, overseerr and prowlarr were never attempted. All six are still exited.
  • radarr_shrek5_monitor.py did its job: at 00:01:20 AEST it classified a definite stopped container inside the window (hour 0, arr_expected_up() true) and emitted the blocked alert rather than swallowing it.
  • Plex crash pairs now 11,671 (2,177 at 21:19, 5,761 at 22:21, 11,671 at 00:03), last pair 2026-09-18T00:03:04 AEST.
  • Guest root disk healthy (1.6G used of 19G, 9%). Guest memory reads 3913 MB total, not the 6 GiB the Colima profile sets; worth a look separately.
  • Host side healthy and readable: FATTY 113Gi free / 91%, FATTY2 38Gi / 95%, FATTY3 247Gi / 75%. ls /Volumes/FATTY{,2,3}/movies works on the host. Guest ls and stat on every one of the three returns Bad file descriptor.
  • Second observation of the ~24 hour offset: guest boot 2026-09-16 20:46, shares dead 2026-09-17 20:40. Treat a virtiofs/hostagent lifetime bug as the leading candidate.
  • New candidate correlate, unproven: /Volumes/ESD-USB (disk5, 30.8 GB removable USB, no file system) appeared at 2026-09-17 21:08:50 AEST, and Finder logged CACHE_DELETE_ERROR = "Bad volume: /Volumes/ESD-USB" at 20:15:16 AEST. The USB appeared after the shares died, so it cannot be the trigger for this occurrence. Recorded only because a host volume event around the failure window is worth checking on the next occurrence.

RESOLVED 2026-09-12 06:22: guest filesystem I/O errors, 6 *arr containers cannot start​

Fix: colima stop then colima start. No store rebuild was needed. All five previously-failing operations now succeed: docker images, docker system df, docker logs radarr, docker exec plex, colima ssh. All 13 containers came back on their unless-stopped policies; the three exited containers (lidarr, sabnzbd, overseerr) were started manually. Plex /identity 200, synapse /health 200, Cinny 200 (it had been returning 500 before the fault and recovered too), mosquitto up. All six *arr ports answer: radarr 17878, sonarr 18989, prowlarr 19696 (302), lidarr 18686, sabnzbd 18090, overseerr 15055 (307). HA re-setup radarr/sonarr/sabnzbd/lidarr integrations at 06:21:01 with no new errors, so the 600s retry backoff cleared. Disk: 20 GiB free / 91%.

Why a restart was sufficient: the fault was the guest root filesystem remounted read-only, not corrupted data. Read-only is consistent with every observation: reads succeeded (docker network ls, docker inspect, docker ps -a all returned correct data) while every write failed (containerd meta.db, container hosts files, runc process files, overlayfs mkdirs), and the VM was up with the Docker daemon answering. The earlier "containerd store needs rebuilding" hypothesis was wrong, and so was the earlier "corruption of already-written state": the guest disk held only ~1.6 GB of a 20 GiB sparse file, so nothing ran out of room inside the VM. A plain stop/start remounts the guest filesystem read-write and restores normal service. Prefer stop/start before considering colima delete.

Root cause of the read-only remount: the host Data volume hit ENOSPC on 2026-09-11 15:43-16:06 AEST. The VM's sparse disk file ~/.colima/_lima/colima/disk stopped growing at exactly 2026-09-11 16:06:44 AEST, inside that window. Guest log ~/.colima/_lima/colima/ha.stderr.log line 27106 onwards records no space left on device at 16:04:35-16:07:12; HA's ~/.homeassistant/camera-vision.log line 127 records OSError: [Errno 28] at the 15:59:41 sweep. Fix the host disk first or the fault returns. Dwayne approved reclaim on 2026-09-12: deleted two oldest automatic HA backups (3,385,036,800 B and 3,394,037,760 B) plus ~/.hermes/backups/pre-update-2026-08-27-024100.zip (2,594,260,684 B) = 8.73 GiB, taking the host from 11 GiB / 95% to 19 GiB / 91%.

Historical fault description (kept as evidence, superseded by the resolution above).

Observed 2026-09-12 01:13 to 01:40 AEST. This is a different fault from the 2026-09-10 memory-pressure wedge: the Colima VM's guest filesystem returns input/output error (EIO) on every write and on some reads. The VM is up, the Docker daemon answers, and the six containers that were already running (Plex, mosquitto, the four MindRoom containers) still serve traffic. Anything that needs to create or open a file on the guest disk fails.

Evidence (all live probes this session):

  • docker start fails for 6 of 6 *arr containers. Radarr/Sonarr/Prowlarr: open /var/lib/docker/containers/<id>/hosts: input/output error. Lidarr/SABnzbd/Overseerr: failed to mount : mkdir /var/lib/docker/rootfs/overlayfs/<id>: input/output error.
  • docker exec plex echo fails: OCI runtime exec failed: open /tmp/runc-process3229729590: input/output error.
  • docker create --name io_probe_test alpine true fails: write /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db: input/output error.
  • docker system df and docker image ls fail on missing containerd blobs (EIO).
  • docker rename fails writing .tmp-config.v2.json.
  • colima ssh -- echo guest-ok fails with Input/output error, exit status 1.
  • Reads still work: docker network ls, docker inspect, docker ps -a all return correct data.

docker ps is stale and cannot be trusted here: it lists radarr, sonarr and prowlarr as Up 18 hours while docker inspect reports exited running=false. Trust docker inspect for container state.

Still healthy (verified 01:13 to 01:40 AEST): Plex HTTP 200 on /identity and /library/sections, synapse /health returns OK, mosquitto TCP 1883 open, Cinny TCP 8080 open but HTTP 500 (Cinny was already returning 500 before this fault, unclear whether related). mindroom-synapse and mindroom-redis report Docker health unhealthy while still serving.

Impact: the *arr stack is down for its whole 00:00-05:00 window. arr-schedule.sh runs set -euo pipefail, so the first failed docker start (radarr, 00:00:03) aborted the script; the log ends there and the remaining five containers were never attempted. Launchd retries only at 05:00 and 17:00.

Suspected cause (assumed, not proven): host Data volume hit 99% full on 2026-09-11 between 15:43 and 16:05, and the Colima disk images were written in that window (datadisk mtime 15:59:42, root disk mtime 16:06:44). A write interrupted by a full host disk can leave a corrupt ext4 guest image. Host is now at 95% (11 GiB free) and the host itself is healthy: dd read and write tests on the host filesystem pass at 1.2 to 2.0 GB/s, vm.swapusage shows 3803 MB of 4096 MB used, load average 5.16. The corrupt state has persisted for about 9 hours, so it will not self-heal.

Not attempted: no recovery was run. colima stop --force && colima start restarts every container including Plex and the MindRoom stack, which is a consequential action needing Dwayne's approval. The hourly ha-health-check job is a read-only status probe by contract and has no authority to restart the VM.

2026-09-10 incident signature (observed 00:30 to 01:20 AEST): the Docker daemon degrades under memory pressure before it stops answering. Colima lists the VM as running and containers show running, but daemon API calls time out unevenly (some succeed in 0.5s, others hang). Radarr keeps accepting TCP on 17878 but closes without responding to HTTP. The Radarr container had been OOM-killed earlier (State.OOMKilled=true, restartCount=0). /api/v3/health and /ping still answer while /api/v3/movie (310 movies, 1.4 MB) times out at 20s and 45s because cold DB reads through virtiofs take 39s+ after a VM restart.

Recovery that works: colima stop --force && colima start (about 60s), wait for containers to come up from restart policy, then warm Radarr once with a long-timeout /api/v3/movie call before any 20s-timeout monitor probe. After recovery, Radarr reported Shrek 5: monitored, announced, no file, empty queue and the monitor went silent.

Hardening applied: ~/.hermes/scripts/radarr_shrek5_monitor.py fetch timeout raised from 20s to 45s so a post-boot cold cache does not produce false "probe failed" alarms.

superseded 2026-09-13: the 2 GiB Colima memory budget was raised to 6 GiB on 2026-09-12, so the OOM-kill root risk below no longer describes the current VM. Kept as evidence for the 2026-09-10 incident signature.

The (now superseded) 2 GiB Colima memory budget for 13 containers (media stack plus MindRoom stack) was the standing root risk for repeat OOM kills. Confirmed cyclic pattern 2026-09-10: after each colima restart the stack is healthy for roughly 5 to 10 minutes, then the VM wedges again (Radarr, SAB, and the Docker daemon all stop answering HTTP while TCP ports stay open). One restart per incident is enough; repeated restarts are a retry storm. The candidate durable fix is a Colima profile increase (e.g. colima stop && colima start --memory 4), which needs Dwayne approval because it restarts every container including Plex and MindRoom.

Sources: live docker-socket, port, and API probes plus Radarr state inspection, 2026-09-10 00:50 to 01:25 AEST; monitor script and its state file.

RESOLVED 2026-09-12 17:12: monitor false alarm from the *arr shutdown window​

radarr_shrek5_monitor.py reported Local Radarr probe failed: <urlopen error [Errno 61] Connection refused> at 17:06 AEST. This was not a fault. arr-schedule.sh stops the six *arr containers at 05:00 AEST and starts them at 00:00 (/Users/dwayne/ARR/arr-schedule.log: 2026-09-12 17:00:06 AEST STOP radarr), so connection refused off-window is the schedule working. Radarr's own docker inspect confirms the clean stop: exited started=2026-09-11T21:57:32Z finished=2026-09-12T07:00:06Z exit=0 oom=false.

Fix: the monitor now classifies the failure instead of alarming on it. In the except branch it calls container_running("radarr"); when docker inspect reports the container stopped AND arr_expected_up() is false (local hour outside 00:00-05:00 Australia/Sydney), it records an off_schedule signature and exits [SILENT]. Every other case still alarms: container stopped inside the window, container running yet unreachable (a real daemon wedge), or container state unreadable (None, treated as unknown, never healthy). The successful-probe branches are untouched, so an off-schedule restart that leaves Radarr running is still polled and a download is still caught.

Verified by falsification against the live environment, 17:12 AEST: window + stopped alarms; window + state unreadable alarms; off-window + running alarms; off-window + stopped is silent; off-window + state unreadable alarms; two consecutive off-schedule runs stay silent; and the no-movie, watch-only, active-queue and downloaded messages all still emit on first sight. py_compile passes and a live run at 17:12 AEST printed [SILENT] and wrote {"off_schedule": true, "container": "radarr", "localTime": "2026-09-12 17:12 AEST"}. The test was then falsified against the pre-fix backup: it exits 1 there, failing exactly the four off-schedule cases (off-window + down, repeat run, 05:00 boundary) while the in-window alarm cases still pass, so the test is demonstrated to catch the bug it guards. Script sha256 071fae1b308545b0610e373d16daa4d654fde92f3067df6c61620e9b389f0e60. Backup of the previous script: ~/.hermes/backups/monitor-patches/radarr_shrek5_monitor.py.20260912-171123.bak. Not a recurrence of the 07:53 AEST timeout event today (that one was a 45s cold-cache timeout that recovered by the 08:23 poll).

Separate trap found while recording this: the kl shim on PATH (/Users/dwayne/.local/bin/kl) executes /Users/dwayne/Projects/knowledgelayer/tools/kl.py, which is a stale second clone (HEAD 5fb5499, 2026-08-24) with no infra/arr-colima-stack.md. Running kl commit from the canonical repo therefore printed nothing to commit and discarded the knowledge write silently. The canonical brain is /Users/dwayne/.hermes/knowledgelayer (origin https://github.com/dwaynehelena/knowledgelayer.git); commit there with python3 tools/kl.py commit or by repointing the shim. Source: live shim inspection, git logs of both trees, 17:15 to 17:18 AEST this session.

RESOLVED 2026-09-16 21:05: Shrek 5 monitor false alarm, plus a docker-context defect that hid the classification​

radarr_shrek5_monitor.py reported Local Radarr probe failed: <urlopen error [Errno 61] Connection refused> at 20:44 AEST. Two separate things were wrong at once.

  1. The Colima VM restarted during the window. Guest uptime was 11 min at 20:57:20 AEST (boot about 20:46), the limactl hostagent for colima started 2026-09-16 20:45:10 AEST, and the docker socket file ~/.colima/default/docker.sock has mtime 20:46. So HTTP 17878 was genuinely refusing at 20:44. The *arr containers themselves were stopped by schedule as designed (arr-schedule.log: 2026-09-16 17:00:02 AEST STOP radarr). After the VM came back, Plex started at 20:47:15 AEST and radarr stayed exited with exit code 0.

  2. The monitor could not tell that apart, for an environment reason. container_running() shells docker inspect, and the CLI had currentContext = default pointing at unix:///var/run/docker.sock, which does not exist on this host. The CLI aborted before reaching the daemon, the helper returned None (unknown), and the script's off-schedule branch needs a definite False, so every occurrence became a "blocked" alert instead of the intended [SILENT]. ~/.docker/config.json now reads currentContext: colima with mtime 20:47:21 AEST, set by colima start; before that restart the context was default.

Fix applied to /Users/dwayne/.hermes/scripts/radarr_shrek5_monitor.py: at import, if DOCKER_HOST is unset, /var/run/docker.sock is absent, and the Colima socket exists, set DOCKER_HOST=unix:///Users/dwayne/.colima/default/docker.sock. That makes the classification independent of whatever context the CLI happens to hold. Backup of the previous script: ~/.hermes/backups/monitor-patches/radarr_shrek5_monitor.py.20260916-205033.bak.

Verified by falsification, 21:05 AEST: with DOCKER_CONFIG pointed at a config whose currentContext is default and DOCKER_HOST unset, the pre-fix script prints the exact 20:44 blocked message while the fixed script prints [SILENT] and writes {"off_schedule": true, "container": "radarr", "localTime": "2026-09-16 21:04 AEST"}. Also confirmed the fix respects a pre-set DOCKER_HOST, returns a real boolean from container_running("radarr"), degrades to None on a dead socket path, and that py_compile passes. Dwayne's stored state file was restored to its live value after the test.

Unresolved: what stopped the VM just before 20:45. The launchd job homebrew.mxcl.colima runs colima start -f in the foreground under KeepAlive, pid 977 since 2026-09-15 09:27, and the replacement VM came up under it. No cron job, script, or prompt on this host references a colima start command. Guest dmesg shows the usual virtiofs workqueue stalls, EXT4-fs (vdb1): error count since last fsck: 6 with the last error at an unrecorded earlier time, and clocksource watchdog delays, none of which date to this window. Next occurrence: capture ~/.colima/_lima/colima/ha.stderr.log before it rotates, since a plain colima start truncates the previous instance's log.

Log​

  • 2026-09-22 09:20 AEST (Plex customConnections, approved follow-up): Fixed a defect that a plain reading of the setting would have missed. customConnections is what the server publishes to plex.tv for client discovery (server-side summary: "A comma-separated list of URLs (http or https) which are published up to plex.tv for server discovery"). Appending the plain-HTTP form http://100.80.168.13:32400 broke the publish: servers.plex.tv answered every PUT /devices/<id>?Connection[][uri]=... with 403, 9 consecutive times, and the connection never appeared in https://plex.tv/api/resources, while Plex logged MyPlex: Updating device connections failed, retrying in 20/40/80/160 seconds. Proved causation by A/B: reverting to LAN-only and restarting gave 0 403s, re-adding the tailnet URL and restarting gave 9 403s again; the 403 log is the only one whose PUTs carried the tailnet URI (18 such PUTs, 9 403s) and neighbouring logs with zero tailnet PUTs had zero 403s. Root cause: the publish PUT sets httpsEnabled=1, so plex.tv accepts HTTPS plex.direct URIs, not bare http:// addressing for this slot. The working form uses the same wildcard cert the server already presents: https://100-80-168-13.2e16709bdfe749aba50aa09a795c2839.plex.direct:32400 (cert CN=*.2e16709bdfe749aba50aa09a795c2839.plex.direct, Let's Encrypt, valid to 2026-12-17). Final state: customConnections = http://192.168.68.79:32400,https://100-80-168-13.<hash>.plex.direct:32400; 0 403s sustained over 5+ minutes; plex.tv now lists the tailnet connection (local=0, correctly classified remote) alongside the two LAN entries. All four entry points return 200: LAN http://192.168.68.79:32400, tailnet http://100.80.168.13:32400, the published plex.direct HTTPS URL (media fetch 76,373 bytes), and MagicDNS http://macbook-pro-3.tail9e4b21.ts.net:32400; 3 libraries intact. Note the plain http://100.80.168.13:32400 URL still works directly for a client that already knows it; only the published discovery slot needs the HTTPS form. Backups: ~/.hermes/state/plex-Preferences.xml.bak-20260922-090404 (pre-change, sha256 ef59e5c3bf7cd4d4) and .bak-withtailnet (the broken intermediate, kept only to document the failed attempt). Source: live /:/prefs (including the server's own summary text for customConnections), the publish PUT URLs extracted from the Plex logs, plex.tv/api/resources, A/B restart test, direct curl on all four addresses., hermes.

  • 2026-09-22 05:53 AEST (Plex over Tailscale, approved live config change): Plex was already reachable on the tailnet; the fix was a bandwidth-classification gap, not exposure. docker ps shows the container publishing 0.0.0.0:32400->32400/tcp (and [::]), so Plex was never localhost-bound. Live probes over the tailnet IP 100.80.168.13 returned 200 on /identity (machine id cc7feeacacbc05b4, version 1.43.3.10896), 200 on /web/index.html (30,184 bytes), the authed /library/sections listing all three libraries (Movies, TV Shows, Music), and a real 76,373-byte media fetch in 12 ms. MagicDNS resolves: http://macbook-pro-3.tail9e4b21.ts.net:32400/identity also 200. The actual defect: allowedNetworks was 192.168.68.0/22,172.20.0.0/16,172.17.0.0/16,127.0.0.1/32 with the Tailscale CGNAT range absent, so Plex classified tailnet clients as WAN and applied remote bandwidth limits (forced transcodes where direct play was possible). Dwayne approved adding the tailnet. Change applied: appended 100.64.0.0/10 to allowedNetworks in /Volumes/FATTY/dockerdata/plex/Library/Application Support/Plex Media Server/Preferences.xml (sha256 f03eb3909fff36df -> afeac640835842b3), then docker restart plex with 0 active sessions. Plex answered HTTP 200 again after 5 seconds. Verified the RUNNING server loaded it via /:/prefs (not just the file): allowedNetworks = 192.168.68.0/22,172.20.0.0/16,172.17.0.0/16,127.0.0.1/32,100.64.0.0/10, and it also reports secureConnections = 1. Post-restart re-test on the tailnet: /identity 200, /web 200, real media 200 / 76,373 bytes, all 3 libraries intact. Rollback: pre-change copy at ~/.hermes/state/plex-Preferences.xml.bak-20260922-055209 (sha256 f03eb3909fff36df, 913 bytes); restore it and restart the container. Note customConnections still points at the LAN address http://192.168.68.79:32400; harmless for tailnet reach but it is the value Plex advertises to clients, so it is worth revisiting if remote clients ever connect via a non-LAN address. Source: live docker ps/docker inspect, lsof, /:/prefs, direct curl over the tailnet, Preferences.xml before/after., hermes.

  • 2026-09-20 06:02 AEST (weekly synthesis, live probes): recovery confirmed end to end. colima status reports running on Virtualization.Framework with virtiofs and the docker socket at /Users/dwayne/.colima/default/docker.sock. docker ps shows only plex Up 23 hours and mosquitto Up 23 hours, which is correct for 06:02 because the *arr containers run only in the 00:00-05:00 window. arr-schedule.log shows the window closed normally: 2026-09-20 05:00:23 AEST STOP prowlarr, 2026-09-20 05:00:27 AEST OK plex already running, 2026-09-20 05:00:27 AEST done. Host disk at the same read: /System/Volumes/Data 187 GiB used of 233 GiB (90%), 21 GiB free. The 2026-09-19 02:12-02:25 outage entry is retained as history and now marked superseded in Now. Source: live colima status, docker ps with DOCKER_HOST=unix:///Users/dwayne/.colima/default/docker.sock, arr-schedule.log tail, df -h, 2026-09-20 06:02 AEST., hermes.

  • 2026-09-19 (cron, 02:12-02:25 AEST): outage unchanged 2h47m after the 23:38 host start. Live: ls /Volumes has only EMTEC C410, FATTY3, Macintosh HD; colima status reports "not running"; launchctl print gui/501/homebrew.mxcl.colima reads state = not running, runs = 1, last exit code = 1; /Volumes/FATTY2 also absent. diskutil apfs list shows exactly two containers (disk1 internal, and disk4 = disk3s2 1.0 TB carrying the single volume FATTY3, 759.0 GB consumed / 264.8 GB free / 74.1%), so FATTY and FATTY2 exist nowhere, not merely unmounted. New cause evidence: /var/log/install.log records SU:macOS Sequoia 15.8 24H23 and an installer boot.efi entry at 23:38:09, so the 23:35:15 halt[85500]<-suhelperd[288] was the Sequoia 15.8 software-update install. Home Assistant itself returned at 23:40:43 (2026.8.2) and serves /api/ 200. Knock-on beyond the media stack: the 00:00-05:00 *arr window is unrecoverable after arr-schedule.log 2026-09-19 00:02:28 AEST ERROR docker not ready after 120s; /Users/dwayne/ARR/spillover_guard.log (mtime 02:10) now ends in a crash loop, df: /Volumes/FATTY: No such file or directory then CalledProcessError on ['df', '-k', '/Volumes/FATTY'], because the guard assumes the mount exists; ~/.homeassistant/colima_monitor/collector.log (mtime 02:21) fails on colima ssh --profile default -- cat /proc/meminfo exit 1; Plex 32400, Synapse 8008 and Cinny 8080 all return HTTP 000. HA unavailable count 172 (sensor x67, button x21, automation x20, light x16, binary_sensor x13), dominated by 52 roborock, 33 *arr family, 21 tuya LED, 12 driveway, 6 colima-memory, 3 Plex; SABnzbd API ERROR repeats on a 10-minute cadence through 02:21:25. radarr_shrek5_monitor.py still classifies the probe as blocked (state file api-key-not-found signature, checkedAt 2026-09-18T16:10:05Z) rather than off-schedule silence. No recovery attempted: FATTY and FATTY2 are physical and a Colima start restarts all containers; this hourly probe is read-only by contract. Source: live ls, diskutil, colima, launchctl, df, curl, install.log, arr-schedule.log and collector.log probes 02:12-02:25 AEST this session., hermes.

  • 2026-09-19 (cron, 00:04-00:08 AEST): host reboot at 2026-09-18 23:38:01 AEST (shutdown initiated 23:35:15 by halt[85500]<-suhelperd[288], cause 3, shutdown_stall_2026-09-18-233535 report) left Colima down and FATTY/FATTY2 detached. launchctl print gui/501/homebrew.mxcl.colima reports not running, last exit code = 1; colima.log shows one attempt at 23:39:12 and fatal ... error starting vm: error at 'starting': exit status 1 at 23:39:22, with ha.stderr.log mkdir /Volumes/FATTY: permission denied at 23:39:23, /Volumes is root:wheel 755 and Colima treats the absent mount sources as fatal. diskutil list external shows only disk2 EMTEC C410 62.9 GB FAT32 and disk3 Seagate BACKUP+ 1.02 TB serial 2HC015KJ, whose sole APFS volume is FATTY3 (759.0 GB used / 264.8 GB free, 74.1%); no FATTY or FATTY2 volume exists. Consequently Plex 32400 and Synapse 8008 answer HTTP 000, the docker socket is missing, and arr-schedule.log logs 2026-09-19 00:00:00 AEST apply hour=0 arr=up plex=up then 2026-09-19 00:02:28 AEST ERROR docker not ready after 120s (is Colima up?), losing the entire 00:00-05:00 window. Radarr's config and every *arr media path sit on FATTY, so the drive must return before Colima or Radarr can. radarr_shrek5_monitor.py correctly emitted the blocked alert (new signature) at 23:53:32 rather than the off-schedule silence. No recovery attempted: reattaching the drives is physical and starting Colima is consequential. Source: live launchctl/colima.log/ha.stderr.log/diskutil/system_profiler/log show/curl/arr-schedule.log probes 23:53-00:08 AEST this session., hermes.

  • 2026-09-13 (weekly synthesis): Verified the 2026-09-12 resource fix is live and stable. colima status reports 4 CPU / 6 GiB (~/.colima/default/colima.yaml cpu 4, memory 6, disk 100, arch x86_64); guest free -m shows 5923 MB total with 3334 free, 5031 available, zero swap, and guest root /dev/root 1.6G used of 19G (9%). 7 containers up 22 hours (plex, mindroom-telegram-bridge, mindroom-synapse healthy, mindroom-postgres healthy, mindroom-redis healthy, mindroom-cinny-local, mosquitto). Host Data volume 20 GiB free / 91%. Source: live colima status, colima ssh -- free -m, docker ps, df -h /System/Volumes/Data, 06:02 to 06:04 AEST this session., hermes.

  • 2026-09-12: Fixed the Shrek 5 monitor false alarm caused by the 05:00 *arr shutdown. The 17:06 AEST Connection refused was the schedule, not a fault (radarr clean exit 0 at 17:00:06 AEST). Monitor now stays silent when docker inspect reports the container stopped off-window, and still alarms for in-window stops, running-but-unreachable, and unreadable state. Falsification-tested against six cases plus the four success branches; live run returned [SILENT]. Source: live docker inspect, arr-schedule.log, cron executions db and output files, 17:07 to 17:14 AEST this session., hermes.

  • 2026-09-12: Full backup cleanup executed on Dwayne's explicit instruction ("cleanup all backups", then confirming the widest option). Removed six backup directories totalling 9.97 GiB plus one loose archive (503.2 MiB): .homeassistant/backups 1.40 GiB, .hermes/backups 3.28 GiB, .openclaw/backups 0.06 GiB, ~/backups 2.40 GiB, .mindroom/backups 1.95 GiB, ha-backups 0.91 GiB, ~/openclaw-backup-20260831.tar.gz 503.2 MiB. Host Data volume went 21.61 GiB free / 91% to 32 GiB free / 85%. Note ~/backups and ~/Backups are the SAME directory on this case-insensitive filesystem (same inode 841946), so the true total is ~10 GiB, not the 12 GiB a naive per-path sum suggests. Pre-deletion checks: no live data paths (grepped .hermes/cron/jobs.json, .mindroom/config.yaml, .openclaw/openclaw.json - no backup-dir references); ~/.hermes/scripts/apply-local-hermes-patches.sh points at ~/.hermes/backups/local-patches/ which did not exist and the script is not scheduled; weekly-ha-backup (Sun 03:30) and backup-integrity-verify (Sun 04:47) both still run. Verified after: 13/13 containers up, Plex 200, radarr 200, HA 200, no HA error lines. HA recreates .homeassistant/backups on its own automatic schedule (automatic_backups_configured: true, create_backup.agent_ids: [backup.local], last completed 2026-09-12T05:21:39), and ha-backup.py only globs that path (line 45) and handles OSError -> returns None, so a missing directory is tolerated rather than fatal. Preserved deliberately: ~/.hermes/state.db (1.3 GB) and ~/.codex/state_5.sqlite (348 MB), which are live databases, not backups. Source: live df/du/stat/inode and container/API probes 06:25 to 06:36 AEST this session; file inspection of the three consumer scripts., hermes.

  • 2026-09-12 04:28: Fault still active 10.5h after onset; no self-heal. Live re-probe: all six *arr containers docker inspect running=false (radarr/sonarr/prowlarr/overseerr exit 137, sabnzbd/lidarr exit 128); docker start radarr still fails with open /var/lib/docker/containers/e443e7.../hosts: input/output error; colima ssh -- echo ok still returns Input/output error; docker exec plex true fails open /tmp/runc-process...: input/output error; docker system df fails on a missing containerd blob. HA logged the SABnzbd failure every 10 min through 04:20:57 AEST, 7 lines in the trailing 70 min. docker ps still mislists radarr/sonarr/prowlarr as Up 21 hours while inspect says exited; no host listeners on 7878/8989/9696/8686/5055. Still serving: Plex and the MindRoom containers. Host Data volume now 96% used, 9.2 GiB free. Note: the hourly ha-health-check alert has now fired at 01:42, 02:17, 03:21 and 04:12 AEST; it is a read-only probe with no restart authority. Source: live docker/colima/curl probes 04:20 to 04:28 AEST this session., hermes.

  • 2026-09-12: Diagnosed Colima guest-filesystem EIO fault blocking all 6 *arr container starts; distinct from the 2026-09-10 memory wedge; Plex/MindRoom/mosquitto still serving; suspected cause is the 2026-09-11 15:43-16:05 host disk-full window corrupting a guest disk image; recovery not attempted pending approval. Source: live docker, colima ssh, host dd and df probes 01:13 to 01:40 AEST this session., hermes.

  • 2026-09-12: Causation confirmed for the guest EIO fault, and host reclaim executed. The VM's sparse disk file ~/.colima/_lima/colima/disk (21,474,836,480 bytes allocated) stopped growing at exactly 2026-09-11 16:06:44 AEST, inside the host Data-volume ENOSPC window. Guest log ~/.colima/_lima/colima/ha.stderr.log line 27106 onwards records Failed to write to log, write /dev/stderr: no space left on device at 16:04:35 to 16:07:12 AEST; HA's ~/.homeassistant/camera-vision.log line 127 records OSError: [Errno 28] No space left on device writing www/camera-stills/vision_analysis.tmp at the 15:59:41 sweep. So the sequence is host volume full -> guest writes fail -> containerd/docker store corrupt. This is corruption of already-written state, not a capacity limit: the guest disk is only ~1.6 GB used of a 20 GiB sparse file, so freeing host space alone cannot repair it. Dwayne approved reclaim; deleted the two oldest automatic HA backups (2026-09-10 3,385,036,800 B and 2026-09-11 3,394,037,760 B) and ~/.hermes/backups/pre-update-2026-08-27-024100.zip (2,594,260,684 B) = 8.73 GiB. Host Data volume went 11 GiB free / 95% -> 19 GiB free / 91%; host writes verified healthy (64 MiB dd probe), HA HTTP 200, camera sweeps completing again. The Colima EIO errors persist unchanged: docker images, docker system df, docker logs radarr, docker exec plex and colima ssh all still fail. Remaining backups: one HA automatic (2026-09-12) and one Hermes pre-update (2026-09-10). Source: live df/dd/curl/docker/colima probes 06:12 to 06:14 AEST this session, file mtimes, log greps., hermes.

  • 2026-09-12: RESOLVED the guest EIO fault with colima stop + colima start; no store rebuild needed. Fault was the guest root filesystem remounted read-only, not corrupted data (reads succeeded, every write failed; guest used only ~1.6 GB of a 20 GiB sparse file). All 13 containers back, all six *arr ports answering, Plex/synapse/Cinny 200, HA arr integrations re-set up 06:21:01. Corrected two earlier wrong hypotheses on record in this file: "containerd store needs rebuilding" and "corruption of already-written state" were both disproved by the successful plain restart. Source: live docker/colima/curl probes and HA log 06:15 to 06:22 AEST this session., hermes.

  • 2026-09-10: Diagnosed Shrek 5 monitor timeout: Colima Docker daemon degraded under memory pressure; Radarr container previously OOM-killed; /api/v3/movie timed out past 45s on cold cache. Recovered with colima restart; all 7 media containers plus Plex verified 200 on their ports; monitor script run returns correct watch-only state and stays silent; monitor fetch timeout hardened 20s to 45s. 2 GiB VM budget flagged as root constraint. Source: live probes this session., hermes.