Colima guest I/O errors are a read-only remount, not store corruption
hermes · confidence high · status active
tags: colima docker disk enospc eio recovery false-hypothesis
When the host volume fills, the Colima VM's guest filesystem remounts read-only. Every
write inside the guest then fails with input/output error (EIO), which reads like disk
or database corruption and is not. Observed 2026-09-12 on macbook-pro-3: all six *arr
containers refused to start (open /var/lib/docker/containers/\<id>/hosts: input/output error, or failed to mount : mkdir /var/lib/docker/rootfs/overlayfs/\<id>),
docker exec plex failed on open /tmp/runc-process..., docker create failed writing
containerd meta.db, and colima ssh -- echo itself returned Input/output error.
Two signals separate this from corruption. Reads still work: docker network ls,
docker inspect and docker ps -a all returned correct data while every write failed.
And the guest had barely used its disk, roughly 1.6 GB of a 20 GiB sparse file, so no
capacity limit was reached inside the VM.
The fix is colima stop then colima start, with no store rebuild. All 13 containers
returned on their unless-stopped policies, Plex and synapse answered 200, and every
*arr port responded. colima delete was never needed. Two hypotheses recorded during
the incident were disproved by that plain restart: "containerd store needs rebuilding"
and "corruption of already-written state".
Fixing the host disk is mandatory, not optional. The guest disk file
~/.colima/_lima/colima/disk stopped growing at exactly 2026-09-11 16:06:44 AEST,
inside the host Data volume ENOSPC window; guest ha.stderr.log recorded no space left on device from 16:04:35 to 16:07:12 AEST and HA logged OSError: [Errno 28]. Without
reclaim the fault returns.
Also worth remembering: docker ps lies here. It listed radarr, sonarr and prowlarr as
Up 18 hours while docker inspect reported exited running=false. Trust
docker inspect for container state.
Rule: on guest EIO, check host free space first, then do one stop/start. Do not rebuild
the Docker store, do not delete the VM, and do not trust docker ps during the fault.
Log
- 2026-09-13 (weekly synthesis): Distilled the 2026-09-12 incident into a reusable
diagnostic rule. Confirmed the fix held: on 2026-09-13 the VM runs 4 CPU / 6 GiB with
the guest root at 1.6 GB of 19 GB, 7 containers up 22 hours, and host Data volume at
20 GiB free / 91%. The 2 GiB memory budget that made the stack fragile was raised to
6 GiB on 2026-09-12. Source:
~/.hermes/knowledgelayer/infra/arr-colima-stack.mdplus live probes., hermes.
Sources:
~/.hermes/knowledgelayer/infra/arr-colima-stack.md · ~/.colima/_lima/colima/ha.stderr.log · ~/.colima/default/colima.yaml · live colima and docker probes 2026-09-12