Skip to main content

Colima guest virtiofs share failing with EBADF looks like a Plex bug and is not

lesson · created 2026-09-17 · actor hermes · confidence high · status active tags: colima docker virtiofs plex ebadf recovery false-hypothesis

When the Colima VM's virtiofs share to a host APFS volume breaks, every process inside the guest gets Bad file descriptor (EBADF, system 9) on any open or readdir under that mount point. It reads like a bug in whichever container touches the share first.

Observed 2026-09-17 20:40 AEST on macbook-pro-3. All three shares were dead at once:

  • colima ssh -- ls /Volumes/FATTY -> Bad file descriptor; same for FATTY2 and FATTY3.
  • python3 -c 'os.listdir("/Volumes/FATTY")' in the guest -> OSError: [Errno 9] Bad file descriptor.
  • stat -f /Volumes/FATTY in the guest -> Operation not supported, even though mount still listed lima-3a32249ef427521e on /Volumes/FATTY type virtiofs (rw,relatime).

The share was mounted but non-functional, so the mount table is not evidence of a working share. Probe with a real open or readdir instead.

Plex was the visible casualty because its /config bind is /Volumes/FATTY/dockerdata/plex. The container logged 2,177 crashes in 40 minutes, each a pair of lines:

Starting Plex Media Server. . . libc++abi: terminating with uncaught exception of type boost::filesystem::filesystem_error: boost::filesystem::create_directories: Bad file descriptor [system:9]: "/config/Library/Application Support/Plex Media Server/Cache"

Two traps in reading this:

  1. docker ps reported Plex as Up 24 hours with RestartCount=0. Both are true and both are misleading: s6 supervises the Plex binary inside a container whose own PID 1 never exits, so the container stays up while the application crash-loops. Read docker logs, not docker ps, to learn whether the app is alive.
  2. Port 32400 is held by the Colima ssh mux forwarder (ssh ... [mux], listening as *:32400). TCP connects succeed and HTTP returns Empty reply from server, so a port-open check passes on a completely dead server. Fetch /identity and inspect the body.

Signals that separate this from the two earlier Colima faults:

  • Not guest read-only remount (2026-09-12): guest root writes succeed, touch /tmp/x works. Only the virtiofs mounts fail.
  • Not host disk-full: FATTY was 113 GiB free / 91%, host mounts writable, touch on the host-side path succeeded while the guest path returned EBADF.
  • MindRoom (Synapse, Postgres, Redis, Cinny, mosquitto) stayed healthy throughout, because its data lives on the guest root disk, not on a virtiofs share. Partial survivability is expected, not a sign the fault is mild.

Downstream effect: Home Assistant's plex integration went not ready yet: ('Connection aborted.', RemoteDisconnected(...)) and backed off to 600s retries from 20:54, with [plexwebsocket] Websocket connection failed ... Server disconnected at 20:41:00, 20:41:15, 20:41:45, 20:42:45 and 20:44:45 (15/30/60/120/240s ladder). The hourly ha-health-check summary surfaces only the websocket lines, which understates the fault.

Also note the timing: the guest booted 2026-09-16 20:46 AEST and the shares died 2026-09-17 20:40 AEST, roughly 24 hours into the VM lifetime. One observation is not a pattern; if it recurs at the same offset, treat a virtiofs/hostagent lifetime bug as the candidate cause.

Rule: on guest EBADF against a /Volumes/* path, test the share with readdir in the guest before blaming any container. Fix the share, not the container. Expect the recovery colima stop + colima start, which restarts all 13 containers and needs Dwayne's approval.

Log​

  • 2026-09-17 22:21 AEST (hermes, ha-health-check run, second hour): Fault still active and growing. Plex crash pairs since onset rose from 2,177 at 21:19 to 5,761 at 22:21 (174 in the last 3 minutes). All three guest shares still return EBADF on readdir (colima ssh -- ls /Volumes/FATTY / FATTY2 / FATTY3). HA plex integration still not ready yet ... RemoteDisconnected, retrying every 600s, last attempt 22:19:25. Host side unchanged and healthy: FATTY 113 GiB free / 91%, FATTY2 38 GiB free / 95%, FATTY3 247 GiB free / 75%. MindRoom (synapse, postgres, redis, cinny, telegram bridge) up 26 hours and healthy. Guest booted 2026-09-16 20:46 AEST; shares died 2026-09-17 ~20:40 AEST, so the ~24 hour offset is now two observations of the same VM lifetime. No recovery attempted; read-only probe contract., hermes.
  • 2026-09-18 00:03 AEST (hermes, radarr-shrek5-monitor run): Fault still active, now 27h+. Guest readdir on all three shares still EBADF. Plex crash pairs 11,671 (2,177 at 21:19, 5,761 at 22:21, 11,671 at 00:03), last pair 2026-09-18T00:03:04 AEST. New downstream casualty: the 00:00 *arr start window failed because arr-schedule.sh aborts on the first error (set -euo pipefail); radarr failed to start on error while creating mount source path '/Volumes/FATTY2/movies': mkdir /Volumes/FATTY2: file exists, so the other five containers were never attempted and the whole 00:00-05:00 window is lost. Second observation of the ~24 hour offset (guest boot 2026-09-16 20:46, shares dead 2026-09-17 20:40). Host side healthy throughout (FATTY 113Gi/91%, FATTY2 38Gi/95%, FATTY3 247Gi/75%, all readable). Recovery still colima stop + colima start, still needs Dwayne's approval; this monitor run has no restart authority., hermes.
  • 2026-09-17 (hermes, ha-health-check run): Diagnosed the 20:40 AEST Plex crash loop as a guest virtiofs failure on all three FATTY shares, not a Plex fault. Confirmed guest root writes healthy, host mounts healthy and writable, Plex crash count 2,177, HA plex integration backed off to 600s. Recovery not attempted; read-only probe contract. Source: live colima ssh / docker inspect / docker logs / curl / HA log probes 21:10 to 21:25 AEST this session., hermes.
Sources: ~/.hermes/knowledgelayer/infra/arr-colima-stack.md · live colima/docker/HA probes 2026-09-17 21:10-21:25 AEST · ~/.homeassistant/home-assistant.log · docker logs plex