A socket timeout does not bound a drip-feeding read; bound the wall clock

_{lesson · created 2026-09-29 · actor hermes · confidence high · status active} _{tags: infra reliability python http}

urllib.request.urlopen(req, timeout=6) sets a per-recv socket timeout, not a budget for the response. A server that keeps sending at least one byte before the timeout expires resets the clock on every recv, so the call never returns and no outer deadline check ever runs. Observed 2026-09-29: the HA camera-proxy still refresh sat in _buffered_readline -> sock_recv_into -> poll for 5h06m, grew to a 9.2 GB footprint, and pushed swap to 18.5 GB of 19.4 GB, while the script's own timeout=6 was set on every request.

Two mistakes combine in this pattern, and fixing only one leaves the hang:

A single timeout= is not a deadline. Read the body in a loop with resp.read1(n) (returns as soon as any data is available) and compare time.monotonic() against an absolute deadline on every pass. Do not use resp.read(n): it blocks until it has n bytes or the socket closes, so a deadline checked around it is never evaluated.
A deadline is not a run budget. Give the whole run a wall-clock bound and each item a smaller one, clamped to what remains of the run. On expiry, mark the item skipped and keep the previous artifact rather than failing the run; a monitor should degrade in freshness, not hang.

Also bound the body: a 32 MiB cap stops a runaway or hostile response from filling disk or memory even when it arrives fast enough to keep the deadline satisfied.

Verification pattern that catches this: stand up a local HTTP server that writes one byte per second and never closes, then assert the client raises at the deadline. A test that only calls the client against a healthy server passes on the broken build. The bug reproduces as a hang past the test timeout, so run the check under an outer limit and treat the timeout as the failure signal.

_{Sources: ~/.hermes/scripts/ha_expose_camera_stills.py · ~/.hermes/cron/executions.db · sample of PID 5186 · 2026-09-29}