Skip to main content

A single slow HTTP sample is not evidence of a down site

entry

Observed: 2026-09-17, ~/.hermes/scripts/public_site_uptime_monitor.py (cron site-uptime).

What happened​

The monitor reported DOWN dwaynehelena.com: slow 24.8s (HTTP 200). The site was healthy.

Reproduced live the same hour: 3 of 5 urllib samples to https://dwaynehelena.com returned 200 in 0.08-0.14s. One sample took 11.5s. The single slow read is what fired the DOWN.

The real mechanism: the host stalls, not the site​

The stall is host-wide, not site-specific. Same window, unrelated hosts:

  • example.com over curl: 5 tries, 2 took 11.8s and 11.2s, the rest 0.04-0.05s.
  • apple.com over curl: 6 tries, all 0.07-0.20s.
  • dwaynehelena.com over curl x13: 12 tries at 0.04-0.4s, one at 1.8s and one curl at 9.7s.
  • dns, tcp and tls all sub-4ms on the fast tries, so it is not DNS.

The stall lands on the TLS/connect step and clears on retry. 76.76.21.21 (Vercel) answered fast on every fast try, and gateway ping to 1.1.1.1 was 6.7-8.4ms with 0% loss.

The fix​

check() now takes two samples and reports DOWN only when they agree. A fast second sample, or any second sample faster than the first, counts as up. A genuinely slow endpoint still reports DOWN (slow 9.9s (HTTP 200)) because both samples stay slow.

Verified with four probes after the change:

  • dead host (*.invalid) still returns DOWN URLError
  • a forced consistently-slow endpoint still returns DOWN slow, and took both samples
  • a forced stall-then-fast sequence returns UP (the false positive is gone)
  • four live runs of the script: one RECOVERED line, then [SILENT] x3

Rule​

Two agreeing samples before a latency-based DOWN. Latency is a property of the path, not the endpoint. When one monitored host stalls against everything, the fault is local; widen the probe to a control host before reporting a site down.

  • lessons/2026-09-14-ha-states-read-timeout-shorter-than-stall.md, same class, read timeout shorter than the observed stall.
  • lessons/2026-09-12-domain-health-timeout-smaller-than-lobe-runtime.md, same class.