Skip to main content

PM2 processes with a deleted cwd lie about their own fleet

entry

Symptom​

/api/swarm/health on Nexus (Zappo) reports observability_halt with PM2 unhealthy: nexus-q status=unknown; PM2 unhealthy: nexus-omega status=unknown; PM2 unhealthy: nexus-aether status=unknown; PM2 unhealthy: nexus-x status=unknown while PM2 itself shows all four engines online with live PIDs and heartbeats seconds old.

Cause​

nexus-api (PID 1284, started 2026-09-15 09:27 AEST) and nexus-health (PID 1322, same start) were launched from /Volumes/FATTY/src/nexus. The /Volumes/FATTY volume was remounted at 2026-09-15 11:20, which orphaned those processes' working-directory inode. lsof -p <pid> -d cwd returns no row for both PIDs, which is the tell.

Node's process.cwd() then throws ENOENT: no such file or directory, uv_cwd. _pm2_status_map() in /Volumes/FATTY/src/nexus/api.py shells out to pm2 jlist; the pm2 CLI loads cwd at bootstrap (pm2/lib/API.js:69) and exits 1 before reading the daemon. The API therefore reports status=unknown for every engine and raises an observability halt.

Reproduced 2026-09-17 on Zappo: chdir into a directory, rmdir it, then run pm2 jlist → exit 1 with the identical uv_cwd traceback. Running pm2 jlist from an intact cwd always exits 0, so this is not a PM2 daemon or PATH problem.

Evidence: 806 Unable to read PM2 status lines in ~/.pm2/logs/nexus-api-error.log; 12,424 PM2 unhealthy lines in ~/.pm2/logs/nexus-health-error.log.

Rule​

A PM2-managed process can stay online and serve HTTP while its cwd is gone. Health probes that shell out to pm2 from inside such a process fail silently and report the whole fleet as unknown. When PM2 status disagrees with heartbeats:

  1. Compare lsof -a -p <pid> -d cwd -Fn against pm2 jlist's pm_cwd. An empty lsof cwd row is the false-positive signature.
  2. Trust the real PID plus the engine heartbeat (/api/health engines.*), not the derived pm2_status field.
  3. Clear it with pm2 restart <app> so PM2 re-spawns the process in the current mount. Do not restart the engines, which are healthy.
  4. restart_time=0 plus an online PID does not clear the fault. Both nexus-api (pid 1284) and nexus-health (pid 1322) have shown 0 restarts and no cwd row at the same time on 2026-09-18, because they started before the 2026-09-15 11:20 remount and never respawned. Only lsof -a -p <pid> -d cwd settles it.
  5. Fix the caller, not just the process. api.py calls subprocess.check_output(["pm2", "jlist"], timeout=6) with no cwd=, so it inherits the dead directory and dies. Passing cwd=BASE_DIR makes the same call succeed from a deleted cwd (verified 2026-09-18: inherited cwd -> CalledProcessError exit 1; cwd="/Volumes/FATTY/src/nexus" -> exit 0, 33 apps parsed). health_check.py already does this in _run() (cwd=str(BASE_DIR), line 122), and its error log has 0 uv_cwd lines as a result. Only nexus-api needs the restart; nexus-health reads PM2 fine and its PM2 unhealthy lines are about genuinely stopped idle apps (nexus-metrics, nexus-printer, nexus-profit-tracker), not the engines.
  6. Scope the blast radius before calling it fail-closed. The observability provider is registered only inside api.py's get_swarm_engine() (line 148), so the is_trading_allowed() gate only holds that state in the API process. The four engine processes never register a provider (set_observability_provider appears nowhere else outside tests), so their _observability_halt stays False and the gate returns early. Engine error logs carry no HEALTH_HALT or OBSERVABILITY HALT lines. Earlier notes calling this halt "fail-closed on new entries" were wrong about the live engines; /api/swarm reporting trading_allowed: false is the API's own view.

Recurrence log​

  • 2026-09-18 18:05 AEST: still engaged, nexus-api pid 1284 unchanged since 2026-09-15 09:27 (0 restarts, empty lsof -a -p 1284 -d cwd). The four engines were restarted 2026-09-17 20:23-20:35, so their pm_cwd is current and PM2 shows them online with live PIDs; only the API is stale. pm2 jlist from a removed cwd still exits 1 with the uv_cwd traceback. Fix is unchanged: pm2 restart nexus-api (API only, engines are healthy and untouched). 1,888 returned non-zero exit status 1 lines in the API error log by 18:06.

  • 2026-09-18 21:05 AEST: unchanged at the next monitor run. The scheduled pro_nexus_finance_monitor_report.py reported swarm degraded: OBSERVABILITY HALT with nexus-q/omega/aether/x status=unknown, checked 2026-09-18 21:05:44. Independently confirmed: PM2 shows all four engines online with live PIDs (nexus-q 62949, nexus-x 67643, nexus-omega 60393, nexus-aether 61251, one restart each, started 2026-09-17 20:23-20:35) and heartbeats seconds old (omega 21:08:09, x 21:07:41, aether 21:07:07, q 21:02:25). nexus-api pid 1284 still unfixed (0 restarts, started 2026-09-15 09:27). Counters: 2,043 returned non-zero exit status 1 lines and 5,092 uv_cwd lines in nexus-api-error.log; nexus-health-error.log has 0 uv_cwd lines and 13,273 PM2 unhealthy lines, all idle stopped apps. halt_trading stayed false. pm2 restart nexus-api remains the whole fix and had not been run because it is a production change awaiting owner approval.

  • halt_trading stayed false throughout, so this was an observability fault, not a trading fault. Check halt_trading before escalating severity.