Skip to main content

Lesson: an unattended sync must decode its diff leniently and never stage credentials

entry

Date: 2026-09-20 Lane: hermes Confidence: high

Context​

Cron job nightly-github-sync (c737c1447aa8, 02:00 daily AEST, deliver=telegram:[redacted-id]) runs ~/.hermes/scripts/github_sync_report.py, which wraps /Users/dwayne/clawd/scripts/github-sync-orchestrator.py. The 2026-09-20 02:00 fire exited 1 with UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 55567476: invalid continuation byte, raised inside the orchestrator's secret leak scan.

The three defects​

  1. Crash: text=True on a byte-unsafe diff. The leak check ran subprocess.run(["git","diff","--cached","-U0","--","."], capture_output=True, text=True). A staged tree containing non-UTF-8 bytes (a legacy-encoded vendored source file, a binary .db, a log with latin-1 bytes) makes Python raise UnicodeDecodeError during decoding. The whole nightly sync aborted before any repo was committed or pushed. Fix: capture bytes (text=False) and .decode("utf-8", errors="replace"); treat a non-zero diff returncode as "cannot scan this repo, skip and report" rather than raising.

  2. Credential exposure risk. A backup tree (~/Projects/removed-mindroom-backup-20260919, 4.76 GB, 21,541 staged files) carried dot-mindroom/.env and dot-mindroom/.env.bak-pre-edge-fleet-20260806-203839 into the staged set. The leak regex inspects git diff text only; it is not a dependable gate for a whole fresh file, and the 50 MB / extension filters let .env through. unstage_excluded() now un-stages .env and .env.* outright, with .env.example / .env.template / .env.sample as explicit exceptions. (Both staged .env files were inspected and found to hold no credential-shaped value; the control was added regardless.)

  3. An unattended multi-GB first commit. The repo had zero commits and a dirty tree, so git commit -m "Claw Automated Sync" would have created a first commit over 21k files and then attempted a 4.76 GB push. The orchestrator now refuses when HEAD does not exist and the tree exceeds FIRST_COMMIT_SIZE_LIMIT (500 MB), printing SKIPPING \<repo>: no commits yet and the tree is N GB... needs a supervised init.

Rule​

A backup/sync job that runs unattended must (a) never assume its own inputs decode as UTF-8, (b) treat credential filenames as a hard un-stage rule rather than relying on content regexes, and (c) refuse a first commit above a stated size threshold instead of staging gigabytes. The leak regex is a content backstop; the filename and size rules are the actual guards.

Verification​

  • python3 -m py_compile clean.
  • Unit-level: is_sensitive_env_file returns True for dot-mindroom/.env and dot-mindroom/.env.bak-pre-edge-fleet-20260806-203839, False for .env.example.
  • unstage_excluded() against the real repo: Unstaging 2 excluded file(s) - exactly the two .env files; staged .env count then 0.
  • sync_project() against the real 4.76 GB repo prints the SKIPPING line, return code 0, and leaves the repo with zero commits and zero staged files.
  • sync_project() against a normal project (leader-briefing-builds) still completes the full stage → leak scan → commit → push path: Successfully synced.
  • The orchestrator was committed to ~/clawd as 24c05b3 (2026-09-20).

Files​

  • /Users/dwayne/clawd/scripts/github-sync-orchestrator.py (fixed)
  • /Users/dwayne/.hermes/scripts/github_sync_report.py (wrapper; unchanged)
  • /Users/dwayne/Projects/removed-mindroom-backup-20260919 (uncommitted, no remote content)

Residual​

  • Superseded 2026-09-25 (hermes): the repo is no longer "uncommitted". It now holds exactly one commit, 848349a4 "chore: autonomous wip backup by Dodge janitor (2026-09-21T10:15:23.704987)", created by /Users/dwayne/clawd/scripts/ProjectJanitor.py (cron weekly-project-janitor, cb63565fd572, 10:15 Mondays) on its dodge-wip-20260921 branch, later renamed to main. That initial commit bypassed the orchestrator's FIRST_COMMIT_SIZE_LIMIT guard entirely, so the guard never fired.
  • Superseded 2026-09-25 (hermes): the push now fails every night with remote: error: GH001: Large files detected ... pre-receive hook declined, and has done so on each 02:00 nightly-github-sync (c737c1447aa8) run since 2026-09-21; the branch rename to main is the only step that succeeds. Remote dwaynehelena/claw-proj-removed-mindroom-backup-20260919 is PRIVATE, isEmpty: true, diskUsage: 0, pushedAt: 2026-09-19T16:01:47Z.
  • Measured 2026-09-25 (hermes): tracked tree is 21,805 files / 2.43 GB (2,317 MB). Eight tracked files exceed 50 MB: dot-mindroom/mindroom_data/logs/ mindroom_20260910_042158.log 424 MB, .../mindroom_20260811_061349.log 191 MB, dot-mindroom/build-tools/zig-x86_64-macos-0.16.0/zig 186 MB, dot-mindroom/mindroom_data/teams/team_aidlc_team_7301a920a5c5/sessions/ team_aidlc_team_7301a920a5c5.db 120 MB, dot-mindroom/mindroom_data/ edge_fleet.db 80 MB, dot-mindroom/telegram-bridge-runtime/mautrix-telegram 70 MB, dot-mindroom/mindroom_data/event_cache.db 57 MB, dot-mindroom/build-tools/zig.tar.xz 55 MB. Directory totals: mindroom_data 1,823 MB, build-tools 418 MB, telegram-bridge-runtime 70 MB.
  • Root cause of the recurring failure: unstage_excluded() only un-stages files newly added by git add .. Files already inside the single existing commit are untouched, so the sync re-pushes the same oversized commit and fails. The 50 MB filter cannot repair an existing history.
  • Clean subset measured 2026-09-25 (hermes): 626 tracked files / 6.0 MB when dot-mindroom/mindroom_data/, dot-mindroom/build-tools/, dot-mindroom/telegram-bridge-runtime/ and .env* are excluded, with zero files over 50 MB. A repaired backup would carry that subset plus a .gitignore.
  • dot-mindroom/.env and dot-mindroom/.env.bak-pre-edge-fleet-20260806-203839 are tracked inside that commit. Re-checked 2026-09-25: no credential-shaped value in either (the unstage_excluded filename rule covers new adds only).
  • Repairing this needs a history rebuild plus a force-push, which is a consequential action and requires Dwayne's explicit approval. Do not rewrite or force-push unattended.
  • ~/Projects/gods-eye-view and ~/Projects/archify have origin pointing at upstreams this account only has READ on (bilawalsidhu/gods-eye-view, tt-a1i/archify, both public). gods-eye-view holds one unpushed Claw Automated Sync commit. The orchestrator's private-visibility gate reads the claw-proj-* repo name and so reports PRIVATE while the real push target is a public upstream it cannot write to. Any fleet commit meant to reach a deployed origin belongs in a claw-proj-* repo or needs a deliberate fork.
  • A scheduled sync must not be re-fired while its own agent phase holds the fire_claim; hermes cron run c737c1447aa8 returns "already being fired". Exercise the fixed code by importing sync_project directly instead.