Skip to main content

YouTube OAuth and analytics

entity · created 2026-09-13 · actor hermes · confidence high · status active tags: youtube oauth analytics cron credentials

Now​

OAuth is RESTORED as at 2026-10-02 17:29 AEST, after 24 days on public fallback. Dwayne completed the browser consent; the loopback listener exchanged the code and wrote a new token with all three scopes. Verified live, not just from the token file: channels.list(mine=true) returns UC1MD4A9Ok_xbTOh7Xmb-3mw / @grand_gold_rl / 731 subs / 89,663 views / 68 videos, and youtube_analytics_fetcher.py prints Source: oauth-mine-true. A second refresh against the new refresh token succeeded (token: refreshed), so it is not a one-shot token.

Two caveats that survive the fix:

  • The report's CTR claim was wrong and I repeated it. The Analytics API has no impressions or thumbnail-CTR metric. impressions, impressionCtr, impressionsClickThroughRate, videoThumbnailImpressions and videoThumbnailImpressionsClickRate are all rejected by the API, and the official metric reference lists only legacy annotationClickThroughRate / annotationCloseRate (which return 0 here). CTR, retention curves and video-level subscriber conversion are Studio-only and no re-auth can produce them. Restored by OAuth: watch minutes (estimatedMinutesWatched = 6), averageViewPercentage (56.36), engaged views, likes, comments, shares, subscribersGained/Lost, and the video dimension.
  • The sprint ledger is still stale, and that is now the only blocker to honest reporting. youtube-growth-sprint.json is unchanged since 2026-05-16; its window ended 2026-05-26. sprint_progress clamps elapsed to duration_days, so every job still prints "day 14/14" and the +645/+22508/+7 deltas are all-time from the 2026-05-16 oauth_baseline, not sprint measures. The ledger also carries a baseline block for a different, real channel: UC1BDGDirPEwEMbKJcMeTQbw (@grandgoldrl, title "Grandgold", 251 subs / 201,003 views / 207 videos). Two distinct channels, handles differing only by case.

Channel tooling for @grand_gold_rl (channel UC1MD4A9Ok_xbTOh7Xmb-3mw) runs across three scheduled jobs: youtube-growth-optimizer (10:30 daily), youtube-weekly-analytics (10:10 Sunday), zappo-youtube-growth-autoresearch (03:17 daily), plus youtube-combined (08:00/20:00, youtube_oauth_status_report.py). All read through ~/.hermes/scripts/youtube_sprint_common.py, which prefers OAuth mine=true and falls back to scraping the public channel page.

The fallback masks OAuth failure. When OAuth fails, every job prints a short Data warning: oauth fetch failed ... line and continues on public stats. Public stats give subscribers, views and video count only.

OAuth state as at 2026-09-13, the three faults that were found then:

  1. invalid_grant: the refresh token is expired or revoked. A direct POST to https://oauth2.googleapis.com/token returns HTTP 400 with {"error": "invalid_grant", "error_description": "Token has been expired or revoked."}. The stored credentials are internally consistent (token client_id and client_secret both match client_secret.json, project gen-lang-client-[redacted-id]), so this is not a config mismatch. No automated retry can fix it. This needs a human re-auth.
  2. The self-heal script was deleted. youtube_oauth_status_report.py called ~/.hermes/scripts/youtube_oauth_self_heal.py, which did not exist, so SELF_HEAL.exists() was False and every refresh attempt silently no-opped. Restored 2026-09-13 from the surviving .pyc strings. It now delegates to ~/clawd/scripts/youtube-oauth.py refresh-silent and logs the real outcome to ~/Library/Logs/youtube-oauth-self-heal.log. It also reports self_heal_unavailable rather than failing silently if the refresh tool ever goes missing again.
  3. The live token has never carried the analytics scope. youtube_oauth_setup.py requested only youtube.readonly and youtube.force-ssl, and the stored token matches that set. analytics_summary() calls youtubeanalytics.googleapis.com/v2/reports, which needs yt-analytics.readonly. So weekly analytics could not have worked even with a valid token. Fixed in youtube_oauth_setup.py (commit 72050a7 in ~/clawd, unpushed).

The token was last written 2026-09-08 03:17 AEST and was rejected by 04:17 the same morning. Failure has recurred daily since 2026-09-08.

How to re-auth​

Use ~/clawd/scripts/youtube-oauth.py, not youtube_oauth_setup.py. The latter still writes its URL to skills/youtube-analytics/oauth-auth-url.txt with a stale localhost:8765 redirect and the old scope set.

  • youtube-oauth.py url prints a consent URL requesting youtube.readonly and yt-analytics.readonly, access_type=offline, prompt=consent select_account, redirect http://localhost:8080/.
  • youtube-oauth.py exchange <URL_OR_CODE> saves the access and refresh token to ~/clawd/skills/youtube-analytics/.youtube-tokens.json (mode 600).
  • youtube-oauth.py refresh-silent refreshes without printing.
  • youtube-oauth.py status prints redacted state as JSON.

All three redirect forms were probed live on 2026-09-13: http://localhost:8080/, http://localhost:8765/ and bare http://localhost each return invalid_grant (the probe code) rather than redirect_uri_mismatch, so the configured http://localhost registration covers all of them.

Facts worth keeping​

  • Analytics needs yt-analytics.readonly. youtube.readonly alone serves the Data API (channels?mine=true) but not the Analytics API. Two different endpoints and two different scopes; conflating them is what hid fault 3.
  • force-ssl is requested but nothing locally uses an upload or write path, so it is not currently load-bearing.
  • The OAuth client is an installed app with redirect_uris: ["http://localhost"] and lives at ~/clawd/skills/youtube-analytics/client_secret.json.
  • Refresh tokens issued by this client have failed repeatedly, including one issued 2026-09-08 that died within an hour. If the next re-auth also dies quickly, suspect the Google Cloud project's OAuth consent screen being in Testing status, which expires refresh tokens after 7 days. That is unverified (it needs the Cloud console) and is the first thing to check.
  • Never print token values, client secrets, or auth URLs in cron output. The self-heal script and its tests assert this.

Log​

  • 2026-10-02 17:29 AEST (resolution, live probes): OAuth restored after 24 days. Re-auth path used was ~/clawd/scripts/youtube_oauth_setup.py listen --port 8765 (not the older youtube-oauth.py recommended on 2026-09-13); it bound 127.0.0.1:8765, wrote the consent URL to skills/youtube-analytics/oauth-auth-url.txt, and on redirect printed token: saved. New token: three scopes (youtube.readonly, youtube.force-ssl, yt-analytics.readonly), updated_at 2026-10-02 17:29:02, mode 600. Pre-change token backed up to .youtube-tokens.json.bak-20261002-171810. A follow-up refresh against the new refresh token returned token: refreshed, so the grant is durable, not one-shot.

    Verified by live API, not by reading the token file: channels.list(mine=true) returned exactly one item, UC1MD4A9Ok_xbTOh7Xmb-3mw / GrandgoldRL / @grand_gold_rl / 731 subs / 89,663 views / 68 videos, matching the public fallback numbers. youtube_analytics_fetcher.py printed Source: oauth-mine-true. The 28-day channel report returned views 35, estimatedMinutesWatched 6, averageViewPercentage 56.36, subscribersGained 1, subscribersLost 4, likes 5, comments 2, shares 0, plus a working video dimension. youtube_oauth_status_report.py (the 08:00/20:00 youtube-combined job, id 04b6378d3971) also prints Source: oauth-mine-true, so tonight's 20:00 run restores private reporting without further action.

    Corrected an inherited claim: CTR is not recoverable by OAuth. impressions, impressionCtr, impressionsClickThroughRate, videoThumbnailImpressions and videoThumbnailImpressionsClickRate are all rejected (Unknown identifier / The query is not supported), and the official metric reference (developers.google.com/youtube/analytics/metrics) lists only legacy annotationClickThroughRate / annotationCloseRate, both 0 here. Thumbnail impressions and CTR, retention curves and video-level subscriber conversion are Studio-only. The ledger's blocked_metrics_until_oauth_fixed list is therefore wrong about the reason as well as the membership: estimatedMinutesWatched and engagedViews are now available, while CTR and the retention curve can never come from the API.

    Sprint ledger remains stale and is now the only blocker to honest framing: youtube-growth-sprint.json unchanged since 2026-05-16, window ended 2026-05-26 (129 days ago), sprint_progress still clamps to day 14/14 and +645/+22508/+7 are all-time from the 2026-05-16 oauth_baseline. Its baseline block names a different real channel, UC1BDGDirPEwEMbKJcMeTQbw / @grandgoldrl / "Grandgold" / 251 subs / 201,003 views / 207 videos, distinct from @grand_gold_rl; handles differ only by case. No edit made to the ledger, because resetting the window is a reporting-progress decision, not a defect repair. Sources: live token-endpoint and Analytics API probes, channels.list, youtube_oauth_setup.py status, youtube_analytics_fetcher.py, youtube_oauth_status_report.py, youtube_sprint_common.py, sprint JSON, cron jobs.json. - hermes-zappo.

  • 2026-09-27 10:30 AEST (daily optimizer run, live probes): Twentieth consecutive day on public fallback. Direct token-endpoint POST still returns HTTP 400 invalid_grant; stored token updated_at decodes to 2026-09-08 03:17 AEST and expires_at to 2026-09-08 04:17 AEST, scope still youtube.readonly + youtube.force-ssl with no yt-analytics.readonly. Self-heal log shows invalid_grant at 2026-09-27 08:00 (twice daily since 2026-09-14). Public log latest rows: 731 subs / 89,663 views / 68 videos at 2026-09-27 03:17:49 and 10:10:23 AEST, against 732 at 2026-09-26 03:17:58. Dated deltas from the same log: subs flat over 7d (731 on 2026-09-20), -3 over 30d, -17 from the 748 peak of 2026-06-22; views +0 over 7d, +21 over 30d; video count 68 unchanged since 2026-06-29 (90 days). No new upload in the last 24h. Stale sprint unchanged: youtube-growth-sprint.json ended 2026-05-26 (124 days ago) and its oauth_baseline (86 subs / 67,155 views, 2026-05-16) is what produces the report's +645/5 and +22508/500 lines, so those are all-time-from-baseline, not sprint measures; sprint_progress clamps elapsed to duration_days and still prints day 14 with the defunct day-14 queue item. No Kanban card exists; human re-auth remains the only fix. Sources: live token-endpoint probe, self-heal log, youtube-oauth-status-state.json, youtube-grandgoldrl-public-log.csv, youtube-growth-log.csv, sprint JSON. - hermes-zappo.

  • 2026-09-27 06:00 AEST (weekly synthesis, live state read): The fault has now run nineteen days (first daily failure 2026-09-08, unchanged since), and the OAuth defect is unchanged in every part. ~/clawd/memory/youtube-oauth-status-state.json reads status: oauth_public_fallback_steady, reason: missing_token, refreshed: true, failed_at 2026-09-26 20:00 (checked_at the same minute), and the last growth line records Source: public-youtube-page with Self-healed: oauth: HTTP Error 400: Bad Request. The stored token still has not been re-authed: exchanged_at decodes to 2026-05-14 02:50 and expires_at to 2026-07-28 04:45, both far behind now. Public log (~/clawd/memory/youtube-grandgoldrl-public-log.csv) latest row is 2026-09-27 03:17:49 AEST: 731 subscribers, 89,663 views, 68 videos, tagged @grand_gold_rl source public_youtube_about_live. Against the 2026-09-26 03:17:58 row (732/89,663/68) that is one subscriber lost, with views and video count flat, and the 2026-09-26 20:00:57 growth line reads a net change of zero on all three fields. Video count has now been 68 for 90 days (unchanged since 2026-06-29). No re-auth has been performed; the yt-analytics.readonly scope gap in youtube_oauth_setup.py (commit 72050a7, unpushed) still means weekly analytics cannot work even after one. Source: ~/clawd/memory/youtube-oauth-status-state.json, ~/clawd/memory/youtube-grandgoldrl-public-log.csv, ~/clawd/memory/youtube-growth-log.csv, cron output ~/.hermes/cron/output/b2e5b4073752/2026-09-26_10-30-53.md and ~/.hermes/cron/output/179f165b0a1b/2026-09-27_03-17-51.md, read 2026-09-27 05:55-06:00 AEST., hermes.

  • 2026-09-24 10:35 AEST (daily optimizer run, live probes): Seventeenth consecutive day on public fallback. Direct token-endpoint POST still returns HTTP 400 invalid_grant; token updated_at decodes to 2026-09-08 03:17 AEST and expires_at to 2026-09-08 04:17 AEST, scope still youtube.readonly + youtube.force-ssl with no yt-analytics.readonly. Self-heal log records invalid_grant twice daily (08:00 and 20:00), last 2026-09-24 08:00. State file ~/clawd/memory/youtube-oauth-status-state.json reads checked_at 2026-09-24 08:00, failed_at 2026-09-24 08:00, reason: missing_token, status: oauth_public_fallback_steady. Public log latest row 2026-09-24 10:30:09 AEST: 731 subs, 89,663 views, 68 videos. Dated deltas from the same log: subs +0 over 1d, +0 over 7d, -3 over 14d, -4 over 30d, -12 over 60d, -16 over 90d (peak 748 on 2026-06-22, so -17 from peak); views +0 over 1d, +10 over 7d, +24 over 14d, +71 over 30d, +120 over 60d, +188 over 90d; video count 68 unchanged since 2026-06-29 (87 days). The report's +645/5 and +22508/500 deltas are all-time from the 2026-05-16 oauth_baseline (86 subs / 67,155 views) and a mid-history 208-video public reading, not sprint measures. Stale sprint unchanged: youtube-growth-sprint.json ended 2026-05-26 (121 days ago) yet elapsed is clamped to duration_days in sprint_progress, so the report still prints "day 14" and emits "ship the slot" from the defunct day-14 queue item. No Kanban card exists (tasks holds 3 done, 1 blocked, none YouTube); human consent remains the only fix. Sources: live token-endpoint probe, self-heal log, state file, public log, sprint JSON, kanban.db. - hermes-zappo.

  • 2026-09-22 10:30 AEST (daily optimizer run, live probes): Fifteenth consecutive day on public fallback. Direct token-endpoint POST returns HTTP 400 invalid_grant again today; token client_id/client_secret still match client_secret.json (project gen-lang-client-[redacted-id]), token updated_at decodes to 2026-09-08 03:17 AEST and expires_at to 2026-09-08 04:17 AEST, scope still youtube.readonly + youtube.force-ssl with no yt-analytics.readonly. The 2026-05-18 token backup also returns invalid_grant, so no on-disk token is recoverable. Self-heal log records invalid_grant at 08:00 and 20:00 daily (last 2026-09-22 08:00). Public log latest row 2026-09-22 10:30:37 AEST: 731 subs, 89,663 views, 68 videos. Dated deltas from the same log: views +10 over 7d, +10 over 14d, +24 over 30d; video count unchanged since 2026-06-29 (85 days); subscribers read 735 at 2026-09-21 03:17 and 10:05, then 731 at 2026-09-22 03:17, so the +4 was a one-day public-page reading that did not hold. The report's own numbers are internally inconsistent: it prints +645/5 subs against a 731 reading while the sprint's oauth_baseline.subscribers is 86 and the public baseline.subscribers is 252, so the +N/target deltas are not tied to the row they accompany. Stale-sprint defect unchanged: youtube-growth-sprint.json ended 2026-05-26 (119 days ago), yet the optimizer still prints "day 14" and emits "ship the slot" from the defunct day-14 queue item. No Kanban card exists for the re-auth. Human consent is the only remaining fix. Sources: live token-endpoint probe, self-heal log, youtube_sprint_common.py, public log, sprint JSON, kanban.db. - hermes-zappo.

  • 2026-09-20 06:00 AEST (weekly synthesis, live state file read): OAuth is still broken and the public fallback is still carrying every reading, 13 days in. ~/clawd/memory/youtube-oauth-status-state.json reads status: oauth_public_fallback_steady, reason: missing_token, last_error empty, refreshed: true, failed_at 2026-09-19 20:00 and checked_at 2026-09-19 20:00, while the stored token's expires_at decodes to 2026-07-28 04:45 and exchanged_at to 2026-05-14 02:50. The last growth line (last_growth, 2026-09-19 20:00:23 AEST) records Source: public-youtube-page and Self-healed: oauth: HTTP Error 400: Bad Request. Public log rows: 731 subscribers, 89,655 views, 68 videos at 2026-09-19 15:00 and 20:00; the 2026-09-20 03:17:06 row reads 731/89,663/68, so views moved +8 overnight while subscribers and video count stayed flat. Every row is tagged @grand_gold_rl with source public_youtube_about_live or public-youtube-page. This is unchanged from the state recorded on 2026-09-14 and 2026-09-15; no re-auth has been performed, and the yt-analytics.readonly scope gap still means weekly analytics cannot work even after one. Source: ~/clawd/memory/youtube-oauth-status-state.json, ~/clawd/memory/youtube-grandgoldrl-public-log.csv, ~/clawd/memory/youtube-growth-log.csv, read 2026-09-20 06:05 AEST., hermes.

  • 2026-09-15 10:30 AEST, Confirmations from the youtube-growth-optimizer run. (a) Live token endpoint probe again returns HTTP 400 invalid_grant ("Token has been expired or revoked."); self-heal log has invalid_grant at 2026-09-14 20:00 and 2026-09-15 08:02 AEST. Stored token expired 2026-09-08 04:17 and its scope set is still only youtube.readonly + youtube.force-ssl - no yt-analytics.readonly, so a re-auth is needed regardless. Eighth consecutive day on public fallback. (b) Public log latest row 2026-09-15 10:30:35 AEST: 733 subs, 89653 views, 68 videos. Dated deltas from the same log: subs -1 over 7d, 0 over 14d, -5 over 30d, -11 over 60d; views +16 / +60 / +70 / +123; videos unchanged since 2026-06-29 (78 days). (c) The stale-sprint defect is worse than the plotter-only note of 2026-09-14: it drives the daily action line too. sprint_progress clamps elapsed to duration_days, so the optimized report prints "day 14/14" for a sprint that ended 2026-05-26 (112 days ago), labels all-time-since-2026-05-16 deltas as sprint progress (+647 subs / +22498 views / +7 uploads), and then emits "ship the slot above" from the defunct day-14 queue item (youtube-grandgoldrl-execution-loop.json). Recommended fix, not yet applied: either retarget youtube-growth-sprint.json or have sprint_progress return expired when today > end_at and have the reports say "since 2026-05-16 baseline" instead of "sprint". Sources: live token probe, self-heal log, youtube_sprint_common.py lines 408-481, public log, sprint JSON. - hermes-zappo.

  • 2026-09-14, Recurrence confirmed by live probe: token endpoint still returns HTTP 400 invalid_grant ("Token has been expired or revoked"). Self-heal log shows invalid_grant at 03:33, 03:34, 03:35, 03:37 and 08:00 AEST. Public fallback has now carried every reading since 2026-09-08 08:00.

  • 2026-09-14, Presentation defect found while reading the weekly plotter output: the youtube-growth-plotter job prints +N/<target> sprint deltas that are measured against the oauth_baseline from 2026-05-16, not the 14-day sprint. The sprint in youtube-growth-sprint.json started 2026-05-12 and ended 2026-05-26, so every "sprint" figure in that line is a ~4-month all-time delta and reads as recent progress. Dated deltas from youtube-grandgoldrl-public-log.csv for the same day: views +15 over 7 days, +59 over 14 days, +69 over 30 days; subscribers flat over 7 days, −4 over 30; uploads unchanged since 2026-06-29. Fix options: retarget the sprint or relabel the line as all-time-since-baseline. Not yet changed. Sources: live token probe, self-heal log, public log. - hermes.

  • 2026-09-13, Diagnosed and partially fixed. Confirmed fault 1 by direct token-endpoint probe (HTTP 400 invalid_grant). Restored ~/.hermes/scripts/youtube_oauth_self_heal.py (sha256 378d8d3c189747f1ae9fe79f5686b0844d15d44cbfa92d5344aa7993ef757eba), which had been deleted and caused silent no-op self-heal; it now logs invalid_grant instead of nothing, proven by a live run appending that line to the self-heal log. Added yt-analytics.readonly to youtube_oauth_setup.py SCOPES (sha256 e6b500ecf92160426f53e783effdf3d363c700a8f5ccd122da357f15b327f0a2), committed locally as 72050a7; not pushed because origin/main is ahead by 1 and a Claw Automated Sync agent is active on that tree. Regression tests ~/.hermes/tests/test_youtube_oauth_self_heal.py, 9 tests, falsified both ways (8 of 9 fail with the self-heal script absent; 1 fails with the analytics scope absent; 9 pass after restore). The re-auth itself was not performed: it needs an interactive Google consent, which is a human gate. Fault 1 remains open. Sources: live probe, git log, pytest runs, self-heal log. - hermes.

  • 2026-09-17 10:30 AEST, Tenth consecutive day on public fallback, and the first day the public series is clearly declining rather than flat. Public log latest row 2026-09-17 10:30:48 AEST: 731 subs, 89653 views, 68 videos. Dated deltas from the same log: subs -2 over 1d, -2 over 2d, -3 over 7d, -3 over 14d, -7 over 30d, -13 over 60d and 80d; peak subs 748 on 2026-06-22, so -17 from peak. Views +0 over 2d, +14 over 7d, +49 over 14d, +70 over 30d, +163 over 80d. Video count unchanged since 2026-06-29 (80 days). OAuth: self-heal log invalid_grant at 2026-09-17 08:00; stored token access expiry 2026-09-08 04:17, scope still lacks yt-analytics.readonly. Stale-sprint defect unchanged and still driving the daily action line: youtube-growth-sprint.json ended 2026-05-26 (114 days ago), yet the optimizer prints "day 14/14" with all-time-since-2026-05-16 deltas (+645 subs / +22498 views / +7 uploads) and emits "ship the slot" from the defunct day-14 queue item. Sources: public log, self-heal log, sprint JSON, live report run. - hermes-zappo.

Sources: ~/.hermes/scripts/youtube_oauth_status_report.py · ~/.hermes/scripts/youtube_sprint_common.py · ~/.hermes/scripts/youtube_oauth_self_heal.py · ~/clawd/scripts/youtube-oauth.py · ~/clawd/scripts/youtube_oauth_setup.py · ~/clawd/skills/youtube-analytics/.youtube-tokens.json · live token-endpoint probe 2026-09-13