Skip to main content

A "no source" finding usually means the source list is incomplete

entry tags: reporting corpus sourcing verification

Lesson​

Two projects reported that their subject had no source material. The subject was covered in detail. The corpus builder simply never listed the files that held it.

candidate_files() gathered 31 files: the last 20 briefing outputs, seven memory artifacts and four domain pages. It omitted wiki/pages/entities/, 117 company-level pages. Those pages carried exactly what the adoption monitor needed:

  • google.md: Gemini Enterprise paid MAU grew 40% QoQ
  • meta.md: Meta Business AI handling ~10M weekly conversations
  • anthropic.md: Claude connectors, across 11 lines

Before reporting a subject as unsourced, check whether the source list is complete. "I searched and found nothing" is a claim about the search, not the world. Measured 2026-09-20: adding the entity pages took the corpus from 31 to 148 files and cut the adoption monitor's unsourced products from 3 to 1.

Trap: a naming gap is not a missing subject​

The one remaining unsourced item was the phrase "Claude connectors". Claude's connectors are covered, under other names ("Claude for Creative Work adding connectors", "Claude for Small Business connector/workflow package"). Reporting that as "no corpus source" implies the capability is unmentioned, which is false. Report the distinction: the tracked phrase is absent, the subject is present.

Trap: widening a source invites false positives​

More material means more chance of a wrong match. Two appeared immediately:

  • grounded matched "business-grounded"; collision matched a namespace collision. Word boundaries help, but a hyphen is itself a word boundary, so "business-grounded" still matches. Scoping the term to a branch that first requires a vehicle subject is what actually removes it.
  • hallucinat matched a vendor improvement claim, "52.5% fewer hallucinated claims than GPT-5.3". An improvement claim is not a failure incident.

For a generic term to count as a failure, require a named subject in the same line. outage alone matched 126 unrelated lines; beside "Anthropic" or "Claude" it matched the genuine Claude AI outage of 23 June.

Trap: one branch for two questions​

The reliability auditor was asked about autonomy failures. The corpus contains no vehicle-autonomy failure (all six vehicle lines are deployment news) but does contain an agent/AI service failure. Merging the two lets a fleet directive count as a vehicle incident: the corpus's 104 "autonomy" lines are directives such as "act autonomously", not vehicles.

Split the branches and label them. A combined count answers neither question.

Corollary: a test that encodes old behaviour is the test to update​

Five tests failed after the corpus grew. Each failure was a test asserting the previous, narrower state. Update them with the reason recorded, and keep the trap-reproduction assertions, so the old behaviour stays documented as wrong.