An extractor must match the format its source actually uses
extraction verification reporting sqlite regex
Lesson
Three extractors written against the same data returned wrong answers because each pattern assumed a format the source did not use. None of them raised an error. Each returned a confident, plausible, wrong result.
1. A Unix epoch is not a date string. sessions.started_at in
~/.hermes/state.db holds a float such as [redacted-id].335724. Comparing it to
'2026-09-06' returns zero rows silently, because a float sorts below a string.
The session count read as zero while 33,404 sessions existed. Check the column's
actual type before filtering on it.
2. A substring marker is not a word. 'repo' matched inside 'report', so
lines about the local OpenClaw gateway were labelled "external ecosystem
references". The ambiguous bucket held 27 entries; 1 was real. Match markers on
word boundaries, or every marker that is a common prefix of another word leaks
false positives.
3. A unit suffix is part of the number. The money pattern listed B, M and
billion but not T, so $1T matched as $1. In a dashboard whose purpose is
scale, that understates the figure by twelve orders of magnitude. Enumerate the
full unit set, and keep the matched string verbatim in the output so the error
is visible.
Rule
Before trusting an extractor, print the raw value it is meant to match and read it. A source's format is a fact about the source, not an assumption to encode.
Trap: a boolean hides partial coverage
The adoption monitor tracks four products. The corpus mentions one of them. A
supported: true/false flag reads as "fine" because one term matched, hiding
that three products have no source at all. Report coverage as full, partial or
none, and name what is missing.
Trap: a truncated sample cannot justify its own label
Mentions were stored truncated to 300 characters. The marker that decided a classification often fell past the cutoff, so the artifact could not show why it said "ambiguous". Record the deciding markers alongside the text. An artifact a reviewer cannot check is not evidence.
Trap: a live source is a snapshot, not a stable fixture
A session still running when the sample was collected gained 21 messages and an
end_reason afterwards. A re-run therefore differed from the artifact. That is
not drift and not a defect. Say the data is point-in-time, or every later
comparison reads as an inconsistency.
Corollary: not every action has material
The autonomy action returned no results, and that was the truthful answer. All 104 lines mentioning autonomy concerned fleet-operating directives ("act autonomously"), not vehicle autonomy, and the corpus's only two robotaxi lines were deployment news. A filter that removes noise can leave nothing, and an empty result must explain itself rather than read as a broken query.