mirror of
https://github.com/n8n-io/n8n.git
synced 2026-10-22 20:41:50 +02:00
449ea18406
Grading context at the moment the request arrived is right for asking whether a fact survived from earlier: the agent restates facts as it works, so the end state would let a request manufacture its own evidence. It is wrong for asking whether the agent went and fetched something. Tool calls land on step two and later while that snapshot is step one, so a retrieval claim graded there fails every time regardless of whether retrieval worked — the anchor excludes the very thing the claim is about. Both claim kinds now carry an optional anchor, defaulting to the arrival moment, so existing cases are unchanged. Claims are grouped by anchor and judged one call per group, each with its own context block, rather than handing the judge both and asking it to infer which moment a claim meant. Verdicts are returned in the order the case wrote them. The judge prompt no longer states a timing of its own; the block says which moment it holds, in its opening section and in every heading, and the prompt explains how to read an absence at each. Getting this wrong was silent, which is the argument for naming it in the case rather than inferring it from wording. Found by re-running the cross-workflow case, where one grader passed on "the agent called workflows(action='get') on two workflow IDs" while the other failed with "the raw message window contains no tool calls or results retrieving any existing workflow's node details". Both were right. Its two retrieval claims are anchored at turn end here, which also means its earlier classification flip from context-ignored to retrieval-gap was an instrumentation artifact — and those two point at opposite conclusions about where the bottleneck is. Also drops the cutover-date assertion from the write-step probe. A date is rewordable, the agent keeps it in prose while regenerating the ISO string, so the literal measured phrasing rather than recall. The upsert key, which survives verbatim, still carries one. Four of the new tests fail when the anchor is ignored, confirmed by reintroducing that. |
||
|---|---|---|
| .. | ||
| aggregator.ts | ||
| build-orchestrator.ts | ||
| case-pipeline.ts | ||
| case-selection.ts | ||
| ci-metadata.ts | ||
| context-outcome.ts | ||
| direct-driver.ts | ||
| eval-session.ts | ||
| iterations.ts | ||
| lane-allocator.ts | ||
| lane-setup.ts | ||
| lane-users.ts | ||
| langsmith-driver.ts | ||
| persist.ts | ||
| reporters.ts | ||
| reshape.ts | ||
| rows.ts | ||
| tiers.ts | ||