mirror of
https://github.com/n8n-io/n8n.git
synced 2026-10-19 16:55:02 +02:00
390b697cd7
Eval cases had two grading surfaces: `processExpectations`, judged on the build conversation, and `outcomeExpectations`, judged on the built workflow. Neither observes the memory subsystem, so a memory comparison could only be measured end-to-end — a score delta was not attributable to retrieval rather than to a luckier build. Add a third kind, `memoryExpectations`, graded against context state the harness already captured and discarded: run debug is stashed on every build, and `parseSystemPromptForDisplay` / `parseUsageSummary` already parse it. No product change is needed. It gets its own judge call with its own rubric, whose only inputs are the final compressed observation block, the raw message window the model received, and the final system prompt. The conversation judge is handed the full transcript, which would let it satisfy "the agent still knew X" from the turn where X was first said — the confound this kind exists to remove. The raw window is not the transcript: evicted messages are absent from it, which is why it can be trusted as a record of what the agent still had. A fact counts as retained if it appears in any of the three tiers. The Observer only compresses past a token threshold, so on a short thread the observation block is legitimately empty while every fact sits in the raw window; grading on the block alone would fail every memory expectation for a structural reason on exactly the short threads the suite is mostly made of. Verdicts carry `kind: "memory"` on every path — judged, ungraded and judge-error — since the report badge and `eval-results.json` filter on it. `stashBuildExpectations` previously early-returned once any expectation was ungraded. With three kinds a case can legitimately have ungraded memory expectations and gradable outcome ones at once, so the verdict sets are now merged; the early return would have dropped the latter and shrunk the pass-rate denominator. `stashRunDebug` now runs just before it at all three call sites, so the memory judge can read the capture. |
||
|---|---|---|
| .. | ||
| @n8n | ||
| cli | ||
| core | ||
| extensions/insights | ||
| frontend | ||
| modules | ||
| node-dev | ||
| nodes-base | ||
| testing | ||
| workflow | ||