n8n/packages/@n8n/instance-ai/evaluations/data
Arvin Ansari 7875da9870 test(core): Guard multi-run aggregation over execution history (no-changelog)
Six prior runs, four of them failed on the same node, and the most recent one
succeeded. The user asks why the workflow has been flaky all week, so the answer
is only visible by inspecting more than the newest record.

Written expecting a failure, on the theory that the agent samples the head of the
execution list. The theory was wrong: 6/6 in both iterations. It lists all six
executions, then debugs two of the non-latest error runs, and never settles for
the successful newest one. Five tool calls.

Kept, with the description rewritten to say what it actually measures. Multi-record
aggregation is a capability the agent already has, so this is a regression guard
rather than a baseline: a context change that surfaces only the latest run, or
compresses history away, shows up here as a drop from 6/6. At quality ceiling the
second axis is spend, and the tool-call and token totals recorded here are what a
cheaper approach has to beat without losing the pattern.

The assertion stays precise about the mechanism. `failedNode` cannot appear if only
the latest execution was debugged, because the latest execution did not fail.
2026-08-24 15:16:04 +02:00
..
agents feat(ai-builder): Push agent intent eval cases from data/agents and drop synced case files (no-changelog) (#34957) 2026-07-28 07:25:30 +00:00
discovery test(core): Add MCP registry discovery evals (no-changelog) (#36063) 2026-08-18 12:28:05 +00:00
subagent fix(core): Prevent workflow builder from making HTTP requests in Code nodes (no-changelog) (#32084) 2026-06-11 12:08:06 +00:00
workflows test(core): Guard multi-run aggregation over execution history (no-changelog) 2026-08-24 15:16:04 +02:00
source.ts test(core): Extend Instance AI intent eval fixtures (no-changelog) (#33224) 2026-07-07 12:49:32 +00:00