n8n/packages/@n8n/instance-ai/evaluations/data/workflows
Arvin Ansari 7875da9870 test(core): Guard multi-run aggregation over execution history (no-changelog)
Six prior runs, four of them failed on the same node, and the most recent one
succeeded. The user asks why the workflow has been flaky all week, so the answer
is only visible by inspecting more than the newest record.

Written expecting a failure, on the theory that the agent samples the head of the
execution list. The theory was wrong: 6/6 in both iterations. It lists all six
executions, then debugs two of the non-latest error runs, and never settles for
the successful newest one. Five tool calls.

Kept, with the description rewritten to say what it actually measures. Multi-record
aggregation is a capability the agent already has, so this is a regression guard
rather than a baseline: a context change that surfaces only the latest run, or
compresses history away, shows up here as a drop from 6/6. At quality ceiling the
second axis is spend, and the tool-call and token totals recorded here are what a
cheaper approach has to beat without losing the pattern.

The assertion stays precise about the mechanism. `failedNode` cannot appear if only
the latest execution was debugged, because the latest execution did not fail.
2026-08-24 15:16:04 +02:00
..
ai-gateway-prefers-covered-scraping-node.json fix(core): Prefer n8n credits covered nodes in Instance AI workflow builds (#35019) 2026-07-28 11:58:42 +00:00
ai-gateway-prefers-supported-service-node.json fix(core): Prefer n8n credits covered nodes in Instance AI workflow builds (#35019) 2026-07-28 11:58:42 +00:00
ai-gateway-respects-named-web-search-tool.json fix(core): Prefer n8n credits covered nodes in Instance AI workflow builds (#35019) 2026-07-28 11:58:42 +00:00
applies-the-house-naming-prefix-across-a-session-boundary.json test(core): Complete the standing-preference case family (no-changelog) 2026-08-21 13:33:41 +02:00
carries-a-standing-preference-across-a-session-boundary.json fix(core): Create the prior-session thread before seeding into it (no-changelog) 2026-08-20 21:45:47 +02:00
checks-the-record-before-accepting-a-false-premise.json test(core): Cover execution history as a context surface (no-changelog) 2026-08-24 14:54:25 +02:00
credential-setup-anthropic-browser.json feat: Add the credential-setup browser eval lane (no-changelog) (#35982) 2026-08-13 15:56:24 +00:00
diagnoses-the-last-run-instead-of-asking.json test(core): Cover execution history as a context surface (no-changelog) 2026-08-24 14:54:25 +02:00
discloses-deleted-credential-on-next-edit.json test(core): Add stale-fact eval cases for memory vs live instance state (no-changelog) 2026-08-20 11:52:30 +02:00
discloses-node-removed-outside-conversation.json test(core): Grade the stale-state claims at the moment they can be true (no-changelog) 2026-08-24 12:56:54 +02:00
discloses-renamed-data-table-column.json test(core): Add stale-fact eval cases for memory vs live instance state (no-changelog) 2026-08-20 11:52:30 +02:00
discloses-workflow-no-longer-active.json test(core): Grade the stale-state claims at the moment they can be true (no-changelog) 2026-08-24 12:56:54 +02:00
finds-the-failure-pattern-across-several-runs.json test(core): Guard multi-run aggregation over execution history (no-changelog) 2026-08-24 15:16:04 +02:00
honors-missing-credential-field-correction.json feat(ai-builder): Collapse the eval case's three seed keys into one seed union (no-changelog) (#35292) 2026-08-03 14:37:06 +00:00
index.ts test(core): Extend Instance AI intent eval fixtures (no-changelog) (#33224) 2026-07-07 12:49:32 +00:00
matches-instance-conventions-when-adding-a-sibling-sync.json fix(core): Let a context claim say which moment it is about (no-changelog) 2026-08-21 14:19:07 +02:00
recalls-agreed-reminder-rules-from-immediately-prior-turn.json test(core): Add needle-probe eval cases for recall at known depths (no-changelog) 2026-08-20 11:52:17 +02:00
recalls-alert-channel-when-two-channels-were-agreed.json test(core): Add deterministic context claims to the recall probes (no-changelog) 2026-08-20 16:43:38 +02:00
recalls-paging-threshold-among-two-similar-percentages.json test(core): Add needle-probe eval cases for recall at known depths (no-changelog) 2026-08-20 11:52:17 +02:00
recalls-report-source-table-among-two-similar-tables.json test(core): Add deterministic context claims to the recall probes (no-changelog) 2026-08-20 16:43:38 +02:00
recalls-retry-and-region-rules-set-several-turns-earlier.json test(core): Add deterministic context claims to the recall probes (no-changelog) 2026-08-20 16:43:38 +02:00
recalls-write-step-rules-locked-in-at-thread-start.json fix(core): Let a context claim say which moment it is about (no-changelog) 2026-08-21 14:19:07 +02:00
routes-through-the-house-gateway-across-a-session-boundary.json test(core): Complete the standing-preference case family (no-changelog) 2026-08-21 13:33:41 +02:00
tags-work-for-review-across-a-session-boundary.json test(core): Complete the standing-preference case family (no-changelog) 2026-08-21 13:33:41 +02:00