n8n/packages/@n8n/instance-ai/evaluations/data/workflows
Arvin Ansari 33ad2d3680 test(core): Grade node usage over a folder structure (no-changelog)
Two cases where the answer needs the node composition of a set of workflows, not
their names or their graphs.

- `finds-the-real-users-of-a-node-type`: inside one folder, three workflows use
  the Postgres node and none says so in its name, while two more carry Postgres
  in the NAME and never use it. A name filter returns exactly the wrong two.
- `reads-a-folders-node-conventions-before-extending-it`: what a folder is built
  out of, before adding to it. Two workflows outside the folder use nodes that
  appear nowhere inside it, so an instance-wide summary gives the wrong
  convention.

Both are folder-scoped rather than instance-wide, for a reason worth recording:
the eval instance is shared and already holds unrelated workflows, so an
instance-wide count is not a fact a case can assert. Scoping also makes them
test the two capabilities composed — the folder bounds the set, the index says
which members actually contain the node.

The out-of-folder pair in the second case deliberately avoids Postgres: cases
share one project and run concurrently, so one case's seeds can otherwise answer
another case's question.
2026-08-27 19:06:06 +02:00
..
ai-gateway-prefers-covered-scraping-node.json fix(core): Prefer n8n credits covered nodes in Instance AI workflow builds (#35019) 2026-07-28 11:58:42 +00:00
ai-gateway-prefers-supported-service-node.json fix(core): Prefer n8n credits covered nodes in Instance AI workflow builds (#35019) 2026-07-28 11:58:42 +00:00
ai-gateway-respects-named-web-search-tool.json fix(core): Prefer n8n credits covered nodes in Instance AI workflow builds (#35019) 2026-07-28 11:58:42 +00:00
applies-the-house-naming-prefix-across-a-session-boundary.json test(core): Complete the standing-preference case family (no-changelog) 2026-08-21 13:33:41 +02:00
carries-a-standing-preference-across-a-session-boundary.json fix(core): Create the prior-session thread before seeding into it (no-changelog) 2026-08-20 21:45:47 +02:00
checks-the-record-before-accepting-a-false-premise.json test(core): Cover execution history as a context surface (no-changelog) 2026-08-24 14:54:25 +02:00
credential-setup-anthropic-browser.json feat: Add the credential-setup browser eval lane (no-changelog) (#35982) 2026-08-13 15:56:24 +00:00
diagnoses-the-last-run-instead-of-asking.json test(core): Cover execution history as a context surface (no-changelog) 2026-08-24 14:54:25 +02:00
discloses-deleted-credential-on-next-edit.json test(core): Add stale-fact eval cases for memory vs live instance state (no-changelog) 2026-08-20 11:52:30 +02:00
discloses-node-removed-outside-conversation.json test(core): Grade the stale-state claims at the moment they can be true (no-changelog) 2026-08-24 12:56:54 +02:00
discloses-renamed-data-table-column.json test(core): Add stale-fact eval cases for memory vs live instance state (no-changelog) 2026-08-20 11:52:30 +02:00
discloses-workflow-no-longer-active.json test(core): Grade the stale-state claims at the moment they can be true (no-changelog) 2026-08-24 12:56:54 +02:00
does-not-report-a-foldered-workflow-as-missing.json test(core): Cover folder membership as a context surface (no-changelog) 2026-08-27 19:06:05 +02:00
finds-the-failure-pattern-across-several-runs.json test(core): Guard multi-run aggregation over execution history (no-changelog) 2026-08-24 15:16:04 +02:00
finds-the-real-users-of-a-node-type.json test(core): Grade node usage over a folder structure (no-changelog) 2026-08-27 19:06:06 +02:00
honors-missing-credential-field-correction.json feat(ai-builder): Collapse the eval case's three seed keys into one seed union (no-changelog) (#35292) 2026-08-03 14:37:06 +00:00
index.ts test(core): Extend Instance AI intent eval fixtures (no-changelog) (#33224) 2026-07-07 12:49:32 +00:00
lists-only-the-workflows-inside-the-named-folder.json test(core): Cover folder membership as a context surface (no-changelog) 2026-08-27 19:06:05 +02:00
matches-instance-conventions-when-adding-a-sibling-sync.json fix(core): Let a context claim say which moment it is about (no-changelog) 2026-08-21 14:19:07 +02:00
opens-the-workflow-the-folder-disambiguates.json test(core): Cover folder membership as a context surface (no-changelog) 2026-08-27 19:06:05 +02:00
reads-a-folders-node-conventions-before-extending-it.json test(core): Grade node usage over a folder structure (no-changelog) 2026-08-27 19:06:06 +02:00
recalls-agreed-reminder-rules-from-immediately-prior-turn.json test(core): Add needle-probe eval cases for recall at known depths (no-changelog) 2026-08-20 11:52:17 +02:00
recalls-alert-channel-when-two-channels-were-agreed.json test(core): Add deterministic context claims to the recall probes (no-changelog) 2026-08-20 16:43:38 +02:00
recalls-paging-threshold-among-two-similar-percentages.json test(core): Add needle-probe eval cases for recall at known depths (no-changelog) 2026-08-20 11:52:17 +02:00
recalls-report-source-table-among-two-similar-tables.json test(core): Add deterministic context claims to the recall probes (no-changelog) 2026-08-20 16:43:38 +02:00
recalls-retry-and-region-rules-set-several-turns-earlier.json test(core): Add deterministic context claims to the recall probes (no-changelog) 2026-08-20 16:43:38 +02:00
recalls-write-step-rules-locked-in-at-thread-start.json fix(core): Let a context claim say which moment it is about (no-changelog) 2026-08-21 14:19:07 +02:00
routes-through-the-house-gateway-across-a-session-boundary.json test(core): Complete the standing-preference case family (no-changelog) 2026-08-21 13:33:41 +02:00
scopes-a-bulk-change-to-the-named-folder.json test(core): Cover folder membership as a context surface (no-changelog) 2026-08-27 19:06:05 +02:00
tags-work-for-review-across-a-session-boundary.json test(core): Complete the standing-preference case family (no-changelog) 2026-08-21 13:33:41 +02:00