Arvin Ansari f0cfdd9fe3 test(core): Cover execution history as a context surface (no-changelog) ... Two cases that can only be answered by reading a prior run, using the pre-turn execution step. Both verified against a live instance. The first: last night's run failed on the HTTP node with a rate limit, and the user says only "it broke again". Nothing about the failure is in the request, so asking the user which node broke or guessing from the workflow's shape both fail. The second is the inverse and the more interesting half — the run SUCCEEDED but the user asserts it failed. Accepting that premise means editing a healthy workflow, which is worse than doing nothing. They are deliberately paired. An agent that always reports success, or always reports failure, passes one and fails the other; only reading the record passes both. Both context claims are anchored at turn end, because the agent learns what happened by calling the executions tool during the turn. Two things the first live run taught, now written into the case: Seeding appends a ` [seed <8hex>]` uniquifier to workflow names, so the name a case authors never matches the name on the instance. Every prior run failed at seed time until the resolver stripped it. The loud refusal earned its keep — the error printed both names side by side and the cause was obvious immediately. Hints steer the failure's nature, not its literal response body. The first version asserted a string from the JSON body the hints asked for; the mock returned an equivalent rate-limit error worded differently, and the assertion failed on a run that was otherwise perfect. It now asserts `failedNode`, a field of the executions debug payload, which cannot come from the workflow definition — so it still proves the record was read, without depending on mock wording.		2026-08-24 14:54:25 +02:00
..
fixtures	fix(ai-builder): Keep model-provider outages out of eval builder scores (no-changelog) (#35293)	2026-08-04 10:52:11 +00:00
agent-report-rendering.test.ts	refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691)	2026-07-22 16:25:44 +00:00
agent-scenario-artifact.test.ts	refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834)	2026-07-27 09:01:09 +00:00
aggregator.test.ts	feat: Add the credential-setup browser eval lane (no-changelog) (#35982)	2026-08-13 15:56:24 +00:00
args.test.ts	docs(ai-builder): Add a source-linking provenance step to the eval-authoring skill (no-changelog) (#34721)	2026-07-22 22:35:13 +00:00
artifact-agent-handler.test.ts	test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888)	2026-07-15 12:34:21 +00:00
artifact-config-eval-handler.test.ts	chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673)	2026-07-22 08:18:41 +00:00
artifact-context.test.ts	test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888)	2026-07-15 12:34:21 +00:00
artifact-workflow-handler.test.ts	chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673)	2026-07-22 08:18:41 +00:00
assertion-judge.test.ts	test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888)	2026-07-15 12:34:21 +00:00
attribution.test.ts	feat: Add the credential-setup browser eval lane (no-changelog) (#35982)	2026-08-13 15:56:24 +00:00
browser-lane-serialisation.test.ts	feat: Add the credential-setup browser eval lane (no-changelog) (#35982)	2026-08-13 15:56:24 +00:00
bucket-from-evaluation.test.ts	fix: Report an unverifiable unit as notVerified, not a silent pass (#34421)	2026-07-17 14:23:10 +00:00
build-cost-report.test.ts	feat(core): Show whether context arrived in the two-arm cost comparison (no-changelog)	2026-08-20 17:49:38 +02:00
build-expectations-verifier.test.ts	test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888)	2026-07-15 12:34:21 +00:00
build-orchestrator.test.ts	feat(core): Grade the agent's context state in Instance AI evals (no-changelog)	2026-08-20 11:51:54 +02:00
build-workflow-conversation-seed.test.ts	fix(core): Remember skipped Instance AI credential setup for the rest of a thread (#36308)	2026-08-17 12:53:22 +00:00
build-workflow-seed-cleanup.test.ts	test(ai-builder): Let a case declare a credential already broken before the conversation started (no-changelog) (#36029)	2026-08-13 13:33:11 +00:00
capture-run-debug.test.ts	ci: Bound and instrument Instance AI eval lane containers (no-changelog) (#33903)	2026-07-14 09:28:46 +00:00
case-files-schema.test.ts	test(core): Extend Instance AI intent eval fixtures (no-changelog) (#33224)	2026-07-07 12:49:32 +00:00
case-pipeline.test.ts	fix(ai-builder): Give a case's scenario data tables per-run names (no-changelog) (#35543)	2026-08-05 12:48:00 +00:00
checklist-verifier.test.ts	feat(core): Improve Instance AI model experiments (no-changelog) (#35034)	2026-08-13 10:15:03 +00:00
ci-metadata.test.ts	refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691)	2026-07-22 16:25:44 +00:00
cleanup-build.test.ts	fix(core): Search the whole context when checking exact values (no-changelog)	2026-08-21 13:17:17 +02:00
collect-expectations.test.ts	feat(core): Make instanceAI evals more mcp-friendly (no-changelog) (#32916)	2026-06-25 06:33:50 +00:00
comparison-compare.test.ts	fix(ai-builder): Keep model-provider outages out of eval builder scores (no-changelog) (#35293)	2026-08-04 10:52:11 +00:00
comparison-format.test.ts	chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673)	2026-07-22 08:18:41 +00:00
comparison-gate.test.ts	ci(ai-builder): Gate the mcp-gate eval tier absolutely (no-changelog) (#35762)	2026-08-06 14:37:04 +00:00
comparison-statistics.test.ts	feat(ai-builder): Add per-PR eval regression detection vs LangSmith baseline (#29456)	2026-05-06 08:15:08 +00:00
context-assertions.test.ts	fix(core): Let a context claim say which moment it is about (no-changelog)	2026-08-21 14:19:07 +02:00
context-outcome.test.ts	feat(core): Classify whether context was present, and whether it was used (no-changelog)	2026-08-20 15:36:10 +02:00
conversation-seed.test.ts	fix(core): Do not promote a seeded agent the live turn never continued (no-changelog)	2026-08-21 15:32:56 +02:00
conversation-text.test.ts	feat(ai-builder): Support seeding agents alongside workflows in eval threads (no-changelog) (#35548)	2026-08-06 12:40:14 +00:00
credential-setup-checks.test.ts	feat: Add the credential-setup browser eval lane (no-changelog) (#35982)	2026-08-13 15:56:24 +00:00
credential-setup-lane.test.ts	feat: Add the credential-setup browser eval lane (no-changelog) (#35982)	2026-08-13 15:56:24 +00:00
data-discovery.test.ts	feat(ai-builder): Harden the eval harness — crash-recovery journal, parity fixes, extension points (no-changelog) (#34747)	2026-07-23 12:25:55 +00:00
data-workflows-schema.test.ts	feat(core): Grade the agent's context state in Instance AI evals (no-changelog)	2026-08-20 11:51:54 +02:00
data-workflows.test.ts	chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673)	2026-07-22 08:18:41 +00:00
dataset-sync.test.ts	feat(ai-builder): Harden the eval harness — crash-recovery journal, parity fixes, extension points (no-changelog) (#34747)	2026-07-23 12:25:55 +00:00
effective-timeout.test.ts	refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834)	2026-07-27 09:01:09 +00:00
eval-artifact-output-dir.test.ts	fix(ai-builder): Route eval report artifacts through --output-dir (no-changelog) (#35083)	2026-07-29 11:04:42 +00:00
eval-results-dispatcher-contract.test.ts	feat(core): Report per-case tokens and tool calls in Instance AI evals (no-changelog)	2026-08-20 12:19:17 +02:00
event-parser.test.ts	fix(core): Remember skipped Instance AI credential setup for the rest of a thread (#36308)	2026-08-17 12:53:22 +00:00
execution-scenario-seed-data-tables.test.ts	feat: Add typed seed data tables with rows on execution scenarios (#34420)	2026-07-17 17:30:19 +00:00
export-latest-verifier-request.test.ts	feat(core): Harden Instance AI eval attribution and diagnostics (no-changelog) (#31951)	2026-06-09 12:10:10 +00:00
external-workflow-edits.test.ts	test(core): Drive external workflow edits from the user proxy (#35736)	2026-08-12 08:42:20 +00:00
fetch-baseline.test.ts	ci: Bound and instrument Instance AI eval lane containers (no-changelog) (#33903)	2026-07-14 09:28:46 +00:00
fixture-server.test.ts	feat: Add the credential-setup browser eval lane (no-changelog) (#35982)	2026-08-13 15:56:24 +00:00
fixtures.ts	feat(core): Extend evaluations to support node groups (no-changelog) (#35629)	2026-08-19 08:42:53 +00:00
iterations.test.ts	refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691)	2026-07-22 16:25:44 +00:00
lane-allocator.test.ts	refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691)	2026-07-22 16:25:44 +00:00
lane-users.test.ts	test(core): Support credential seeding in MCP workflow evals (no-changelog) (#35782)	2026-08-10 19:56:03 +00:00
langsmith-seed.test.ts	refactor(core): Consolidate agent workspace string replacement tools (no-changelog) (#36178)	2026-08-17 11:11:51 +00:00
langtracer-client.test.ts	feat(core): Add eval:langtracer-push to push eval cases to a suite (no-changelog) (#33781)	2026-07-08 07:52:23 +00:00
langtracer-normalize.test.ts	feat(ai-builder): Collapse the eval case's three seed keys into one seed union (no-changelog) (#35292)	2026-08-03 14:37:06 +00:00
langtracer-provider.test.ts	feat(ai-builder): Collapse the eval case's three seed keys into one seed union (no-changelog) (#35292)	2026-08-03 14:37:06 +00:00
langtracer-push.test.ts	feat(ai-builder): Support seeding agents alongside workflows in eval threads (no-changelog) (#35548)	2026-08-06 12:40:14 +00:00
langtracer-to-exported.test.ts	feat(ai-builder): Support seeding agents alongside workflows in eval threads (no-changelog) (#35548)	2026-08-06 12:40:14 +00:00
mcp-builder-timeout.test.ts	test: Parallelize MCP workflow evals across build lanes (no-changelog) (#33468)	2026-07-03 08:33:42 +00:00
mcp-builder.test.ts	test(core): Support credential seeding in MCP workflow evals (no-changelog) (#35782)	2026-08-10 19:56:03 +00:00
mcp-registry-seeder.test.ts	fix(core): Require an authenticated session for the MCP registry test seed endpoint (no-changelog) (#34880)	2026-07-27 13:20:38 +00:00
memory-context.test.ts	fix(core): Search the whole context when checking exact values (no-changelog)	2026-08-21 13:17:17 +02:00
memory-verifier.test.ts	fix(core): Let a context claim say which moment it is about (no-changelog)	2026-08-21 14:19:07 +02:00
missing-workflow-error.test.ts	refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834)	2026-07-27 09:01:09 +00:00
n8n-client-data-tables.test.ts	fix(ai-builder): Give a case's scenario data tables per-run names (no-changelog) (#35543)	2026-08-05 12:48:00 +00:00
normalize-workflow.test.ts	chore: Migrate instance-ai from Jest to Vitest (#31463)	2026-06-03 06:48:27 +00:00
pair-records.test.ts	feat: Add pairwise workflow eval pipeline (#29123)	2026-05-04 13:26:27 +00:00
partial-results-abort.test.ts	refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834)	2026-07-27 09:01:09 +00:00
prebuilt-coverage.test.ts	feat(ai-builder): Source eval test cases from LangTracer (no-changelog) (#33067)	2026-06-29 10:17:49 +00:00
prebuilt-workflows.test.ts	feat(core): Allow deleting pre-built workflows after evaluating (no-changelog) (#31960)	2026-06-09 08:50:31 +00:00
prior-runs.test.ts	test(core): Cover execution history as a context surface (no-changelog)	2026-08-24 14:54:25 +02:00
redact.test.ts	feat: Add Simplified Custom Auth credential with recipe-guided setup in the AI Assistant (#35056)	2026-08-07 09:22:36 +00:00
relay-connection.test.ts	feat: Add the credential-setup browser eval lane (no-changelog) (#35982)	2026-08-13 15:56:24 +00:00
reshape.test.ts	fix(ai-builder): Keep model-provider outages out of eval builder scores (no-changelog) (#35293)	2026-08-04 10:52:11 +00:00
row-sink.test.ts	feat(ai-builder): Harden the eval harness — crash-recovery journal, parity fixes, extension points (no-changelog) (#34747)	2026-07-23 12:25:55 +00:00
rows.test.ts	feat(ai-builder): Harden the eval harness — crash-recovery journal, parity fixes, extension points (no-changelog) (#34747)	2026-07-23 12:25:55 +00:00
run-debug-report.test.ts	feat(ai-builder): Collapse the eval case's three seed keys into one seed union (no-changelog) (#35292)	2026-08-03 14:37:06 +00:00
runner-thread-id.test.ts	refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834)	2026-07-27 09:01:09 +00:00
runner-workflow-checks.test.ts	refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834)	2026-07-27 09:01:09 +00:00
sandbox-config.test.ts	feat(core): Use n8n default sandbox for Instance AI (no-changelog) (#31335)	2026-06-04 08:31:27 +00:00
scenario-budget-stop.test.ts	fix(ai-builder): Bound eval lane requests and stop abandoned mocked executions (no-changelog) (#35228)	2026-08-06 08:23:37 +00:00
scenario-seed-tables.test.ts	fix(ai-builder): Give a case's scenario data tables per-run names (no-changelog) (#35543)	2026-08-05 12:48:00 +00:00
select-expectations.test.ts	feat(core): Grade the agent's context state in Instance AI evals (no-changelog)	2026-08-20 11:51:54 +02:00
session-boundary.test.ts	feat(core): Run seeded workflows before the graded turn (no-changelog)	2026-08-24 14:35:47 +02:00
stub-services.test.ts	feat: Add pairwise workflow eval pipeline (#29123)	2026-05-04 13:26:27 +00:00
stub-workspace.test.ts	test(core): Add MCP registry discovery evals (no-changelog) (#36063)	2026-08-18 12:28:05 +00:00
subagent-runner.test.ts	refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834)	2026-07-27 09:01:09 +00:00
summary.test.ts	chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673)	2026-07-22 08:18:41 +00:00
transcript-from-events.test.ts	fix(core): Remember skipped Instance AI credential setup for the rest of a thread (#36308)	2026-08-17 12:53:22 +00:00
transient-error.test.ts	fix(ai-builder): Bound eval lane requests and stop abandoned mocked executions (no-changelog) (#35228)	2026-08-06 08:23:37 +00:00
usage-summary.test.ts	feat(core): Report per-case tokens and tool calls in Instance AI evals (no-changelog)	2026-08-20 12:19:17 +02:00
user-proxy.test.ts	feat(ai-builder): Let the eval harness mint Simplified Custom Auth credentials from setup-card recipes (no-changelog) (#36192)	2026-08-19 08:44:23 +00:00
verification-artifact.test.ts	refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834)	2026-07-27 09:01:09 +00:00
verifier.test.ts	feat(core): Harden Instance AI eval attribution and diagnostics (no-changelog) (#31951)	2026-06-09 12:10:10 +00:00
workflow-checks-applicability.test.ts	chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673)	2026-07-22 08:18:41 +00:00
workflow-checks-errors.test.ts	fix: Improve AI Assistant post-build follow-ups (#33419)	2026-07-02 11:34:21 +00:00
workflow-checks-feedback.test.ts	test(ai-builder): Rename response-matches eval check and fix multi-turn wiring (no-changelog) (#32810)	2026-07-10 12:58:13 +00:00
workflow-context.test.ts	feat(core): Extend evaluations to support node groups (no-changelog) (#35629)	2026-08-19 08:42:53 +00:00
workflow-discovery.test.ts	ci: Run PR pipeline on Node 26 (#34032)	2026-08-14 07:54:25 +00:00
workflow-expected.test.ts	refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834)	2026-07-27 09:01:09 +00:00
workflow-report.test.ts	fix(ai-builder): Keep model-provider outages out of eval builder scores (no-changelog) (#35293)	2026-08-04 10:52:11 +00:00
workspace-tool-contract.test.ts	feat(core): Add writable workspaces to agents (#36033)	2026-08-12 12:24:33 +00:00
write-eval-results.test.ts	refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691)	2026-07-22 16:25:44 +00:00
write-partial-results.test.ts	refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834)	2026-07-27 09:01:09 +00:00