n8n/packages/@n8n/instance-ai/evaluations/__tests__
Riqwan Thamir 91ebeecb4e
feat(core): Improve Instance AI model experiments (no-changelog) (#35034)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-13 10:15:03 +00:00
..
fixtures fix(ai-builder): Keep model-provider outages out of eval builder scores (no-changelog) (#35293) 2026-08-04 10:52:11 +00:00
agent-report-rendering.test.ts refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691) 2026-07-22 16:25:44 +00:00
agent-scenario-artifact.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
aggregator.test.ts refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691) 2026-07-22 16:25:44 +00:00
args.test.ts docs(ai-builder): Add a source-linking provenance step to the eval-authoring skill (no-changelog) (#34721) 2026-07-22 22:35:13 +00:00
artifact-agent-handler.test.ts test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888) 2026-07-15 12:34:21 +00:00
artifact-config-eval-handler.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
artifact-context.test.ts test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888) 2026-07-15 12:34:21 +00:00
artifact-workflow-handler.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
assertion-judge.test.ts test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888) 2026-07-15 12:34:21 +00:00
attribution.test.ts fix(ai-builder): Keep model-provider outages out of eval builder scores (no-changelog) (#35293) 2026-08-04 10:52:11 +00:00
bucket-from-evaluation.test.ts fix: Report an unverifiable unit as notVerified, not a silent pass (#34421) 2026-07-17 14:23:10 +00:00
build-cost-report.test.ts feat(ai-builder): Report per-case claude build cost for MCP eval builds (no-changelog) (#34777) 2026-07-23 18:42:09 +00:00
build-expectations-verifier.test.ts test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888) 2026-07-15 12:34:21 +00:00
build-orchestrator.test.ts fix(ai-builder): Bound eval lane requests and stop abandoned mocked executions (no-changelog) (#35228) 2026-08-06 08:23:37 +00:00
build-workflow-conversation-seed.test.ts feat(ai-builder): Support seeding agents alongside workflows in eval threads (no-changelog) (#35548) 2026-08-06 12:40:14 +00:00
build-workflow-seed-cleanup.test.ts test(ai-builder): Let seeded eval credentials pass their connection test (no-changelog) (#35675) 2026-08-06 09:40:27 +00:00
capture-run-debug.test.ts ci: Bound and instrument Instance AI eval lane containers (no-changelog) (#33903) 2026-07-14 09:28:46 +00:00
case-files-schema.test.ts test(core): Extend Instance AI intent eval fixtures (no-changelog) (#33224) 2026-07-07 12:49:32 +00:00
case-pipeline.test.ts fix(ai-builder): Give a case's scenario data tables per-run names (no-changelog) (#35543) 2026-08-05 12:48:00 +00:00
checklist-verifier.test.ts feat(core): Improve Instance AI model experiments (no-changelog) (#35034) 2026-08-13 10:15:03 +00:00
ci-metadata.test.ts refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691) 2026-07-22 16:25:44 +00:00
cleanup-build.test.ts feat(ai-builder): Support seeding agents alongside workflows in eval threads (no-changelog) (#35548) 2026-08-06 12:40:14 +00:00
collect-expectations.test.ts feat(core): Make instanceAI evals more mcp-friendly (no-changelog) (#32916) 2026-06-25 06:33:50 +00:00
comparison-compare.test.ts fix(ai-builder): Keep model-provider outages out of eval builder scores (no-changelog) (#35293) 2026-08-04 10:52:11 +00:00
comparison-format.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
comparison-gate.test.ts ci(ai-builder): Gate the mcp-gate eval tier absolutely (no-changelog) (#35762) 2026-08-06 14:37:04 +00:00
comparison-statistics.test.ts feat(ai-builder): Add per-PR eval regression detection vs LangSmith baseline (#29456) 2026-05-06 08:15:08 +00:00
conversation-seed.test.ts feat(ai-builder): Support seeding agents alongside workflows in eval threads (no-changelog) (#35548) 2026-08-06 12:40:14 +00:00
conversation-text.test.ts feat(ai-builder): Support seeding agents alongside workflows in eval threads (no-changelog) (#35548) 2026-08-06 12:40:14 +00:00
data-discovery.test.ts feat(ai-builder): Harden the eval harness — crash-recovery journal, parity fixes, extension points (no-changelog) (#34747) 2026-07-23 12:25:55 +00:00
data-workflows-schema.test.ts feat(ai-builder): Push a seeded case, and hand the agent its workflow (no-changelog) (#35304) 2026-08-05 16:09:01 +00:00
data-workflows.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
dataset-sync.test.ts feat(ai-builder): Harden the eval harness — crash-recovery journal, parity fixes, extension points (no-changelog) (#34747) 2026-07-23 12:25:55 +00:00
effective-timeout.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
eval-artifact-output-dir.test.ts fix(ai-builder): Route eval report artifacts through --output-dir (no-changelog) (#35083) 2026-07-29 11:04:42 +00:00
eval-results-dispatcher-contract.test.ts fix(ai-builder): Keep model-provider outages out of eval builder scores (no-changelog) (#35293) 2026-08-04 10:52:11 +00:00
event-parser.test.ts feat(ai-builder): Tidy up tool names and agent builder tool UX (#34646) 2026-07-24 15:26:08 +00:00
execution-scenario-seed-data-tables.test.ts feat: Add typed seed data tables with rows on execution scenarios (#34420) 2026-07-17 17:30:19 +00:00
export-latest-verifier-request.test.ts feat(core): Harden Instance AI eval attribution and diagnostics (no-changelog) (#31951) 2026-06-09 12:10:10 +00:00
external-workflow-edits.test.ts test(core): Drive external workflow edits from the user proxy (#35736) 2026-08-12 08:42:20 +00:00
fetch-baseline.test.ts ci: Bound and instrument Instance AI eval lane containers (no-changelog) (#33903) 2026-07-14 09:28:46 +00:00
fixtures.ts test(core): Support credential seeding in MCP workflow evals (no-changelog) (#35782) 2026-08-10 19:56:03 +00:00
iterations.test.ts refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691) 2026-07-22 16:25:44 +00:00
lane-allocator.test.ts refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691) 2026-07-22 16:25:44 +00:00
lane-users.test.ts test(core): Support credential seeding in MCP workflow evals (no-changelog) (#35782) 2026-08-10 19:56:03 +00:00
langsmith-seed.test.ts feat(ai-builder): Collapse the eval case's three seed keys into one seed union (no-changelog) (#35292) 2026-08-03 14:37:06 +00:00
langtracer-client.test.ts feat(core): Add eval:langtracer-push to push eval cases to a suite (no-changelog) (#33781) 2026-07-08 07:52:23 +00:00
langtracer-normalize.test.ts feat(ai-builder): Collapse the eval case's three seed keys into one seed union (no-changelog) (#35292) 2026-08-03 14:37:06 +00:00
langtracer-provider.test.ts feat(ai-builder): Collapse the eval case's three seed keys into one seed union (no-changelog) (#35292) 2026-08-03 14:37:06 +00:00
langtracer-push.test.ts feat(ai-builder): Support seeding agents alongside workflows in eval threads (no-changelog) (#35548) 2026-08-06 12:40:14 +00:00
langtracer-to-exported.test.ts feat(ai-builder): Support seeding agents alongside workflows in eval threads (no-changelog) (#35548) 2026-08-06 12:40:14 +00:00
mcp-builder-timeout.test.ts test: Parallelize MCP workflow evals across build lanes (no-changelog) (#33468) 2026-07-03 08:33:42 +00:00
mcp-builder.test.ts test(core): Support credential seeding in MCP workflow evals (no-changelog) (#35782) 2026-08-10 19:56:03 +00:00
mcp-registry-seeder.test.ts fix(core): Require an authenticated session for the MCP registry test seed endpoint (no-changelog) (#34880) 2026-07-27 13:20:38 +00:00
missing-workflow-error.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
n8n-client-data-tables.test.ts fix(ai-builder): Give a case's scenario data tables per-run names (no-changelog) (#35543) 2026-08-05 12:48:00 +00:00
normalize-workflow.test.ts chore: Migrate instance-ai from Jest to Vitest (#31463) 2026-06-03 06:48:27 +00:00
pair-records.test.ts feat: Add pairwise workflow eval pipeline (#29123) 2026-05-04 13:26:27 +00:00
partial-results-abort.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
prebuilt-coverage.test.ts feat(ai-builder): Source eval test cases from LangTracer (no-changelog) (#33067) 2026-06-29 10:17:49 +00:00
prebuilt-workflows.test.ts feat(core): Allow deleting pre-built workflows after evaluating (no-changelog) (#31960) 2026-06-09 08:50:31 +00:00
redact.test.ts feat: Add Simplified Custom Auth credential with recipe-guided setup in the AI Assistant (#35056) 2026-08-07 09:22:36 +00:00
reshape.test.ts fix(ai-builder): Keep model-provider outages out of eval builder scores (no-changelog) (#35293) 2026-08-04 10:52:11 +00:00
row-sink.test.ts feat(ai-builder): Harden the eval harness — crash-recovery journal, parity fixes, extension points (no-changelog) (#34747) 2026-07-23 12:25:55 +00:00
rows.test.ts feat(ai-builder): Harden the eval harness — crash-recovery journal, parity fixes, extension points (no-changelog) (#34747) 2026-07-23 12:25:55 +00:00
run-debug-report.test.ts feat(ai-builder): Collapse the eval case's three seed keys into one seed union (no-changelog) (#35292) 2026-08-03 14:37:06 +00:00
runner-thread-id.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
runner-workflow-checks.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
sandbox-config.test.ts feat(core): Use n8n default sandbox for Instance AI (no-changelog) (#31335) 2026-06-04 08:31:27 +00:00
scenario-budget-stop.test.ts fix(ai-builder): Bound eval lane requests and stop abandoned mocked executions (no-changelog) (#35228) 2026-08-06 08:23:37 +00:00
scenario-seed-tables.test.ts fix(ai-builder): Give a case's scenario data tables per-run names (no-changelog) (#35543) 2026-08-05 12:48:00 +00:00
select-expectations.test.ts fix(ai-builder): Keep model-provider outages out of eval builder scores (no-changelog) (#35293) 2026-08-04 10:52:11 +00:00
stub-services.test.ts feat: Add pairwise workflow eval pipeline (#29123) 2026-05-04 13:26:27 +00:00
subagent-runner.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
summary.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
transcript-from-events.test.ts feat(ai-builder): Emit transcriptPerRun and buildErrorPerRun in eval-results.json (no-changelog) (#35078) 2026-07-28 19:25:22 +00:00
transient-error.test.ts fix(ai-builder): Bound eval lane requests and stop abandoned mocked executions (no-changelog) (#35228) 2026-08-06 08:23:37 +00:00
user-proxy.test.ts test(ai-builder): Let seeded eval credentials pass their connection test (no-changelog) (#35675) 2026-08-06 09:40:27 +00:00
verification-artifact.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
verifier.test.ts feat(core): Harden Instance AI eval attribution and diagnostics (no-changelog) (#31951) 2026-06-09 12:10:10 +00:00
workflow-checks-applicability.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
workflow-checks-errors.test.ts fix: Improve AI Assistant post-build follow-ups (#33419) 2026-07-02 11:34:21 +00:00
workflow-checks-feedback.test.ts test(ai-builder): Rename response-matches eval check and fix multi-turn wiring (no-changelog) (#32810) 2026-07-10 12:58:13 +00:00
workflow-discovery.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
workflow-expected.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
workflow-report.test.ts fix(ai-builder): Keep model-provider outages out of eval builder scores (no-changelog) (#35293) 2026-08-04 10:52:11 +00:00
workspace-tool-contract.test.ts feat(core): Add writable workspaces to agents (#36033) 2026-08-12 12:24:33 +00:00
write-eval-results.test.ts refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691) 2026-07-22 16:25:44 +00:00
write-partial-results.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00