n8n/packages/@n8n/instance-ai/evaluations/__tests__
n8n-assistant[bot] ed9fa260fa
chore: Resolve master→3.x sync conflict (#34758)
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
Co-authored-by: Stephen Wright <sjw948@gmail.com>
Co-authored-by: n8n-assistant[bot] <100856346+n8n-assistant[bot]@users.noreply.github.com>
Co-authored-by: Charlie Kolb <charlie@n8n.io>
Co-authored-by: Guillaume Jacquart <jacquart.guillaume@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Bernhard Wittmann <bernhard.wittmann@n8n.io>
Co-authored-by: Nikhil Kuriakose <nikhil.kuriakose@n8n.io>
Co-authored-by: Albert Alises <albert.alises@gmail.com>
Co-authored-by: Michael Kret <88898367+michael-radency@users.noreply.github.com>
Co-authored-by: Andreas Fitzek <andreas.fitzek@n8n.io>
Co-authored-by: Elias Meire <elias@meire.dev>
Co-authored-by: Robin Braumann <50590409+bjorger@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: yehorkardash <yehor.kardash@n8n.io>
Co-authored-by: Mike Repeć <mike.repec@n8n.io>
Co-authored-by: Yuliia Pominchuk <31064937+yuliia-pominchuk@users.noreply.github.com>
Co-authored-by: RomanDavydchuk <roman.davydchuk@n8n.io>
Co-authored-by: Kai <kai.hartlage@n8n.io>
Co-authored-by: José Braulio González Valido <jose.gonzalez@n8n.io>
Co-authored-by: Michael Drury <me@michaeldrury.co.uk>
Co-authored-by: Alex Grozav <alex@grozav.com>
Co-authored-by: Lorent Lempereur <looorent@users.noreply.github.com>
Co-authored-by: Rodrigo Santos da Silva <rodrigo.silva@n8n.io>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Co-authored-by: Dmitrii <dmitrii.kulikov@n8n.io>
Co-authored-by: Savelii <savelii.sychov@n8n.io>
Co-authored-by: Arvin A <51036481+DeveloperTheExplorer@users.noreply.github.com>
Co-authored-by: Raúl Gómez Morales <raul00gm@gmail.com>
Co-authored-by: multica-agent <github@multica.ai>
Co-authored-by: Yen Su <49794855+yens1@users.noreply.github.com>
Co-authored-by: Nour Alhadi Mahmoud <nour.mahmoud@n8n.io>
Co-authored-by: Eugene <eugene@n8n.io>
Co-authored-by: LevSky22 <56281588+LevSky22@users.noreply.github.com>
Co-authored-by: Riqwan Thamir <rmthamir@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Sam Wooler <swooler592@gmail.com>
Co-authored-by: Soham Ingle <sohamingle@gmail.com>
Co-authored-by: Vishesh Gupta <vishesh.gupta12@outlook.com>
Co-authored-by: Declan Carroll <declan@n8n.io>
Co-authored-by: Jan Kalkan <jan.kalkan@n8n.io>
Co-authored-by: n8n-assistant[bot] <n8n-assistant[bot]@users.noreply.github.com>
2026-07-23 10:05:19 +02:00
..
fixtures fix: Report an unverifiable unit as notVerified, not a silent pass (#34421) 2026-07-17 14:23:10 +00:00
agent-report-rendering.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
agent-scenario-artifact.test.ts feat(ai-builder): Support execution scenarios for first-class agents in evals (#34384) 2026-07-21 13:54:06 +00:00
aggregator.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
args.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
artifact-agent-handler.test.ts test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888) 2026-07-15 12:34:21 +00:00
artifact-config-eval-handler.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
artifact-context.test.ts test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888) 2026-07-15 12:34:21 +00:00
artifact-workflow-handler.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
assertion-judge.test.ts test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888) 2026-07-15 12:34:21 +00:00
bucket-from-evaluation.test.ts fix: Report an unverifiable unit as notVerified, not a silent pass (#34421) 2026-07-17 14:23:10 +00:00
build-expectations-verifier.test.ts test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888) 2026-07-15 12:34:21 +00:00
build-orchestrator.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
build-workflow-seed-cleanup.test.ts feat: Add typed seed data tables with rows on execution scenarios (#34420) 2026-07-17 17:30:19 +00:00
capture-run-debug.test.ts ci: Bound and instrument Instance AI eval lane containers (no-changelog) (#33903) 2026-07-14 09:28:46 +00:00
case-files-schema.test.ts test(core): Extend Instance AI intent eval fixtures (no-changelog) (#33224) 2026-07-07 12:49:32 +00:00
case-pipeline.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
checklist-verifier.test.ts fix(ai-builder): Make eval verifier resilient to stalls and exclude no-verdict runs from scoring (no-changelog) (#33562) 2026-07-07 08:29:53 +00:00
ci-metadata.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
cleanup-build.test.ts feat(ai-builder): Support execution scenarios for first-class agents in evals (#34384) 2026-07-21 13:54:06 +00:00
collect-expectations.test.ts feat(core): Make instanceAI evals more mcp-friendly (no-changelog) (#32916) 2026-06-25 06:33:50 +00:00
comparison-compare.test.ts feat(ai-builder): Persist eval expectation verdicts to LangSmith run outputs (no-changelog) (#33788) 2026-07-09 22:21:46 +00:00
comparison-format.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
comparison-gate.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
comparison-statistics.test.ts feat(ai-builder): Add per-PR eval regression detection vs LangSmith baseline (#29456) 2026-05-06 08:15:08 +00:00
conversation-seed.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
conversation-text.test.ts test(ai-builder): Rename response-matches eval check and fix multi-turn wiring (no-changelog) (#32810) 2026-07-10 12:58:13 +00:00
data-discovery.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
data-workflows-schema.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
data-workflows.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
dataset-sync.test.ts test: Reduce non-builder noise in MCP workflow evals (no-changelog) (#33565) 2026-07-06 13:48:11 +00:00
effective-timeout.test.ts feat(ai-builder): Scale eval budgets by case complexity and bound the heaviest prompts (no-changelog) (#33950) 2026-07-10 12:58:17 +00:00
eval-results-dispatcher-contract.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
event-parser.test.ts test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888) 2026-07-15 12:34:21 +00:00
execution-scenario-seed-data-tables.test.ts feat: Add typed seed data tables with rows on execution scenarios (#34420) 2026-07-17 17:30:19 +00:00
export-latest-verifier-request.test.ts feat(core): Harden Instance AI eval attribution and diagnostics (no-changelog) (#31951) 2026-06-09 12:10:10 +00:00
fetch-baseline.test.ts ci: Bound and instrument Instance AI eval lane containers (no-changelog) (#33903) 2026-07-14 09:28:46 +00:00
fixtures.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
iterations.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
lane-allocator.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
langsmith-seed.test.ts feat(ai-builder): Capture compiled workflow JSON as a trace event for eval seed reconstruction (no-changelog) (#33302) 2026-07-06 15:03:38 +00:00
langtracer-client.test.ts feat(core): Add eval:langtracer-push to push eval cases to a suite (no-changelog) (#33781) 2026-07-08 07:52:23 +00:00
langtracer-normalize.test.ts feat(ai-builder): Dual-tenant LangSmith reads for Instance AI evals (no-changelog) (#33230) 2026-06-29 17:17:14 +00:00
langtracer-provider.test.ts feat(ai-builder): Source eval test cases from LangTracer (no-changelog) (#33067) 2026-06-29 10:17:49 +00:00
langtracer-push.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
langtracer-to-exported.test.ts feat(core): Add eval:langtracer-push to push eval cases to a suite (no-changelog) (#33781) 2026-07-08 07:52:23 +00:00
mcp-builder-timeout.test.ts test: Parallelize MCP workflow evals across build lanes (no-changelog) (#33468) 2026-07-03 08:33:42 +00:00
mcp-builder.test.ts test(core): Extend Instance AI intent eval fixtures (no-changelog) (#33224) 2026-07-07 12:49:32 +00:00
missing-workflow-error.test.ts test(ai-builder): Add advanced user-behavior multi-turn eval cases (no-changelog) (#31758) 2026-07-09 18:13:42 +00:00
normalize-workflow.test.ts chore: Migrate instance-ai from Jest to Vitest (#31463) 2026-06-03 06:48:27 +00:00
pair-records.test.ts feat: Add pairwise workflow eval pipeline (#29123) 2026-05-04 13:26:27 +00:00
partial-results-abort.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
prebuilt-coverage.test.ts feat(ai-builder): Source eval test cases from LangTracer (no-changelog) (#33067) 2026-06-29 10:17:49 +00:00
prebuilt-workflows.test.ts feat(core): Allow deleting pre-built workflows after evaluating (no-changelog) (#31960) 2026-06-09 08:50:31 +00:00
redact.test.ts feat(ai-builder): Author-level conversation expectations on eval test cases (no-changelog) (#31787) 2026-06-09 10:06:07 +00:00
reshape.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
run-debug-report.test.ts fix(ai-builder): Guard optional conversation in run-debug report label (no-changelog) (#32883) 2026-06-25 12:24:26 +00:00
runner-thread-id.test.ts fix(core): Teach the builder the SDK language subset and stop repeated invalid repairs (#31911) 2026-06-09 07:42:28 +00:00
runner-workflow-checks.test.ts chore: Migrate instance-ai from Jest to Vitest (#31463) 2026-06-03 06:48:27 +00:00
sandbox-config.test.ts feat(core): Use n8n default sandbox for Instance AI (no-changelog) (#31335) 2026-06-04 08:31:27 +00:00
scenario-seed-tables.test.ts feat: Add typed seed data tables with rows on execution scenarios (#34420) 2026-07-17 17:30:19 +00:00
select-expectations.test.ts feat(core): Make instanceAI evals more mcp-friendly (no-changelog) (#32916) 2026-06-25 06:33:50 +00:00
stub-services.test.ts feat: Add pairwise workflow eval pipeline (#29123) 2026-05-04 13:26:27 +00:00
subagent-runner.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
summary.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
transcript-from-events.test.ts test(ai-builder): Add advanced user-behavior multi-turn eval cases (no-changelog) (#31758) 2026-07-09 18:13:42 +00:00
transient-error.test.ts fix(core): Harden MCP eval execution against transient network errors (no-changelog) (#33398) 2026-07-02 08:51:49 +00:00
user-proxy.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
verification-artifact.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
verifier.test.ts feat(core): Harden Instance AI eval attribution and diagnostics (no-changelog) (#31951) 2026-06-09 12:10:10 +00:00
workflow-checks-applicability.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
workflow-checks-errors.test.ts fix: Improve AI Assistant post-build follow-ups (#33419) 2026-07-02 11:34:21 +00:00
workflow-checks-feedback.test.ts test(ai-builder): Rename response-matches eval check and fix multi-turn wiring (no-changelog) (#32810) 2026-07-10 12:58:13 +00:00
workflow-discovery.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
workflow-expected.test.ts test(ai-builder): Treat answer-only cases as valid without a built workflow (no-changelog) (#33942) 2026-07-10 21:54:14 +00:00
workflow-report.test.ts feat: Remove dead nodes (#34327) 2026-07-22 10:43:54 +02:00
workspace-tool-contract.test.ts perf(core): Reduce Instance AI initial tool context (no-changelog) (#33815) 2026-07-10 07:59:35 +00:00
write-eval-results.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00
write-partial-results.test.ts chore: Resolve master→3.x sync conflict (#34758) 2026-07-23 10:05:19 +02:00