n8n/packages/@n8n/instance-ai/evaluations/__tests__
José Braulio González Valido b0a10229a0
refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 09:01:09 +00:00
..
fixtures fix: Report an unverifiable unit as notVerified, not a silent pass (#34421) 2026-07-17 14:23:10 +00:00
agent-report-rendering.test.ts refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691) 2026-07-22 16:25:44 +00:00
agent-scenario-artifact.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
aggregator.test.ts refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691) 2026-07-22 16:25:44 +00:00
args.test.ts docs(ai-builder): Add a source-linking provenance step to the eval-authoring skill (no-changelog) (#34721) 2026-07-22 22:35:13 +00:00
artifact-agent-handler.test.ts test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888) 2026-07-15 12:34:21 +00:00
artifact-config-eval-handler.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
artifact-context.test.ts test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888) 2026-07-15 12:34:21 +00:00
artifact-workflow-handler.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
assertion-judge.test.ts test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888) 2026-07-15 12:34:21 +00:00
bucket-from-evaluation.test.ts fix: Report an unverifiable unit as notVerified, not a silent pass (#34421) 2026-07-17 14:23:10 +00:00
build-cost-report.test.ts feat(ai-builder): Report per-case claude build cost for MCP eval builds (no-changelog) (#34777) 2026-07-23 18:42:09 +00:00
build-expectations-verifier.test.ts test(ai-builder): Support evals for agent building and for config evals (no-changelog) (#33888) 2026-07-15 12:34:21 +00:00
build-orchestrator.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
build-workflow-seed-cleanup.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
capture-run-debug.test.ts ci: Bound and instrument Instance AI eval lane containers (no-changelog) (#33903) 2026-07-14 09:28:46 +00:00
case-files-schema.test.ts test(core): Extend Instance AI intent eval fixtures (no-changelog) (#33224) 2026-07-07 12:49:32 +00:00
case-pipeline.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
checklist-verifier.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
ci-metadata.test.ts refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691) 2026-07-22 16:25:44 +00:00
cleanup-build.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
collect-expectations.test.ts feat(core): Make instanceAI evals more mcp-friendly (no-changelog) (#32916) 2026-06-25 06:33:50 +00:00
comparison-compare.test.ts feat(ai-builder): Persist eval expectation verdicts to LangSmith run outputs (no-changelog) (#33788) 2026-07-09 22:21:46 +00:00
comparison-format.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
comparison-gate.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
comparison-statistics.test.ts feat(ai-builder): Add per-PR eval regression detection vs LangSmith baseline (#29456) 2026-05-06 08:15:08 +00:00
conversation-seed.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
conversation-text.test.ts test(ai-builder): Rename response-matches eval check and fix multi-turn wiring (no-changelog) (#32810) 2026-07-10 12:58:13 +00:00
data-discovery.test.ts feat(ai-builder): Harden the eval harness — crash-recovery journal, parity fixes, extension points (no-changelog) (#34747) 2026-07-23 12:25:55 +00:00
data-workflows-schema.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
data-workflows.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
dataset-sync.test.ts feat(ai-builder): Harden the eval harness — crash-recovery journal, parity fixes, extension points (no-changelog) (#34747) 2026-07-23 12:25:55 +00:00
effective-timeout.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
eval-results-dispatcher-contract.test.ts feat(ai-builder): Report per-case claude build cost for MCP eval builds (no-changelog) (#34777) 2026-07-23 18:42:09 +00:00
event-parser.test.ts feat(ai-builder): Tidy up tool names and agent builder tool UX (#34646) 2026-07-24 15:26:08 +00:00
execution-scenario-seed-data-tables.test.ts feat: Add typed seed data tables with rows on execution scenarios (#34420) 2026-07-17 17:30:19 +00:00
export-latest-verifier-request.test.ts feat(core): Harden Instance AI eval attribution and diagnostics (no-changelog) (#31951) 2026-06-09 12:10:10 +00:00
fetch-baseline.test.ts ci: Bound and instrument Instance AI eval lane containers (no-changelog) (#33903) 2026-07-14 09:28:46 +00:00
fixtures.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
iterations.test.ts refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691) 2026-07-22 16:25:44 +00:00
lane-allocator.test.ts refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691) 2026-07-22 16:25:44 +00:00
langsmith-seed.test.ts feat(ai-builder): Capture compiled workflow JSON as a trace event for eval seed reconstruction (no-changelog) (#33302) 2026-07-06 15:03:38 +00:00
langtracer-client.test.ts feat(core): Add eval:langtracer-push to push eval cases to a suite (no-changelog) (#33781) 2026-07-08 07:52:23 +00:00
langtracer-normalize.test.ts feat(ai-builder): Dual-tenant LangSmith reads for Instance AI evals (no-changelog) (#33230) 2026-06-29 17:17:14 +00:00
langtracer-provider.test.ts feat(ai-builder): Source eval test cases from LangTracer (no-changelog) (#33067) 2026-06-29 10:17:49 +00:00
langtracer-push.test.ts fix(ai-builder): Detect datasets changes when re-pushing eval cases to LangTracer (no-changelog) (#34658) 2026-07-22 11:55:48 +00:00
langtracer-to-exported.test.ts feat(core): Add eval:langtracer-push to push eval cases to a suite (no-changelog) (#33781) 2026-07-08 07:52:23 +00:00
mcp-builder-timeout.test.ts test: Parallelize MCP workflow evals across build lanes (no-changelog) (#33468) 2026-07-03 08:33:42 +00:00
mcp-builder.test.ts feat(ai-builder): Report per-case claude build cost for MCP eval builds (no-changelog) (#34777) 2026-07-23 18:42:09 +00:00
missing-workflow-error.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
normalize-workflow.test.ts chore: Migrate instance-ai from Jest to Vitest (#31463) 2026-06-03 06:48:27 +00:00
pair-records.test.ts feat: Add pairwise workflow eval pipeline (#29123) 2026-05-04 13:26:27 +00:00
partial-results-abort.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
prebuilt-coverage.test.ts feat(ai-builder): Source eval test cases from LangTracer (no-changelog) (#33067) 2026-06-29 10:17:49 +00:00
prebuilt-workflows.test.ts feat(core): Allow deleting pre-built workflows after evaluating (no-changelog) (#31960) 2026-06-09 08:50:31 +00:00
redact.test.ts feat(ai-builder): Author-level conversation expectations on eval test cases (no-changelog) (#31787) 2026-06-09 10:06:07 +00:00
reshape.test.ts feat(ai-builder): Report per-case claude build cost for MCP eval builds (no-changelog) (#34777) 2026-07-23 18:42:09 +00:00
row-sink.test.ts feat(ai-builder): Harden the eval harness — crash-recovery journal, parity fixes, extension points (no-changelog) (#34747) 2026-07-23 12:25:55 +00:00
rows.test.ts feat(ai-builder): Harden the eval harness — crash-recovery journal, parity fixes, extension points (no-changelog) (#34747) 2026-07-23 12:25:55 +00:00
run-debug-report.test.ts fix(ai-builder): Guard optional conversation in run-debug report label (no-changelog) (#32883) 2026-06-25 12:24:26 +00:00
runner-thread-id.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
runner-workflow-checks.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
sandbox-config.test.ts feat(core): Use n8n default sandbox for Instance AI (no-changelog) (#31335) 2026-06-04 08:31:27 +00:00
scenario-seed-tables.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
select-expectations.test.ts feat(core): Make instanceAI evals more mcp-friendly (no-changelog) (#32916) 2026-06-25 06:33:50 +00:00
stub-services.test.ts feat: Add pairwise workflow eval pipeline (#29123) 2026-05-04 13:26:27 +00:00
subagent-runner.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
summary.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
transcript-from-events.test.ts test(ai-builder): Add advanced user-behavior multi-turn eval cases (no-changelog) (#31758) 2026-07-09 18:13:42 +00:00
transient-error.test.ts fix(core): Harden MCP eval execution against transient network errors (no-changelog) (#33398) 2026-07-02 08:51:49 +00:00
user-proxy.test.ts fix(ai-builder): Scope user-proxy decisions to the conversation moment (no-changelog) (#34719) 2026-07-23 00:04:17 +00:00
verification-artifact.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
verifier.test.ts feat(core): Harden Instance AI eval attribution and diagnostics (no-changelog) (#31951) 2026-06-09 12:10:10 +00:00
workflow-checks-applicability.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
workflow-checks-errors.test.ts fix: Improve AI Assistant post-build follow-ups (#33419) 2026-07-02 11:34:21 +00:00
workflow-checks-feedback.test.ts test(ai-builder): Rename response-matches eval check and fix multi-turn wiring (no-changelog) (#32810) 2026-07-10 12:58:13 +00:00
workflow-discovery.test.ts chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673) 2026-07-22 08:18:41 +00:00
workflow-expected.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
workflow-report.test.ts test: Reduce non-builder noise in MCP workflow evals (no-changelog) (#33565) 2026-07-06 13:48:11 +00:00
workspace-tool-contract.test.ts perf(core): Reduce Instance AI initial tool context (no-changelog) (#33815) 2026-07-10 07:59:35 +00:00
write-eval-results.test.ts refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691) 2026-07-22 16:25:44 +00:00
write-partial-results.test.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00