Mutasem Aldmour
22c8117321
fix(ai-builder): Route eval report artifacts through --output-dir (no-changelog) ( #35083 )
...
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 11:04:42 +00:00
José Braulio González Valido
b93a99945b
feat(ai-builder): Harden the eval harness — crash-recovery journal, parity fixes, extension points (no-changelog) ( #34747 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:25:55 +00:00
José Braulio González Valido
463d50b1bd
refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) ( #34691 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 16:25:44 +00:00
José Braulio González Valido
5fd800de20
feat(ai-builder): Support execution scenarios for first-class agents in evals ( #34384 )
...
Build: Benchmark Image / build (push) Waiting to run
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.22.3) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.18.0) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Util: Sync API Docs / sync-public-api (push) Waiting to run
Test: E2E Performance / prepare-docker (push) Has been cancelled
Release: Storybook / Cloudflare Pages (push) Has been cancelled
Test: E2E Performance / build-and-test-performance (push) Has been cancelled
Test: E2E Performance / Canvas Perf Sentinels (push) Has been cancelled
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 13:54:06 +00:00
Mutasem Aldmour
02af123574
feat: Add typed seed data tables with rows on execution scenarios ( #34420 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 17:30:19 +00:00
Mutasem Aldmour
7d6fd4ca9c
fix: Report an unverifiable unit as notVerified, not a silent pass ( #34421 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 14:23:10 +00:00
Mutasem Aldmour
40c5b76708
fix: Do not lose scenario results on a budget/timeout abort ( #34419 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 11:15:28 +00:00
Benjamin Schroth
d6dcf3b749
test(ai-builder): Support evals for agent building and for config evals (no-changelog) ( #33888 )
...
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
2026-07-15 12:34:21 +00:00
José Braulio González Valido
2233d720ae
ci: Bound and instrument Instance AI eval lane containers (no-changelog) ( #33903 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 09:28:46 +00:00
José Braulio González Valido
3bc2292c78
test(ai-builder): Treat answer-only cases as valid without a built workflow (no-changelog) ( #33942 )
...
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.22.3) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.16.0) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 21:54:14 +00:00
José Braulio González Valido
7dd065a77b
feat(ai-builder): Scale eval budgets by case complexity and bound the heaviest prompts (no-changelog) ( #33950 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 12:58:17 +00:00
José Braulio González Valido
8e6018b0a5
fix(ai-builder): Return experimentUrl from the empty-selection eval early exit (no-changelog) ( #33947 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 12:50:43 +00:00
José Braulio González Valido
a88231544a
feat(ai-builder): Persist eval expectation verdicts to LangSmith run outputs (no-changelog) ( #33788 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 22:21:46 +00:00
Arvin A
e6dee743e6
test(ai-builder): Add advanced user-behavior multi-turn eval cases (no-changelog) ( #31758 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: José Braulio González Valido <jose.gonzalez@n8n.io>
Co-authored-by: José Braulio González Valido <josebragv@gmail.com>
2026-07-09 18:13:42 +00:00
José Braulio González Valido
a89f85b052
fix(ai-builder): Make eval verifier resilient to stalls and exclude no-verdict runs from scoring (no-changelog) ( #33562 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 08:29:53 +00:00
Milorad FIlipović
c5d65998da
test: Reduce non-builder noise in MCP workflow evals (no-changelog) ( #33565 )
2026-07-06 13:48:11 +00:00
Albert Alises
2be4092c4d
fix(core): Improve Instance AI workflow-building reliability ( #33530 )
2026-07-03 13:27:45 +00:00
Milorad FIlipović
579656db2e
test: Parallelize MCP workflow evals across build lanes (no-changelog) ( #33468 )
2026-07-03 08:33:42 +00:00
oleg
6f0d30fbb1
fix: Improve AI Assistant post-build follow-ups ( #33419 )
2026-07-02 11:34:21 +00:00
Milorad FIlipović
3f3b3a756c
fix(core): Harden MCP eval execution against transient network errors (no-changelog) ( #33398 )
2026-07-02 08:51:49 +00:00
José Braulio González Valido
dbadc276a2
test(ai-builder): Pin eval experiments to the Staging LangSmith workspace (no-changelog) ( #33397 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-02 07:17:26 +00:00
José Braulio González Valido
65f375bbd4
feat(ai-builder): Make eval executionScenarios optional for build-only cases (no-changelog) ( #33252 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 09:41:14 +00:00
José Braulio González Valido
23d70d1bb9
feat(ai-builder): Source eval test cases from LangTracer (no-changelog) ( #33067 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 10:17:49 +00:00
José Braulio González Valido
c3741ce12a
feat(ai-builder): Re-run Instance AI evals against the PR head on demand (no-changelog) ( #33148 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 06:00:43 +00:00
José Braulio González Valido
d78aba164e
test(ai-builder): Add absolute green-gate verdict for pr-tier evals (no-changelog) ( #32984 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 10:00:06 +00:00
Milorad FIlipović
e12ce0ead0
feat(core): Enable LangSmith for mcp evaluations (no-changelog) ( #32995 )
2026-06-25 09:33:22 +00:00
Milorad FIlipović
d3dd105aae
feat(core): Make instanceAI evals more mcp-friendly (no-changelog) ( #32916 )
2026-06-25 06:33:50 +00:00
José Braulio González Valido
fe11536df1
feat(ai-builder): Add conversation pre-seeding to workflow evals (no-changelog) ( #32196 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-23 17:30:30 +00:00
José Braulio González Valido
13239d1cfe
feat(ai-builder): Add per-test-case eval credentials via thread credential-view pinning (no-changelog) ( #32161 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-22 14:02:57 +00:00
Riqwan Thamir
b60bc923d4
feat(core): Add ability to view agent traces locally under a flag (no-changelog) ( #32397 )
2026-06-17 08:36:38 +00:00
José Braulio González Valido
5fa988104b
test(ai-builder): Richer multi-turn eval framework (no-changelog) ( #32054 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 12:57:08 +00:00
oleg
77b95c611d
feat(core): Harden Instance AI eval attribution and diagnostics (no-changelog) ( #31951 )
2026-06-09 12:10:10 +00:00
José Braulio González Valido
7791a940dd
feat(ai-builder): Author-level conversation expectations on eval test cases (no-changelog) ( #31787 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 10:06:07 +00:00
Milorad FIlipović
9bb77cc55e
feat(core): Allow deleting pre-built workflows after evaluating (no-changelog) ( #31960 )
2026-06-09 08:50:31 +00:00
José Braulio González Valido
46a222a15b
ci: Re-enable Instance AI evals on PRs with cost-bounded trigger (no-changelog) ( #31430 )
...
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-05 08:40:23 +00:00
Bernhard Wittmann
f4e998f245
fix(core): Prefer MCP registry nodes when wiring AI Agent tools ( #30774 )
...
Co-authored-by: Roman Davydchuk <roman.davydchuk@n8n.io>
2026-06-04 12:24:13 +00:00
José Braulio González Valido
f25e12de87
ci(ai-builder): Add iterations + experiment-name inputs to instance-ai eval dispatch (no-changelog) ( #31631 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 09:57:08 +00:00
José Braulio González Valido
2a1f3d6b2d
feat(ai-builder): Add datasets field for PR-tier suite (no-changelog) ( #31429 )
...
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-02 13:30:31 +00:00
José Braulio González Valido
64f25b2ad8
feat(core): Show eval-mock executions in canvas list (no-changelog) ( #31244 )
...
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-02 13:14:51 +00:00
Arvin A
55d8b59a48
feat(core): Stream tool calls and ship M3 fixtures from LLM eval wire server (no-changelog) ( #30983 )
2026-05-27 14:53:43 +01:00
José Braulio González Valido
700b32237f
feat(ai-builder): Surface WHAT-dimension binary checks per built workflow (no-changelog) ( #30932 )
...
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 12:18:52 +01:00
José Braulio González Valido
81ea56fa6b
test(ai-builder): Add multi-turn capability for IAI evals (no-changelog) ( #30586 )
...
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:03:35 +00:00
José Braulio González Valido
30d9a168bc
feat(ai-builder): Add --prebuilt-workflows flag for eval CLI (no-changelog) ( #29830 )
...
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 11:47:29 +00:00
Albert Alises
869dc32c15
feat(ai-builder): Speeds up Instance AI eval by parallelizing iterations and trimming mock handler (no-changelog) ( #29839 )
2026-05-06 08:15:33 +00:00
José Braulio González Valido
bbe3e2d148
feat(ai-builder): Add per-PR eval regression detection vs LangSmith baseline ( #29456 )
...
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 08:15:08 +00:00
José Braulio González Valido
4fd68bfc99
ci(ai-builder): Parallelize Instance AI eval CI across multiple n8n containers ( #29545 )
...
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 07:22:28 +00:00
José Braulio González Valido
54d9286d92
fix(ai-builder): Filter LangSmith eval dataset by local file slugs ( #29507 )
...
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 13:30:29 +00:00
José Braulio González Valido
29cdd011b0
fix(ai-builder): Honor --timeout-ms across the eval harness (no-changelog) ( #29219 )
...
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 12:49:01 +00:00
José Braulio González Valido
16e5f9572f
feat(ai-builder): Add LangSmith integration for workflow eval tracking (no-changelog) ( #28835 )
...
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 08:47:02 +00:00
Luca Mattiazzi
714981eea3
feat: Add multiple runs to instanceAI eval (no-changelog) ( #28493 )
...
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
2026-04-22 07:11:10 +00:00
