José Braulio González Valido
b0a10229a0
refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) ( #34834 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 09:01:09 +00:00
Anne Aguirre
decb3228c5
feat(ai-builder): Tidy up tool names and agent builder tool UX ( #34646 )
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.22.3) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.18.0) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Build: Benchmark Image / build (push) Has been cancelled
Util: Sync API Docs / sync-public-api (push) Has been cancelled
2026-07-24 15:26:08 +00:00
José Braulio González Valido
cbe5710c4c
feat(ai-builder): Report per-case claude build cost for MCP eval builds (no-changelog) ( #34777 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 18:42:09 +00:00
José Braulio González Valido
b93a99945b
feat(ai-builder): Harden the eval harness — crash-recovery journal, parity fixes, extension points (no-changelog) ( #34747 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:25:55 +00:00
José Braulio González Valido
3f63b73d38
fix(ai-builder): Scope user-proxy decisions to the conversation moment (no-changelog) ( #34719 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 00:04:17 +00:00
José Braulio González Valido
ce6416fe04
docs(ai-builder): Add a source-linking provenance step to the eval-authoring skill (no-changelog) ( #34721 )
...
Build: Benchmark Image / build (push) Waiting to run
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.22.3) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.18.0) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Util: Sync API Docs / sync-public-api (push) Waiting to run
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 22:35:13 +00:00
José Braulio González Valido
463d50b1bd
refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) ( #34691 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 16:25:44 +00:00
José Braulio González Valido
5a1aa3feef
fix(ai-builder): Detect datasets changes when re-pushing eval cases to LangTracer (no-changelog) ( #34658 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 11:55:48 +00:00
José Braulio González Valido
ee46e64d50
chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) ( #34673 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 08:18:41 +00:00
José Braulio González Valido
5fd800de20
feat(ai-builder): Support execution scenarios for first-class agents in evals ( #34384 )
...
Build: Benchmark Image / build (push) Waiting to run
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.22.3) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.18.0) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Util: Sync API Docs / sync-public-api (push) Waiting to run
Test: E2E Performance / prepare-docker (push) Has been cancelled
Release: Storybook / Cloudflare Pages (push) Has been cancelled
Test: E2E Performance / build-and-test-performance (push) Has been cancelled
Test: E2E Performance / Canvas Perf Sentinels (push) Has been cancelled
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 13:54:06 +00:00
Mutasem Aldmour
02af123574
feat: Add typed seed data tables with rows on execution scenarios ( #34420 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 17:30:19 +00:00
Mutasem Aldmour
7d6fd4ca9c
fix: Report an unverifiable unit as notVerified, not a silent pass ( #34421 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 14:23:10 +00:00
Mutasem Aldmour
40c5b76708
fix: Do not lose scenario results on a budget/timeout abort ( #34419 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 11:15:28 +00:00
Benjamin Schroth
d6dcf3b749
test(ai-builder): Support evals for agent building and for config evals (no-changelog) ( #33888 )
...
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
2026-07-15 12:34:21 +00:00
José Braulio González Valido
2233d720ae
ci: Bound and instrument Instance AI eval lane containers (no-changelog) ( #33903 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 09:28:46 +00:00
José Braulio González Valido
3bc2292c78
test(ai-builder): Treat answer-only cases as valid without a built workflow (no-changelog) ( #33942 )
...
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.22.3) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.16.0) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 21:54:14 +00:00
José Braulio González Valido
13c95f2c05
fix(ai-builder): Eliminate non-builder noise families in workflow evals (no-changelog) ( #33944 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 13:25:29 +00:00
José Braulio González Valido
7dd065a77b
feat(ai-builder): Scale eval budgets by case complexity and bound the heaviest prompts (no-changelog) ( #33950 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 12:58:17 +00:00
José Braulio González Valido
1d61db2252
test(ai-builder): Rename response-matches eval check and fix multi-turn wiring (no-changelog) ( #32810 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 12:58:13 +00:00
José Braulio González Valido
35632ce3ab
fix(ai-builder): Sync execution scenarios on lang-tracer push updates (no-changelog) ( #33841 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 12:50:12 +00:00
Riqwan Thamir
345b7aaabe
perf(core): Reduce Instance AI initial tool context (no-changelog) ( #33815 )
...
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-10 07:59:35 +00:00
José Braulio González Valido
a88231544a
feat(ai-builder): Persist eval expectation verdicts to LangSmith run outputs (no-changelog) ( #33788 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 22:21:46 +00:00
Arvin A
e6dee743e6
test(ai-builder): Add advanced user-behavior multi-turn eval cases (no-changelog) ( #31758 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: José Braulio González Valido <jose.gonzalez@n8n.io>
Co-authored-by: José Braulio González Valido <josebragv@gmail.com>
2026-07-09 18:13:42 +00:00
Anne Aguirre
5c6544af4a
feat(core): Port agent builder skills and tools to Instance AI (no-changelog) ( #33384 )
...
Co-authored-by: Robin Braumann <robin.braumann@n8n.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: bjorger <50590409+bjorger@users.noreply.github.com>
Co-authored-by: Michael Drury <michael.drury@n8n.io>
Co-authored-by: Michael Drury <me@michaeldrury.co.uk>
2026-07-08 12:41:12 +00:00
Mutasem Aldmour
c79184b5a5
feat(core): Add eval:langtracer-push to push eval cases to a suite (no-changelog) ( #33781 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 07:52:23 +00:00
Robin Braumann
f25699fe2c
test(core): Extend Instance AI intent eval fixtures (no-changelog) ( #33224 )
...
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-07 12:49:32 +00:00
José Braulio González Valido
a89f85b052
fix(ai-builder): Make eval verifier resilient to stalls and exclude no-verdict runs from scoring (no-changelog) ( #33562 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 08:29:53 +00:00
José Braulio González Valido
eb59191076
feat(ai-builder): Capture compiled workflow JSON as a trace event for eval seed reconstruction (no-changelog) ( #33302 )
...
Build: Benchmark Image / build (push) Waiting to run
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.22.3) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.16.0) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Util: Sync API Docs / sync-public-api (push) Waiting to run
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 15:03:38 +00:00
Milorad FIlipović
c5d65998da
test: Reduce non-builder noise in MCP workflow evals (no-changelog) ( #33565 )
2026-07-06 13:48:11 +00:00
José Braulio González Valido
7fc1e4e791
fix(ai-builder): Track consolidated-tool data table creations for eval cleanup (no-changelog) ( #33580 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 09:51:36 +00:00
Albert Alises
2be4092c4d
fix(core): Improve Instance AI workflow-building reliability ( #33530 )
2026-07-03 13:27:45 +00:00
Milorad FIlipović
579656db2e
test: Parallelize MCP workflow evals across build lanes (no-changelog) ( #33468 )
2026-07-03 08:33:42 +00:00
José Braulio González Valido
a8f1b2c1b8
ci: Scope Instance AI evals dispatcher to PR gating (no-changelog) ( #33499 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 07:29:09 +00:00
oleg
6f0d30fbb1
fix: Improve AI Assistant post-build follow-ups ( #33419 )
2026-07-02 11:34:21 +00:00
Milorad FIlipović
3f3b3a756c
fix(core): Harden MCP eval execution against transient network errors (no-changelog) ( #33398 )
2026-07-02 08:51:49 +00:00
José Braulio González Valido
65f375bbd4
feat(ai-builder): Make eval executionScenarios optional for build-only cases (no-changelog) ( #33252 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 09:41:14 +00:00
José Braulio González Valido
7ea6900e58
feat(ai-builder): Honor seedThread.liveTurnRunId in eval reconstruction (no-changelog) ( #33251 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 09:34:10 +00:00
José Braulio González Valido
b8a993a2ed
test(ai-builder): Render full judge text for eval failures, warn only on barely-passed units (no-changelog) ( #33127 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 09:30:35 +00:00
José Braulio González Valido
1d28330d18
feat(ai-builder): Dual-tenant LangSmith reads for Instance AI evals (no-changelog) ( #33230 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 17:17:14 +00:00
José Braulio González Valido
23d70d1bb9
feat(ai-builder): Source eval test cases from LangTracer (no-changelog) ( #33067 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 10:17:49 +00:00
Arvin A
9b7451f387
ci: Record git provenance metadata on eval experiments (no-changelog) ( #33024 )
2026-06-29 09:12:31 +00:00
José Braulio González Valido
c3741ce12a
feat(ai-builder): Re-run Instance AI evals against the PR head on demand (no-changelog) ( #33148 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 06:00:43 +00:00
oleg
a00ac2722c
fix(core): Make AI Assistant workflow verification and setup repeatable ( #33085 )
...
Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>
2026-06-26 13:39:28 +00:00
José Braulio González Valido
3380ca3790
fix(ai-builder): Reconstruct file-based seeded workflows and judge the whole conversation (no-changelog) ( #33048 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 10:06:20 +00:00
José Braulio González Valido
d78aba164e
test(ai-builder): Add absolute green-gate verdict for pr-tier evals (no-changelog) ( #32984 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 10:00:06 +00:00
José Braulio González Valido
5fc00bf17d
fix(ai-builder): Guard optional conversation in run-debug report label (no-changelog) ( #32883 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 12:24:26 +00:00
Milorad FIlipović
5b77a717b0
fix(core): Handle array-style conversations in mcp evals (no-changelog) ( #33013 )
2026-06-25 11:53:57 +00:00
José Braulio González Valido
974e7be09b
fix(ai-builder): Scope seedThread reconstruction and add per-turn tool counts for the judge (no-changelog) ( #32962 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 10:56:59 +00:00
Milorad FIlipović
e12ce0ead0
feat(core): Enable LangSmith for mcp evaluations (no-changelog) ( #32995 )
2026-06-25 09:33:22 +00:00
Milorad FIlipović
d3dd105aae
feat(core): Make instanceAI evals more mcp-friendly (no-changelog) ( #32916 )
2026-06-25 06:33:50 +00:00
