Commit Graph

233 Commits

Author SHA1 Message Date
Bernhard Wittmann
62d5de3ec7
feat: Add the credential-setup browser eval lane (no-changelog) (#35982)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 15:56:24 +00:00
Rodrigo Santos da Silva
2d37afc13d
test(ai-builder): Let a case declare a credential already broken before the conversation started (no-changelog) (#36029)
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-13 13:33:11 +00:00
Riqwan Thamir
91ebeecb4e
feat(core): Improve Instance AI model experiments (no-changelog) (#35034)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-13 10:15:03 +00:00
Ricardo Espinoza
b9496b2fa6
refactor(core): Consolidate the node search engine into @n8n/ai-utilities (no-changelog) (#35546) 2026-08-13 09:28:06 +00:00
Robin Braumann
01ce4490d0
feat(core): Add writable workspaces to agents (#36033)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-12 12:24:33 +00:00
Arvin A
45f243419d
test(core): Drive external workflow edits from the user proxy (#35736) 2026-08-12 08:42:20 +00:00
Svetoslav Dekov
9205eb3f19
test(core): Support credential seeding in MCP workflow evals (no-changelog) (#35782)
Some checks failed
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.23.2) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.18.1) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Build: v3 Nightly Docker Images / Compute date tag (push) Has been cancelled
Build: v3 Nightly Docker Images / Build and push v3-nightly images (push) Has been cancelled
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Ricardo Espinoza <ricardo@n8n.io>
2026-08-10 19:56:03 +00:00
Raúl Gómez Morales
53841f1b0e
feat: Add Simplified Custom Auth credential with recipe-guided setup in the AI Assistant (#35056) 2026-08-07 09:22:36 +00:00
Svetoslav Dekov
ff37ef5b3f
ci(ai-builder): Gate the mcp-gate eval tier absolutely (no-changelog) (#35762)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-06 14:37:04 +00:00
José Braulio González Valido
9b8f964a55
feat(ai-builder): Support seeding agents alongside workflows in eval threads (no-changelog) (#35548)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 12:40:14 +00:00
José Braulio González Valido
9ac12d212f
test(ai-builder): Let seeded eval credentials pass their connection test (no-changelog) (#35675)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 09:40:27 +00:00
José Braulio González Valido
1575fce711
fix(ai-builder): Bound eval lane requests and stop abandoned mocked executions (no-changelog) (#35228)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 08:23:37 +00:00
José Braulio González Valido
166771c77d
feat(ai-builder): Push a seeded case, and hand the agent its workflow (no-changelog) (#35304)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 16:09:01 +00:00
oleg
f9bddc7ff3
fix: Include node behavior flags in instance-ai eval judge context (no-changelog) (#35644) 2026-08-05 14:59:37 +00:00
José Braulio González Valido
7c61f2826a
fix(ai-builder): Give a case's scenario data tables per-run names (no-changelog) (#35543)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 12:48:00 +00:00
José Braulio González Valido
d57a81ac60
fix(ai-builder): Keep model-provider outages out of eval builder scores (no-changelog) (#35293)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 10:52:11 +00:00
José Braulio González Valido
5d8f2353bf
fix(ai-builder): Give a restored seed workflow a unique name and evict leftovers (no-changelog) (#35299)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 17:26:21 +00:00
José Braulio González Valido
e62417356f
feat(ai-builder): Collapse the eval case's three seed keys into one seed union (no-changelog) (#35292)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 14:37:06 +00:00
Mutasem Aldmour
05a809867e
feat(ai-builder): Add eval check for secrets placed in HTTP node parameters (no-changelog) (#35421)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 13:43:24 +00:00
José Braulio González Valido
c6917a194f
fix(ai-builder): Record the served wire envelope in the eval LLM mock ledger (no-changelog) (#34929)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-03 13:16:45 +00:00
José Braulio González Valido
a2b8383bf9
feat(ai-builder): Let eval cases set up a working credential (no-changelog) (#35410)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 11:21:11 +00:00
Rodrigo Santos da Silva
a25a1ec608
feat: Add manual credential-setup path to instance-ai eval harness (no-changelog) (#35197)
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-03 09:11:25 +00:00
José Braulio González Valido
be12a4ccdc
feat(ai-builder): Let eval runs exercise the builder's web search (no-changelog) (#35274)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 07:31:10 +00:00
Riqwan Thamir
bbd7b7db88
feat(core): Add workflow SDK validation framework for Instance AI (#35045)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-30 15:18:11 +00:00
José Braulio González Valido
c3b50283fd
fix(ai-builder): Report an unparseable LLM check verdict as errored (no-changelog) (#35198)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 08:18:36 +00:00
José Braulio González Valido
aad2fa5577
feat(ai-builder): Add a Google Drive credential template to the eval seeder (no-changelog) (#35200)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 22:17:11 +00:00
Mutasem Aldmour
22c8117321
fix(ai-builder): Route eval report artifacts through --output-dir (no-changelog) (#35083)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 11:04:42 +00:00
José Braulio González Valido
2ae18ad9f6
feat(ai-builder): Validate the eval conversation-seed message envelope (no-changelog) (#35138)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:58:55 +00:00
Luka Zivkovic
650d893151
feat(ai-builder): Emit transcriptPerRun and buildErrorPerRun in eval-results.json (no-changelog) (#35078)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 19:25:22 +00:00
Tomi Turtiainen
b964ed6f4b
refactor(core): Move sleep into @n8n/utils and unify sleep/sleepWithAbort (#35057) 2026-07-28 18:47:18 +00:00
José Braulio González Valido
632ef22c0c
feat(ai-builder): Carry an eval case's conversation seed in the case body (no-changelog) (#35074)
Some checks are pending
Build: Benchmark Image / build (push) Waiting to run
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.22.3) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.18.0) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Util: Sync API Docs / sync-public-api (push) Waiting to run
2026-07-28 16:26:08 +00:00
José Braulio González Valido
6d8b01dc79
fix(ai-builder): Enforce declared field names on eval pin data (no-changelog) (#34960)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 13:11:02 +00:00
Michael Kret
b2af755502
fix(core): Prefer n8n credits covered nodes in Instance AI workflow builds (#35019) 2026-07-28 11:58:42 +00:00
José Braulio González Valido
66408a0cd5
feat(ai-builder): Push agent intent eval cases from data/agents and drop synced case files (no-changelog) (#34957)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 07:25:30 +00:00
José Braulio González Valido
9a96941d81
fix(ai-builder): Pin zero-item premises as empty arrays and attribute pinned-fixture faults (no-changelog) (#34953)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 13:20:52 +00:00
José Braulio González Valido
ba31860da7
fix(core): Require an authenticated session for the MCP registry test seed endpoint (no-changelog) (#34880)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 13:20:38 +00:00
Anne Aguirre
3301e12014
fix(ai-builder): MCP Connections failures are non-blocking for agents (#34722) 2026-07-27 12:47:54 +00:00
José Braulio González Valido
b0a10229a0
refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 09:01:09 +00:00
Anne Aguirre
decb3228c5
feat(ai-builder): Tidy up tool names and agent builder tool UX (#34646)
Some checks failed
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.22.3) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.18.0) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Build: Benchmark Image / build (push) Has been cancelled
Util: Sync API Docs / sync-public-api (push) Has been cancelled
2026-07-24 15:26:08 +00:00
José Braulio González Valido
cbe5710c4c
feat(ai-builder): Report per-case claude build cost for MCP eval builds (no-changelog) (#34777)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 18:42:09 +00:00
José Braulio González Valido
b93a99945b
feat(ai-builder): Harden the eval harness — crash-recovery journal, parity fixes, extension points (no-changelog) (#34747)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:25:55 +00:00
José Braulio González Valido
3f63b73d38
fix(ai-builder): Scope user-proxy decisions to the conversation moment (no-changelog) (#34719)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 00:04:17 +00:00
José Braulio González Valido
ce6416fe04
docs(ai-builder): Add a source-linking provenance step to the eval-authoring skill (no-changelog) (#34721)
Some checks are pending
Build: Benchmark Image / build (push) Waiting to run
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.22.3) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.18.0) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Util: Sync API Docs / sync-public-api (push) Waiting to run
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 22:35:13 +00:00
José Braulio González Valido
463d50b1bd
refactor(ai-builder): Decompose the eval CLI into phase modules with typed seams (no-changelog) (#34691)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 16:25:44 +00:00
Riqwan Thamir
c0fe8ba4db
feat(editor): Track Instance AI entry point as action_source on builder messages (#34671)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-22 13:24:29 +00:00
José Braulio González Valido
5a1aa3feef
fix(ai-builder): Detect datasets changes when re-pushing eval cases to LangTracer (no-changelog) (#34658)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 11:55:48 +00:00
José Braulio González Valido
ee46e64d50
chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 08:18:41 +00:00
José Braulio González Valido
5fd800de20
feat(ai-builder): Support execution scenarios for first-class agents in evals (#34384)
Some checks failed
Build: Benchmark Image / build (push) Waiting to run
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.22.3) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.18.0) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Util: Sync API Docs / sync-public-api (push) Waiting to run
Test: E2E Performance / prepare-docker (push) Has been cancelled
Release: Storybook / Cloudflare Pages (push) Has been cancelled
Test: E2E Performance / build-and-test-performance (push) Has been cancelled
Test: E2E Performance / Canvas Perf Sentinels (push) Has been cancelled
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 13:54:06 +00:00
José Braulio González Valido
32d4c1a123
ci(ai-builder): Point eval workflows at the consolidated LangTracer baseline suite (no-changelog) (#34402)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 13:55:41 +00:00
Mutasem Aldmour
02af123574
feat: Add typed seed data tables with rows on execution scenarios (#34420)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 17:30:19 +00:00