Commit Graph

30 Commits

Author SHA1 Message Date
Rodrigo Santos da Silva
69ba5a8618
feat(ai-builder): Let the eval harness mint Simplified Custom Auth credentials from setup-card recipes (no-changelog) (#36192)
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
2026-08-19 08:44:23 +00:00
Jaakko Husso
6ace2a5b4f
fix(core): Remember skipped Instance AI credential setup for the rest of a thread (#36308)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 12:53:22 +00:00
Arvin A
45f243419d
test(core): Drive external workflow edits from the user proxy (#35736) 2026-08-12 08:42:20 +00:00
José Braulio González Valido
9ac12d212f
test(ai-builder): Let seeded eval credentials pass their connection test (no-changelog) (#35675)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 09:40:27 +00:00
José Braulio González Valido
166771c77d
feat(ai-builder): Push a seeded case, and hand the agent its workflow (no-changelog) (#35304)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 16:09:01 +00:00
José Braulio González Valido
e62417356f
feat(ai-builder): Collapse the eval case's three seed keys into one seed union (no-changelog) (#35292)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 14:37:06 +00:00
José Braulio González Valido
a2b8383bf9
feat(ai-builder): Let eval cases set up a working credential (no-changelog) (#35410)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 11:21:11 +00:00
Rodrigo Santos da Silva
a25a1ec608
feat: Add manual credential-setup path to instance-ai eval harness (no-changelog) (#35197)
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-03 09:11:25 +00:00
José Braulio González Valido
be12a4ccdc
feat(ai-builder): Let eval runs exercise the builder's web search (no-changelog) (#35274)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 07:31:10 +00:00
José Braulio González Valido
632ef22c0c
feat(ai-builder): Carry an eval case's conversation seed in the case body (no-changelog) (#35074)
Some checks are pending
Build: Benchmark Image / build (push) Waiting to run
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.22.3) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.18.0) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Util: Sync API Docs / sync-public-api (push) Waiting to run
2026-07-28 16:26:08 +00:00
José Braulio González Valido
3f63b73d38
fix(ai-builder): Scope user-proxy decisions to the conversation moment (no-changelog) (#34719)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 00:04:17 +00:00
José Braulio González Valido
ee46e64d50
chore(ai-builder): Typecheck the eval harness and pin its external contracts (no-changelog) (#34673)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 08:18:41 +00:00
José Braulio González Valido
1d61db2252
test(ai-builder): Rename response-matches eval check and fix multi-turn wiring (no-changelog) (#32810)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 12:58:13 +00:00
Robin Braumann
f25699fe2c
test(core): Extend Instance AI intent eval fixtures (no-changelog) (#33224)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-07 12:49:32 +00:00
José Braulio González Valido
4d11639cf1
test(ai-builder): Add plan-rejection eval and harden multi-turn eval consistency (no-changelog) (#33433)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 06:11:22 +00:00
Tomi Turtiainen
7fb04cdb98
refactor: Remove @n8n/utils barrel export (no-changelog) (#33082) 2026-06-30 16:05:24 +03:00
oleg
a00ac2722c
fix(core): Make AI Assistant workflow verification and setup repeatable (#33085)
Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>
2026-06-26 13:39:28 +00:00
José Braulio González Valido
3380ca3790
fix(ai-builder): Reconstruct file-based seeded workflows and judge the whole conversation (no-changelog) (#33048)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 10:06:20 +00:00
José Braulio González Valido
974e7be09b
fix(ai-builder): Scope seedThread reconstruction and add per-turn tool counts for the judge (no-changelog) (#32962)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 10:56:59 +00:00
Milorad FIlipović
d3dd105aae
feat(core): Make instanceAI evals more mcp-friendly (no-changelog) (#32916) 2026-06-25 06:33:50 +00:00
José Braulio González Valido
d7a331afd4
test(ai-builder): Detect excessive build-workflow failures in instance-ai evals (no-changelog) (#32808)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 09:56:59 +00:00
José Braulio González Valido
fe11536df1
feat(ai-builder): Add conversation pre-seeding to workflow evals (no-changelog) (#32196)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-23 17:30:30 +00:00
yehorkardash
d501854409
fix(core): Deduplicate isRecord (no-changelog) (#32469) 2026-06-22 08:35:03 +00:00
José Braulio González Valido
5fa988104b
test(ai-builder): Richer multi-turn eval framework (no-changelog) (#32054)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 12:57:08 +00:00
oleg
77b95c611d
feat(core): Harden Instance AI eval attribution and diagnostics (no-changelog) (#31951) 2026-06-09 12:10:10 +00:00
José Braulio González Valido
7791a940dd
feat(ai-builder): Author-level conversation expectations on eval test cases (no-changelog) (#31787)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 10:06:07 +00:00
José Braulio González Valido
700b32237f
feat(ai-builder): Surface WHAT-dimension binary checks per built workflow (no-changelog) (#30932)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 12:18:52 +01:00
José Braulio González Valido
ada126d9b7
test(ai-builder): Validate user-proxy tool outputs against api-types schemas (no-changelog) (#30905)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 14:15:44 +00:00
José Braulio González Valido
81ea56fa6b
test(ai-builder): Add multi-turn capability for IAI evals (no-changelog) (#30586)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:03:35 +00:00
Bernhard Wittmann
68560fbb9a
refactor: Extract shared eval helpers (no-changelog) (#29800) 2026-05-07 08:05:01 +00:00