Commit Graph

69 Commits

Author SHA1 Message Date
oleg
a1df115aef
fix(core): Report auto-bound credentials from AI Assistant workflow builds (#33802) 2026-07-08 13:01:29 +00:00
oleg
2d42b3fecf
fix(core): Reconcile stale AI Assistant mocked-credential simulation plan before workflow verification (#33808) 2026-07-08 13:01:08 +00:00
Benjamin Schroth
48e7900db0
test(ai-builder): Add eval cases for evaluation generation (#33719)
Some checks are pending
Build: Benchmark Image / build (push) Waiting to run
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.22.3) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.16.0) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Util: Sync API Docs / sync-public-api (push) Waiting to run
2026-07-07 15:29:42 +00:00
Albert Alises
e83512dfd8
test(core): Add Instance AI eval cases for recently merged builder changes (no-changelog) (#33763) 2026-07-07 14:12:08 +00:00
Robin Braumann
f25699fe2c
test(core): Extend Instance AI intent eval fixtures (no-changelog) (#33224)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-07 12:49:32 +00:00
Albert Alises
3aa3ffd13b
fix(core): Report node errors in workflow verification (#33576) 2026-07-07 08:21:38 +00:00
Riqwan Thamir
d21a499462
test(core): Add Instance AI loop pattern workflow eval cases (no-changelog) (#33712)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-07 07:26:51 +00:00
Milorad FIlipović
c5d65998da
test: Reduce non-builder noise in MCP workflow evals (no-changelog) (#33565) 2026-07-06 13:48:11 +00:00
José Braulio González Valido
4d11639cf1
test(ai-builder): Add plan-rejection eval and harden multi-turn eval consistency (no-changelog) (#33433)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 06:11:22 +00:00
Albert Alises
2be4092c4d
fix(core): Improve Instance AI workflow-building reliability (#33530) 2026-07-03 13:27:45 +00:00
oleg
6f0d30fbb1
fix: Improve AI Assistant post-build follow-ups (#33419) 2026-07-02 11:34:21 +00:00
Albert Alises
4192c9554e
feat(core): Prefer predefined credentials over generic auth on HTTP Request nodes (#33298) 2026-07-02 07:50:49 +00:00
José Braulio González Valido
2ad33a1303
fix(ai-builder): Fix AI-node eval mock-execution timeouts (no-changelog) (#33323)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 08:33:51 +00:00
José Braulio González Valido
65f375bbd4
feat(ai-builder): Make eval executionScenarios optional for build-only cases (no-changelog) (#33252)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 09:41:14 +00:00
José Braulio González Valido
7ea6900e58
feat(ai-builder): Honor seedThread.liveTurnRunId in eval reconstruction (no-changelog) (#33251)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 09:34:10 +00:00
Mutasem Aldmour
b67ef426e8
test: Add Instance AI flight-status-change workflow eval (#33228)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 06:55:40 +00:00
José Braulio González Valido
1d28330d18
feat(ai-builder): Dual-tenant LangSmith reads for Instance AI evals (no-changelog) (#33230)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 17:17:14 +00:00
Riqwan Thamir
6d550faf7d
chore(core): Add test on voice based agent (#33226) 2026-06-29 14:23:40 +00:00
José Braulio González Valido
23d70d1bb9
feat(ai-builder): Source eval test cases from LangTracer (no-changelog) (#33067)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 10:17:49 +00:00
Albert Alises
de8f2984ed
fix(core): Stop the builder from re-running a workflow it just verified (no-changelog) (#33141) 2026-06-29 08:10:44 +00:00
Jaakko Husso
691d39d9c8
fix(core): Ground WhatsApp Trigger verify-token guidance and add regression evals (no-changelog) (#33095)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 07:39:08 +00:00
José Braulio González Valido
1f012f8cb5
test(ai-builder): Reproduce explicit scenario values in eval mocks (no-changelog) (#33166)
Some checks failed
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.22.3) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.16.0) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Util: Update Node Popularity / update-popularity (push) Has been cancelled
Util: Update Node Popularity / approve-and-automerge (push) Has been cancelled
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 15:15:15 +00:00
oleg
a00ac2722c
fix(core): Make AI Assistant workflow verification and setup repeatable (#33085)
Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>
2026-06-26 13:39:28 +00:00
Jaakko Husso
3b27616b41
fix(core): Settle failed workflow verification instead of looping (no-changelog) (#33111)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 13:14:04 +00:00
José Braulio González Valido
d78aba164e
test(ai-builder): Add absolute green-gate verdict for pr-tier evals (no-changelog) (#32984)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 10:00:06 +00:00
Albert Alises
9161865a72
fix(core): Improve builder handling of errored expressions (no-changelog) (#33059) 2026-06-26 09:23:39 +00:00
Jaakko Husso
713203a1c5
fix(core): Surface infeasible core capabilities instead of silently downgrading (no-changelog) (#33063)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 09:19:17 +00:00
Albert Alises
cf8ef2b1b7
test(core): Add eval ensuring builder decides technical choices instead of asking (no-changelog) (#33053)
Some checks failed
Build: Benchmark Image / build (push) Waiting to run
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.22.3) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.16.0) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Util: Sync API Docs / sync-public-api (push) Waiting to run
Release: Schedule Patch Release PRs / Create patch release PR (${{ matrix.track }}) (beta) (push) Has been cancelled
Release: Schedule Patch Release PRs / Create patch release PR (${{ matrix.track }}) (stable) (push) Has been cancelled
Release: Schedule Patch Release PRs / Create patch release PR (${{ matrix.track }}) (v1) (push) Has been cancelled
2026-06-25 22:33:38 +00:00
Albert Alises
02df832dd2
fix: Prefer placeholders over pre-build setup questions (#32969)
Some checks are pending
Build: Benchmark Image / build (push) Waiting to run
CI: Master (Build, Test, Lint) / Build for Github Cache (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (22.22.3) (push) Waiting to run
CI: Master (Build, Test, Lint) / Unit tests (24.16.0) (push) Waiting to run
CI: Master (Build, Test, Lint) / Lint (push) Waiting to run
CI: Master (Build, Test, Lint) / Performance (push) Waiting to run
CI: Master (Build, Test, Lint) / Notify Slack on failure (push) Blocked by required conditions
Util: Sync API Docs / sync-public-api (push) Waiting to run
2026-06-25 17:39:39 +00:00
Riqwan Thamir
58fb6ca742
feat(core): Add more use case evaluations to instance ai (no-changelog) (#32963) 2026-06-25 14:15:30 +00:00
Milorad FIlipović
5b77a717b0
fix(core): Handle array-style conversations in mcp evals (no-changelog) (#33013) 2026-06-25 11:53:57 +00:00
Riqwan Thamir
b3a399553f
fix(core): Broken workflow verification loop in iAI (no-changelog) (#32979)
Co-authored-by: Oleg Ivaniv <me@olegivaniv.com>
2026-06-25 08:20:46 +00:00
Milorad FIlipović
d3dd105aae
feat(core): Make instanceAI evals more mcp-friendly (no-changelog) (#32916) 2026-06-25 06:33:50 +00:00
Riqwan Thamir
436bfbc875
feat(core): Add test for whatsapp support agent (no-changelog) (#32953) 2026-06-24 15:49:50 +00:00
José Braulio González Valido
d7a331afd4
test(ai-builder): Detect excessive build-workflow failures in instance-ai evals (no-changelog) (#32808)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 09:56:59 +00:00
José Braulio González Valido
365fb7ec6d
test(ai-builder): Add multi-turn context-carry eval case (no-changelog) (#32868)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 08:51:49 +00:00
Albert Alises
5f4d196001
fix(core): Route clarifying questions through ask-user (#32871) 2026-06-24 06:48:14 +00:00
José Braulio González Valido
fe11536df1
feat(ai-builder): Add conversation pre-seeding to workflow evals (no-changelog) (#32196)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-23 17:30:30 +00:00
Albert Alises
846e2cb36d
fix(core): Require verification evidence before Instance AI claims a fix works (no-changelog) (#32835) 2026-06-23 17:30:03 +00:00
Albert Alises
842b01f08e
fix(core): Guide AI builder to create exact Data Table schemas and repair mismatches (no-changelog) (#32811) 2026-06-23 13:56:10 +00:00
José Braulio González Valido
13239d1cfe
feat(ai-builder): Add per-test-case eval credentials via thread credential-view pinning (no-changelog) (#32161)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-22 14:02:57 +00:00
Albert Alises
9ce241b8bf
fix(core): Stop re-asking answered or skipped clarifying questions (#32755) 2026-06-22 13:30:45 +00:00
Albert Alises
d5dd4923a5
fix: Handle HTTP raw body validation loops (#32277) 2026-06-18 12:05:40 +00:00
Riqwan Thamir
9d7ed01e95
feat(core): Add eval + trim builder prompt (no-changelog) (#32484) 2026-06-18 07:55:33 +00:00
Albert Alises
15da27bc52
fix(core): Guide workflow builder to handle list-shaped HTTP responses (no-changelog) (#32144) 2026-06-16 15:15:32 +00:00
José Braulio González Valido
4573969629
fix(ai-builder): Harden eval mock layer for text responses, trigger binary and lookup attribution (no-changelog) (#32179)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 11:04:59 +00:00
José Braulio González Valido
efc8e28967
test(ai-builder): Behaviour eval cases + authoring docs (no-changelog) (#32055)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 18:30:54 +00:00
José Braulio González Valido
5fa988104b
test(ai-builder): Richer multi-turn eval framework (no-changelog) (#32054)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 12:57:08 +00:00
José Braulio González Valido
7791a940dd
feat(ai-builder): Author-level conversation expectations on eval test cases (no-changelog) (#31787)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 10:06:07 +00:00
José Braulio González Valido
e5d4aa9836
fix(ai-builder): Get the agent-with-notion-mcp eval case running (no-changelog) (#31788)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 07:33:13 +00:00