From 52db75973ed977cc8cd46b61ca8590a78afcf1a2 Mon Sep 17 00:00:00 2001 From: Mutasem Aldmour <4711238+mutdmour@users.noreply.github.com> Date: Wed, 30 Sep 2026 09:35:38 +0000 Subject: [PATCH] docs: Keep Instance AI eval expectations free of internal mechanics (no-changelog) (#39967) Co-authored-by: Claude Sonnet 5.5 --- .../skills/create-agent-builder-eval/SKILL.md | 37 +++++++++++++ .../skills/create-instance-ai-eval/SKILL.md | 53 ++++++++++++++++++- 2 files changed, 88 insertions(+), 2 deletions(-) diff --git a/.agents/skills/create-agent-builder-eval/SKILL.md b/.agents/skills/create-agent-builder-eval/SKILL.md index 9b941375875..477ef53907a 100644 --- a/.agents/skills/create-agent-builder-eval/SKILL.md +++ b/.agents/skills/create-agent-builder-eval/SKILL.md @@ -70,6 +70,41 @@ or outcome expectations can prove the behavior. Use the substitution test for every expectation. A correct alternative build must pass. A build that misses the requested behavior must fail. +## Write expectations about the Agent, not the builder's internals + +Agent Builder runs as a delegated sub-agent architecture today. That design +can change. Do not write an expectation that depends on it. Follow +"Keep expectations free of internal mechanics" in +[create-instance-ai-eval](../create-instance-ai-eval/SKILL.md). + +Do not write: + +- "The sub-agent runs/tests the Agent before it responds." +- "Instance AI delegates to `build-agent`" or "calls `build-agent` once." +- "The builder calls `call_agent` to verify the Agent." +- "The sub-agent reads the skill, then sets the model." + +Write what a user or a reviewer can observe: + +| ❌ mechanical | ✅ intent | +|---|---| +| "The sub-agent tests the Agent before it reports" | "The final response says the Agent was tested only if a test ran, and it reports the result of that test honestly" | +| "`build-agent` is called with the Slack credential" | "The Agent uses the Slack credential the user has" | +| "The sub-agent adds a tool for the lookup" | "The Agent can look up the order status when a user asks for it" | +| "The builder sets `model` after the catalog lookup" | "The Agent uses a model that the connected credential supports" | + +Put each one where it belongs: + +- The Agent's configuration, tools, skills, and behavior go in + `outcomeExpectations`. +- What the user is told, asked, or shown goes in `processExpectations`. +- The final response is the place to check claims ("says it is ready", + "lists the tools it added"). Do not check which component made the claim. + +The calibration steps below ask you to confirm that `build-agent` ran. That is +a check for you, to be sure the case reached Agent Builder. Do not add it to the +case. + For multi-turn, seeded, or capability-gap cases, follow the case-shape and calibration rules in [create-instance-ai-eval](../create-instance-ai-eval/SKILL.md). This skill overrides its workflow directory and suite guidance for Agent cases. @@ -184,6 +219,8 @@ after a successful push. - A local run captures a standalone Agent. - The transcript proves each process expectation was exercised. - The rendered Agent artifact proves each outcome expectation. +- No expectation names a sub-agent, an internal tool, or a delegation step. + Each one passes for a correct build from a different architecture. - Repeated runs are stable enough for the selected tier. - The case is pushed to `Instance AI capabilities — agents` with suite slug `agents`. - Agent Builder PR changes select the `agents` suite in CI. diff --git a/.agents/skills/create-instance-ai-eval/SKILL.md b/.agents/skills/create-instance-ai-eval/SKILL.md index bdc6a8f783e..2f80183fb6b 100644 --- a/.agents/skills/create-instance-ai-eval/SKILL.md +++ b/.agents/skills/create-instance-ai-eval/SKILL.md @@ -712,8 +712,9 @@ Both are natural-language assertions graded by the same Sonnet judge, and each They run everywhere, including prebuilt/MCP runs (no transcript needed). - **`processExpectations`** — **how the agent behaved during the build**, judged from the transcript. Assert clarifying questions asked (or not re-asked), - tool-call behaviour, plan/approval handling, batching, honouring a correction, - ordering. They need a transcript, so they're **skipped in prebuilt/MCP runs**. + user-visible behaviour, plan/approval handling, batching, honouring a + correction, ordering. Do not assert internal mechanics such as sub-agents or + tool names (see "Keep expectations free of internal mechanics"). They need a transcript, so they're **skipped in prebuilt/MCP runs**. Rule of thumb: an assertion about *the artifact* is an outcome expectation; an assertion about *the conversation or the agent's choices along the way* is a @@ -751,6 +752,54 @@ case, where the source and channel were **left unspecified**): Put intent the conversation only *implied* (a preferred but unstated channel) in `processExpectations`, not `outcomeExpectations`. +## Keep expectations free of internal mechanics + +An expectation describes what the user or the artifact sees. It does not +describe how Instance AI is built inside. The internal architecture changes: +sub-agents, tool names, delegation, planning steps, and run ordering all move. +An expectation that names them goes red when the design changes, although the +behavior is still correct. It also stops proving the behavior: a build that +solves the problem by another route fails it. + +**Do not name these in an expectation:** + +- Sub-agents, delegation, or hand-off ("the sub-agent runs ...", "the + orchestrator delegates ..."). +- Internal tool names or call counts ("calls `build-agent`", "calls + `search-nodes` twice"). +- Internal stages, task lists, or step order that the user cannot see. +- Which component produced a result. + +**Do name these:** + +- What the user sees in the conversation: the question asked, the plan shown + for approval, the claim in the final response. +- What the artifact contains: the workflow or Agent, its configuration, and + how it behaves. +- A user-visible sequence, only when the user asked for it ("asks before it + builds"). + +Rewrite test. Delete the internal term. State the result the term produced. + +| ❌ mechanical | ✅ intent | +|---|---| +| "The sub-agent runs the workflow before it reports success" | "The final response reports only results that the build verified; it does not claim an untested step works" | +| "The orchestrator calls `ask-user` before delegating" | "The agent asks for the missing Slack channel before it builds, and does not guess one" | +| "The builder calls `search-nodes` for the trigger" | "The workflow starts from a trigger that matches the requested schedule" | + +Second test. Imagine the internals are replaced by one agent with different +tools. A correct build must still pass. If the expectation would fail or become +meaningless, rewrite it. + +Some mechanics are a legitimate subject: a case whose purpose is to guard a +specific tool contract (for example, a regression in how a tool reports an +error). Keep these rare. State the reason in `description`. Do not mix them with +intent expectations in the same case. + +**Calibration steps are not expectations.** Reading the transcript to confirm +that a build ran, or that a sub-agent was reached, is a good calibration check. +Keep it in your own run notes. Do not copy it into the case JSON. + ## Robust design vs harness flakiness Two different things — keep them apart: