diff --git a/.agents/skills/create-agent-builder-eval/SKILL.md b/.agents/skills/create-agent-builder-eval/SKILL.md index c999dc1805c..465d319f779 100644 --- a/.agents/skills/create-agent-builder-eval/SKILL.md +++ b/.agents/skills/create-agent-builder-eval/SKILL.md @@ -45,9 +45,8 @@ value. Rerun the check after the user confirms. Continue only when it passes. - Author the case at `packages/@n8n/instance-ai/evaluations/data/agents/.json`. - Set `"datasets": ["agents"]`. -- Push general Agent Builder cases to - [Instance AI capabilities — agents](https://lang-tracer.n8n-maintenance.workers.dev/suites/10). - Its suite slug is `agents`. +- Ask the driver for the target suite before you do anything else. See + "Ask for the target suite first" below. Do not assume `agents`. - Do not commit the case JSON. LangTracer is the durable source of truth. - Commit changes to this skill, the harness, and CI when applicable. @@ -111,6 +110,22 @@ This skill overrides its workflow directory and suite guidance for Agent cases. The sections below add the Agent-specific parts. Do not skip the general rules they point to. +## Ask for the target suite first + +Follow "Ask for the target suite first" in +[create-instance-ai-eval](../create-instance-ai-eval/SKILL.md). Do it after the +LangTracer preflight and before you source or draft anything. If the request +already names a suite, state it and do not ask. Otherwise ask, in autonomous +mode too, in the same message as the autonomy question. + +For Agent cases, recommend +[Instance AI capabilities — agents](https://lang-tracer.n8n-maintenance.workers.dev/suites/10) +(slug `agents`). Offer `baseline` as well: the Instance AI (INS) team monitors +it and it runs nightly. Get the full list from `list_suites`. Use the suite the +driver picks in every push command, in place of `agents` in the examples below. +Keep the `agents` dataset and the `data/agents/` directory. They do not depend +on the suite. + ## Set the autonomy level first Follow "Set the autonomy level first" in @@ -325,14 +340,14 @@ credential such as `EVAL_OPENAI_API_KEY`. A build-only case does not need one. 4. Classify each red as a product gap, harness limitation, or non-determinism. 5. Use `--iterations 5` before adding a case to a gating tier. 6. Preview the LangTracer change. -7. Push it to `agents`. +7. Push it to the suite the driver chose (`` below). ```bash pnpm exec dotenvx run -f ../../../.env.eval -- \ - pnpm eval:langtracer-push --suite agents --dry-run --changed + pnpm eval:langtracer-push --suite --dry-run --changed pnpm exec dotenvx run -f ../../../.env.eval -- \ - pnpm eval:langtracer-push --suite agents --changed + pnpm eval:langtracer-push --suite --changed ``` The push needs `LANGTRACER_URL` and `LANGTRACER_API_KEY`. Generate a key on the @@ -343,6 +358,7 @@ after a successful push. ## Completion checklist - The LangTracer environment preflight passed before authoring. +- The driver chose the target suite before any work, including in autonomous mode. - The autonomy level was set and stated before any work. - A sourced case links to its source thread, and the source turn was read raw. - Each conditional expectation has proof that its precondition fired. @@ -355,6 +371,7 @@ after a successful push. - No expectation names a sub-agent, an internal tool, or a delegation step. Each one passes for a correct build from a different architecture. - Repeated runs are stable enough for the selected tier. -- The case is pushed to `Instance AI capabilities — agents` with suite slug `agents`. -- Agent Builder PR changes select the `agents` suite in CI. +- The case is pushed to the suite the driver chose, and the report links to it. +- Agent Builder PR changes select the `agents` suite in CI. A case pushed to + another suite does not run in that lane. - The local case JSON is not committed. diff --git a/.agents/skills/create-instance-ai-eval/SKILL.md b/.agents/skills/create-instance-ai-eval/SKILL.md index 2eb19de562b..f3e484ded03 100644 --- a/.agents/skills/create-instance-ai-eval/SKILL.md +++ b/.agents/skills/create-instance-ai-eval/SKILL.md @@ -25,7 +25,9 @@ exhaustive field reference; this skill is the opinionated *how*. > approach.** Author the file locally (uncommitted), calibrate it against a real > build, then **push it to a lang-tracer suite** with `eval:langtracer-push` > (see [Push to a lang-tracer suite](#push-to-a-lang-tracer-suite)) — -> `--suite baseline` for the consolidated corpus n8n CI runs, or a dedicated +> the suite the driver chose (see +> [Ask for the target suite first](#ask-for-the-target-suite-first)) — +> `baseline` for the consolidated corpus n8n CI runs, or a dedicated > capability suite like `agents`. > The suite is the home for the case; the eval CLI reads it back via > `--source langtracer`. You still write the JSON file — it's just the input to @@ -38,6 +40,34 @@ exhaustive field reference; this skill is the opinionated *how*. > pruned and has no durable home. Don't commit a replay case either — derive a > synthetic case from it. See [`case-shapes.md`](case-shapes.md). +## Ask for the target suite first + +**Before you source, draft, or run anything, ask the driver which LangTracer +suite receives every eval you create in this session.** Ask this in both +autonomy levels, including autonomous mode. Never pick the suite yourself and +never default silently. A case pushed to the wrong suite runs in the wrong CI +lane and alerts the wrong team. + +Ask the suite question and the autonomy question together, in one message. +Recommend `baseline`: + +- `baseline` (recommended): the general corpus. The Instance AI (INS) team + monitors it, and it runs nightly. +- Any other suite, such as `agents` or `node-gauntlet`: call `list_suites` and + offer the live list. Do not use a hardcoded list, because suites are added + and retired. + +Rules: + +- The driver's answer applies to all cases in the session. Do not ask again per + case. +- If the driver already named a suite in the request, state it back in one line + and continue. Do not ask again. +- Check that the suite kind matches the case. Push a case with a wrong build + (`capability_gap`) only into a suite that allows that kind. If the suite does + not allow it, tell the driver and ask again. +- Put the chosen suite slug in the final decision log, as a link. + ## Set the autonomy level first **Before you source, draft, or run anything, decide how hands-on the driver @@ -896,10 +926,11 @@ drifted, leaves the rest unchanged, and never prunes. It's the inverse of ```bash cd packages/@n8n/instance-ai +# is the slug the driver chose (e.g. baseline) # preview first — no writes: -pnpm exec dotenvx run -f .env.eval -- pnpm eval:langtracer-push --suite baseline --dry-run --changed +pnpm exec dotenvx run -f .env.eval -- pnpm eval:langtracer-push --suite --dry-run --changed # then push (drop --dry-run): -pnpm exec dotenvx run -f .env.eval -- pnpm eval:langtracer-push --suite baseline --changed +pnpm exec dotenvx run -f .env.eval -- pnpm eval:langtracer-push --suite --changed ``` - **Selectors** (at least one required — no accidental push-all): positional