From f6663940edd73ecd6a0696ceb18cd32dcc11d90e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jos=C3=A9=20Braulio=20Gonz=C3=A1lez=20Valido?= Date: Wed, 5 Aug 2026 09:32:55 +0100 Subject: [PATCH] docs(ai-builder): Add the scrub recipe for seeded eval cases (no-changelog) (#35536) Co-authored-by: Claude Opus 5 (1M context) --- .../skills/create-instance-ai-eval/SKILL.md | 2 +- .../create-instance-ai-eval/case-shapes.md | 46 ++-- .../create-instance-ai-eval/sourcing-cases.md | 219 +++++++++++++++++- 3 files changed, 231 insertions(+), 36 deletions(-) diff --git a/.agents/skills/create-instance-ai-eval/SKILL.md b/.agents/skills/create-instance-ai-eval/SKILL.md index 53f76cb0cde..56fc0c6dea5 100644 --- a/.agents/skills/create-instance-ai-eval/SKILL.md +++ b/.agents/skills/create-instance-ai-eval/SKILL.md @@ -104,7 +104,7 @@ case can still assert outcome), but the primary shape drives the work. | **Build** (default) | Does the workflow the agent builds actually *work*? | `outcomeExpectations` + `executionScenarios` | | **Behaviour / process** | Does the agent *converse* correctly (ask the right clarifying question, not re-ask, honour a correction, respect plan approval)? | `processExpectations` + multi-turn director script; often **build-only** | | **Credential** | Does the build behave correctly given a specific credential view? | `credentials[]` | -| **Seeded** | Reproduce a conversation mid-thread and drive the turn under test | `seed` (`mode: "inline"` or `"replay"`) | +| **Seeded** | Start mid-thread, with prior work already in place, and drive the turn under test | `seed` (authored `mode: "inline"`; `"replay"` for a local check) | **Build** is documented in full below. The other three, the director-script vocabulary, and the seeding modes are in [`case-shapes.md`](case-shapes.md). diff --git a/.agents/skills/create-instance-ai-eval/case-shapes.md b/.agents/skills/create-instance-ai-eval/case-shapes.md index bd237e06177..91809f75ae7 100644 --- a/.agents/skills/create-instance-ai-eval/case-shapes.md +++ b/.agents/skills/create-instance-ai-eval/case-shapes.md @@ -345,19 +345,19 @@ assertion: two credentials with one scripted invalid should show exactly `1`. ## Seeded cases (start mid-conversation) -A seeded case restores prior history into the build thread *before* the live -turn, so the eval drives only the turn under test. Use it to reproduce a real -situation — a conversation up to some point, then a message that should trigger -(or correct) a behaviour. +A seeded case puts prior history into the build thread *before* the live turn, so +the eval drives only the turn under test. Use it to set up the situation you want +to test — history up to some point, then a message that should trigger (or +correct) a behaviour. Pick the lightest mode that fits: | Situation | Mode | Pairs with | |---|---|---| -| Reproduce a real conversation (common case) | `seed.mode: "replay"` — fetch + reconstruct its LangSmith trace at run time; nothing committed | supplies its own live turn (omit `conversation`) | | Prior work already exists (a workflow to repair) | `seed.mode: "inline"` — prior messages + the workflows they reference, in the case body | a normal `conversation` for the live turn | | Prelude is just "what was discussed" (no tool calls, no workflows) | `seed.mode: "inline"` with `{role, text}` shorthand messages | a normal `conversation` for the live turn | | Shallow 2–3 turn prelude where the agent's live replies matter | none — a plain multi-turn `conversation` re-drives it live | — | +| Confirming a real failure locally, before authoring the case | `seed.mode: "replay"` — rebuilds a thread from its LangSmith trace at run time; nothing committed, expires with the trace | supplies its own live turn (omit `conversation`) | Both modes are implemented and wired (`harness/conversation-seed.ts` + `harness/langsmith-seed.ts`, threaded through the runner). The literals match @@ -365,13 +365,12 @@ lang-tracer's `metadata.seed` verbatim, so nothing translates between the repos. ### What the seed does — and does not — exercise -**The seeded portion is restored, not re-run.** The message log is written into -the thread verbatim (marked `seeded: true` so the judge and checks can tell it -apart), and the workflows and data tables the history references are **recreated -on the instance** — so when the live turn runs, the agent sees the same -workspace the original conversation left behind. Data tables are recreated -**schema-only, no rows** (row values are the most sensitive part of a trace and -are kept out of the eval instance). +**The seeded portion is replayed, not re-run.** The prior messages are written into +the thread as they stand (marked `seeded: true` so the judge and checks can tell +them apart), and the workflows and data tables they reference are **created on the +instance** — so when the live turn runs, the agent sees the workspace the case says +it should. Data tables are created **schema-only, no rows**: row values are the most +sensitive thing a table holds, and they stay off the eval instance. The consequence to internalise: **nothing you assert can change what already happened in the seeded turns** — the agent didn't produce them, it's only @@ -391,18 +390,21 @@ the issue is in a *later* turn. (A turn-0 issue can't be isolated by seeding: it lands inside the seed, so you'd bake the bug into the prelude.) Two standing costs keep it a last resort, not a default: -- **Data handling.** It recreates a real conversation on the eval instance. The - most sensitive content is scrubbed first — data-table row values are kept out - and redacted from the restored history, node credentials stripped — but that - isn't guaranteed exhaustive, so treat reproduced content as if it may carry - user data and follow your team's data-handling policy. +- **Data handling.** A replay stands someone's own conversation up on the eval + instance. The most sensitive parts are removed first — data-table row values are + kept out and redacted from the history, node credentials stripped — but that pass + can't be assumed exhaustive, so treat what comes back as potentially personal data + and follow your team's data-handling policy. Cleanup deletes the thread, workflows + and tables when the build finishes, though it is best-effort: a crashed run can + leave them on the instance. Scrubbing the workflow into an `inline` seed is how the + whole concern goes away for good. - **Transience.** It depends on LangSmith trace retention (~14 days); the case stops running once the source trace ages out (tag it `seeded`, keep it out of `full`/`pr`). If a plain prompt + director script can reproduce the situation, prefer that. -### `mode: "replay"` — reproduce a real conversation +### `mode: "replay"` — rebuild a thread for a local run ```json "seed": { "mode": "replay", "threadId": "", "project": "instance-ai" } @@ -416,13 +418,11 @@ sent live. `project` defaults to `instance-ai`. Optional `endpoint` pins a US-tenant source host during the US→EU migration; optional `liveTurnRunId` pins which user turn goes live. -- **Cross-workspace, zero config.** A prod thread can be reproduced in a staging +- **Cross-workspace, zero config.** A production thread can be replayed in a staging eval — the harness enumerates the workspaces your `LANGSMITH_API_KEY` can reach and finds the one holding the thread. It only *reads* the source; the eval - writes its own traces/datasets to its own workspace. Reproducing a real thread - recreates its conversation on the eval instance; the most sensitive content is - scrubbed first (see the data-handling note above), and it's still worth - handling per your team's data policy. + writes its own traces/datasets to its own workspace. What it rebuilds still lands + on the eval instance, so the data-handling note above applies. - **Continue past the live turn.** Add a `conversation` to keep driving after the trace's last message replays (first authored turn = expected assistant reply as proxy reference; subsequent `user` turns become follow-ups). Omit it to replay diff --git a/.agents/skills/create-instance-ai-eval/sourcing-cases.md b/.agents/skills/create-instance-ai-eval/sourcing-cases.md index 4e25de9680c..19e69825243 100644 --- a/.agents/skills/create-instance-ai-eval/sourcing-cases.md +++ b/.agents/skills/create-instance-ai-eval/sourcing-cases.md @@ -1,17 +1,24 @@ -# Sourcing cases from real failures (LangTracer + LangSmith) +# Sourcing cases from real conversations (LangTracer + LangSmith) -The strongest cases encode a **real** failure, not an invented premise. Two -connections help you find one and confirm it — and neither is usually the -durable artifact. You author a synthetic case from what you learn (reach for -`seed.mode: "replay"` only per [`case-shapes.md`](case-shapes.md)). +The strongest cases encode a **real** failure, not an invented premise. But the +case you keep is always one you **wrote**: a real thread tells you what to test, +and you author the case — prompt, workflow, seed and all — from what you learned. +That is what makes the committed fixture durable and free of anyone's personal +data. The [scrub recipe](#scrubbing-a-real-workflow-into-a-synthetic-seed) below is +that path, and it's the default. -- **LangTracer — discover.** It ingests real Instance AI conversations and - clusters them into **capability-gap themes** ("what fails, at scale"), and - stores each analysed conversation. Use it to find high-frequency real failures - worth encoding, instead of guessing a failure mode. -- **LangSmith — verify.** Eval runs and prod conversations land in LangSmith as - raw traces. When a finding or a flaky result is ambiguous, read the raw trace - to confirm exactly what happened — which tool calls fired, and their payloads. +Replaying a thread at run time (`seed.mode: "replay"`) is a **secondary mode**: a +local check that the failure is real, not something to commit. See +[Replaying a thread first](#replaying-a-thread-first-the-secondary-mode). + +Two connections make either possible: + +- **LangTracer — discover.** It analyses Instance AI conversations and clusters + them into **capability-gap themes** ("what fails, at scale"). Use it to find + high-frequency real failures worth encoding, instead of guessing a failure mode. +- **LangSmith — verify.** Eval runs and production conversations land in LangSmith + as traces. When a finding or a flaky result is ambiguous, read the trace to + confirm what happened — which tool calls fired, and their payloads. ## Connect the LangTracer MCP @@ -62,6 +69,194 @@ connected.) its trace is pruned or deleted — so it has no durable home; that's exactly why step 4 turns the confirmed failure into a durable synthetic case. +## Scrubbing a real workflow into a synthetic seed + +A seeded case carries a workflow inside it, and that file is committed and runs for +as long as the case lives. So the workflow has to be **written, not copied**: read +the real one, then author a stand-in that keeps its technical shape and none of its +personal data. One careful pass, with a person reading it, and the committed fixture +is clean from the start — nothing further down the line will clean it up for you. + +### What to look for + +Identifying data hides in more places than the obvious ones: + +- **Parameters** — email addresses, phone numbers, channel names, folder paths, + spreadsheet / document / table ids, internal hostnames, and any token someone + pasted into a header or a query string. A token sitting in a parameter is not a + credential, so nothing strips it for you. +- **Free text inside the workflow** — sticky notes and node notes, string literals in + a Code node, sample rows in a Set node. Sample rows are often real records. +- **The workflow's own name**, which often carries a company or a person + ("Acme lead sync", "Chase invoices — Dana"). This repo is public, so a real + customer's name must not appear anywhere in a case; use a placeholder like + `Acme Corp`. +- **The earlier messages you seed alongside it** — someone's own wording, their name, + their employer, links they pasted. Write those turns yourself, in a user's voice. +- **Recorded tool calls, if you keep any.** Measured on a real thread, this is where + the leaks actually were — the workflow was clean, while the outputs of + `workflows[list]` / `get` / `get-as-code` and `data-tables[list]` carried a live + credential id, the project id, the person's entire workflow inventory, table schemas + down to every column name, and a webhook id. Each `toolCallId` also carries the + source run id. Read the messages: a regex pass over that seed caught one category out + of about eight. + +Provenance is the one thing worth keeping: note the source thread id in the case's +`description` so anyone can find the original later. An id points at the conversation +without carrying any of it. + +### Which version of the workflow to start from + +Take the workflow as it stood when the turn you're testing began, not the state it +ended the thread in. + +- **The assistant built it.** `list_conversation_workflow_builds` lists each build + with a `seq` and `turnRunId`, so you can find the one just before your turn, and + `get_conversation_workflow_build` returns it. Its `contentStatus` says what you + actually get: `stored-json` is ready to work with; `compilable-source` means only + the builder's code was kept, which + [`parseSeedWorkflowCode`](../../../packages/@n8n/instance-ai/evaluations/harness/parse-seed-workflow.ts) + turns back into a workflow; `unrecoverable` and `unrecoverable-intermediate` mean + there is nothing to read. About one build in five has no stored JSON, so don't + assume. And don't reach for a **later** build instead — that one is the workflow + after the fix, so the case would test the repair rather than the problem. +- **The person already had it.** When the editor hands a workflow to the assistant, + the assistant looks it up by id — so the workflow you want is the *result* of the + `workflows[get]` call whose input mentions that id. +- **Nothing was stored.** Older threads pre-date this being kept at all, and the read + says so in a `note` rather than returning an empty list. An empty answer never means + "there was no workflow". Two ways out: replay the thread locally and take the + workflow the harness rebuilds from the trace (see below), or write the stand-in + yourself — the simplest set of nodes that can still show the problem. Hand-authored, + there's no original to compare against, so the before-you-ship checks are the only + safety net left. + +### What to keep, and what to replace + +Keep the technical shape, since that is what the case tests. Replace the values that +point at a real person, company or account. + +| Keep exactly as it is | Safe to replace | +|---|---| +| node `type` and `typeVersion` | URLs, hostnames, webhook paths | +| how the nodes are wired (`connections`), and the node names it refers to | email addresses, people's names, channel names | +| parameter names, and the shape of an expression (`={{ $json.url }}`, `$('Prepare rows')`) | spreadsheet / document / data-table ids | +| settings that change behaviour: `batchSize`, `resource` / `operation` / `mode`, the shape of a condition | sticky notes, node notes, Code-node text, sample data | + +**Swap like for like** — a URL stays a URL, an id keeps the same look and length. +That's not only about readability: + +- Loading a seed checks every node for `id`, `name`, `type`, a numeric `typeVersion`, + a two-number `position` and a `parameters` object, and rejects the whole seed if one + is missing. So don't delete fields while you're in there. +- Ids get swapped for fresh ones by a plain search-and-replace across the seed. + Anything under 8 characters is rejected outright, and a short id that squeaks + through is exactly how you end up rewriting unrelated text that happens to contain + it. + +On one node that comes out as: the host, the id and the key change; the expression, +the reference to another node and the parameter names don't. + +```jsonc +// before +"url": "=https://acme-internal.example/v2/orders/{{ $('Get order').item.json.orderId }}", +"headerParameters": { "parameters": [{ "name": "X-Api-Key", "value": "sk_live_9f2c8b…" }] } + +// after +"url": "=https://api.example.com/v2/orders/{{ $('Get order').item.json.orderId }}", +"headerParameters": { "parameters": [{ "name": "X-Api-Key", "value": "sk_test_placeholder" }] } +``` + +**Trim as you go.** The seed is one field on the case, with a 256KB ceiling — +`load_skill` bodies alone can account for most of it. Trimming has a rule of its own: +**keep the shape here too.** Filtering a list output changes what the agent believes +exists, and a `{ "note": "…" }` where a result belongs invents a shape no tool ever +returns. Shortening the text inside a block is fine; to get rid of a call, drop the +whole tool-call block. If you keep a build-workflow call, its `output.workflowId` has +to match a workflow the seed declares — see [`case-shapes.md`](case-shapes.md). + +### Three easy mistakes + +**Renaming a node.** A node's name is how the rest of the workflow points at it — the +wiring is keyed by name, and expressions call it by name (`$('Old Name')`). It can +also turn up in recorded builder code and in the prose. Miss one reference and the +workflow quietly stops working, which is the kind of over-cleaning that leaves the +case testing nothing. Node names are rarely identifying on their own, so the default +is to leave them be. If you do rename one, `grep -c 'Old Name' .json` should +come back 0. (Workflow names are a different story: each run gets its own copy with a +`[seed …]` suffix, and a seed declaring two workflows with the same name is refused, +because there's no way to tell which one a mention refers to.) + +**Replacing the value the case is about.** In +`http-keep-generic-credential-unknown-service`, the host `queue.fal.run` matters +precisely because n8n has no built-in credential for it — swap in a well-known host +and the case tests the opposite thing. Same with the invented column names in +`flags-unverified-sql-identifiers`. Expectations quote values too ("posts to +`#growth`", a particular model name), so a replacement has to be made in both places, +or the expectation loosened the way others already are for values that don't matter +("an unset or placeholder value is acceptable — it's something to fill in at setup, +not a build mistake"). Knowing what the case asserts is part of scrubbing it, which +is why this is a person's job and not a script's. + +**Tidying up a node's version.** Split In Batches v2 lists its outputs as +`['loop', 'done']`; v3 lists them as `['done', 'loop']`. Same two outputs, opposite +meaning — so bumping the version quietly sends the loop the other way while the wiring +still looks untouched. Leave versions as the conversation had them. + +### Check that only values changed + +Compare the JSON you started from with the workflow you put in the case: + +```bash +leaves() { jq -r '{nodes,connections} | paths(scalars) as $p | "\($p|map(tostring)|join("."))\t\(getpath($p)|tojson)"' | sort; } +diff <(leaves < original-workflow.json) <(jq '.seed.workflows[0]' .json | leaves) +``` + +`leaves` prints one line per value in the file, so the diff is a plain list of what +you changed. Every line should be a value you meant to replace. A line mentioning +`type`, `typeVersion`, `connections` or a behaviour setting means the shape moved, not +just the values; a line on one side only means something was added or dropped. Keys +that aren't part of the graph (`settings`, `meta`, `pinData`) are ignored, so the +original compares cleanly against the trimmed version. + +This is the third of +[Before you ship a seeded case](case-shapes.md#before-you-ship-a-seeded-case--three-checks). +The other two — the problem is still there, and the case fails without the seed — are +what catch a clean-up that kept the shape but lost the point. + +### What the harness already handles + +You don't need to do these by hand: + +- **Credential references on nodes are dropped as the seed loads.** The case's own + `credentials[]` decides what the assistant can see, and a credential's display name + goes with the reference. +- **Data tables are columns only.** The seed format has no place for rows, and a + `rows` key is rejected rather than quietly removed — so table contents can't come + along by accident. +- **A seed workflow is only `id`, `name`, `nodes` and `connections`.** Pinned example + data, instance metadata and settings never travel. +- **Ids and workflow names are per run.** Ids are replaced with fresh ones and names + get a `[seed <8hex>]` suffix, with mentions updated in the prose — though not inside + node definitions or recorded tool calls. + +Everything else in the seed is yours to check. + +### Replaying a thread first (the secondary mode) + +Scrubbing takes work, and sometimes you want to know the failure is real before you +put that work in. That's what `seed.mode: "replay"` is for: give it a thread id and +the harness rebuilds the conversation from its trace at run time, then drives the +turn you care about. Nothing about the thread lands in the repo — the case holds only +the id. + +Treat it as a local check, not a case. The trace it depends on ages out in about two +weeks, so a committed replay case stops working; and it stands up someone's real +conversation on the eval instance, which is exactly what scrubbing exists to avoid. +So run it, confirm the failure, then scrub the workflow it hands you into a seeded +case that will still be there next quarter. Details and limits in +[`case-shapes.md`](case-shapes.md). + ## Sourcing a regression baseline (successful builds) The discover→verify→encode flow above hunts **failures**. The complementary need