docs(ai-builder): Add the scrub recipe for seeded eval cases (no-changelog) (#35536)

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

.agents/skills/create-instance-ai-eval/SKILL.md

@@ -104,7 +104,7 @@ case can still assert outcome), but the primary shape drives the work.
 | **Build** (default) | Does the workflow the agent builds actually *work*? | `outcomeExpectations` + `executionScenarios` |
 | **Behaviour / process** | Does the agent *converse* correctly (ask the right clarifying question, not re-ask, honour a correction, respect plan approval)? | `processExpectations` + multi-turn director script; often **build-only** |
 | **Credential** | Does the build behave correctly given a specific credential view? | `credentials[]` |
-| **Seeded** | Reproduce a conversation mid-thread and drive the turn under test | `seed` (`mode: "inline"` or `"replay"`) |
+| **Seeded** | Start mid-thread, with prior work already in place, and drive the turn under test | `seed` (authored `mode: "inline"`; `"replay"` for a local check) |
 
 **Build** is documented in full below. The other three, the director-script
 vocabulary, and the seeding modes are in [`case-shapes.md`](case-shapes.md).

.agents/skills/create-instance-ai-eval/case-shapes.md

@@ -345,19 +345,19 @@ assertion: two credentials with one scripted invalid should show exactly `1`.
 
 ## Seeded cases (start mid-conversation)
 
-A seeded case restores prior history into the build thread *before* the live
-turn, so the eval drives only the turn under test. Use it to reproduce a real
-situation — a conversation up to some point, then a message that should trigger
-(or correct) a behaviour.
+A seeded case puts prior history into the build thread *before* the live turn, so
+the eval drives only the turn under test. Use it to set up the situation you want
+to test — history up to some point, then a message that should trigger (or
+correct) a behaviour.
 
 Pick the lightest mode that fits:
 
 | Situation | Mode | Pairs with |
 |---|---|---|
-| Reproduce a real conversation (common case) | `seed.mode: "replay"` — fetch + reconstruct its LangSmith trace at run time; nothing committed | supplies its own live turn (omit `conversation`) |
 | Prior work already exists (a workflow to repair) | `seed.mode: "inline"` — prior messages + the workflows they reference, in the case body | a normal `conversation` for the live turn |
 | Prelude is just "what was discussed" (no tool calls, no workflows) | `seed.mode: "inline"` with `{role, text}` shorthand messages | a normal `conversation` for the live turn |
 | Shallow 2–3 turn prelude where the agent's live replies matter | none — a plain multi-turn `conversation` re-drives it live | — |
+| Confirming a real failure locally, before authoring the case | `seed.mode: "replay"` — rebuilds a thread from its LangSmith trace at run time; nothing committed, expires with the trace | supplies its own live turn (omit `conversation`) |
 
 Both modes are implemented and wired (`harness/conversation-seed.ts` +
 `harness/langsmith-seed.ts`, threaded through the runner). The literals match
@@ -365,13 +365,12 @@ lang-tracer's `metadata.seed` verbatim, so nothing translates between the repos.
 
 ### What the seed does — and does not — exercise
 
-**The seeded portion is restored, not re-run.** The message log is written into
-the thread verbatim (marked `seeded: true` so the judge and checks can tell it
-apart), and the workflows and data tables the history references are **recreated
-on the instance** — so when the live turn runs, the agent sees the same
-workspace the original conversation left behind. Data tables are recreated
-**schema-only, no rows** (row values are the most sensitive part of a trace and
-are kept out of the eval instance).
+**The seeded portion is replayed, not re-run.** The prior messages are written into
+the thread as they stand (marked `seeded: true` so the judge and checks can tell
+them apart), and the workflows and data tables they reference are **created on the
+instance** — so when the live turn runs, the agent sees the workspace the case says
+it should. Data tables are created **schema-only, no rows**: row values are the most
+sensitive thing a table holds, and they stay off the eval instance.
 
 The consequence to internalise: **nothing you assert can change what already
 happened in the seeded turns** — the agent didn't produce them, it's only
@@ -391,18 +390,21 @@ the issue is in a *later* turn. (A turn-0 issue can't be isolated by seeding: it
 lands inside the seed, so you'd bake the bug into the prelude.) Two standing
 costs keep it a last resort, not a default:
 
-- **Data handling.** It recreates a real conversation on the eval instance. The
-  most sensitive content is scrubbed first — data-table row values are kept out
-  and redacted from the restored history, node credentials stripped — but that
-  isn't guaranteed exhaustive, so treat reproduced content as if it may carry
-  user data and follow your team's data-handling policy.
+- **Data handling.** A replay stands someone's own conversation up on the eval
+  instance. The most sensitive parts are removed first — data-table row values are
+  kept out and redacted from the history, node credentials stripped — but that pass
+  can't be assumed exhaustive, so treat what comes back as potentially personal data
+  and follow your team's data-handling policy. Cleanup deletes the thread, workflows
+  and tables when the build finishes, though it is best-effort: a crashed run can
+  leave them on the instance. Scrubbing the workflow into an `inline` seed is how the
+  whole concern goes away for good.
 - **Transience.** It depends on LangSmith trace retention (~14 days); the case
   stops running once the source trace ages out (tag it `seeded`, keep it out of
   `full`/`pr`).
 
 If a plain prompt + director script can reproduce the situation, prefer that.
 
-### `mode: "replay"` — reproduce a real conversation
+### `mode: "replay"` — rebuild a thread for a local run
 
 ```json
 "seed": { "mode": "replay", "threadId": "<thread-id>", "project": "instance-ai" }
@@ -416,13 +418,11 @@ sent live. `project` defaults to `instance-ai`. Optional `endpoint` pins a
 US-tenant source host during the US→EU migration; optional `liveTurnRunId` pins
 which user turn goes live.
 
-- **Cross-workspace, zero config.** A prod thread can be reproduced in a staging
+- **Cross-workspace, zero config.** A production thread can be replayed in a staging
   eval — the harness enumerates the workspaces your `LANGSMITH_API_KEY` can reach
   and finds the one holding the thread. It only *reads* the source; the eval
-  writes its own traces/datasets to its own workspace. Reproducing a real thread
-  recreates its conversation on the eval instance; the most sensitive content is
-  scrubbed first (see the data-handling note above), and it's still worth
-  handling per your team's data policy.
+  writes its own traces/datasets to its own workspace. What it rebuilds still lands
+  on the eval instance, so the data-handling note above applies.
 - **Continue past the live turn.** Add a `conversation` to keep driving after the
   trace's last message replays (first authored turn = expected assistant reply as
   proxy reference; subsequent `user` turns become follow-ups). Omit it to replay

.agents/skills/create-instance-ai-eval/sourcing-cases.md

@@ -1,17 +1,24 @@
-# Sourcing cases from real failures (LangTracer + LangSmith)
+# Sourcing cases from real conversations (LangTracer + LangSmith)
 
-The strongest cases encode a **real** failure, not an invented premise. Two
-connections help you find one and confirm it — and neither is usually the
-durable artifact. You author a synthetic case from what you learn (reach for
-`seed.mode: "replay"` only per [`case-shapes.md`](case-shapes.md)).
+The strongest cases encode a **real** failure, not an invented premise. But the
+case you keep is always one you **wrote**: a real thread tells you what to test,
+and you author the case — prompt, workflow, seed and all — from what you learned.
+That is what makes the committed fixture durable and free of anyone's personal
+data. The [scrub recipe](#scrubbing-a-real-workflow-into-a-synthetic-seed) below is
+that path, and it's the default.
 
-- **LangTracer — discover.** It ingests real Instance AI conversations and
-  clusters them into **capability-gap themes** ("what fails, at scale"), and
-  stores each analysed conversation. Use it to find high-frequency real failures
-  worth encoding, instead of guessing a failure mode.
-- **LangSmith — verify.** Eval runs and prod conversations land in LangSmith as
-  raw traces. When a finding or a flaky result is ambiguous, read the raw trace
-  to confirm exactly what happened — which tool calls fired, and their payloads.
+Replaying a thread at run time (`seed.mode: "replay"`) is a **secondary mode**: a
+local check that the failure is real, not something to commit. See
+[Replaying a thread first](#replaying-a-thread-first-the-secondary-mode).
+
+Two connections make either possible:
+
+- **LangTracer — discover.** It analyses Instance AI conversations and clusters
+  them into **capability-gap themes** ("what fails, at scale"). Use it to find
+  high-frequency real failures worth encoding, instead of guessing a failure mode.
+- **LangSmith — verify.** Eval runs and production conversations land in LangSmith
+  as traces. When a finding or a flaky result is ambiguous, read the trace to
+  confirm what happened — which tool calls fired, and their payloads.
 
 ## Connect the LangTracer MCP
 
@@ -62,6 +69,194 @@ connected.)
    its trace is pruned or deleted — so it has no durable home; that's exactly why
    step 4 turns the confirmed failure into a durable synthetic case.
 
+## Scrubbing a real workflow into a synthetic seed
+
+A seeded case carries a workflow inside it, and that file is committed and runs for
+as long as the case lives. So the workflow has to be **written, not copied**: read
+the real one, then author a stand-in that keeps its technical shape and none of its
+personal data. One careful pass, with a person reading it, and the committed fixture
+is clean from the start — nothing further down the line will clean it up for you.
+
+### What to look for
+
+Identifying data hides in more places than the obvious ones:
+
+- **Parameters** — email addresses, phone numbers, channel names, folder paths,
+  spreadsheet / document / table ids, internal hostnames, and any token someone
+  pasted into a header or a query string. A token sitting in a parameter is not a
+  credential, so nothing strips it for you.
+- **Free text inside the workflow** — sticky notes and node notes, string literals in
+  a Code node, sample rows in a Set node. Sample rows are often real records.
+- **The workflow's own name**, which often carries a company or a person
+  ("Acme lead sync", "Chase invoices — Dana"). This repo is public, so a real
+  customer's name must not appear anywhere in a case; use a placeholder like
+  `Acme Corp`.
+- **The earlier messages you seed alongside it** — someone's own wording, their name,
+  their employer, links they pasted. Write those turns yourself, in a user's voice.
+- **Recorded tool calls, if you keep any.** Measured on a real thread, this is where
+  the leaks actually were — the workflow was clean, while the outputs of
+  `workflows[list]` / `get` / `get-as-code` and `data-tables[list]` carried a live
+  credential id, the project id, the person's entire workflow inventory, table schemas
+  down to every column name, and a webhook id. Each `toolCallId` also carries the
+  source run id. Read the messages: a regex pass over that seed caught one category out
+  of about eight.
+
+Provenance is the one thing worth keeping: note the source thread id in the case's
+`description` so anyone can find the original later. An id points at the conversation
+without carrying any of it.
+
+### Which version of the workflow to start from
+
+Take the workflow as it stood when the turn you're testing began, not the state it
+ended the thread in.
+
+- **The assistant built it.** `list_conversation_workflow_builds` lists each build
+  with a `seq` and `turnRunId`, so you can find the one just before your turn, and
+  `get_conversation_workflow_build` returns it. Its `contentStatus` says what you
+  actually get: `stored-json` is ready to work with; `compilable-source` means only
+  the builder's code was kept, which
+  [`parseSeedWorkflowCode`](../../../packages/@n8n/instance-ai/evaluations/harness/parse-seed-workflow.ts)
+  turns back into a workflow; `unrecoverable` and `unrecoverable-intermediate` mean
+  there is nothing to read. About one build in five has no stored JSON, so don't
+  assume. And don't reach for a **later** build instead — that one is the workflow
+  after the fix, so the case would test the repair rather than the problem.
+- **The person already had it.** When the editor hands a workflow to the assistant,
+  the assistant looks it up by id — so the workflow you want is the *result* of the
+  `workflows[get]` call whose input mentions that id.
+- **Nothing was stored.** Older threads pre-date this being kept at all, and the read
+  says so in a `note` rather than returning an empty list. An empty answer never means
+  "there was no workflow". Two ways out: replay the thread locally and take the
+  workflow the harness rebuilds from the trace (see below), or write the stand-in
+  yourself — the simplest set of nodes that can still show the problem. Hand-authored,
+  there's no original to compare against, so the before-you-ship checks are the only
+  safety net left.
+
+### What to keep, and what to replace
+
+Keep the technical shape, since that is what the case tests. Replace the values that
+point at a real person, company or account.
+
+| Keep exactly as it is | Safe to replace |
+|---|---|
+| node `type` and `typeVersion` | URLs, hostnames, webhook paths |
+| how the nodes are wired (`connections`), and the node names it refers to | email addresses, people's names, channel names |
+| parameter names, and the shape of an expression (`={{ $json.url }}`, `$('Prepare rows')`) | spreadsheet / document / data-table ids |
+| settings that change behaviour: `batchSize`, `resource` / `operation` / `mode`, the shape of a condition | sticky notes, node notes, Code-node text, sample data |
+
+**Swap like for like** — a URL stays a URL, an id keeps the same look and length.
+That's not only about readability:
+
+- Loading a seed checks every node for `id`, `name`, `type`, a numeric `typeVersion`,
+  a two-number `position` and a `parameters` object, and rejects the whole seed if one
+  is missing. So don't delete fields while you're in there.
+- Ids get swapped for fresh ones by a plain search-and-replace across the seed.
+  Anything under 8 characters is rejected outright, and a short id that squeaks
+  through is exactly how you end up rewriting unrelated text that happens to contain
+  it.
+
+On one node that comes out as: the host, the id and the key change; the expression,
+the reference to another node and the parameter names don't.
+
+```jsonc
+// before
+"url": "=https://acme-internal.example/v2/orders/{{ $('Get order').item.json.orderId }}",
+"headerParameters": { "parameters": [{ "name": "X-Api-Key", "value": "sk_live_9f2c8b…" }] }
+
+// after
+"url": "=https://api.example.com/v2/orders/{{ $('Get order').item.json.orderId }}",
+"headerParameters": { "parameters": [{ "name": "X-Api-Key", "value": "sk_test_placeholder" }] }
+```
+
+**Trim as you go.** The seed is one field on the case, with a 256KB ceiling —
+`load_skill` bodies alone can account for most of it. Trimming has a rule of its own:
+**keep the shape here too.** Filtering a list output changes what the agent believes
+exists, and a `{ "note": "…" }` where a result belongs invents a shape no tool ever
+returns. Shortening the text inside a block is fine; to get rid of a call, drop the
+whole tool-call block. If you keep a build-workflow call, its `output.workflowId` has
+to match a workflow the seed declares — see [`case-shapes.md`](case-shapes.md).
+
+### Three easy mistakes
+
+**Renaming a node.** A node's name is how the rest of the workflow points at it — the
+wiring is keyed by name, and expressions call it by name (`$('Old Name')`). It can
+also turn up in recorded builder code and in the prose. Miss one reference and the
+workflow quietly stops working, which is the kind of over-cleaning that leaves the
+case testing nothing. Node names are rarely identifying on their own, so the default
+is to leave them be. If you do rename one, `grep -c 'Old Name' <case>.json` should
+come back 0. (Workflow names are a different story: each run gets its own copy with a
+`[seed …]` suffix, and a seed declaring two workflows with the same name is refused,
+because there's no way to tell which one a mention refers to.)
+
+**Replacing the value the case is about.** In
+`http-keep-generic-credential-unknown-service`, the host `queue.fal.run` matters
+precisely because n8n has no built-in credential for it — swap in a well-known host
+and the case tests the opposite thing. Same with the invented column names in
+`flags-unverified-sql-identifiers`. Expectations quote values too ("posts to
+`#growth`", a particular model name), so a replacement has to be made in both places,
+or the expectation loosened the way others already are for values that don't matter
+("an unset or placeholder value is acceptable — it's something to fill in at setup,
+not a build mistake"). Knowing what the case asserts is part of scrubbing it, which
+is why this is a person's job and not a script's.
+
+**Tidying up a node's version.** Split In Batches v2 lists its outputs as
+`['loop', 'done']`; v3 lists them as `['done', 'loop']`. Same two outputs, opposite
+meaning — so bumping the version quietly sends the loop the other way while the wiring
+still looks untouched. Leave versions as the conversation had them.
+
+### Check that only values changed
+
+Compare the JSON you started from with the workflow you put in the case:
+
+```bash
+leaves() { jq -r '{nodes,connections} | paths(scalars) as $p | "\($p|map(tostring)|join("."))\t\(getpath($p)|tojson)"' | sort; }
+diff <(leaves < original-workflow.json) <(jq '.seed.workflows[0]' <case>.json | leaves)
+```
+
+`leaves` prints one line per value in the file, so the diff is a plain list of what
+you changed. Every line should be a value you meant to replace. A line mentioning
+`type`, `typeVersion`, `connections` or a behaviour setting means the shape moved, not
+just the values; a line on one side only means something was added or dropped. Keys
+that aren't part of the graph (`settings`, `meta`, `pinData`) are ignored, so the
+original compares cleanly against the trimmed version.
+
+This is the third of
+[Before you ship a seeded case](case-shapes.md#before-you-ship-a-seeded-case--three-checks).
+The other two — the problem is still there, and the case fails without the seed — are
+what catch a clean-up that kept the shape but lost the point.
+
+### What the harness already handles
+
+You don't need to do these by hand:
+
+- **Credential references on nodes are dropped as the seed loads.** The case's own
+  `credentials[]` decides what the assistant can see, and a credential's display name
+  goes with the reference.
+- **Data tables are columns only.** The seed format has no place for rows, and a
+  `rows` key is rejected rather than quietly removed — so table contents can't come
+  along by accident.
+- **A seed workflow is only `id`, `name`, `nodes` and `connections`.** Pinned example
+  data, instance metadata and settings never travel.
+- **Ids and workflow names are per run.** Ids are replaced with fresh ones and names
+  get a `[seed <8hex>]` suffix, with mentions updated in the prose — though not inside
+  node definitions or recorded tool calls.
+
+Everything else in the seed is yours to check.
+
+### Replaying a thread first (the secondary mode)
+
+Scrubbing takes work, and sometimes you want to know the failure is real before you
+put that work in. That's what `seed.mode: "replay"` is for: give it a thread id and
+the harness rebuilds the conversation from its trace at run time, then drives the
+turn you care about. Nothing about the thread lands in the repo — the case holds only
+the id.
+
+Treat it as a local check, not a case. The trace it depends on ages out in about two
+weeks, so a committed replay case stops working; and it stands up someone's real
+conversation on the eval instance, which is exactly what scrubbing exists to avoid.
+So run it, confirm the failure, then scrub the workflow it hands you into a seeded
+case that will still be there next quarter. Details and limits in
+[`case-shapes.md`](case-shapes.md).
+
 ## Sourcing a regression baseline (successful builds)
 
 The discover→verify→encode flow above hunts **failures**. The complementary need