mirror of
https://github.com/n8n-io/n8n.git
synced 2026-10-17 05:29:44 +02:00
feat(ai-builder): Collapse the eval case's three seed keys into one seed union (no-changelog) (#35292)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
José Braulio González Valido
GitHub
parent
fd5ecfff6d
commit
e62417356f
|
|
@ -29,11 +29,11 @@ exhaustive field reference; this skill is the opinionated *how*.
|
|||
> `--source langtracer`. You still write the JSON file — it's just the input to
|
||||
> the push, not a committed artifact.
|
||||
>
|
||||
> **Exception — seeded cases.** The case-write API can't hold any seeding mode, so
|
||||
> seeded cases are never pushed. A `seedThread` case is a local throwaway (don't
|
||||
> commit it either — it dies when its trace is pruned); a `conversationSeed` or
|
||||
> `priorConversation` case isn't transient and has no suite home, so it's the one
|
||||
> sanctioned exception — it lives as committed JSON. See [`case-shapes.md`](case-shapes.md).
|
||||
> **Exception — seeded cases.** The case-write API has no `seed` field, so seeded
|
||||
> cases are never pushed. A `seed.mode: "replay"` case is a local throwaway (don't
|
||||
> commit it either — it dies when its trace is pruned); a `seed.mode: "inline"`
|
||||
> case isn't transient and has no suite home, so it's the one sanctioned
|
||||
> exception — it lives as committed JSON. See [`case-shapes.md`](case-shapes.md).
|
||||
|
||||
## Set the autonomy level first
|
||||
|
||||
|
|
@ -88,7 +88,7 @@ connections help you find and verify one: **LangTracer** clusters real
|
|||
conversations into capability-gap themes (discover what actually fails, at
|
||||
scale), and **LangSmith** holds the raw traces (verify exactly what happened in a
|
||||
run). LangTracer is the discovery layer; the durable artifact is almost always a
|
||||
synthetic case you author from what you learn (use `seedThread` only per
|
||||
synthetic case you author from what you learn (use `seed.mode: "replay"` only per
|
||||
[`case-shapes.md`](case-shapes.md)). See
|
||||
[`sourcing-cases.md`](sourcing-cases.md) for connecting the MCPs and the
|
||||
discover → verify → encode workflow.
|
||||
|
|
@ -104,7 +104,7 @@ case can still assert outcome), but the primary shape drives the work.
|
|||
| **Build** (default) | Does the workflow the agent builds actually *work*? | `outcomeExpectations` + `executionScenarios` |
|
||||
| **Behaviour / process** | Does the agent *converse* correctly (ask the right clarifying question, not re-ask, honour a correction, respect plan approval)? | `processExpectations` + multi-turn director script; often **build-only** |
|
||||
| **Credential** | Does the build behave correctly given a specific credential view? | `credentials[]` |
|
||||
| **Seeded** | Reproduce a conversation mid-thread and drive the turn under test | `seedThread` / `priorConversation` / `conversationSeed` |
|
||||
| **Seeded** | Reproduce a conversation mid-thread and drive the turn under test | `seed` (`mode: "inline"` or `"replay"`) |
|
||||
|
||||
**Build** is documented in full below. The other three, the director-script
|
||||
vocabulary, and the seeding modes are in [`case-shapes.md`](case-shapes.md).
|
||||
|
|
@ -206,8 +206,8 @@ calibration you hand the driver the thread link + login to review the real build
|
|||
[Push to a lang-tracer suite](#push-to-a-lang-tracer-suite)); the suite is the
|
||||
case's home, not the repo. Leave the `data/workflows/*.json` file uncommitted
|
||||
(or delete it once it's in the suite). Committing new case JSONs into the repo
|
||||
is no longer the approach. (Exception: seeded cases can't be pushed — a
|
||||
`conversationSeed`/`priorConversation` case stays committed JSON, a `seedThread` case is
|
||||
is no longer the approach. (Exception: seeded cases can't be pushed — an
|
||||
`inline`-seeded case stays committed JSON, a `replay` case is
|
||||
a local throwaway; see [`case-shapes.md`](case-shapes.md).) For a sourced case,
|
||||
finish by **linking it to its source thread/finding** over the MCP — see
|
||||
[Link the pushed case to its source](#link-the-pushed-case-to-its-source-provenance-step--always-do-this).
|
||||
|
|
@ -667,13 +667,12 @@ npx dotenvx run -f .env.eval -- pnpm eval:langtracer-push --suite baseline --cha
|
|||
lang-tracer #48), so scenario edits re-push like any other field. A lang-tracer
|
||||
deployment predating that change silently ignores the key; if a pushed scenario
|
||||
edit doesn't land, update the scenario in the lang-tracer UI.
|
||||
- **Seeded cases can't be pushed:** the case-write API rejects every seeding mode
|
||||
(`seedThread` / `conversationSeed` / `priorConversation`), so the push lists them under
|
||||
`skipped:` and they never reach the suite. A `seedThread` case shouldn't be
|
||||
committed either — it dies when its trace is pruned or deleted — so derive a
|
||||
durable synthetic case as the artifact instead. A `conversationSeed`/`priorConversation`
|
||||
case isn't transient and has no suite home, so it's the one exception to
|
||||
"don't commit the JSON" — it lives as a committed artifact.
|
||||
- **Seeded cases can't be pushed:** the case-write API has no `seed` field, so the
|
||||
push lists any seeded case under `skipped:` and it never reaches the suite. A
|
||||
`replay` case shouldn't be committed either — it dies when its trace is pruned
|
||||
or deleted — so derive a durable synthetic case as the artifact instead. An
|
||||
`inline`-seeded case isn't transient and has no suite home, so it's the one
|
||||
exception to "don't commit the JSON" — it lives as a committed artifact.
|
||||
|
||||
### Link the pushed case to its source (provenance step — always do this)
|
||||
|
||||
|
|
|
|||
|
|
@ -16,10 +16,10 @@ The schema
|
|||
([`harness/schema.ts`](../../../packages/@n8n/instance-ai/evaluations/harness/schema.ts))
|
||||
enforces the rules you must respect:
|
||||
|
||||
- `conversationSeed`, `priorConversation`, `seedThread` are **mutually exclusive** — pick
|
||||
one seeding mode.
|
||||
- A case needs a `conversation` **or** a `seedThread` (which supplies the live
|
||||
turn from the trace).
|
||||
- Seeding lives in **one** slot, `seed`, whose `mode` is `inline` or `replay` — so
|
||||
the modes are mutually exclusive by construction.
|
||||
- A case needs a `conversation` **or** a `seed` with `mode: "replay"` (which
|
||||
supplies the live turn from the trace).
|
||||
- A case needs **at least one** `executionScenario`, `processExpectation`, or
|
||||
`outcomeExpectation`.
|
||||
- `buildExpectations` is a **forbidden key** (fails loudly) — it was split into
|
||||
|
|
@ -354,13 +354,14 @@ Pick the lightest mode that fits:
|
|||
|
||||
| Situation | Mode | Pairs with |
|
||||
|---|---|---|
|
||||
| Reproduce a real conversation (common case) | `seedThread` — fetch + reconstruct its LangSmith trace at run time; nothing committed | supplies its own live turn (omit `conversation`) |
|
||||
| Prelude is just "what was discussed" (no tool calls, no workflows) | `priorConversation` — prose turns, authored inline | a normal `conversation` for the live turn |
|
||||
| Prior work already exists (a workflow to repair) | `conversationSeed` — prior messages + the workflows they reference, in the case body | a normal `conversation` for the live turn |
|
||||
| Reproduce a real conversation (common case) | `seed.mode: "replay"` — fetch + reconstruct its LangSmith trace at run time; nothing committed | supplies its own live turn (omit `conversation`) |
|
||||
| Prior work already exists (a workflow to repair) | `seed.mode: "inline"` — prior messages + the workflows they reference, in the case body | a normal `conversation` for the live turn |
|
||||
| Prelude is just "what was discussed" (no tool calls, no workflows) | `seed.mode: "inline"` with `{role, text}` shorthand messages | a normal `conversation` for the live turn |
|
||||
| Shallow 2–3 turn prelude where the agent's live replies matter | none — a plain multi-turn `conversation` re-drives it live | — |
|
||||
|
||||
All three modes are implemented and wired (`harness/conversation-seed.ts` +
|
||||
`harness/langsmith-seed.ts`, threaded through the runner).
|
||||
Both modes are implemented and wired (`harness/conversation-seed.ts` +
|
||||
`harness/langsmith-seed.ts`, threaded through the runner). The literals match
|
||||
lang-tracer's `metadata.seed` verbatim, so nothing translates between the repos.
|
||||
|
||||
### What the seed does — and does not — exercise
|
||||
|
||||
|
|
@ -379,11 +380,11 @@ everything built or said after the seed**: what the agent does with the restored
|
|||
state, how it responds to the triggering message, what the workflow looks like
|
||||
after the correction. Asserting on the seeded prelude itself proves nothing.
|
||||
|
||||
### Which mode — and when to avoid seedThread
|
||||
### Which mode — and when to avoid `replay`
|
||||
|
||||
Default to a **synthetic** case (an authored prompt + director script, or a
|
||||
`priorConversation` / `conversationSeed` prelude): it's durable, carries no real user
|
||||
data, never expires, and you control the setup exactly. Reach for **`seedThread`** only when
|
||||
Default to a **synthetic** case (an authored prompt + director script, or an
|
||||
`inline` seed prelude): it's durable, carries no real user
|
||||
data, never expires, and you control the setup exactly. Reach for **`replay`** only when
|
||||
the misbehaviour genuinely needs real prior context that's impractical to
|
||||
synthesize — a long accumulated thread, specific built workflows/tables — **and**
|
||||
the issue is in a *later* turn. (A turn-0 issue can't be isolated by seeding: it
|
||||
|
|
@ -401,10 +402,10 @@ costs keep it a last resort, not a default:
|
|||
|
||||
If a plain prompt + director script can reproduce the situation, prefer that.
|
||||
|
||||
### `seedThread` — reproduce a real conversation
|
||||
### `mode: "replay"` — reproduce a real conversation
|
||||
|
||||
```json
|
||||
"seedThread": { "threadId": "<thread-id>", "project": "instance-ai" }
|
||||
"seed": { "mode": "replay", "threadId": "<thread-id>", "project": "instance-ai" }
|
||||
```
|
||||
|
||||
The case carries only the opaque **thread id** — no conversation content lands
|
||||
|
|
@ -427,7 +428,7 @@ which user turn goes live.
|
|||
proxy reference; subsequent `user` turns become follow-ups). Omit it to replay
|
||||
just the live turn and stop.
|
||||
- **Transient — don't commit it, keep out of CI.** LangSmith base-tier traces
|
||||
retain ~14 days and threads can be deleted or pruned, so a committed `seedThread`
|
||||
retain ~14 days and threads can be deleted or pruned, so a committed `replay`
|
||||
case goes dead the moment its trace disappears. Treat it as a **local, throwaway
|
||||
reproduction**: don't commit it — run it to confirm the failure, then encode a
|
||||
durable synthetic case as the artifact. If you do keep one for a local run, tag
|
||||
|
|
@ -442,35 +443,22 @@ which user turn goes live.
|
|||
seeds with **no workflow to inspect**. Reproduce the target workflow yourself
|
||||
(a synthetic case whose `executionScenarios` precondition builds the stand-in),
|
||||
or grade the live turn with `processExpectations` only.
|
||||
- **Can't be pushed to a lang-tracer suite either.** The case-write API rejects
|
||||
every seeding mode (`seedThread` / `conversationSeed` / `priorConversation`), so
|
||||
`eval:langtracer-push` silently lists them under `skipped:`. Combined with the
|
||||
don't-commit rule above, a `seedThread` case has **no durable home by design** —
|
||||
the durable artifact is always the synthetic case you derive from it. (`conversationSeed`
|
||||
and `priorConversation` carry no thread dependency and can't be pushed either, so
|
||||
— unlike a normal case — they're the one exception to the skill's "push, don't
|
||||
commit the JSON" rule: they live as committed artifacts.)
|
||||
- **Can't be pushed to a lang-tracer suite either.** The case-write API has no
|
||||
`seed` field, so `eval:langtracer-push` silently lists any seeded case under
|
||||
`skipped:`. Combined with the don't-commit rule above, a `replay` case has **no
|
||||
durable home by design** — the durable artifact is always the synthetic case you
|
||||
derive from it. (An `inline` seed carries no thread dependency and can't be
|
||||
pushed either, so — unlike a normal case — it's the one exception to the skill's
|
||||
"push, don't commit the JSON" rule: it lives as a committed artifact.)
|
||||
|
||||
### `priorConversation` — prose prelude
|
||||
|
||||
```json
|
||||
"priorConversation": [
|
||||
{ "role": "user", "text": "We agreed: digests go to #growth, daily at 9am." },
|
||||
{ "role": "assistant", "text": "Noted — #growth, daily at 9am." }
|
||||
]
|
||||
```
|
||||
|
||||
Plain text only — no tool calls, no restored workflows. Paired with a normal
|
||||
`conversation` for the live turn.
|
||||
|
||||
### `conversationSeed` — durable synthetic fixture
|
||||
### `mode: "inline"` — durable synthetic fixture
|
||||
|
||||
For a **synthetic, sanitised** seed pinned in git (never a real user's
|
||||
conversation): author the prior messages, plus the workflows they reference, in
|
||||
the case body (schema in
|
||||
[`harness/conversation-seed.ts`](../../../packages/@n8n/instance-ai/evaluations/harness/conversation-seed.ts)
|
||||
— `messages` + optional `workflows` + `dataTables`). Real conversations belong in
|
||||
`seedThread`, which keeps their content out of the repo.
|
||||
`replay`, which keeps their content out of the repo.
|
||||
|
||||
Two constraints that bite: a workflow `id` must be ≥8 characters (the id remap
|
||||
refuses shorter ones), and a seeded `build-workflow` tool call's
|
||||
|
|
@ -480,3 +468,29 @@ separates them and the agent can't find the workflow it should act on.
|
|||
The seed sits in the case body, not a sibling file, so it travels with the case
|
||||
whether it comes off disk, out of a LangTracer suite, or from a dispatched case
|
||||
body.
|
||||
|
||||
#### `{role, text}` shorthand — a prose prelude
|
||||
|
||||
When the prelude is just "what was discussed" — no tool calls, no workflows —
|
||||
write a message as `{role, text}` and the schema expands it to a full envelope:
|
||||
|
||||
```json
|
||||
"seed": {
|
||||
"mode": "inline",
|
||||
"messages": [
|
||||
{ "role": "user", "text": "We agreed: digests go to #growth, daily at 9am." },
|
||||
{ "role": "assistant", "text": "Noted — #growth, daily at 9am." }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`text` also takes an array of lines (newline-joined), like a `conversation` turn.
|
||||
The expansion stamps `createdAt` itself — ascending, in the past — so a shorthand
|
||||
message can't order *after* the live turn. Shorthand and full envelopes mix freely
|
||||
in one array; a full envelope keeps its authored `createdAt` — **unless any message
|
||||
in the array is in the FUTURE**, in which case the whole sequence is restamped onto
|
||||
ascending pre-live slots (a future stamp would sort a seeded turn after the live
|
||||
turn, and moving only that one entry would reorder it against the array the
|
||||
transcript is graded from). A near-miss (say
|
||||
`text: 123`) is deliberately **not** expanded — it fails at load instead of
|
||||
becoming a message the transcript builder would silently drop.
|
||||
|
|
|
|||
|
|
@ -188,7 +188,7 @@ server-side filter. The two that matter for CI:
|
|||
capability diversity.
|
||||
|
||||
Other values group cases logically (e.g. `behaviour` for conversation-behaviour
|
||||
cases, `seeded` for transient `seedThread` cases kept out of CI). For a new
|
||||
cases, `seeded` for transient `replay` cases kept out of CI). For a new
|
||||
local case, put the value in its `datasets` array before pushing; for a case
|
||||
already in LangTracer, edit `datasets` there — `eval:langtracer-push`
|
||||
deliberately does not re-sync tier-only edits to an existing case. **Only
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
The strongest cases encode a **real** failure, not an invented premise. Two
|
||||
connections help you find one and confirm it — and neither is usually the
|
||||
durable artifact. You author a synthetic case from what you learn (reach for
|
||||
`seedThread` only per [`case-shapes.md`](case-shapes.md)).
|
||||
`seed.mode: "replay"` only per [`case-shapes.md`](case-shapes.md)).
|
||||
|
||||
- **LangTracer — discover.** It ingests real Instance AI conversations and
|
||||
clusters them into **capability-gap themes** ("what fails, at scale"), and
|
||||
|
|
@ -56,9 +56,9 @@ connected.)
|
|||
5. **Push it to a curated suite** (don't commit the JSON) with
|
||||
`eval:langtracer-push` — see
|
||||
[Push to a lang-tracer suite](SKILL.md#push-to-a-lang-tracer-suite). Exception:
|
||||
seeded cases (`seedThread` / `conversationSeed` / `priorConversation`) can't be pushed —
|
||||
the case-write API rejects every seeding mode, so the push lists them under
|
||||
`skipped:`. And a `seedThread` case shouldn't be committed either — it dies when
|
||||
seeded cases (any `seed` mode) can't be pushed — the case-write API has no
|
||||
`seed` field, so the push lists them under
|
||||
`skipped:`. And a `replay` case shouldn't be committed either — it dies when
|
||||
its trace is pruned or deleted — so it has no durable home; that's exactly why
|
||||
step 4 turns the confirmed failure into a durable synthetic case.
|
||||
|
||||
|
|
|
|||
|
|
@ -673,7 +673,7 @@ To record an isolated cohort without touching the shared dataset or baseline —
|
|||
|
||||
## Adding test cases
|
||||
|
||||
The corpus lives in **LangTracer** — suite `baseline` is what CI runs. Author a case as a local JSON file in `evaluations/data/workflows/` (disk mode picks it up, no registration step), calibrate it against a real build, then push it to the suite with `pnpm eval:langtracer-push --suite baseline <slug>` and delete the local file rather than committing it. Seeded cases (`priorConversation`/`conversationSeed`) are the exception — the case-write API can't represent them yet, so they stay as committed JSON. Every case is validated against `harness/schema.ts`.
|
||||
The corpus lives in **LangTracer** — suite `baseline` is what CI runs. Author a case as a local JSON file in `evaluations/data/workflows/` (disk mode picks it up, no registration step), calibrate it against a real build, then push it to the suite with `pnpm eval:langtracer-push --suite baseline <slug>` and delete the local file rather than committing it. Seeded cases (the `seed` slot) are the exception — the case-write API can't represent them yet, so they stay as committed JSON. Every case is validated against `harness/schema.ts`.
|
||||
|
||||
> The essentials are below. For the full authoring guide — picking a case archetype, sizing assertions so a wrong build fails, multi-turn director scripts, seeding vs synthetic, and calibrating against a real build — follow the [`create-instance-ai-eval` skill](../../../../.agents/skills/create-instance-ai-eval/SKILL.md) (with [`case-shapes.md`](../../../../.agents/skills/create-instance-ai-eval/case-shapes.md) and [`running-evals.md`](../../../../.agents/skills/create-instance-ai-eval/running-evals.md)).
|
||||
|
||||
|
|
@ -755,21 +755,27 @@ Each type needs a data template in `credentials/seeder.ts`; declaring an unknown
|
|||
|
||||
A seeded case starts **mid-conversation**: prior history is restored into the build thread before the live turn, so the eval drives only the turn under test. Use it to replicate a real misbehaviour — restore the conversation up to the moment it went wrong, re-drive that turn, and assert what should happen instead.
|
||||
|
||||
Pick the lightest path that fits:
|
||||
The seed lives in **one slot** — `seed` — whose `mode` says where it comes from, so
|
||||
a case cannot declare two seeding modes by accident. Pick the lightest path that fits:
|
||||
|
||||
| Situation | Path |
|
||||
|---|---|
|
||||
| Reproduce a real conversation (the common case) | `seedThread` — fetch + reconstruct its LangSmith trace at run time; nothing committed |
|
||||
| Prelude is just "what was discussed" (no tool calls, no workflows) | `priorConversation` — prose turns, authored inline |
|
||||
| Prior work already exists (a workflow to repair) | `conversationSeed` — prior messages + the workflows they reference, in the case body |
|
||||
| Reproduce a real conversation (the common case) | `seed.mode: "replay"` — fetch + reconstruct its LangSmith trace at run time; nothing committed |
|
||||
| Prior work already exists (a workflow to repair) | `seed.mode: "inline"` — prior messages + the workflows they reference, in the case body |
|
||||
| Prelude is just "what was discussed" (no tool calls, no workflows) | `seed.mode: "inline"` with `{role, text}` shorthand messages |
|
||||
| Shallow 2–3 turn prelude where the agent's live replies matter | Neither — a plain multi-turn `conversation` script re-drives it live |
|
||||
|
||||
#### `seedThread` — reproduce a real conversation (no repo content)
|
||||
The literals match lang-tracer's `metadata.seed` verbatim, so nothing translates
|
||||
between the two repos. They read asymmetrically — `inline` names where the seed
|
||||
lives, `replay` names what the harness does with it — which is a cost we took
|
||||
knowingly: renaming `replay` would break a lang-tracer API contract.
|
||||
|
||||
#### `mode: "replay"` — reproduce a real conversation (no repo content)
|
||||
|
||||
The case carries only a **thread id**. At run time the harness pulls that thread's runs from LangSmith, reconstructs the message log (user/assistant text + resolved tool-call blocks, deduped across suspend/resume), and splits at the **last user message**: everything before it is restored as the seed, that last message is sent live. The seed workflow is compiled from the build/patch tool's captured SDK code **as of the seed boundary**, so it matches what the live turn first saw.
|
||||
|
||||
```json
|
||||
"seedThread": { "threadId": "<thread-id>", "project": "instance-ai" }
|
||||
"seed": { "mode": "replay", "threadId": "<thread-id>", "project": "instance-ai" }
|
||||
```
|
||||
|
||||
No `conversation` field needed — the live turn comes from the trace. `project` is optional (defaults to `instance-ai`). No conversation content lands in the repo — only the opaque thread id.
|
||||
|
|
@ -779,7 +785,7 @@ No `conversation` field needed — the live turn comes from the trace. `project`
|
|||
**Continuing past the live turn.** Add a `conversation` to keep driving *after* the trace's last message is replayed — the effective conversation becomes `[<trace live turn>, ...conversation]`, so the live turn is sent for real and your authored turns become proxy-driven follow-ups (multi-turn). Use it to push a reproduced conversation further (e.g. "now also add error handling", or pressure-test the next decision):
|
||||
|
||||
```json
|
||||
"seedThread": { "threadId": "…", "project": "instance-ai" },
|
||||
"seed": { "mode": "replay", "threadId": "…", "project": "instance-ai" },
|
||||
"conversation": [
|
||||
{ "role": "assistant", "text": "Updated the schedule to every 30 minutes." },
|
||||
{ "role": "user", "text": "Now also send a copy to #ops." }
|
||||
|
|
@ -790,38 +796,55 @@ No `conversation` field needed — the live turn comes from the trace. `project`
|
|||
|
||||
**Cross-workspace, zero config (e.g. prod traces, staging eval).** A source thread can live in a different LangSmith **workspace** than the eval writes to. You don't declare which, and there are no extra env vars — the harness enumerates the workspaces your `LANGSMITH_API_KEY` can access and finds the one holding the thread (the workspace is selected per request via the `x-tenant-id` header; a personal access token typically spans staging/prod/feature). Reads use the ambient key; the eval still writes its own traces/datasets to its own workspace, so **nothing is ever written to the source workspace**. The resolved workspace is logged (`[Prod/instance-ai]`).
|
||||
|
||||
`seedThread.project` overrides the source project name (default `instance-ai`); the same name is searched in every workspace, so if prod and staging share it you need nothing. Reconstruction recreates the source conversation on the eval instance (and in its model calls, traces, and local report artifacts): the most sensitive content is scrubbed first — see *What's restored* above (no data-table rows, values redacted from the restored history, credentials stripped) — but scrubbing isn't guaranteed exhaustive, so treat a reproduced conversation as if it may carry user data and follow your team's data-handling policy for real threads.
|
||||
`seed.project` overrides the source project name (default `instance-ai`); the same name is searched in every workspace, so if prod and staging share it you need nothing. Reconstruction recreates the source conversation on the eval instance (and in its model calls, traces, and local report artifacts): the most sensitive content is scrubbed first — see *What's restored* above (no data-table rows, values redacted from the restored history, credentials stripped) — but scrubbing isn't guaranteed exhaustive, so treat a reproduced conversation as if it may carry user data and follow your team's data-handling policy for real threads.
|
||||
|
||||
> **Transient.** LangSmith base-tier traces retain ~14 days, so a `seedThread` case is runnable only while its trace lives. Keep these out of CI datasets (tag them `["seeded"]`, not `full`/`pr`) until durable seed snapshots land; the resolver fails loudly when a trace has aged out. Durable snapshotting (e.g. materialising the reconstructed seed into a private LangSmith dataset on first resolve) is a planned follow-up.
|
||||
> **Transient.** LangSmith base-tier traces retain ~14 days, so a `replay` case is runnable only while its trace lives. Keep these out of CI datasets (tag them `["seeded"]`, not `full`/`pr`) until durable seed snapshots land; the resolver fails loudly when a trace has aged out. Durable snapshotting (e.g. materialising the reconstructed seed into a private LangSmith dataset on first resolve) is a planned follow-up.
|
||||
|
||||
To find the thread id, open the conversation's trace in LangSmith (or read it from the instance the conversation happened on) and copy its `thread_id`.
|
||||
|
||||
#### `priorConversation` — prose prelude
|
||||
#### `mode: "inline"` — durable synthetic fixture
|
||||
|
||||
For a **synthetic, sanitized** seed you want pinned in git (never a real user's conversation): author the prior messages, plus the workflows they reference, in the case body. Real conversations belong in `replay`, which keeps their content out of the repo entirely. Paired with a normal `conversation` for the live turn.
|
||||
|
||||
```json
|
||||
"priorConversation": [
|
||||
{ "role": "user", "text": "We agreed: digests go to #growth, daily at 9am." },
|
||||
{ "role": "assistant", "text": "Noted — #growth, daily at 9am." }
|
||||
]
|
||||
```
|
||||
|
||||
Paired with a normal `conversation` for the live turn. Plain text only — no tool calls, no restored workflows.
|
||||
|
||||
#### `conversationSeed` — durable synthetic fixture
|
||||
|
||||
For a **synthetic, sanitized** seed you want pinned in git (never a real user's conversation): author the prior messages, plus the workflows they reference, in the case body. Real conversations belong in `seedThread`, which keeps their content out of the repo entirely. Paired with a normal `conversation` for the live turn.
|
||||
|
||||
```json
|
||||
"conversationSeed": {
|
||||
"seed": {
|
||||
"mode": "inline",
|
||||
"messages": [ … ],
|
||||
"workflows": [ { "id": "wKk3RmT9xQ2bVn7L", "name": "Batch loop", "nodes": [], "connections": {} } ]
|
||||
}
|
||||
```
|
||||
|
||||
Schema in `harness/conversation-seed.ts` — `messages` plus optional `workflows` and `dataTables`. Two constraints worth knowing: a workflow `id` must be ≥8 characters (`remapSeedWorkflowIds` refuses to rewrite shorter ids safely), and a seeded `build-workflow` tool call's `output.workflowId` must match the seeded workflow's `id`, or the remap separates them and the agent can't find the workflow it's meant to act on.
|
||||
Schema in `harness/conversation-seed.ts` — `messages` plus optional `workflows` and `dataTables` (both default to `[]`, so a messages-only seed is valid). Two constraints worth knowing: a workflow `id` must be ≥8 characters (`remapSeedWorkflowIds` refuses to rewrite shorter ids safely), and a seeded `build-workflow` tool call's `output.workflowId` must match the seeded workflow's `id`, or the remap separates them and the agent can't find the workflow it's meant to act on.
|
||||
|
||||
**Each message must carry the envelope** — `id`, `role` (`user` or `assistant`), `type` (`llm`, `custom`, …), `createdAt` (a parseable timestamp; ordering before the live turn depends on it), and `content` as an array of blocks each with a `type`. Only the envelope is validated: **unknown block types are accepted**, because block shapes belong to the agent's message store rather than to the harness, and unknown keys are preserved rather than stripped. A `type: 'custom'` message is the one exception — it's stored but never rendered, so it may omit `role` and carry any `content` shape. The envelope is checked because a malformed message would otherwise be stored verbatim *and* skipped by `transcriptPrefixFromSeed`, leaving the case graded against a transcript that doesn't match what the agent saw.
|
||||
|
||||
##### `{role, text}` shorthand — a prose prelude
|
||||
|
||||
When the prelude is just "what was discussed" — no tool calls, no workflows — write a
|
||||
message as `{role, text}` and the schema expands it into a full envelope for you:
|
||||
|
||||
```json
|
||||
"seed": {
|
||||
"mode": "inline",
|
||||
"messages": [
|
||||
{ "role": "user", "text": "We agreed: digests go to #growth, daily at 9am." },
|
||||
{ "role": "assistant", "text": "Noted — #growth, daily at 9am." }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`text` also takes an array of lines (newline-joined), same as a `conversation` turn.
|
||||
The expansion stamps `createdAt` itself — ascending and in the past — so a shorthand
|
||||
message can't accidentally order *after* the live turn. Shorthand and full envelopes
|
||||
can be mixed in one `messages` array; a full envelope keeps its authored `createdAt`
|
||||
— **unless any message in the array is stamped in the FUTURE**, in which case the
|
||||
whole sequence is restamped onto the same ascending pre-live slots. A future stamp
|
||||
would sort a seeded turn after the live turn, and clamping only the offending entry
|
||||
would reorder it relative to the array the transcript is graded from.
|
||||
A near-miss (say `text: 123`) is deliberately **not** expanded: it falls through to
|
||||
the envelope rules above and fails at load, rather than becoming a message the
|
||||
transcript builder would silently drop.
|
||||
|
||||
The seed lives **in the case body** rather than in a sibling file, so it travels with the case whatever the source — a JSON on disk, a suite pulled with `--source langtracer`, or a case body handed to a dispatcher. (There used to be a `seedFile` path pointing at a sibling JSON. Only the disk loader could resolve it, so a case delivered any other way lost its seed; the key is gone and a case still carrying it fails at load.)
|
||||
|
||||
#### How restore works (all paths)
|
||||
|
|
@ -831,7 +854,8 @@ At build time the seed is restored right after the credential pin: seeded workfl
|
|||
Rules of thumb:
|
||||
|
||||
- **A seeded case is only worth shipping with `buildExpectations` that detect the misbehaviour recurring** — without them it passes vacuously. Sanity-check by running the case once with the seed removed: it should fail.
|
||||
- `seedThread`, `priorConversation` and `conversationSeed` are mutually exclusive; all order strictly before the live turn. `seedThread` provides its own live turn (omit `conversation`); the other two pair with `conversation`.
|
||||
- **One slot means the modes are mutually exclusive by construction** — there is no way to declare two. Both order strictly before the live turn: `replay` provides its own live turn (omit `conversation`), `inline` pairs with one.
|
||||
- **A suite-sourced case carrying a pre-union key (`conversationSeed`, `priorConversation`, `seedThread`, `seedFile`) fails the suite pull by name.** The normalizer whitelists a hosted case down to the schema's keys, so without that guard the key would be stripped and the case would run *unseeded* — passing or failing for reasons that have nothing to do with what it tests.
|
||||
|
||||
## Failure categories
|
||||
|
||||
|
|
|
|||
|
|
@ -81,7 +81,7 @@ const baseConfig = {
|
|||
logger: silentLogger,
|
||||
};
|
||||
|
||||
describe('buildWorkflow with a conversationSeed', () => {
|
||||
describe('buildWorkflow with an inline seed', () => {
|
||||
it('restores the inline seed before the live turn', async () => {
|
||||
const restoreThread = vi
|
||||
.fn()
|
||||
|
|
@ -90,7 +90,7 @@ describe('buildWorkflow with a conversationSeed', () => {
|
|||
const build = await buildWorkflow({
|
||||
client: makeClient(restoreThread),
|
||||
...baseConfig,
|
||||
conversationSeed: inlineSeed(),
|
||||
seed: { mode: 'inline' as const, ...inlineSeed() },
|
||||
});
|
||||
|
||||
expect(build.success).toBe(true);
|
||||
|
|
@ -115,7 +115,7 @@ describe('buildWorkflow with a conversationSeed', () => {
|
|||
const build = await buildWorkflow({
|
||||
client: makeClient(restoreThread),
|
||||
...baseConfig,
|
||||
conversationSeed: inlineSeed(),
|
||||
seed: { mode: 'inline' as const, ...inlineSeed() },
|
||||
});
|
||||
|
||||
expect(build.success).toBe(false);
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
import {
|
||||
ConversationSeedSchema,
|
||||
expandSeedMessageShorthand,
|
||||
remapSeedWorkflowIds,
|
||||
seedFromProse,
|
||||
transcriptPrefixFromSeed,
|
||||
type ConversationSeed,
|
||||
} from '../harness/conversation-seed';
|
||||
|
|
@ -137,12 +137,14 @@ describe('ConversationSeedSchema message envelope', () => {
|
|||
});
|
||||
});
|
||||
|
||||
it('accepts what seedFromProse produces', () => {
|
||||
const seed = seedFromProse([
|
||||
it('accepts what the shorthand expansion produces', () => {
|
||||
// The case schema expands shorthand BEFORE this schema validates it, so an
|
||||
// expansion that stopped satisfying the envelope would fail every seed.
|
||||
const messages = expandSeedMessageShorthand([
|
||||
{ role: 'user', text: 'hi' },
|
||||
{ role: 'assistant', text: 'hello' },
|
||||
]);
|
||||
expect(ConversationSeedSchema.safeParse(seed).success).toBe(true);
|
||||
expect(ConversationSeedSchema.safeParse({ messages }).success).toBe(true);
|
||||
});
|
||||
|
||||
it('still requires at least one message', () => {
|
||||
|
|
@ -150,16 +152,15 @@ describe('ConversationSeedSchema message envelope', () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe('seedFromProse', () => {
|
||||
it('converts turns to llm text messages with ascending past timestamps', () => {
|
||||
const seed = seedFromProse([
|
||||
describe('expandSeedMessageShorthand', () => {
|
||||
it('converts shorthand turns to llm text messages with ascending past timestamps', () => {
|
||||
const messages = expandSeedMessageShorthand([
|
||||
{ role: 'user', text: 'Digest to #cosmic-otter-alerts please' },
|
||||
{ role: 'assistant', text: 'Done — daily at 9am.' },
|
||||
]);
|
||||
|
||||
expect(seed.workflows).toEqual([]);
|
||||
expect(seed.messages).toHaveLength(2);
|
||||
const [first, second] = seed.messages;
|
||||
expect(messages).toHaveLength(2);
|
||||
const [first, second] = messages as Array<Record<string, unknown>>;
|
||||
expect(first).toMatchObject({
|
||||
type: 'llm',
|
||||
role: 'user',
|
||||
|
|
@ -172,6 +173,29 @@ describe('seedFromProse', () => {
|
|||
expect(t1).toBeGreaterThan(t0);
|
||||
expect(t1).toBeLessThan(Date.now());
|
||||
});
|
||||
|
||||
it('passes a full envelope through untouched', () => {
|
||||
const envelope = {
|
||||
id: 'm1',
|
||||
type: 'llm',
|
||||
role: 'assistant' as const,
|
||||
createdAt: '2026-06-29T09:00:00.000Z',
|
||||
content: [{ type: 'text', text: 'authored' }],
|
||||
};
|
||||
expect(expandSeedMessageShorthand([envelope])[0]).toBe(envelope);
|
||||
});
|
||||
|
||||
it('leaves a near-miss shorthand alone so the envelope schema reports it', () => {
|
||||
// Expanding these would produce a message the transcript builder silently
|
||||
// drops; passing them through means the envelope schema rejects them loudly.
|
||||
const nearMisses = [
|
||||
{ role: 'user', text: 42 },
|
||||
{ role: 'system', text: 'hi' },
|
||||
{ role: 'user', text: 'hi', extra: true },
|
||||
{ role: 'user' },
|
||||
];
|
||||
expect(expandSeedMessageShorthand(nearMisses)).toEqual(nearMisses);
|
||||
});
|
||||
});
|
||||
|
||||
describe('remapSeedWorkflowIds', () => {
|
||||
|
|
@ -189,7 +213,19 @@ describe('remapSeedWorkflowIds', () => {
|
|||
});
|
||||
|
||||
it('returns the seed untouched when there are no workflows', () => {
|
||||
const seed = seedFromProse([{ role: 'user', text: 'hi' }]);
|
||||
const seed: ConversationSeed = {
|
||||
messages: [
|
||||
{
|
||||
id: 'm1',
|
||||
type: 'llm',
|
||||
role: 'user',
|
||||
createdAt: '2026-06-29T09:00:00.000Z',
|
||||
content: [{ type: 'text', text: 'hi' }],
|
||||
},
|
||||
],
|
||||
workflows: [],
|
||||
dataTables: [],
|
||||
};
|
||||
expect(remapSeedWorkflowIds(seed)).toBe(seed);
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -65,7 +65,7 @@ describe('conversationUserTurnsAsText', () => {
|
|||
expect(conversationUserTurnsAsText([])).toBe('');
|
||||
});
|
||||
|
||||
it('returns empty string when conversation is undefined (seedThread-only case)', () => {
|
||||
it('returns empty string when conversation is undefined (replay-seeded case)', () => {
|
||||
expect(conversationUserTurnsAsText(undefined)).toBe('');
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ vi.mock('fs', () => ({
|
|||
import { readdirSync, readFileSync } from 'fs';
|
||||
|
||||
import { loadWorkflowTestCasesWithFiles } from '../data/workflows';
|
||||
import { EvalTestCaseSchema, conversationTurnTextSchema } from '../harness/schema';
|
||||
import { EvalTestCaseSchema, conversationTurnTextSchema, type CaseSeed } from '../harness/schema';
|
||||
|
||||
const mockedReaddir = vi.mocked(readdirSync);
|
||||
const mockedReadFile = vi.mocked(readFileSync);
|
||||
|
|
@ -28,6 +28,21 @@ const validFixture = () => ({
|
|||
],
|
||||
});
|
||||
|
||||
/** Narrow a parsed case's `seed` to one arm so its fields are readable. Throwing
|
||||
* (rather than casting) keeps a wrong-arm parse a test failure, not a silent
|
||||
* `undefined` compared against `undefined`. */
|
||||
function inlineSeedOf(parsed: { seed?: CaseSeed }) {
|
||||
if (parsed.seed?.mode !== 'inline')
|
||||
throw new Error(`expected an inline seed, got ${String(parsed.seed?.mode)}`);
|
||||
return parsed.seed;
|
||||
}
|
||||
|
||||
function replaySeedOf(parsed: { seed?: CaseSeed }) {
|
||||
if (parsed.seed?.mode !== 'replay')
|
||||
throw new Error(`expected a replay seed, got ${String(parsed.seed?.mode)}`);
|
||||
return parsed.seed;
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
mockedReaddir.mockReturnValue(['demo.json'] as unknown as ReturnType<typeof readdirSync>);
|
||||
|
|
@ -82,24 +97,23 @@ describe('EvalTestCaseSchema', () => {
|
|||
expect(() => EvalTestCaseSchema.parse({ ...validFixture(), complexity: 'gigantic' })).toThrow();
|
||||
});
|
||||
|
||||
it('accepts a prose priorConversation prelude', () => {
|
||||
// `.strict()` rejects every key the schema no longer declares, so a disk case
|
||||
// carrying a pre-union seed key fails loudly. (A SUITE-sourced case is a
|
||||
// different path — the normalizer strips unknown keys, so provider.ts guards
|
||||
// the raw body instead; see langtracer-provider.test.ts.)
|
||||
it.each(['seedFile', 'conversationSeed', 'priorConversation', 'seedThread'])(
|
||||
'rejects a disk case still carrying the legacy %s key',
|
||||
(key) => {
|
||||
// `.strict()` names the offending key: "Unrecognized key(s) in object: 'seedFile'".
|
||||
expect(() => EvalTestCaseSchema.parse({ ...validFixture(), [key]: 'anything' })).toThrow(key);
|
||||
},
|
||||
);
|
||||
|
||||
it('accepts an inline seed and defaults its optional arrays', () => {
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
priorConversation: [{ role: 'user', text: 'We already agreed on #cosmic-otter-alerts' }],
|
||||
});
|
||||
expect(parsed.priorConversation).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('rejects a case still pointing at a seedFile', () => {
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({ ...validFixture(), seedFile: 'seeds/some-thread.seed.json' }),
|
||||
).toThrow(/seedFile/);
|
||||
});
|
||||
|
||||
it('accepts an inline conversationSeed and defaults its optional arrays', () => {
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
conversationSeed: {
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [
|
||||
{
|
||||
id: 'm1',
|
||||
|
|
@ -111,109 +125,282 @@ describe('EvalTestCaseSchema', () => {
|
|||
],
|
||||
},
|
||||
});
|
||||
expect(parsed.conversationSeed?.messages).toHaveLength(1);
|
||||
expect(parsed.conversationSeed?.workflows).toEqual([]);
|
||||
expect(parsed.conversationSeed?.dataTables).toEqual([]);
|
||||
const seed = inlineSeedOf(parsed);
|
||||
expect(seed.messages).toHaveLength(1);
|
||||
expect(seed.workflows).toEqual([]);
|
||||
expect(seed.dataTables).toEqual([]);
|
||||
});
|
||||
|
||||
it('rejects an inline conversationSeed with no messages', () => {
|
||||
it('rejects an inline seed with no messages', () => {
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({ ...validFixture(), conversationSeed: { messages: [] } }),
|
||||
EvalTestCaseSchema.parse({ ...validFixture(), seed: { mode: 'inline', messages: [] } }),
|
||||
).toThrow();
|
||||
});
|
||||
|
||||
it('rejects conversationSeed combined with another seeding mode', () => {
|
||||
it('rejects an unknown seed mode', () => {
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({ ...validFixture(), seed: { mode: 'prose', messages: [] } }),
|
||||
).toThrow(/mode/);
|
||||
});
|
||||
|
||||
// A future stamp would sort the seeded turn after the live turn, so the agent
|
||||
// sees its own history out of order and the judge grades a transcript that
|
||||
// never happened.
|
||||
it('pulls a future envelope createdAt back before the live turn', () => {
|
||||
const future = new Date(Date.now() + 60 * 60 * 1000).toISOString();
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [
|
||||
{
|
||||
id: 'm1',
|
||||
type: 'llm',
|
||||
role: 'user',
|
||||
createdAt: future,
|
||||
content: [{ type: 'text', text: 'build it' }],
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
const seed = inlineSeedOf(parsed);
|
||||
expect(Date.parse(String(seed.messages[0].createdAt))).toBeLessThan(Date.now());
|
||||
});
|
||||
|
||||
// A per-message clamp is not enough: with [future A, past B] only A moves, so
|
||||
// the DB orders B then A while `transcriptPrefixFromSeed` still grades array
|
||||
// order (A then B). The sequence has to stay coherent as a whole.
|
||||
it('restamps the whole sequence in array order when any timestamp is future', () => {
|
||||
const msg = (id: string, createdAt: string) => ({
|
||||
id,
|
||||
type: 'llm',
|
||||
role: 'user' as const,
|
||||
createdAt,
|
||||
content: [{ type: 'text', text: id }],
|
||||
});
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [
|
||||
msg('a', new Date(Date.now() + 60 * 60 * 1000).toISOString()),
|
||||
msg('b', '2026-06-29T09:00:00.000Z'),
|
||||
],
|
||||
},
|
||||
});
|
||||
|
||||
const at = inlineSeedOf(parsed).messages.map((m) => Date.parse(String(m.createdAt)));
|
||||
// Ascending in ARRAY order, and entirely before the live turn.
|
||||
expect(at[0]).toBeLessThan(at[1]);
|
||||
expect(at[1]).toBeLessThan(Date.now());
|
||||
});
|
||||
|
||||
it('leaves an authored past createdAt exactly as written', () => {
|
||||
const authored = '2026-06-29T09:00:00.000Z';
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [
|
||||
{
|
||||
id: 'm1',
|
||||
type: 'llm',
|
||||
role: 'user',
|
||||
createdAt: authored,
|
||||
content: [{ type: 'text', text: 'build it' }],
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
expect(inlineSeedOf(parsed).messages[0].createdAt).toBe(authored);
|
||||
});
|
||||
|
||||
// Both arms are strict, so a seed mixing them fails instead of having the
|
||||
// wrong-arm field stripped — which would run the case unseeded and grade it
|
||||
// as a build from scratch.
|
||||
it('rejects a replay seed carrying inline fields', () => {
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
conversationSeed: {
|
||||
messages: [
|
||||
{
|
||||
id: 'm1',
|
||||
type: 'llm',
|
||||
role: 'user',
|
||||
createdAt: '2026-06-29T09:00:00.000Z',
|
||||
content: [{ type: 'text', text: 'build it' }],
|
||||
},
|
||||
],
|
||||
seed: {
|
||||
mode: 'replay',
|
||||
threadId: 'thread-1',
|
||||
messages: [{ role: 'user', text: 'build it' }],
|
||||
},
|
||||
priorConversation: [{ role: 'user', text: 'prelude' }],
|
||||
}),
|
||||
).toThrow(/mutually exclusive/);
|
||||
).toThrow(/messages/);
|
||||
});
|
||||
|
||||
it('accepts a seedThread case with no conversation (live turn from the trace)', () => {
|
||||
it('rejects an inline seed carrying replay fields', () => {
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [{ role: 'user', text: 'build it' }],
|
||||
threadId: 'thread-1',
|
||||
},
|
||||
}),
|
||||
).toThrow(/threadId/);
|
||||
});
|
||||
|
||||
it('expands a {role, text} shorthand message into a full envelope', () => {
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [
|
||||
{ role: 'user', text: 'We already agreed on #cosmic-otter-alerts' },
|
||||
{ role: 'assistant', text: ['Understood.', 'Posting there.'] },
|
||||
],
|
||||
},
|
||||
});
|
||||
const { messages } = inlineSeedOf(parsed);
|
||||
expect(messages).toHaveLength(2);
|
||||
expect(messages[0]).toMatchObject({
|
||||
role: 'user',
|
||||
type: 'llm',
|
||||
content: [{ type: 'text', text: 'We already agreed on #cosmic-otter-alerts' }],
|
||||
});
|
||||
expect(messages[0].id).toEqual(expect.any(String));
|
||||
// Array-form text is newline-joined, same as an authored conversation turn.
|
||||
expect(messages[1]).toMatchObject({
|
||||
content: [{ type: 'text', text: 'Understood.\nPosting there.' }],
|
||||
});
|
||||
});
|
||||
|
||||
it('stamps shorthand timestamps ascending and in the past (before the live turn)', () => {
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [
|
||||
{ role: 'user', text: 'first' },
|
||||
{ role: 'assistant', text: 'second' },
|
||||
{ role: 'user', text: 'third' },
|
||||
],
|
||||
},
|
||||
});
|
||||
const stamps = inlineSeedOf(parsed).messages.map((m) => Date.parse(String(m.createdAt)));
|
||||
expect(stamps).toEqual([...stamps].sort((a, b) => a - b));
|
||||
expect(new Set(stamps).size).toBe(3);
|
||||
expect(Math.max(...stamps)).toBeLessThan(Date.now());
|
||||
});
|
||||
|
||||
it('keeps an authored createdAt when shorthand and full envelopes are mixed', () => {
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [
|
||||
{ role: 'user', text: 'prose prelude' },
|
||||
{
|
||||
id: 'm2',
|
||||
type: 'llm',
|
||||
role: 'assistant',
|
||||
createdAt: '2026-06-29T09:00:00.000Z',
|
||||
content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'build-workflow' }],
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
const { messages } = inlineSeedOf(parsed);
|
||||
expect(messages[1].createdAt).toBe('2026-06-29T09:00:00.000Z');
|
||||
// The tool-call block's own keys survive (`.passthrough()`), so the seeded
|
||||
// history the agent reads isn't gutted.
|
||||
expect(messages[1].content).toEqual([
|
||||
{ type: 'tool-call', toolCallId: 'c1', toolName: 'build-workflow' },
|
||||
]);
|
||||
});
|
||||
|
||||
it('rejects a near-miss shorthand rather than expanding it into a droppable message', () => {
|
||||
// `text: 123` is not shorthand, so it falls through to the envelope schema —
|
||||
// which fails loudly. Expanding it would produce a text block the transcript
|
||||
// builder silently skips (the failure TRUST-357 exists to prevent).
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
seed: { mode: 'inline', messages: [{ role: 'user', text: 123 }] },
|
||||
}),
|
||||
).toThrow(/full envelope[\s\S]*shorthand/);
|
||||
});
|
||||
|
||||
it('accepts a replay seed with no conversation (live turn from the trace)', () => {
|
||||
const { conversation: _omit, ...rest } = validFixture();
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
...rest,
|
||||
seedThread: { threadId: 'example-thread-id' },
|
||||
seed: { mode: 'replay', threadId: 'example-thread-id' },
|
||||
});
|
||||
expect(parsed.seedThread?.threadId).toBe('example-thread-id');
|
||||
expect(replaySeedOf(parsed).threadId).toBe('example-thread-id');
|
||||
expect(parsed.conversation).toBeUndefined();
|
||||
});
|
||||
|
||||
it('accepts seedThread WITH a conversation (continuation after the live turn)', () => {
|
||||
it('accepts a replay seed WITH a conversation (continuation after the live turn)', () => {
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
seedThread: { threadId: 't1' },
|
||||
seed: { mode: 'replay', threadId: 't1' },
|
||||
conversation: [{ role: 'user', text: 'now also add error handling' }],
|
||||
});
|
||||
expect(parsed.seedThread?.threadId).toBe('t1');
|
||||
expect(replaySeedOf(parsed).threadId).toBe('t1');
|
||||
expect(parsed.conversation).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('accepts a seedThread carrying a dual-tenant endpoint (US-sourced case)', () => {
|
||||
it('accepts a replay seed carrying a dual-tenant endpoint (US-sourced case)', () => {
|
||||
// Cross-repo contract (TRUST-212): LangTracer's buildExportedTestCase emits
|
||||
// seedThread.endpoint for a US-sourced replay; the harness must retain it
|
||||
// (the inner seedThread object isn't .strict(), so an un-modelled field
|
||||
// would be silently stripped and the read would wrongly target home/EU).
|
||||
// the endpoint for a US-sourced replay; the harness must retain it (the arm
|
||||
// isn't .strict(), so an un-modelled field would be silently stripped and
|
||||
// the read would wrongly target home/EU).
|
||||
const { conversation: _omit, ...rest } = validFixture();
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
...rest,
|
||||
seedThread: { threadId: 't1', endpoint: 'https://api.smith.langchain.com' },
|
||||
seed: { mode: 'replay', threadId: 't1', endpoint: 'https://api.smith.langchain.com' },
|
||||
});
|
||||
expect(parsed.seedThread?.endpoint).toBe('https://api.smith.langchain.com');
|
||||
expect(replaySeedOf(parsed).endpoint).toBe('https://api.smith.langchain.com');
|
||||
});
|
||||
|
||||
it('rejects a seedThread endpoint that is not a URL', () => {
|
||||
const { conversation: _omit, ...rest } = validFixture();
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({ ...rest, seedThread: { threadId: 't1', endpoint: 'us' } }),
|
||||
).toThrow();
|
||||
});
|
||||
|
||||
it('retains seedThread.liveTurnRunId through parse (LangTracer live-turn pin)', () => {
|
||||
// Regression guard: the inner seedThread object is non-strict, so before the field
|
||||
// was modelled it was silently stripped on parse and never reached the reconstructor.
|
||||
const { conversation: _omit, ...rest } = validFixture();
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
...rest,
|
||||
seedThread: { threadId: 't1', liveTurnRunId: 'run-abc-123' },
|
||||
});
|
||||
expect(parsed.seedThread?.liveTurnRunId).toBe('run-abc-123');
|
||||
});
|
||||
|
||||
it('rejects an empty-string seedThread.liveTurnRunId', () => {
|
||||
const { conversation: _omit, ...rest } = validFixture();
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({ ...rest, seedThread: { threadId: 't1', liveTurnRunId: '' } }),
|
||||
).toThrow();
|
||||
});
|
||||
|
||||
it('rejects seedThread combined with another seeding mode', () => {
|
||||
it('rejects a replay endpoint that is not a URL', () => {
|
||||
const { conversation: _omit, ...rest } = validFixture();
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({
|
||||
...rest,
|
||||
seedThread: { threadId: 't1' },
|
||||
priorConversation: [{ role: 'user', text: 'prelude' }],
|
||||
seed: { mode: 'replay', threadId: 't1', endpoint: 'us' },
|
||||
}),
|
||||
).toThrow(/mutually exclusive/);
|
||||
).toThrow();
|
||||
});
|
||||
|
||||
it('rejects a non-seedThread case that omits conversation', () => {
|
||||
it('retains liveTurnRunId through parse (LangTracer live-turn pin)', () => {
|
||||
// Regression guard: the arm is non-strict, so before the field was modelled it
|
||||
// was silently stripped on parse and never reached the reconstructor.
|
||||
const { conversation: _omit, ...rest } = validFixture();
|
||||
expect(() => EvalTestCaseSchema.parse(rest)).toThrow(/needs a conversation, or a seedThread/);
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
...rest,
|
||||
seed: { mode: 'replay', threadId: 't1', liveTurnRunId: 'run-abc-123' },
|
||||
});
|
||||
expect(replaySeedOf(parsed).liveTurnRunId).toBe('run-abc-123');
|
||||
});
|
||||
|
||||
it('rejects an empty-string liveTurnRunId', () => {
|
||||
const { conversation: _omit, ...rest } = validFixture();
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({
|
||||
...rest,
|
||||
seed: { mode: 'replay', threadId: 't1', liveTurnRunId: '' },
|
||||
}),
|
||||
).toThrow();
|
||||
});
|
||||
|
||||
it('rejects a case that omits conversation without a replay seed', () => {
|
||||
const { conversation: _omit, ...rest } = validFixture();
|
||||
expect(() => EvalTestCaseSchema.parse(rest)).toThrow(
|
||||
/needs a conversation, or a seed with mode: replay/,
|
||||
);
|
||||
// An inline seed carries no live turn, so it needs one too.
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({
|
||||
...rest,
|
||||
seed: { mode: 'inline', messages: [{ role: 'user', text: 'prelude' }] },
|
||||
}),
|
||||
).toThrow(/needs a conversation, or a seed with mode: replay/);
|
||||
});
|
||||
|
||||
it('accepts the optional triggerType field', () => {
|
||||
|
|
|
|||
|
|
@ -962,7 +962,7 @@ describe('reconstructSeedFromThread — workspace auto-discovery', () => {
|
|||
// Dual-tenant READS (US→EU migration): a seed ref's `endpoint` selects which
|
||||
// LangSmith tenant to read from. Writes are unaffected (they stay on the home
|
||||
// tenant elsewhere). The endpoint→key mapping is the cross-repo contract that
|
||||
// LangTracer's exported `seedThread.endpoint` rides on (TRUST-212).
|
||||
// LangTracer's exported `seed.endpoint` rides on (TRUST-212).
|
||||
describe('configFor — dual-tenant read resolution', () => {
|
||||
const EU = 'https://eu.api.smith.langchain.com';
|
||||
const US = 'https://api.smith.langchain.com';
|
||||
|
|
|
|||
|
|
@ -69,13 +69,14 @@ describe('normalizeExportedCase', () => {
|
|||
}
|
||||
});
|
||||
|
||||
it('keeps a real schema field (seedThread, incl. its endpoint) through the whitelist', () => {
|
||||
it('keeps a real schema field (seed, incl. the replay endpoint) through the whitelist', () => {
|
||||
const out = normalizeExportedCase({
|
||||
seedThread: { threadId: 't1', endpoint: 'https://api.smith.langchain.com' },
|
||||
seed: { mode: 'replay', threadId: 't1', endpoint: 'https://api.smith.langchain.com' },
|
||||
complexity: 'medium',
|
||||
id: 1,
|
||||
}) as Record<string, unknown>;
|
||||
expect(out.seedThread).toEqual({
|
||||
expect(out.seed).toEqual({
|
||||
mode: 'replay',
|
||||
threadId: 't1',
|
||||
endpoint: 'https://api.smith.langchain.com',
|
||||
});
|
||||
|
|
|
|||
|
|
@ -29,11 +29,12 @@ describe('casesFromExportedFiles', () => {
|
|||
expect(cases[0].testCase.outcomeExpectations).toEqual(['has a trigger']);
|
||||
});
|
||||
|
||||
it('keeps an inline conversationSeed on a suite-sourced case', () => {
|
||||
it('keeps an inline seed on a suite-sourced case', () => {
|
||||
const cases = casesFromExportedFiles(
|
||||
{
|
||||
'repair-it.json': validCase({
|
||||
conversationSeed: {
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [
|
||||
{
|
||||
id: 'm1',
|
||||
|
|
@ -49,16 +50,36 @@ describe('casesFromExportedFiles', () => {
|
|||
},
|
||||
{ suite: 'demo' },
|
||||
);
|
||||
expect(cases[0].testCase.conversationSeed?.workflows[0].id).toBe('wKk3RmT9xQ2bVn7L');
|
||||
const seed = cases[0].testCase.seed;
|
||||
expect(seed?.mode === 'inline' && seed.workflows[0].id).toBe('wKk3RmT9xQ2bVn7L');
|
||||
});
|
||||
|
||||
it('refuses a suite-sourced case that points at a seed FILE', () => {
|
||||
// The normalizer whitelists to the schema's keys, so a hosted case carrying a
|
||||
// pre-union seed key would be STRIPPED and run unseeded — silently grading a
|
||||
// seeded case as build-from-scratch. Each removed key needs a raw-body guard.
|
||||
it.each([
|
||||
['seedFile', 'repair-it.seed'],
|
||||
['conversationSeed', { messages: [] }],
|
||||
['priorConversation', [{ role: 'user', text: 'earlier' }]],
|
||||
['seedThread', { threadId: 't1' }],
|
||||
])('refuses a suite-sourced case carrying the legacy %s key', (key, value) => {
|
||||
expect(() =>
|
||||
casesFromExportedFiles({ 'repair-it.json': validCase({ [key]: value }) }, { suite: 'demo' }),
|
||||
).toThrow(`${key}: no longer supported`);
|
||||
});
|
||||
|
||||
it('names every legacy seed key a case carries, not just the first', () => {
|
||||
expect(() =>
|
||||
casesFromExportedFiles(
|
||||
{ 'repair-it.json': validCase({ seedFile: 'repair-it.seed' }) },
|
||||
{
|
||||
'repair-it.json': validCase({
|
||||
priorConversation: [{ role: 'user', text: 'earlier' }],
|
||||
seedThread: { threadId: 't1' },
|
||||
}),
|
||||
},
|
||||
{ suite: 'demo' },
|
||||
),
|
||||
).toThrow(/seedFile/);
|
||||
).toThrow(/repair-it\.json[\s\S]*priorConversation[\s\S]*seedThread/);
|
||||
});
|
||||
|
||||
it('aggregates validation errors and names the offending file', () => {
|
||||
|
|
|
|||
|
|
@ -55,10 +55,10 @@ describe('planPush', () => {
|
|||
});
|
||||
|
||||
it('skips a case that uses an unsupported seeding mode', () => {
|
||||
const plan = planPush([item('c', { seedThread: { threadId: 't' } })], {}, {});
|
||||
const plan = planPush([item('c', { seed: { mode: 'replay', threadId: 't' } })], {}, {});
|
||||
expect(plan.skipped).toHaveLength(1);
|
||||
expect(plan.skipped[0].fileSlug).toBe('c');
|
||||
expect(plan.skipped[0].reason).toMatch(/seedThread/);
|
||||
expect(plan.skipped[0].reason).toMatch(/seed \(mode: replay\)/);
|
||||
expect(plan.toCreate).toEqual([]);
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -110,34 +110,34 @@ describe('unsupportedPushReason', () => {
|
|||
expect(unsupportedPushReason(diskCase())).toBeNull();
|
||||
});
|
||||
|
||||
it('flags seedThread and priorConversation as unsupported', () => {
|
||||
expect(unsupportedPushReason(diskCase({ seedThread: { threadId: 't' } }))).toMatch(
|
||||
/seedThread/,
|
||||
);
|
||||
expect(
|
||||
unsupportedPushReason(diskCase({ priorConversation: [{ role: 'user', text: 'earlier' }] })),
|
||||
).toMatch(/priorConversation/);
|
||||
// Keyed off the discriminant, so a new mode is flagged without editing the
|
||||
// mapper — the point of the one-slot union.
|
||||
it('flags a replay seed as unsupported, naming the mode', () => {
|
||||
const reason = unsupportedPushReason(diskCase({ seed: { mode: 'replay', threadId: 't' } }));
|
||||
expect(reason).toMatch(/seed/);
|
||||
expect(reason).toMatch(/replay/);
|
||||
});
|
||||
|
||||
it('flags an inline conversationSeed as unsupported', () => {
|
||||
expect(
|
||||
unsupportedPushReason(
|
||||
diskCase({
|
||||
conversationSeed: {
|
||||
messages: [
|
||||
{
|
||||
id: 'm1',
|
||||
type: 'llm',
|
||||
role: 'user',
|
||||
createdAt: '2026-06-29T09:00:00.000Z',
|
||||
content: [{ type: 'text', text: 'build it' }],
|
||||
},
|
||||
],
|
||||
workflows: [],
|
||||
dataTables: [],
|
||||
},
|
||||
}),
|
||||
),
|
||||
).toMatch(/conversationSeed/);
|
||||
it('flags an inline seed as unsupported, naming the mode', () => {
|
||||
const reason = unsupportedPushReason(
|
||||
diskCase({
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [
|
||||
{
|
||||
id: 'm1',
|
||||
type: 'llm',
|
||||
role: 'user',
|
||||
createdAt: '2026-06-29T09:00:00.000Z',
|
||||
content: [{ type: 'text', text: 'build it' }],
|
||||
},
|
||||
],
|
||||
workflows: [],
|
||||
dataTables: [],
|
||||
},
|
||||
}),
|
||||
);
|
||||
expect(reason).toMatch(/seed/);
|
||||
expect(reason).toMatch(/inline/);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -133,12 +133,16 @@ describe('unsupportedMcpBuildSetupFields', () => {
|
|||
expect(unsupportedMcpBuildSetupFields(testCase({ messageBudget: 6 }))).toEqual([]);
|
||||
});
|
||||
|
||||
it.each<[string, Partial<WorkflowTestCase>]>([
|
||||
['credentials', { credentials: [{ type: 'slackApi' }] }],
|
||||
// One `seed` entry covers every mode: the classification keys off the slot, not
|
||||
// the mode, so a new arm needs no edit here.
|
||||
it.each<[string, string, Partial<WorkflowTestCase>]>([
|
||||
['credentials', 'credentials', { credentials: [{ type: 'slackApi' }] }],
|
||||
[
|
||||
'conversationSeed',
|
||||
'an inline seed',
|
||||
'seed',
|
||||
{
|
||||
conversationSeed: {
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [
|
||||
{
|
||||
id: 'm1',
|
||||
|
|
@ -153,9 +157,8 @@ describe('unsupportedMcpBuildSetupFields', () => {
|
|||
},
|
||||
},
|
||||
],
|
||||
['priorConversation', { priorConversation: [user('We already agreed on #alerts')] }],
|
||||
['seedThread', { seedThread: { threadId: 't1' } }],
|
||||
])('flags %s', (field, overrides) => {
|
||||
['a replay seed', 'seed', { seed: { mode: 'replay', threadId: 't1' } }],
|
||||
])('flags %s', (_label, field, overrides) => {
|
||||
expect(unsupportedMcpBuildSetupFields(testCase(overrides))).toEqual([field]);
|
||||
});
|
||||
|
||||
|
|
@ -164,10 +167,10 @@ describe('unsupportedMcpBuildSetupFields', () => {
|
|||
unsupportedMcpBuildSetupFields(
|
||||
testCase({
|
||||
credentials: [{ type: 'slackApi' }],
|
||||
priorConversation: [user('prelude')],
|
||||
seed: { mode: 'replay', threadId: 't1' },
|
||||
}),
|
||||
),
|
||||
).toEqual(['credentials', 'priorConversation']);
|
||||
).toEqual(['credentials', 'seed']);
|
||||
});
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -67,13 +67,13 @@ describe('run debug report', () => {
|
|||
expect(getTestCaseAnchorId(result, 0)).toBe('tc-slack-notifier');
|
||||
});
|
||||
|
||||
it('labels a seedThread case (no authored conversation) without throwing', () => {
|
||||
it('labels a replay-seeded case (no authored conversation) without throwing', () => {
|
||||
const seedCase: WorkflowTestCase = {
|
||||
complexity: 'simple',
|
||||
tags: ['seeded'],
|
||||
executionScenarios: [{ name: 's', description: 'd', dataSetup: '', successCriteria: 'ok' }],
|
||||
datasets: ['seeded'],
|
||||
seedThread: { threadId: 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee' },
|
||||
seed: { mode: 'replay' as const, threadId: 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee' },
|
||||
};
|
||||
const html = generateRunDebugReport([
|
||||
resultWithRunDebug(
|
||||
|
|
|
|||
|
|
@ -53,10 +53,10 @@ describe('selectAuthorExpectations', () => {
|
|||
expect(warnings).toEqual([]);
|
||||
});
|
||||
|
||||
it('synthesizes an empty-prompt transcript for a seedThread-style case with no authored conversation', () => {
|
||||
it('synthesizes an empty-prompt transcript for a replay-seeded case with no authored conversation', () => {
|
||||
const { logger, warnings } = makeLogger();
|
||||
const { expectations, transcript } = selectAuthorExpectations({
|
||||
// seedThread cases carry no authored `conversation`; on the prebuilt/no-transcript
|
||||
// A replay seed carries no authored `conversation`; on the prebuilt/no-transcript
|
||||
// path this must not crash (regression: conversationUserTurnsAsText(undefined)).
|
||||
testCase: { outcomeExpectations: ['o1'] },
|
||||
transcript: undefined,
|
||||
|
|
|
|||
|
|
@ -504,7 +504,7 @@ async function main(): Promise<void> {
|
|||
casesBySlug.set(slug, testCaseSchema.parse(readJson(file, `test case ${slug}`)).conversation);
|
||||
}
|
||||
}
|
||||
// Drop cases with no user turn to build from (e.g. seedThread-only).
|
||||
// Drop cases with no user turn to build from (e.g. `replay`-seeded only).
|
||||
for (const [slug, conv] of [...casesBySlug]) {
|
||||
if (!conv.some((t) => t.role === 'user' && t.text.trim().length > 0)) {
|
||||
console.log(` [${slug}] skip: no user turn to build from`);
|
||||
|
|
|
|||
|
|
@ -166,9 +166,7 @@ export const MCP_BUILD_KEY_SUPPORT: Record<
|
|||
// reach this check; classified only to keep the map schema-complete.
|
||||
buildExpectations: 'supported',
|
||||
credentials: 'orchestrator-only',
|
||||
conversationSeed: 'orchestrator-only',
|
||||
priorConversation: 'orchestrator-only',
|
||||
seedThread: 'orchestrator-only',
|
||||
seed: 'orchestrator-only',
|
||||
datasets: 'supported',
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -1,15 +1,18 @@
|
|||
{
|
||||
"description": "INS-642 regression (the more damaging failure: overriding a correct user correction). The seeded history plants the bug as it happened — the agent told the user to set a 'Verify Token' field in the WhatsApp credential and type any string. The live turn is the user's correct correction: there is no such field, only Client ID and Client Secret. The agent must honor the correction and NOT pivot to an unrelated success claim ('activated' / 'done'). Ground truth: no verify-token credential field exists; the verify token is the trigger node's auto-generated id and n8n auto-registers the Meta webhook with it on activation. Q&A/debug, not a build — judged on processExpectations.",
|
||||
"priorConversation": [
|
||||
{
|
||||
"role": "user",
|
||||
"text": "My WhatsApp Trigger still isn't receiving messages from Meta. The Meta webhook setup needs a 'Verify Token' — how do I set that up in n8n?"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"text": "Open your 'WhatsApp OAuth account' credential, find the Verify Token field, and type any string you like (e.g. myverifytoken123). Then paste that same value into Meta's webhook 'Verify token' box and click Verify and Save."
|
||||
}
|
||||
],
|
||||
"seed": {
|
||||
"mode": "inline",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"text": "My WhatsApp Trigger still isn't receiving messages from Meta. The Meta webhook setup needs a 'Verify Token' — how do I set that up in n8n?"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"text": "Open your 'WhatsApp OAuth account' credential, find the Verify Token field, and type any string you like (e.g. myverifytoken123). Then paste that same value into Meta's webhook 'Verify token' box and click Verify and Save."
|
||||
}
|
||||
]
|
||||
},
|
||||
"conversation": [
|
||||
{
|
||||
"role": "user",
|
||||
|
|
|
|||
|
|
@ -22,12 +22,12 @@ import {
|
|||
import { runWorkflowChecks, summarizeMissingWorkflowError } from './cleanup';
|
||||
import {
|
||||
remapSeedWorkflowIds,
|
||||
seedFromProse,
|
||||
transcriptPrefixFromSeed,
|
||||
type ConversationSeed,
|
||||
} from './conversation-seed';
|
||||
import { reconstructSeedFromThread, type SeedThreadRef } from './langsmith-seed';
|
||||
import { reconstructSeedFromThread } from './langsmith-seed';
|
||||
import type { EvalLogger } from './logger';
|
||||
import type { CaseSeed } from './schema';
|
||||
import { buildSeededTablesNote, dedupeScenarioSeedTables } from './seed-tables';
|
||||
import type { CheckOutcome } from '../binaryChecks/types';
|
||||
import { N8nApiError, type N8nClient, type WorkflowResponse } from '../clients/n8n-client';
|
||||
|
|
@ -204,7 +204,7 @@ export interface BuildResult {
|
|||
export interface BuildWorkflowConfig {
|
||||
client: N8nClient;
|
||||
/** Hand-authored conversation (≥1 turn, first `user`; one user turn →
|
||||
* auto-approve, more → proxy). Optional when `seedThread` derives the live turn. */
|
||||
* auto-approve, more → proxy). Optional when a `replay` seed derives the live turn. */
|
||||
conversation?: ConversationTurn[];
|
||||
/** Max follow-up messages the proxy will send. Ignored in auto-approve mode. */
|
||||
messageBudget?: number;
|
||||
|
|
@ -212,13 +212,10 @@ export interface BuildWorkflowConfig {
|
|||
credentials?: TestCaseCredential[];
|
||||
/** Run-level registry the created credential IDs are added to for cleanup. */
|
||||
createdCredentialIds?: Set<string>;
|
||||
/** Prior messages + workflows restored before the live message. */
|
||||
conversationSeed?: ConversationSeed;
|
||||
/** Prose turns seeded as plain-text history. */
|
||||
priorConversation?: ConversationTurn[];
|
||||
/** Reproduce a real conversation from its LangSmith trace (seed = before the
|
||||
* last user message, live = that message). */
|
||||
seedThread?: SeedThreadRef;
|
||||
/** History restored before the live message — carried in the case
|
||||
* (`mode: 'inline'`) or reconstructed from a trace (`mode: 'replay'`, which
|
||||
* also supplies the live turn). */
|
||||
seed?: CaseSeed;
|
||||
/** Execution scenarios whose declared `seedDataTables` are created + row-seeded
|
||||
* after a successful build, before any scenario runs (TRUST-311). */
|
||||
executionScenarios?: ExecutionScenario[];
|
||||
|
|
@ -295,34 +292,47 @@ export async function buildWorkflow(config: BuildWorkflowConfig): Promise<BuildR
|
|||
try {
|
||||
const buildStart = Date.now();
|
||||
|
||||
// `seedThread` derives both seed and live turn from a trace; otherwise the
|
||||
// seed (if any) is a file/prose prelude and the conversation is authored.
|
||||
// `replay` derives both seed and live turn from a trace; `inline` carries the
|
||||
// seed in the case and the conversation is authored.
|
||||
let seed: ConversationSeed | undefined;
|
||||
let conversation = config.conversation ?? [];
|
||||
try {
|
||||
if (config.seedThread) {
|
||||
const reconstructed = await reconstructSeedFromThread(config.seedThread);
|
||||
seed = reconstructed.seed;
|
||||
// The trace's last user message is the live opening; any authored
|
||||
// `conversation` continues from there (proxy-driven follow-ups).
|
||||
conversation = [
|
||||
{ role: 'user', text: reconstructed.liveTurn },
|
||||
...(config.conversation ?? []),
|
||||
];
|
||||
const contSuffix =
|
||||
(config.conversation?.length ?? 0) > 0
|
||||
? ` + ${String(config.conversation!.length)} continuation turn(s)`
|
||||
: '';
|
||||
const wsLabel = reconstructed.sourceWorkspace
|
||||
? `${reconstructed.sourceWorkspace}/${reconstructed.sourceProject}`
|
||||
: reconstructed.sourceProject;
|
||||
logger.info(
|
||||
` Reconstructed seed from thread ${config.seedThread.threadId}: ${String(reconstructed.runCount)} runs → ${String(seed.messages.length)} message(s), ${String(seed.workflows.length)} workflow(s)${contSuffix} [${wsLabel}]${config.laneTag ?? ''}`,
|
||||
);
|
||||
} else if (config.conversationSeed) {
|
||||
seed = config.conversationSeed;
|
||||
} else if (config.priorConversation && config.priorConversation.length > 0) {
|
||||
seed = seedFromProse(config.priorConversation);
|
||||
switch (config.seed?.mode) {
|
||||
case 'replay': {
|
||||
const reconstructed = await reconstructSeedFromThread(config.seed);
|
||||
seed = reconstructed.seed;
|
||||
// The trace's last user message is the live opening; any authored
|
||||
// `conversation` continues from there (proxy-driven follow-ups).
|
||||
conversation = [
|
||||
{ role: 'user', text: reconstructed.liveTurn },
|
||||
...(config.conversation ?? []),
|
||||
];
|
||||
const contSuffix =
|
||||
(config.conversation?.length ?? 0) > 0
|
||||
? ` + ${String(config.conversation!.length)} continuation turn(s)`
|
||||
: '';
|
||||
const wsLabel = reconstructed.sourceWorkspace
|
||||
? `${reconstructed.sourceWorkspace}/${reconstructed.sourceProject}`
|
||||
: reconstructed.sourceProject;
|
||||
logger.info(
|
||||
` Reconstructed seed from thread ${config.seed.threadId}: ${String(reconstructed.runCount)} runs → ${String(seed.messages.length)} message(s), ${String(seed.workflows.length)} workflow(s)${contSuffix} [${wsLabel}]${config.laneTag ?? ''}`,
|
||||
);
|
||||
break;
|
||||
}
|
||||
case 'inline':
|
||||
// The arm is a superset of the restore payload — `mode` is the case
|
||||
// schema's discriminant and never reaches restore-thread.
|
||||
seed = config.seed;
|
||||
break;
|
||||
case undefined:
|
||||
break;
|
||||
default: {
|
||||
// A new arm must decide what to restore here; without this the case
|
||||
// would silently run UNSEEDED, which is the failure this slot exists
|
||||
// to make impossible.
|
||||
const unhandled: never = config.seed;
|
||||
throw new Error(`Unhandled seed mode: ${JSON.stringify(unhandled)}`);
|
||||
}
|
||||
}
|
||||
} catch (error: unknown) {
|
||||
// A seed that can't be resolved is a harness/framework problem, not an
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
// Conversation seeding for eval builds — backs the `conversationSeed` (synthetic)
|
||||
// and `priorConversation` (prose) paths. Real conversations use `seedThread`
|
||||
// (reconstructed from a LangSmith trace; see langsmith-seed.ts).
|
||||
// Conversation seeding for eval builds — the restore payload behind the case
|
||||
// schema's `seed` slot. `mode: 'inline'` carries this payload in the case body;
|
||||
// `mode: 'replay'` reconstructs one from a LangSmith trace at run time (see
|
||||
// langsmith-seed.ts). Either way the shape below is what reaches restore-thread.
|
||||
|
||||
import { generateNanoId } from '@n8n/utils/generate-nano-id';
|
||||
import { isRecord } from '@n8n/utils/is-record';
|
||||
|
|
@ -60,7 +61,15 @@ const SeedMessageBlockSchema = z
|
|||
* the store's contract, not ours. */
|
||||
const seedMessageObjectSchema = z
|
||||
.object({
|
||||
id: z.string().min(1),
|
||||
// The required_error names both authoring forms: a message that is neither a
|
||||
// full envelope nor a well-formed `{role, text}` shorthand lands here, and
|
||||
// "Required" alone wouldn't hint that the shorthand exists.
|
||||
id: z
|
||||
.string({
|
||||
required_error:
|
||||
'is required — a prior message is either a full envelope (id/role/type/createdAt/content) or the `{role, text}` shorthand',
|
||||
})
|
||||
.min(1),
|
||||
// Restricted to the two roles the transcript builder renders: any other
|
||||
// value is guaranteed to vanish from the judge transcript, which is the
|
||||
// exact silent failure this schema exists to catch. If the message store
|
||||
|
|
@ -86,7 +95,7 @@ const seedMessageObjectSchema = z
|
|||
* CI's type-aware lint (same reason as `EvalTestCaseInput` in schema.ts). */
|
||||
export type SeedMessage = z.infer<typeof seedMessageObjectSchema>;
|
||||
|
||||
const SeedMessageSchema = seedMessageObjectSchema.superRefine((message, ctx) => {
|
||||
export const SeedMessageSchema = seedMessageObjectSchema.superRefine((message, ctx) => {
|
||||
// `custom` messages are stored but never rendered (no role, any content
|
||||
// shape). Everything else is read by the transcript builder, which needs a
|
||||
// role it renders and an array of blocks.
|
||||
|
|
@ -123,24 +132,95 @@ export const ConversationSeedSchema = z.object({
|
|||
export type ConversationSeed = z.infer<typeof ConversationSeedSchema>;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Prose prior turns → seed messages
|
||||
// `{role, text}` shorthand → seed messages
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Convert authored prose turns into native llm messages, stamped slightly in
|
||||
* the past (ascending) so they order before the live turn. */
|
||||
export function seedFromProse(turns: ConversationTurn[]): ConversationSeed {
|
||||
const base = Date.now() - (turns.length + 1) * 1000;
|
||||
return {
|
||||
messages: turns.map((turn, index) => ({
|
||||
/** A `{role, text}` prior message — the authoring sugar the `priorConversation`
|
||||
* key used to be. Recognised by its EXACT key set (and valid values), so a full
|
||||
* envelope is never mistaken for one, and a near-miss falls through to the
|
||||
* envelope schema and gets a real error instead of being expanded into a
|
||||
* message the transcript builder would silently drop. */
|
||||
function isShorthandTurn(
|
||||
value: unknown,
|
||||
): value is { role: ConversationTurn['role']; text: string | string[] } {
|
||||
if (!isRecord(value)) return false;
|
||||
const keys = Object.keys(value);
|
||||
if (keys.length !== 2 || !keys.includes('role') || !keys.includes('text')) return false;
|
||||
if (value.role !== 'user' && value.role !== 'assistant') return false;
|
||||
return (
|
||||
typeof value.text === 'string' ||
|
||||
(Array.isArray(value.text) && value.text.every((line) => typeof line === 'string'))
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Expand `{role, text}` shorthand messages into native llm envelopes; anything
|
||||
* else passes through for the envelope schema to validate.
|
||||
*
|
||||
* Array-level rather than per-message because the stamped timestamps ascend by
|
||||
* position — slightly in the past, so seeded history always orders before the
|
||||
* live turn and a shorthand author cannot get the ordering wrong. A full
|
||||
* envelope keeps its own authored `createdAt`.
|
||||
*/
|
||||
export function expandSeedMessageShorthand(messages: unknown[]): unknown[] {
|
||||
const base = Date.now() - (messages.length + 1) * 1000;
|
||||
return messages.map((message, index) => {
|
||||
if (!isShorthandTurn(message)) return message;
|
||||
return {
|
||||
id: randomUUID(),
|
||||
type: 'llm',
|
||||
role: turn.role,
|
||||
content: [{ type: 'text', text: turn.text }],
|
||||
role: message.role,
|
||||
// Array form = lines, joined — same normalization as
|
||||
// `conversationTurnTextSchema` applies to an authored conversation turn.
|
||||
content: [
|
||||
{
|
||||
type: 'text',
|
||||
text: Array.isArray(message.text) ? message.text.join('\n') : message.text,
|
||||
},
|
||||
],
|
||||
createdAt: new Date(base + index * 1000).toISOString(),
|
||||
})),
|
||||
workflows: [],
|
||||
dataTables: [],
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Pull an inline seed's timestamps back into the past when the author put any of
|
||||
* them in the future.
|
||||
*
|
||||
* The shorthand stamps its own ascending pre-live timestamps, so it can't get
|
||||
* this wrong; a full envelope keeps what it was authored with, and a future
|
||||
* stamp sorts the seeded turn AFTER the live turn — the agent then sees its own
|
||||
* history out of order, and the judge grades a transcript that never happened.
|
||||
*
|
||||
* Restamps the WHOLE sequence, not just the offending entry. A per-message clamp
|
||||
* reorders relative to the array: `[future A, past B]` leaves B alone and moves A
|
||||
* to ~now, so the store presents B then A while `transcriptPrefixFromSeed` still
|
||||
* grades array order. Array order is the authority, so the rewrite reproduces it
|
||||
* on the same ascending slots the shorthand uses. Authored timestamps are
|
||||
* therefore only preserved when every one of them is already in the past.
|
||||
*
|
||||
* Inline (hand-authored) seeds only — a `replay` seed is reconstructed from a
|
||||
* real trace and never reaches this schema, so no real timestamp can be moved.
|
||||
*/
|
||||
export function clampFutureSeedTimestamps(messages: unknown[]): unknown[] {
|
||||
const now = Date.now();
|
||||
const stampOf = (message: unknown): number | undefined => {
|
||||
if (!isRecord(message) || typeof message.createdAt !== 'string') return undefined;
|
||||
const at = Date.parse(message.createdAt);
|
||||
// Unparseable is the envelope schema's error to report, not ours to paper over.
|
||||
return Number.isNaN(at) ? undefined : at;
|
||||
};
|
||||
const anyFuture = messages.some((message) => (stampOf(message) ?? -Infinity) >= now);
|
||||
if (!anyFuture) return messages;
|
||||
|
||||
const base = now - (messages.length + 1) * 1000;
|
||||
return messages.map((message, index) => {
|
||||
if (stampOf(message) === undefined) return message;
|
||||
return {
|
||||
...(message as Record<string, unknown>),
|
||||
createdAt: new Date(base + index * 1000).toISOString(),
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ import { parseSeedWorkflowCode } from './parse-seed-workflow';
|
|||
import { COMPILED_WORKFLOW_TRACE_RUN_NAME, DOMAIN_TOOL_IDS } from '../../src/tools/tool-ids';
|
||||
|
||||
/** Default project that instance-ai conversations are traced to (same name in
|
||||
* every workspace). Override per case with `seedThread.project` if it differs. */
|
||||
* every workspace). Override per case with `seed.project` if it differs. */
|
||||
const DEFAULT_SOURCE_PROJECT = 'instance-ai';
|
||||
|
||||
// Reference the live tool-id so a rename there follows here (or breaks the import).
|
||||
|
|
@ -64,7 +64,7 @@ export const IGNORED_WORKSPACE_TOOLS: readonly string[] = [
|
|||
// enumerate them and find the one holding the thread; reads are read-only.
|
||||
//
|
||||
// Reads are also dual-tenant: during the US→EU migration a US-sourced case
|
||||
// carries `seedThread.endpoint` = the US host, while results always write to the
|
||||
// carries `seed.endpoint` = the US host, while results always write to the
|
||||
// home (EU) tenant elsewhere. `configFor` maps an endpoint → host + key via env.
|
||||
|
||||
/** The langsmith SDK's default endpoint is the US tenant, so an unset
|
||||
|
|
@ -100,13 +100,13 @@ export function configFor(endpoint?: string): { apiUrl: string; apiKey: string }
|
|||
const usKey = process.env.LANGSMITH_API_KEY_US ?? '';
|
||||
if (!usKey) {
|
||||
throw new Error(
|
||||
`seedThread.endpoint "${endpoint}" is the secondary (US) tenant but LANGSMITH_API_KEY_US is not set — refusing to read it with the home key. Set LANGSMITH_API_KEY_US.`,
|
||||
`seed.endpoint "${endpoint}" is the secondary (US) tenant but LANGSMITH_API_KEY_US is not set — refusing to read it with the home key. Set LANGSMITH_API_KEY_US.`,
|
||||
);
|
||||
}
|
||||
return { apiUrl: usHost, apiKey: usKey };
|
||||
}
|
||||
throw new Error(
|
||||
`seedThread.endpoint "${endpoint}" matches no configured LangSmith tenant (home: ${homeHost}; secondary: ${usHost}). Set LANGSMITH_ENDPOINT_US + LANGSMITH_API_KEY_US, or omit endpoint to use the home tenant.`,
|
||||
`seed.endpoint "${endpoint}" matches no configured LangSmith tenant (home: ${homeHost}; secondary: ${usHost}). Set LANGSMITH_ENDPOINT_US + LANGSMITH_API_KEY_US, or omit endpoint to use the home tenant.`,
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -321,7 +321,7 @@ async function discoverAndReconstruct(
|
|||
}
|
||||
}
|
||||
throw new Error(
|
||||
`Thread ${ref.threadId} not found in project "${project}" across ${String(workspaces.length)} workspace(s): ${tried.join(', ')}. The trace may have aged out (~14-day base retention), or the project name differs (set seedThread.project).`,
|
||||
`Thread ${ref.threadId} not found in project "${project}" across ${String(workspaces.length)} workspace(s): ${tried.join(', ')}. The trace may have aged out (~14-day base retention), or the project name differs (set seed.project).`,
|
||||
);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,12 @@
|
|||
import { instanceAiEvalSeedDataTableSchema } from '@n8n/api-types';
|
||||
import { z } from 'zod';
|
||||
|
||||
import { ConversationSeedSchema } from './conversation-seed';
|
||||
import {
|
||||
ConversationSeedSchema,
|
||||
SeedMessageSchema,
|
||||
clampFutureSeedTimestamps,
|
||||
expandSeedMessageShorthand,
|
||||
} from './conversation-seed';
|
||||
import { SUPPORTED_CREDENTIAL_TYPES } from '../credentials/seeder';
|
||||
|
||||
/** Default `datasets` grouping for a case that omits the field — the single
|
||||
|
|
@ -44,11 +49,69 @@ const ExecutionScenarioSchema = z.object({
|
|||
seedDataTables: z.array(instanceAiEvalSeedDataTableSchema).max(20).optional(),
|
||||
});
|
||||
|
||||
/** Prior messages for an inline seed. Accepts a full envelope or the `{role, text}`
|
||||
* shorthand, expanded BEFORE validation so the envelope rules apply to the
|
||||
* expansion and error paths stay per-message (`seed.messages.2.createdAt`).
|
||||
* Future `createdAt` values are pulled back after expansion, so seeded history
|
||||
* can never sort after the live turn. */
|
||||
const inlineSeedMessagesSchema = z.preprocess(
|
||||
(raw) => (Array.isArray(raw) ? clampFutureSeedTimestamps(expandSeedMessageShorthand(raw)) : raw),
|
||||
z.array(SeedMessageSchema).min(1),
|
||||
);
|
||||
|
||||
/**
|
||||
* Where a case's seeded history comes from — ONE slot, so the modes are mutually
|
||||
* exclusive by construction instead of by a refine, and `mode` carries a real
|
||||
* either/or: is the seed in the case, or fetched at run time?
|
||||
*
|
||||
* The literals match lang-tracer's `metadata.seed` verbatim (TRUST-358);
|
||||
* diverging rebuilds the old→new translation layer this union exists to delete.
|
||||
* `replay` names an action next to `inline`'s location — an asymmetry we take
|
||||
* knowingly, because renaming it would break an LT HTTP body contract.
|
||||
*
|
||||
* Both arms are `.strict()`, matching the case schema: exclusivity has to fail
|
||||
* loudly, not by stripping. `{ mode: replay, threadId, messages }` would
|
||||
* otherwise parse as a valid replay seed with `messages` silently dropped —
|
||||
* the case runs unseeded and grades as if it were a build from scratch, which
|
||||
* is the misgrading the one-slot union exists to prevent.
|
||||
*/
|
||||
export const CaseSeedSchema = z.discriminatedUnion('mode', [
|
||||
/** Prior messages plus the workflows/tables they reference, carried in the case
|
||||
* body. Synthetic fixtures only — a real conversation belongs in `replay`,
|
||||
* which keeps its content out of the repo. Pairs with `conversation`, which
|
||||
* supplies the live turn. */
|
||||
ConversationSeedSchema.extend({
|
||||
mode: z.literal('inline'),
|
||||
messages: inlineSeedMessagesSchema,
|
||||
}).strict(),
|
||||
/** Reproduce a real conversation from its LangSmith trace at run time (seed =
|
||||
* before the live turn, live = that turn). Commits only the thread id;
|
||||
* workspace auto-discovered. Supplies the live turn itself, so `conversation`
|
||||
* is optional and continues after it. Transient (~14d trace retention). */
|
||||
z
|
||||
.object({
|
||||
mode: z.literal('replay'),
|
||||
threadId: z.string().min(1),
|
||||
project: z.string().min(1).optional(),
|
||||
/** LangSmith host the source trace lives on (dual-tenant reads during the
|
||||
* US→EU migration). Omit ⇒ the eval's home (EU) tenant, so existing cases
|
||||
* are unchanged. A US-sourced case carries the US host; the harness maps
|
||||
* host→key via env (LANGSMITH_API_KEY_US). */
|
||||
endpoint: z.string().url().optional(),
|
||||
/** Pin which user turn is sent live (its LangSmith run id); everything before
|
||||
* it is seeded. Omit ⇒ the thread's last user turn (default). */
|
||||
liveTurnRunId: z.string().min(1).optional(),
|
||||
})
|
||||
.strict(),
|
||||
]);
|
||||
|
||||
export type CaseSeed = z.infer<typeof CaseSeedSchema>;
|
||||
|
||||
const evalTestCaseObjectSchema = z
|
||||
.object({
|
||||
/** Optional human-readable note on what this case is testing (esp. for behaviour cases). */
|
||||
description: z.string().optional(),
|
||||
// Optional only because `seedThread` derives the live turn from the trace;
|
||||
// Optional only because a `replay` seed derives the live turn from the trace;
|
||||
// a refine() below requires it for every other case.
|
||||
conversation: z.array(ConversationTurnSchema).min(1).optional(),
|
||||
complexity: z.enum(['simple', 'medium', 'complex']),
|
||||
|
|
@ -95,29 +158,9 @@ const evalTestCaseObjectSchema = z
|
|||
}),
|
||||
)
|
||||
.optional(),
|
||||
/** Prior messages + the workflows they reference, restored before the live turn.
|
||||
* Synthetic fixtures only; real conversations use `seedThread`. */
|
||||
conversationSeed: ConversationSeedSchema.optional(),
|
||||
/** Prose turns seeded as plain-text history (no tool calls / workflows). */
|
||||
priorConversation: z.array(ConversationTurnSchema).min(1).optional(),
|
||||
/** Reproduce a real conversation from its LangSmith trace at run time (seed =
|
||||
* before the last user message, live = that message). Commits only the thread
|
||||
* id; workspace auto-discovered. Supplies the live turn, so `conversation` is
|
||||
* optional (continues after it). */
|
||||
seedThread: z
|
||||
.object({
|
||||
threadId: z.string().min(1),
|
||||
project: z.string().min(1).optional(),
|
||||
/** LangSmith host the source trace lives on (dual-tenant reads during the
|
||||
* US→EU migration). Omit ⇒ the eval's home (EU) tenant, so existing cases
|
||||
* are unchanged. A US-sourced case carries the US host; the harness maps
|
||||
* host→key via env (LANGSMITH_API_KEY_US). */
|
||||
endpoint: z.string().url().optional(),
|
||||
/** Pin which user turn is sent live (its LangSmith run id); everything before it
|
||||
* is seeded. Omit ⇒ the thread's last user turn (default). */
|
||||
liveTurnRunId: z.string().min(1).optional(),
|
||||
})
|
||||
.optional(),
|
||||
/** History restored before the live turn — one slot, `mode` says where it
|
||||
* comes from. See `CaseSeedSchema`. */
|
||||
seed: CaseSeedSchema.optional(),
|
||||
/**
|
||||
* Logical groupings this case belongs to (e.g. `['pr', 'full']`). Used by
|
||||
* the eval CLI's `--tier` flag and propagated to LangSmith as example
|
||||
|
|
@ -137,18 +180,13 @@ const evalTestCaseObjectSchema = z
|
|||
* set is robust where blacklisting the few keys we happen to know today is not. */
|
||||
export const WORKFLOW_TEST_CASE_KEYS = Object.keys(evalTestCaseObjectSchema.shape);
|
||||
|
||||
// At most one seeding mode, and a source for the live turn.
|
||||
// A source for the live turn. (Seeding modes are exclusive by construction — the
|
||||
// `seed` union — so the old two-refine pair is down to this one rule, keyed off
|
||||
// the discriminant: only `replay` brings its own live turn.)
|
||||
export const EvalTestCaseSchema = evalTestCaseObjectSchema
|
||||
.refine(
|
||||
(c) => [c.conversationSeed, c.priorConversation, c.seedThread].filter(Boolean).length <= 1,
|
||||
{
|
||||
message:
|
||||
'conversationSeed, priorConversation and seedThread are mutually exclusive — pick one seeding mode',
|
||||
},
|
||||
)
|
||||
.refine((c) => c.seedThread !== undefined || c.conversation !== undefined, {
|
||||
.refine((c) => c.seed?.mode === 'replay' || c.conversation !== undefined, {
|
||||
message:
|
||||
'a case needs a conversation, or a seedThread (which supplies the live turn from the trace)',
|
||||
'a case needs a conversation, or a seed with mode: replay (which supplies the live turn from the trace)',
|
||||
})
|
||||
.superRefine((c, ctx) => {
|
||||
// Note: this message avoids double quotes — ZodError.message is a JSON.stringify of
|
||||
|
|
|
|||
|
|
@ -20,6 +20,20 @@ export interface LangTracerLoadOptions {
|
|||
logger: EvalLogger;
|
||||
}
|
||||
|
||||
/**
|
||||
* Seeding keys this schema no longer accepts, and what replaced them. Checked on
|
||||
* the RAW export body because `normalizeExportedCase` whitelists to the schema's
|
||||
* keys: a removed key is STRIPPED, so a hosted case still carrying one would run
|
||||
* UNSEEDED rather than fail — silently grading a seeded case as if it were a
|
||||
* build-from-scratch. Every removal needs an entry here (#35074 added the first).
|
||||
*/
|
||||
const REMOVED_SEED_KEYS: Record<string, string> = {
|
||||
seedFile: 'carry the seed in the case body as `seed: { mode: "inline", … }`',
|
||||
conversationSeed: 'now `seed: { mode: "inline", messages, workflows, dataTables }`',
|
||||
priorConversation: 'now `seed: { mode: "inline", messages: [{ role, text }, …] }`',
|
||||
seedThread: 'now `seed: { mode: "replay", threadId, … }`',
|
||||
};
|
||||
|
||||
/** Normalize + validate an `export_suite` payload into n8n test cases (same
|
||||
* filter/exclude/tier selection as the disk loader); failures are aggregated so
|
||||
* a drift surfaces every bad case at once. */
|
||||
|
|
@ -32,11 +46,14 @@ export function casesFromExportedFiles(
|
|||
|
||||
for (const [filename, raw] of Object.entries(files)) {
|
||||
const fileSlug = filename.replace(/\.json$/i, '');
|
||||
// Checked on the RAW body: the normalizer whitelists to the schema's keys, so a
|
||||
// leftover `seedFile` would be stripped and the case would run unseeded.
|
||||
if (isRecord(raw) && raw.seedFile !== undefined) {
|
||||
const removed = isRecord(raw)
|
||||
? Object.entries(REMOVED_SEED_KEYS).filter(([key]) => raw[key] !== undefined)
|
||||
: [];
|
||||
if (removed.length > 0) {
|
||||
errors.push(
|
||||
`${filename}:\n - seedFile: no longer supported — carry the seed in the case body as \`conversationSeed\``,
|
||||
`${filename}:\n${removed
|
||||
.map(([key, hint]) => ` - ${key}: no longer supported — ${hint}`)
|
||||
.join('\n')}`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -39,17 +39,12 @@ export interface ToLangTracerOptions {
|
|||
synthetic: boolean;
|
||||
}
|
||||
|
||||
/** Seeding modes the case-write API can't represent (no fields for them), so a case
|
||||
* using any of them can't be pushed. Returns a human-readable reason, else null. */
|
||||
/** The case-write API has no `seed` field, so no seeded case can be pushed yet —
|
||||
* keyed off the discriminant so a new mode needs no edit here. Returns a
|
||||
* human-readable reason, else null. */
|
||||
export function unsupportedPushReason(testCase: EvalTestCaseInput): string | null {
|
||||
if (testCase.seedThread) return 'uses seedThread (not supported by the case-write API)';
|
||||
if (testCase.conversationSeed) {
|
||||
return 'uses conversationSeed (not supported by the case-write API)';
|
||||
}
|
||||
if (testCase.priorConversation) {
|
||||
return 'uses priorConversation (not supported by the case-write API)';
|
||||
}
|
||||
return null;
|
||||
if (!testCase.seed) return null;
|
||||
return `uses seed (mode: ${testCase.seed.mode}) — not supported by the case-write API`;
|
||||
}
|
||||
|
||||
/** Map a schema-parsed disk case to a create-case body. `conversation.text` is already
|
||||
|
|
|
|||
|
|
@ -498,7 +498,7 @@ export function seededTurnCounters(seededTurns: TranscriptTurn[]): TurnCounter[]
|
|||
});
|
||||
}
|
||||
|
||||
/** Prepend the seeded prefix's counters to live metrics so a seedThread case's
|
||||
/** Prepend the seeded prefix's counters to live metrics so a seeded case's
|
||||
* metrics span the whole conversation (matching the unified transcript). Live
|
||||
* `reachedRunFinishCleanly` is preserved (it describes the evaluated run); an
|
||||
* empty prefix returns metrics deep-equal to the live ones. */
|
||||
|
|
|
|||
|
|
@ -75,9 +75,7 @@ export type BuildArgs = Pick<
|
|||
| 'conversation'
|
||||
| 'messageBudget'
|
||||
| 'credentials'
|
||||
| 'conversationSeed'
|
||||
| 'priorConversation'
|
||||
| 'seedThread'
|
||||
| 'seed'
|
||||
| 'executionScenarios'
|
||||
| 'outcomeExpectations'
|
||||
> & { timeoutMs: number };
|
||||
|
|
@ -450,9 +448,7 @@ export function createBuildOrchestrator(deps: BuildOrchestratorDeps): BuildOrche
|
|||
conversation: entry.conversation,
|
||||
messageBudget: entry.messageBudget,
|
||||
credentials: entry.credentials,
|
||||
conversationSeed: entry.conversationSeed,
|
||||
priorConversation: entry.priorConversation,
|
||||
seedThread: entry.seedThread,
|
||||
seed: entry.seed,
|
||||
executionScenarios: entry.executionScenarios,
|
||||
outcomeExpectations: entry.outcomeExpectations,
|
||||
timeoutMs,
|
||||
|
|
|
|||
|
|
@ -140,9 +140,7 @@ export function createEvalSession(config: EvalSessionConfig): EvalSession {
|
|||
conversation: buildArgs.conversation,
|
||||
messageBudget: buildArgs.messageBudget,
|
||||
credentials: buildArgs.credentials,
|
||||
conversationSeed: buildArgs.conversationSeed,
|
||||
priorConversation: buildArgs.priorConversation,
|
||||
seedThread: buildArgs.seedThread,
|
||||
seed: buildArgs.seed,
|
||||
executionScenarios: buildArgs.executionScenarios,
|
||||
createdCredentialIds: lane.createdCredentialIds,
|
||||
timeoutMs: buildArgs.timeoutMs,
|
||||
|
|
|
|||
|
|
@ -57,8 +57,8 @@ function flattenRunsForReport(evaluation: MultiRunEvaluation): WorkflowTestCaseR
|
|||
}
|
||||
return evaluation.testCases.flatMap((tc) =>
|
||||
tc.runs.map((run, iter) => {
|
||||
// seedThread cases carry no authored conversation (the live turn comes
|
||||
// from the trace) — nothing to relabel.
|
||||
// A `replay`-seeded case carries no authored conversation (the live turn
|
||||
// comes from the trace) — nothing to relabel.
|
||||
if (!run.testCase.conversation?.length) return run;
|
||||
const [opening, ...rest] = run.testCase.conversation;
|
||||
return {
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ import type {
|
|||
|
||||
import type { CheckOutcome } from './binaryChecks/types';
|
||||
import type { WorkflowResponse } from './clients/n8n-client';
|
||||
import type { ConversationSeed } from './harness/conversation-seed';
|
||||
import type { CaseSeed } from './harness/schema';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Checklist items and verification
|
||||
|
|
@ -212,7 +212,7 @@ export interface WorkflowTestCase {
|
|||
/**
|
||||
* Hand-authored conversation that drives the build (≥1 turn, first `user`).
|
||||
* One user turn → auto-approve single-prompt build; more → multi-turn proxy.
|
||||
* Required unless `seedThread` is set, in which case it's optional and
|
||||
* Required unless `seed.mode` is `replay`, in which case it's optional and
|
||||
* continues after the trace's live turn (`[<live turn>, ...conversation]`).
|
||||
*/
|
||||
conversation?: ConversationTurn[];
|
||||
|
|
@ -239,18 +239,12 @@ export interface WorkflowTestCase {
|
|||
* field build with an empty view (everything mocks).
|
||||
*/
|
||||
credentials?: TestCaseCredential[];
|
||||
/** Prior messages + the workflows they reference, restored before the live turn.
|
||||
* Mutually exclusive with the other seeds. */
|
||||
conversationSeed?: ConversationSeed;
|
||||
/** Prose turns seeded as plain-text history (no tool calls/workflows).
|
||||
* Mutually exclusive with the other seeds. */
|
||||
priorConversation?: ConversationTurn[];
|
||||
/** Reproduce a real conversation from its LangSmith trace at run time: restore
|
||||
* up to the live turn (the last user message, or one pinned by `liveTurnRunId`)
|
||||
* and send that live. Commits only the thread id (workspace auto-discovered;
|
||||
* `project`/`endpoint` override the source project/tenant). Supplies the live
|
||||
* turn, so `conversation` is optional. Transient (~14d). */
|
||||
seedThread?: { threadId: string; project?: string; endpoint?: string; liveTurnRunId?: string };
|
||||
/** History restored before the live turn, in one slot so the modes can't
|
||||
* overlap: `mode: 'inline'` carries the messages (and the workflows/tables
|
||||
* they reference) in the case body; `mode: 'replay'` reconstructs them from a
|
||||
* LangSmith trace at run time and supplies the live turn itself, which is why
|
||||
* `conversation` is optional for it. See `harness/schema.ts`. */
|
||||
seed?: CaseSeed;
|
||||
/** Logical groupings this case belongs to (e.g. `['pr', 'full']`). Defaults to `['full']`. */
|
||||
datasets: string[];
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,21 +1,23 @@
|
|||
import { isRecord } from '@n8n/utils/is-record';
|
||||
|
||||
import type { CaseSeed } from '../harness/schema';
|
||||
import type { ConversationTurn, ToolInteraction, TranscriptStep, TranscriptTurn } from '../types';
|
||||
|
||||
/**
|
||||
* Human-readable prompt label for a test case. Authored cases use their first
|
||||
* turn; seedThread cases carry no authored conversation, so fall back to the
|
||||
* turn; a `replay` seed carries no authored conversation, so fall back to the
|
||||
* live (non-seeded) user turn captured in the transcript, then to the thread id.
|
||||
*/
|
||||
export function caseDisplayPrompt(
|
||||
testCase: { conversation?: ConversationTurn[]; seedThread?: { threadId: string } },
|
||||
testCase: { conversation?: ConversationTurn[]; seed?: CaseSeed },
|
||||
transcript?: TranscriptTurn[],
|
||||
): string {
|
||||
const authored = testCase.conversation?.[0]?.text;
|
||||
if (authored) return authored;
|
||||
const liveTurn = transcript?.find((t) => !t.seeded && t.userMessage)?.userMessage;
|
||||
if (liveTurn) return liveTurn;
|
||||
return testCase.seedThread ? `[seeded] thread ${testCase.seedThread.threadId.slice(0, 8)}` : '';
|
||||
const { seed } = testCase;
|
||||
return seed?.mode === 'replay' ? `[seeded] thread ${seed.threadId.slice(0, 8)}` : '';
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -38,8 +40,8 @@ export function userTurnsAsText(transcript: TranscriptTurn[]): string {
|
|||
* so prompt-aware binary checks (e.g. fulfills_user_request) source the request
|
||||
* text from the authored conversation instead of receiving an empty prompt.
|
||||
*
|
||||
* Accepts `undefined` because `testCase.conversation` is optional (seedThread-only
|
||||
* cases carry none) and callers pass it straight through — no conversation → ''.
|
||||
* Accepts `undefined` because `testCase.conversation` is optional (a `replay`-seeded
|
||||
* case carries none) and callers pass it straight through — no conversation → ''.
|
||||
*/
|
||||
export function conversationUserTurnsAsText(conversation: ConversationTurn[] | undefined): string {
|
||||
if (!conversation) return '';
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user