Two cases where the answer needs the node composition of a set of workflows, not
their names or their graphs.
- `finds-the-real-users-of-a-node-type`: inside one folder, three workflows use
the Postgres node and none says so in its name, while two more carry Postgres
in the NAME and never use it. A name filter returns exactly the wrong two.
- `reads-a-folders-node-conventions-before-extending-it`: what a folder is built
out of, before adding to it. Two workflows outside the folder use nodes that
appear nowhere inside it, so an instance-wide summary gives the wrong
convention.
Both are folder-scoped rather than instance-wide, for a reason worth recording:
the eval instance is shared and already holds unrelated workflows, so an
instance-wide count is not a fact a case can assert. Scoping also makes them
test the two capabilities composed — the folder bounds the set, the index says
which members actually contain the node.
The out-of-folder pair in the second case deliberately avoids Postgres: cases
share one project and run concurrently, so one case's seeds can otherwise answer
another case's question.
The MCP server builds its own tools against WorkflowService and never touched the
Instance AI adapter, so it carried the same gap: `search_folders` could resolve a
folder name to an id, and then nothing would accept that id.
- `search_workflows` gains `folderId`, `recursive` and `nodeTypes`, and every row
now carries `folder {id, name, path}` in the same `path` shape `search_folders`
returns, so the two are directly comparable.
- A `folderId` the caller cannot read aborts the search with an explanation
rather than returning the unscoped set, which is indistinguishable from the
folder being empty and would be presented as its contents.
- Subtree expansion goes through `findFolderSubtreesForUser`, so the access check
covers the descendants in the same query that finds them.
- `get_workflow_details` reports `folder {id, name}` alongside the existing bare
`parentFolderId`, which required a second lookup to be worth anything.
Folder reads are wired only when folders are licensed, matching how the folder
tools are already registered.
The dependency index already records a row per node type per workflow, kept
current on every save, and the workflow repository already resolves a
`nodeTypes` filter against it. No caller ever supplied one, so "which workflows
use Postgres" meant fetching every workflow and parsing its nodes, and "what is
this folder built out of" was not answerable at all.
- `workflows(list)` gains `nodeTypes`. It matches the node a workflow actually
contains, so a workflow merely NAMED for a service is correctly excluded and
one that uses it under an unrelated name is correctly found.
- `workflows(node-usage)` returns the node-type histogram for a scope, most-used
first, and composes with the folder arguments. Offered only where the adapter
can answer it, so the agent is never shown a summary it would get an error from.
- Counts DISTINCT workflows rather than node instances: three HTTP Request nodes
in one workflow are one user of the type, which is the question being asked.
- The scan is capped, and a capped scan says its counts are a lower bound —
silently truncating would read as a total, and these are numbers a caller
quotes back to the user.
- Gated separately from folder context. The two answer different questions
(where something lives vs what it is made of), and one switch could not tell
which of them moved a result.
Shared field descriptions are hoisted to constants because the schema sanitizer
requires a single description per field name across the action union.
Both adapter suites replace `@n8n/instance-ai` wholesale, so a new export the
adapter reads has to be added by hand or every listing test throws
`No "isFolderContextEnabled" export is defined on the mock`. Imported from the
real module rather than stubbed, so the tests exercise the shipped default.
Adds the `folders` tier to the "testing X, run Y" table and writes it up as a
worked example of a two-arm capability experiment: one binary switched by an env
gate, cases built so the wrong heuristic fails rather than accidentally
succeeds, and grading on a value only the correct path can produce.
Also records the idempotent-folder-seeding constraint, which is not obvious and
silently corrupts any run with more than one iteration.
A case could not previously state that a workflow lives in a folder, so the
nearest expressible thing was "workflows whose names share a prefix" — which is
the wrong signal the agent was already reading, and would have graded green on a
name guess.
- `seed.workflows[].folder` takes a slash path and creates it on restore.
Creation is IDEMPOTENT: unlike workflow names, folder names are not uniquified
per run (a case's prose names the folder the way a user would), so without
reuse five iterations leave five identically named folders and the folder under
test resolves as ambiguous. Folders are not rolled back with the workflows —
they are empty containers, and deleting one another restore just filled is the
more destructive of the two options.
- Four cases under a `folders` tier, each taken from a real thread: a search
workflow reported as missing while it sat in the named folder; a folder
inventory; a folder disambiguating two similarly named workflows; and the scope
of a bulk folder edit.
Each case is built so the wrong heuristic fails rather than accidentally
succeeds — in-folder names share no token with the folder, and a same-named
non-member sits outside it. The disambiguation case grades on a node name unique
to the in-folder workflow (and asserts absence of one unique to the decoy): node
names only reach context once a workflow is opened, which separates opening the
right one from listing both.
A listing could not say which folder a workflow was in, and `list` had no way
to ask for one. The only move available for "the workflows in my X folder" was
a substring guess on workflow NAMES — which both misses members named
differently and sweeps in non-members that share the name.
- `WorkflowSummary.folder` carries `{id, name, path}`. The repository's default
select already joined `parentFolder`; the adapter was discarding it. Paths are
batch-resolved per page through the existing recursive-CTE repository method,
so attribution costs one query for a whole page and nothing when a page has no
foldered workflows.
- `list` accepts `folderPath` / `folderId` / `recursive`. Resolution is staged
and strict — exact path, then exact folder name, then the last segment (a user
writing "personal/raglog" is naming the project, which is not part of a folder
path), then a segment-boundary suffix. It never partially matches.
- An unresolved folder is reported, never silently widened: the result carries
`folderResolution` and a note saying the rows are NOT that folder, listing the
folders that do exist, and refusing the name-filter fallback by name.
- Subtrees are expanded here rather than by the repository. Only the
workflows-and-folders query expands `parentFolderId`; this listing uses the
plain path, where it is an exact match. Expansion walks parent links rather
than path prefixes, because a folder name may itself contain a slash.
- `N8N_INSTANCE_AI_FOLDER_CONTEXT_ENABLED=false` removes the fields from the
advertised schema and suppresses attribution, so both arms of a measurement
run come off one binary.
The integration test runs the real queries against a real database: the folder
scan's column selection has to satisfy the ORDER BY that pagination generates,
and a mocked repository cannot see that constraint.
Three additions to the recipe card and the agent guide, all aimed at the moment a
developer stops running the existing cases and starts building their own.
The env inventory, grouped by what breaks without it. Required-or-nothing-runs is
the useful grouping: the module is off by default, the judges run on a different
key from the builder, and run-debug has to be set on the backend or every context
claim degrades to incomplete. Every var listed was checked against the code that
reads it, and the two tuning knobs carry their real defaults with the note that no
case ever reaches them.
The declarable surface, from WORKFLOW_TEST_CASE_KEYS rather than from memory, since
the schema is strict and an unknown key fails the load instead of being ignored.
Split by what it does — establishing state before the graded turn, versus grading —
because that is the decision an author is actually making.
What the harness cannot do, which is the half that saves the most time. Nine gaps
with their consequence and ticket where one exists: no window control so volume is
unaskable, binary grading with no partial credit, no cross-thread context, prior
runs cost real executions, and presence rather than attention — we can prove a fact
was in the window but not that the model used it, so "had it and ignored it" is
inferred by crossing verdicts rather than observed.
Two entry points for engineers who want to test a context-management idea against
this suite, next to the code so they stay true rather than in a doc that drifts.
CONTEXT-EVALS.md is the recipe card: a one-pager built around an "if you're
testing X, run Y, watch Z" table. It leads with the thing most likely to be got
wrong — that measurement is three numbers, not one. Someone who runs the suite,
reads a single aggregate pass rate and concludes nothing changed has mismeasured
their own work: the red baseline is where an improvement shows up, the guards are
where damage shows up, and cost is where a win hides when quality is already at
ceiling.
AGENTS.md is the fuller guide: what each capability does and why, what each case
family is FOR, the authoring rules, and links out to the Linear project, the PR
and the Notion team page. It also states which deficits can produce a red case at
all, so nobody spends a week authoring cases over context the agent already has.
Both say to branch off this branch rather than master, and that the two failing CI
checks are expected consequences of one large PR and are not to be worked around.
Every command in the recipe card was run before publishing. That caught one: the
case filter matches comma-separated substrings, not globs, so the pattern first
written for the red family selected zero cases and would have looked like a
passing suite.
Six prior runs, four of them failed on the same node, and the most recent one
succeeded. The user asks why the workflow has been flaky all week, so the answer
is only visible by inspecting more than the newest record.
Written expecting a failure, on the theory that the agent samples the head of the
execution list. The theory was wrong: 6/6 in both iterations. It lists all six
executions, then debugs two of the non-latest error runs, and never settles for
the successful newest one. Five tool calls.
Kept, with the description rewritten to say what it actually measures. Multi-record
aggregation is a capability the agent already has, so this is a regression guard
rather than a baseline: a context change that surfaces only the latest run, or
compresses history away, shows up here as a drop from 6/6. At quality ceiling the
second axis is spend, and the tool-call and token totals recorded here are what a
cheaper approach has to beat without losing the pattern.
The assertion stays precise about the mechanism. `failedNode` cannot appear if only
the latest execution was debugged, because the latest execution did not fail.
Two cases that can only be answered by reading a prior run, using the pre-turn
execution step. Both verified against a live instance.
The first: last night's run failed on the HTTP node with a rate limit, and the
user says only "it broke again". Nothing about the failure is in the request, so
asking the user which node broke or guessing from the workflow's shape both fail.
The second is the inverse and the more interesting half — the run SUCCEEDED but
the user asserts it failed. Accepting that premise means editing a healthy
workflow, which is worse than doing nothing.
They are deliberately paired. An agent that always reports success, or always
reports failure, passes one and fails the other; only reading the record passes
both.
Both context claims are anchored at turn end, because the agent learns what
happened by calling the executions tool during the turn.
Two things the first live run taught, now written into the case:
Seeding appends a ` [seed <8hex>]` uniquifier to workflow names, so the name a
case authors never matches the name on the instance. Every prior run failed at
seed time until the resolver stripped it. The loud refusal earned its keep — the
error printed both names side by side and the cause was obvious immediately.
Hints steer the failure's nature, not its literal response body. The first
version asserted a string from the JSON body the hints asked for; the mock
returned an equivalent rate-limit error worded differently, and the assertion
failed on a run that was otherwise perfect. It now asserts `failedNode`, a field
of the executions debug payload, which cannot come from the workflow definition —
so it still proves the record was read, without depending on mock wording.
Execution history was unreachable as a context surface. The harness only ever
executed a workflow after the build, so a turn could not ask about "the last
run" — there was no last run. A case can now declare `seed.priorRuns`, naming
seeded workflows to execute before the live turn, which leaves real execution
records in history for the agent to look up.
No new endpoint: this reuses the mocked-execution path the post-build scenarios
already use, so the only new code is orchestration.
Two decisions worth keeping:
A failed prior run is usually the point, not a fault. A case establishes that
last night's run died on the HTTP node, then says only "it broke again", and the
honest answer requires reading the record. So a red run is logged and returned
rather than thrown, and only genuine transient aborts are retried — re-running a
legitimately red workflow would waste minutes and could flip the state the case
is trying to establish.
Names resolve to ids through the input-order guarantee restoreThread already
documents, and a length mismatch refuses rather than guesses: running the wrong
workflow would seed the wrong history silently, which is worse than not running.
Unknown names are refused at parse time, where the message reads as a typo
instead of an infrastructure fault.
The name check lives on the case-level refine rather than the seed arm. A refine
on a discriminated-union member turns it into a ZodEffects, which is not a valid
member and collapses the inferred type across every consumer of the union.
Two stale-fact cases asserted that the agent's context records the live instance
state, and named only the observation block and the system prompt as places to
look. Both parts were wrong.
The tier list omitted the raw message window, which is where the fact actually
lands. Since compression does not run at default thresholds the observation block
is always empty, so the claim could only pass if compression happened to fire —
and it contradicted the rule that a fact counts when present in any of the three
tiers. The judge said as much while failing it: it found the workflow described
as an unpublished draft in the window, then failed the claim because the claim
told it not to look there.
The anchor was wrong too. The agent learns the live state by checking the
instance during the turn, so this is a retrieval claim rather than a retention
one; at the moment the request arrives the information does not exist yet. Both
now grade at turn end.
Also rewords the restore expectation on the node-removal case. It required the
agent to wait for the user to ask before putting back a step it had found
missing, which penalises noticing and fixing. What the case cares about is that
the user is told: restoring on its own initiative is fine, the step silently
reappearing as though it had been there all along is not.
The harness puts the agent a seeded history last targeted at the front of the
restored ids, because the first ref is the one an agent-anchored case is graded
and executed against, and seed-array order is an authoring artifact. The
justification is that the server binds a thread to the agent its history last
targeted.
Under a session boundary that premise is false. The history is restored into the
separate seed thread, so the binding lands there and the live thread continues
nothing — promoting on that basis asserts a link the live turn never had. The
promotion is now skipped when a boundary is active, leaving seed-declaration
order, which is at least an explicit authoring choice.
Not refused at the schema level: agents are instance-scoped and crossing the
boundary is the documented intent. What does not cross is the binding. The
harness warns instead, and the warning says what the code actually does — with
more than one seeded agent the author decides which is graded by listing it
first — rather than claiming the live turn continues none, which would overstate
what is enforced.
The ordering moves into a pure function next to activeSeedAgentId so it can be
tested directly; a review found the guard was reachable by no test, and
reverting it left the whole suite green. It now fails one.
Grading context at the moment the request arrived is right for asking whether a
fact survived from earlier: the agent restates facts as it works, so the end
state would let a request manufacture its own evidence. It is wrong for asking
whether the agent went and fetched something. Tool calls land on step two and
later while that snapshot is step one, so a retrieval claim graded there fails
every time regardless of whether retrieval worked — the anchor excludes the very
thing the claim is about.
Both claim kinds now carry an optional anchor, defaulting to the arrival moment,
so existing cases are unchanged. Claims are grouped by anchor and judged one
call per group, each with its own context block, rather than handing the judge
both and asking it to infer which moment a claim meant. Verdicts are returned in
the order the case wrote them.
The judge prompt no longer states a timing of its own; the block says which
moment it holds, in its opening section and in every heading, and the prompt
explains how to read an absence at each. Getting this wrong was silent, which is
the argument for naming it in the case rather than inferring it from wording.
Found by re-running the cross-workflow case, where one grader passed on "the
agent called workflows(action='get') on two workflow IDs" while the other failed
with "the raw message window contains no tool calls or results retrieving any
existing workflow's node details". Both were right. Its two retrieval claims are
anchored at turn end here, which also means its earlier classification flip from
context-ignored to retrieval-gap was an instrumentation artifact — and those two
point at opposite conclusions about where the bottleneck is.
Also drops the cutover-date assertion from the write-step probe. A date is
rewordable, the agent keeps it in prose while regenerating the ISO string, so
the literal measured phrasing rather than recall. The upsert key, which survives
verbatim, still carries one.
Four of the new tests fail when the anchor is ignored, confirmed by
reintroducing that.
Three more cases stating one preference in a prior session and probing for it
in a new one, taking the family to the four the ticket scopes: a routing
convention (gateway host), a naming convention (workflow name prefix), and a
standing review instruction (tag).
All three run 2/2 built at 33%, and every verdict lands where the design says:
the preference is absent from context, the outcome misses it, and the process
expectation passes because the agent behaves honestly without it. Each asserts
a verbatim-stable literal rather than a date or quantity, since exact matching
cannot see paraphrase.
They classify as retrieval-gap in all six iterations, as does the first case in
the family. That is the cell worth having: the rest of the suite has been
landing in context-ignored, where the fact was already in context and a memory
system would not move the failure. Here the fact is genuinely unreachable, the
agent does the best available thing, and the request still cannot be satisfied
— so these are the failures cross-session memory would actually convert.
Two authoring notes carried into the case bodies. The preference is chosen so
nothing in the graded request could produce the asserted value by coincidence,
which is why a pass can only come from the preference crossing the boundary.
And the review case deliberately does not ask for the workflow to go live: that
would pit a standing instruction against an explicit in-session one and make
the correct behaviour genuinely ambiguous.
The extractor bounds each tool payload at 4,000 characters so one large tool
result cannot crowd the interesting content out of the judge's attention. The
deterministic assertion path shared that bounded output while documenting itself
as searching untruncated context, so a value the model demonstrably received —
sitting past the cap inside a fetched workflow, execution or table schema —
was reported absent. Those payloads routinely exceed the cap, so this hit
exactly the retrieved content the check exists to verify.
The cap is now a `capPayloads` option: on for the judge, whose attention is the
scarce resource, off for assertions, which have no attention budget.
Also in this commit, both found by review of the same range:
- The memory-expectations prompt still told the judge its snapshot was the end
of the thread ("final state", "on its last step") alongside the newer text
saying it was taken as the request arrived. The block it actually receives is
headed "as the probe arrived", so it was reading correct content under
contradictory instructions. One stale comment in the extractor claimed the
same thing.
- A case declaring a session boundary runs a second thread for its seeded
history. That id never reached the build result, so cleanup deleted only the
live thread and every such iteration leaked the backend state the deletion
exists to reclaim. Both are deleted now, deduped rather than trusting the two
ids to differ, since a double delete would 404 and report the cleanup unclean.
Each fix carries a test confirmed by reintroducing the defect and watching it
fail: 3 for the cap, 2 for the leak. Adds direct coverage of the at-probe
snapshot selection, which had only been exercised indirectly.
The session-boundary arm generated a second thread id but never created the
thread, so every boundary case died at seeding with a 404: `restore-thread`
asserts access to an existing thread rather than creating one. The single-thread
path never hit this because the live thread is created by `ensureThread` before
the restore runs; the seeded thread had no equivalent.
Create it in the same project, so its artifacts stay visible from the graded
thread — that is the half of the boundary that is supposed to carry over.
Only an end-to-end run could catch this. The unit tests cover the flag, the
transcript labelling and the schema, none of which touch thread lifecycle.
Also adds the first boundary case, as a deliberate red baseline. In a prior
session the user states a standing preference; the graded turn runs in a new
session and asks for a workflow without restating it. Verified against a live
instance: the preference does not cross, the agent builds email alerts instead,
and the judge attributes it correctly — "the agent had no knowledge of the
previous session's standing preference". The process expectation passes, because
the agent asks where alerts should go rather than inventing a destination, which
is the honest behaviour when it has no memory to draw on.
Drops an assertion that was measuring the wrong thing. It checked whether a
seeded workflow's name reached the agent's context, and read as a control on
artifacts crossing the boundary — but reaching context requires the agent to call
the workflows tool, and nothing in this request asks it to. It could only ever
fail, and its failure said nothing about artifact scope. Reachability needs a case
whose request requires consulting siblings.
A seed restored prior messages into the same thread the graded turn ran in, so
"long-term memory" was really earlier turns of one conversation. That conflates
window eviction, compression and genuine cross-session recall, and nothing could
tell them apart.
`seed.sessionBoundary` restores the messages into their own thread and runs the
graded turn in a fresh one. Artifacts cross the boundary and the conversation does
not: seeded workflows, data tables and agents are instance-scoped so the agent's
tools still find them, while the messages are thread-scoped so nothing about the
prior session is recoverable from the conversation. That is the split the arm
exists to create.
Uses calls that already exist — restore takes the thread id, and a thread is
created implicitly — so there is no product change and no new endpoint.
Seeded turns from a prior session are labelled as such in the graded transcript.
Same-thread seeded turns stay deliberately unlabelled, because there it genuinely
is one conversation; across a boundary the judge would otherwise read them as
earlier turns here and penalise the agent for not remembering something it never
received. That trap is why the flag reaches the transcript rather than staying in
the harness.
Refused on a `replay` seed, which reconstructs one real thread and where a
boundary is incoherent — the strict arm rejects it instead of dropping the flag.
Worth stating plainly: on today's product every memory tier is thread-scoped, so a
boundary case expecting recall fails 100%. That is a deliberate red baseline rather
than a regression, and the guidance says to write expectations around the honest
fallback of asking rather than guessing.
The comparison report priced two arms against each other but said nothing about
whether the context layer delivered. Cost alone cannot tell a better context
system from a cheaper one that fetches nothing, which is the comparison the report
exists to support.
Read the per-iteration context/build classification the harness now persists, and
print a per-arm tally under each arm's cost summary, ordered so it reads as a
diagnosis: never retrieved, present but unused, context used, passed without it.
The last cell is there deliberately. A working approach moves iterations out of
`never retrieved`; it must not move them into `passed without it`, which is the
build succeeding while the context lacked the fact — something other than the
context system carried it, and crediting the system there would flatter it.
The field is optional, and absent renders nothing rather than an empty tally: a
results file recorded before the harness wrote the classification genuinely has
none, and "not measured" must not read as "measured, found nothing". A cell name
the report has not been taught to order is still printed rather than dropped, so a
future classification cannot go silently missing.
Each recall probe graded its planted facts only through the judge. Where a fact
has a literal form, a substring check answers the same question without a model
in the path, so add one per fact to three of the six cases.
Applied selectively, following the rule the last run taught: assert tokens that
survive verbatim, and leave anything a model will reword to the judge. Channel
names and table identifiers travel unchanged, so `#payments-eu`,
`#payments-weekly`, `orders_clean`, `orders_raw`, `sync_failures` and `EMEA` are
good claims. The two cases whose facts are purely numeric or temporal — a 14-day
cutoff, an 07:00 schedule, a 2% versus 20% threshold — get none, because an
assertion on those would report "not retained" every time the agent said the same
thing in different words.
Each competing-fact case asserts BOTH values, so the pair also covers
discrimination: the wanted value present is only half the claim when a
near-identical one sits beside it.
The date assertion on the deep case keeps its deliberately literal form, with the
note rewritten to say so. That fact was measured surviving as prose while the ISO
string was regenerated, so a failure there means the wording travelled and the
value did not — which is worth recording rather than tuning away.
Context claims were graded against the end-of-thread state, which let a probe
manufacture its own evidence. The agent restates facts while it works, and
whatever it restates lands back in the window the grader reads — so asking it to
apply a rule makes it narrate the rule, and the narration was then found and
scored as retention. That is why every depth-parameterised recall case sat at
100% even with compression firing on each turn.
Capture the context as the model held it at the start of its final run, before it
produced anything in the turn being graded, and grade both the judged
expectations and the deterministic assertions against that. The end state is
still captured, but only to tell "never had it" apart from "re-derived it while
answering" — a distinction that is invisible when you grade the end state alone.
A deterministic assertion that finds its value only in the end state now fails
and says which happened.
The snapshot is anchored structurally, on the first step of the last run with
steps, rather than by matching the probe text: a follow-up turn is paraphrased by
the user proxy, so the sent message need not match the authored one. Output from
earlier turns remains in the snapshot, which is correct — a fact that travelled
across turns was retained. What is excluded is the response being graded.
Verified against a live run, which immediately showed the two graders disagreeing
on one fact: the judged expectation passed on "March 2026 cutoff filter" while
the assertion on the exact ISO value reported it re-derived. Both were right —
the fact survived in prose and the literal string did not. Exact matching cannot
see paraphrase, so the authoring guidance now says to assert tokens that survive
verbatim (identifiers, keys, channel names) and leave rewordable values to the
judge, and to read a disagreement between the two as a finding rather than a bug.
A context verdict and a build verdict sat side by side for a human to cross. That
cross is where the actionable signal is, and four situations collapse into the
same pass rate:
context present + build right -> working
context absent + build right -> passed without it (a context system did nothing)
context present + build wrong -> had it and ignored it
context absent + build wrong -> never retrieved it
Only the last is a retrieval problem. The third is a prompting or attention
problem, and building retrieval to fix it would be wasted work — so reporting
both as "failed" hides which project to start.
Derive the classification per iteration from verdicts already persisted, and
surface it in `eval-results.json` and on each case card in the HTML report. Pure
arithmetic over existing data: no judge, no new eval run, and earlier runs can be
reclassified without re-running them.
Both axes take the strictest reading — context counts as present only if every
graded context claim passed, and the build as correct only if every graded build
claim passed. Cases carry several of each, and averaging would turn the
interesting cells into a shrug; the point is to name a situation, and "some
context arrived and some of the build was right" names nothing actionable. The
alternative, pairing each context claim to the build claim it backs, needs
case-authoring support that does not exist yet.
`incomplete` verdicts are excluded from both axes. They mean "not graded", so
counting them either way would manufacture a finding out of a missing capture,
and a case with no gradable claim on both axes reports unclassified rather than
working — otherwise a case with no context claim at all would read as evidence
that the context layer did its job.
Grading whether a value reached the model was LLM-judged, which puts a model on
the load-bearing path for the claims where a wrong verdict misleads most: if the
judge says a fact was absent, the conclusion is "the context system failed", and
that is an expensive thing to get wrong.
For a concrete value — a date, a channel, a column name, a parameter key — the
question is a substring search over data the harness already captures. Add
`contextAssertions`, checked without an LLM: it cannot hallucinate, costs nothing,
and needs no rubric. `mustAppear: false` covers the inverse, a stale value that
should have been dropped.
It also searches the untruncated context. The judge's view is capped so the part
worth grading stays in its attention, which means a value living in an elided
region reads as absent; this check has no such limit.
Matching is case-insensitive and callers are told to assert atomic values rather
than formatted phrases, because the same value is serialised with different
spacing and quoting depending on which context tier carries it.
Judged `memoryExpectations` stay for claims a string search cannot express — that
a retrieved sibling was the right *kind* of artifact, or that a value is present
but altered. Verdicts from both share `kind: "memory"`, so the report and results
file group them together, and both degrade to `incomplete` on the same missing
capture.
Nothing measured whether a new build fits the instance it lands on, so retrieval
over sibling workflows was unverifiable.
Seeds ten workflows sharing one deliberately legible house style — names prefixed
`[Ops] <Domain> — <Action>`, every schedule at 06:00, every outbound fetch with
retryOnFail and maxTries 3, every failure routed to a Slack node named
`Notify Ops` on #ops-alerts. The live turn asks for one more sync of the same
kind and states none of those conventions, so the only way to match them is to
read the siblings off the instance. The seeded conversation is a two-line
greeting carrying no convention detail, so nothing is recoverable from the
conversation itself.
The siblings are generated from one template rather than hand-written: a
convention that drifted across ten examples would be a broken case rather than a
finding, so identical structure is a correctness property here.
Two memory expectations grade retrieval rather than the build — that sibling node
detail actually reached the context, and that the retrieved precedent was a
same-kind sync rather than whatever was listed first. Naming is graded on the
prefix only, because restore appends ` [seed <hex>]` to every seeded workflow
name; the other three conventions are structural and unaffected.
Context-gathering tools return their findings as tool results, so a retrieval
system's whole output arrives that way. The context extractor rendered tool
interactions as a bare `[tool-result: name]` placeholder and dropped the payload,
which would have made the judge report retrieved facts as absent — reporting a
retrieval system as having delivered nothing precisely when it worked.
Render tool-call arguments and tool-result payloads in full, bounded per segment
so one large result cannot crowd out every other fetched fact, and tell the judge
that retrieved content counts exactly as content the user typed.
The window is now middle-truncated rather than head-truncated. Presence claims
point at both ends: a fact established early sits at the head, content just
fetched sits at the tail, so cutting either end would let a truncation read as
absence. The elision is labelled and the judge is told to call such evidence
inconclusive instead of concluding the fact was missing. Its budget doubles,
since the window is where fetched context lands.
Test fixtures use the real AI SDK message shape (`toolName` with `input`/`output`
parts) rather than pre-built segments, so they exercise the actual parser the
harness runs against instead of a shape the product never produces.
Two findings from running the new memory grading against a live instance, both
non-obvious enough to cost the next author real time.
Compression does not run by default. The Observer fires past 30K message tokens
and the Reflector past 40K observation tokens, and no case in the suite reaches
either — every observational-memory task logs `skipped`, so nothing is ever
evicted. Records the local override values observed to work, and notes that the
Tier-2 sliding window is not tunable at all: the env var the docs name for it is
not bound to anything.
Eviction is also not sufficient for a fact to be lost. The agent restates facts
while it works, and whatever it restates lands back in the raw message window the
judge reads — so a probe that asks the agent to narrate the rules it is applying
partly manufactures its own evidence. A fact planted twenty messages back, with
compression firing every turn, still passed for that reason. Depth measured in
messages is therefore a much weaker lever than it looks.
Also documents two stage-direction placement rules. A turn mixing user content
with a direction is read by the proxy as instruction and its content is never
sent, which surfaces as an expectation failing against a turn the agent never
received. And the opening turn reaches the builder verbatim, so a direction there
leaks proxy instructions into the agent's context.
Two deep needle cases bundled a whole-script sequencing direction into the same
turn as real user content. The user proxy read those turns as instruction and
never sent the refinement they carried, so expectations asserting that
refinement graded against a turn the agent was never given — a case defect
reported as a product miss.
Observed on a local run: four authored live turns, three delivered, the mixed one
silently absent from the transcript. The judge correctly reported that the
refinement "was never requested anywhere in the conversation".
Drop the direction rather than relocate it. There is nowhere safe to put it: the
opening turn is sent to the agent verbatim (`openingMessage =
conversation[0].text`, with no stage-direction stripping), so parking it there
would leak proxy instructions into the agent's context, and any other
content-bearing turn risks being swallowed the same way. It is also unnecessary —
the proxy delivers the remaining turns in order without it.
Final-turn directions are left alone. Those carry no content ahead of the
bracket and are delivered correctly today.
Pass/fail is the wrong primary metric for a context or memory change. An agent
with weaker memory can compensate by re-reading instance state — which for some
cases is the desired behaviour — so two approaches can tie on pass rate while
differing sharply in what they spent. That difference only shows up in tokens,
and the harness reported none.
Sum cache-aware token totals and a tool-call count per iteration into
`eval-results.json`. Nothing new needed collecting: run debug is already captured
per build and already reaches the run result, so this is extraction plus emission.
Cache reads are kept separate from uncached input rather than folded together.
They are an order of magnitude cheaper, and cache hit rate is usually the term a
context change moves most, so combining them would hide the effect.
`build-cost-report` gains a third cost source that prices these counts locally,
preferred over the existing LangSmith thread join. A two-arm quality-per-token
comparison now reads straight from two result files with no credentials, no
network and no trace-retention window; the join remains the fallback for runs
recorded before these fields existed. Rates default to Sonnet-class list prices
and are overridable per token class.
Tool calls count every non-narration step rather than only `kind: 'tool-call'`.
Several tools render as their own step kind and never fall through — a
`create-tasks` call becomes `plan`, an `ask-user` call becomes `ask-user` — so
the narrower filter dropped orchestration calls most multi-step conversations
make. This matches `seededTurnCounters`, which produces the per-turn count
reported in the same artifact, so the two numbers cannot disagree. Seeded turns
are excluded: restored fixture is not work the evaluated run did.
The new fields are omitted rather than emitted all-null when a run captured
nothing, since this file is a cross-repo contract and the report already treats
an all-null array as absent.
Fact coverage is skewed toward "don't lose a fact", and nearly all of it sits at
ceiling. Only two cases test the opposite failure — over-trusting a remembered
fact that is no longer true — and one of those is at 100%, so it guards rather
than measures. A persistent memory store makes that failure mode more likely,
not less, which leaves the risk side of any memory change essentially unmeasured.
Add four cases on the same "seed asserts X, live instance says not-X" template:
a credential deleted since the conversation referenced it, a data-table column
renamed, a workflow no longer published, and a node removed outside the
conversation.
Each distinguishes noticing and disclosing the divergence from silently
proceeding on the remembered state. Disclosure is graded as a process
expectation, since it is about what the agent told the user; the reconciliation
itself is a memory expectation, so a miss is attributable to memory-vs-state
reconciliation rather than to the build.
The node-removed case is deliberately built so the final workflow is identical
whether or not the agent disclosed, leaving process and memory as the only
discriminators.
Two harness limits shaped these and are recorded in the case descriptions:
restore always recreates workflows unpublished, so the inverse divergence is not
authorable; and seeded data-table names get a unique suffix that seeded history
is not rewritten to match, so the cases never grade the table name.
Scoped to a `memory` dataset, so they stay out of `full` until calibrated
against a real build.
Outcome-graded cases cannot distinguish "remembered the fact" from "recovered
without it": an agent can build correctly having forgotten a fact, by
re-deriving it or asking again. Measuring recall needs facts planted at known
depths and a terse probe that resolves only via fact k.
Only one case in the suite has that shape, with a single shallow needle. Add six
parameterised over depth (1, ~5, ~20 messages back) and distractors (with and
without a competing similar fact), each carrying one `memoryExpectation` per
needle so a miss is attributable to recall rather than to the build.
Every needle value lives only in the inline seed, never in the authored
conversation. The user proxy is shown the authored script but seeds its
transcript from the opening turn alone, so it is structurally incapable of
leaking a needle into the probe. The director notes also forbid it from auditing
or correcting the agent — a push-back would repair the very recall failure the
case exists to observe.
In the four deeper cases the fact-dependent step is deferred on the opening
turn, so the agent cannot answer the probe by reading back the workflow it
already built.
All six are within-thread. The cross-session variant is deliberately omitted:
inline seeds restore prior messages into the same thread and replay
reconstructs one thread from a trace, so neither crosses a session boundary.
Scoped to a `memory` dataset, so they stay out of `full` until calibrated
against a real build.
Eval cases had two grading surfaces: `processExpectations`, judged on the build
conversation, and `outcomeExpectations`, judged on the built workflow. Neither
observes the memory subsystem, so a memory comparison could only be measured
end-to-end — a score delta was not attributable to retrieval rather than to a
luckier build.
Add a third kind, `memoryExpectations`, graded against context state the harness
already captured and discarded: run debug is stashed on every build, and
`parseSystemPromptForDisplay` / `parseUsageSummary` already parse it. No product
change is needed.
It gets its own judge call with its own rubric, whose only inputs are the final
compressed observation block, the raw message window the model received, and the
final system prompt. The conversation judge is handed the full transcript, which
would let it satisfy "the agent still knew X" from the turn where X was first
said — the confound this kind exists to remove. The raw window is not the
transcript: evicted messages are absent from it, which is why it can be trusted
as a record of what the agent still had.
A fact counts as retained if it appears in any of the three tiers. The Observer
only compresses past a token threshold, so on a short thread the observation
block is legitimately empty while every fact sits in the raw window; grading on
the block alone would fail every memory expectation for a structural reason on
exactly the short threads the suite is mostly made of.
Verdicts carry `kind: "memory"` on every path — judged, ungraded and
judge-error — since the report badge and `eval-results.json` filter on it.
`stashBuildExpectations` previously early-returned once any expectation was
ungraded. With three kinds a case can legitimately have ungraded memory
expectations and gradable outcome ones at once, so the verdict sets are now
merged; the early return would have dropped the latter and shrunk the pass-rate
denominator. `stashRunDebug` now runs just before it at all three call sites, so
the memory judge can read the capture.
