Three pieces so the surfaces built earlier can actually be judged.
The `instance-awareness` skill teaches the ladder rather than the tools: ask
about intent, look up state; read cheapest-first and stop at the first rung that
answers the question; treat an absence in the node-usage histogram as evidence
rather than as a gap. It also says where each surface stops — the log records
that a save happened and the node count changed, never which nodes, so anything
about contents is a workflow read.
Four eval cases on a new `activity` tier, which also carries `memory` so they
run under both. Two are authored from real reviewed traces rather than invented:
lang-tracer observation 3870 and its siblings, where the agent restarted work it
had just finished and re-elicited requirements it already had. Their A/B
discriminator is a probe-anchored context assertion — with the feed off nothing
carries the prior workflow into the turn, so it can only arrive via a tool call,
which is turn-end and not probe. That distinction is the measurement.
Every case is scored on behaviour, never on narration. A probe that asks the
agent to describe its own context manufactures its own evidence, which is the
restatement confound recorded on this branch. So the checks ask whether it
re-elicited a known requirement, whether it named the broken workflow, whether
it picked the house node — not whether it can recite the block.
The fourth case is the negative control, and the one that should stop a rollout:
loud recent activity plus an unrelated request. It deliberately does not assert
that the recent workflow is absent from context — it will be there, the feed put
it there. What is scored is whether the agent acted on it. Having a fact and
using it are different, and that difference is the whole point.
scripts/ab-activity-eval.sh drives both arms. They differ by a flag on the
instance under test, not by a runner argument, so each arm needs its own start —
the flag is read once when the relay registers. The script says what to read and
in what order, with context-ignored first.
Also opens the activity preamble up. It forbade narrating the feed but never
granted permission to act on it, which left the agent with a list of work it was
implicitly told to ignore. It now says this is work you can pick up, that a
vague opener usually means the most recent thing, and points at the skill.
Two honest limits recorded in the README. `node-usage` is not gated by the flag,
so the preference case measures a capability both arms have and a difference
there is noise. And that case has no historical precedent: a sweep of 16,262
traces found no conversation where a user stated a node preference their own
workflows already showed.
Adds the cheap bottom rung of preference discovery. `workflows(action="list")`
filters on the workflow name only — its own description says so — so learning
what a project reaches for meant fetching every workflow and reading its nodes.
On ten workflows that is ~6,500 tokens; the same answer from the dependency
index is ~85.
The index was already there. `WorkflowIndexService` records a `nodeType` row per
node per workflow and keeps it current from the same workflow events the
activity relay listens to — one records that something changed, the other what
it now contains. Nothing exposed it to the agent.
Two reads, both project-scoped from the start rather than retrofitted:
countNodeTypesForProjects returns the histogram with the scope total, and
findWorkflowsByNodeType names the workflows using one, most recently updated
first so the freshest example is the one worth reading. Draft rows only —
what someone is building says more about preference than what happens to be
published — and archived workflows excluded, since those are what the project
used to do.
Surfaced as `workflows(action="node-usage")`. The description states what it
cannot do rather than leaving the agent to over-trust it: node types only, and
parameter-level house style still needs a get. The response carries a note that
a type absent from the list is used by no workflow in scope, so an absence reads
as evidence rather than as a gap in the answer.
The activity block gains one line pointing at it, because the feed answers what
happened and not what the user tends to build. A sentence, not an injected
digest — discovery stays on demand.
The repository returns finished rows rather than ids. The first cut had the
adapter resolving names with In(...), which trips
misplaced-n8n-typeorm-import; pushing the join down fixed the boundary and
removed two round-trips, since the query already joins workflow_entity.
Verified: two integration tests assert a query cannot reach a project it was not
given. The ordering test was checked by inverting the ORDER BY and confirming it
fails. Its first version backdated a row with raw `UPDATE ... $1`, which SQLite
rejects outright — TypeORM was rewriting it, so the assertion was riding on a
driver detail; replaced with a query builder update that is driver-agnostic and
bypasses the @BeforeUpdate hook.
Missing tokens no longer abort the run. The credential is still created and
still attached to every node that needs it, holding a placeholder — so the
workflows open without a missing-credential warning and nothing is left
half-wired. They just cannot run until a real token is supplied.
The placeholder is labelled in both places someone will look: the credential
name becomes "Anthropic (seed, fake key)", which is what shows in the
credential list and on every node, and the stored key is
SEEDED-FAKE-KEY-not-a-real-credential, so a 401 in a log explains itself rather
than raising a question about which key was used. The name is resolved once and
used for the row, the cached name on each node, and the report, so those three
cannot disagree.
The report ends with what to do about it: which token each placeholder needs,
where to generate it, and every place it can be set, each marked exists or
create it. Re-running with the token set replaces the credential in place and
keeps its id, so every node keeps pointing at it and the label drops away —
verified by seeding placeholders and then re-seeding with real tokens.
SEED_PLACEHOLDER_CREDENTIALS=1 forces placeholders even where a real token is
available, for producing a database with no real keys in it or for seeing the
estate as it looks before anything is configured.
`pnpm inspect:activity` serves every column of `activity_event` on
127.0.0.1:5699 — paginated, sortable on any column, with a free-text filter
that also searches the JSON `data` column, so a run id or a failing node name
finds its entry without knowing which field holds it. Category and action get
their own dropdowns, listing what is actually in the table.
Standalone rather than a view in the editor UI, on purpose. An in-app page
would need a route, a store, an API client, a scope-gated endpoint, i18n
strings and tests — a lot of product surface for a table you want to eyeball
while working on it. This is one file, no build step and no dependency, and it
reads the database directly so what you see is what is stored.
Read-only in three ways that matter: the connection is opened with readOnly, so
a write is rejected by SQLite rather than by convention; every query is a
SELECT; and the sort column comes from an allowlist, because a column name
cannot be a bound parameter. Verified — a sort of `1; DROP TABLE
activity_event` falls back to id and leaves the table intact.
Bound to 127.0.0.1 only, and unauthenticated by design: it is a debug surface,
and the whole table including who did what in which project is served to
anything that can reach the port. Confirmed unreachable on the LAN address.
Verified in a browser against a seeded database: filters, header sorting, page
size and paging all drive correctly, the free-text search finds both failing
runs by their failing node name, and the console is clean.
"this build has no activity_event table" misdiagnosed the common case. The
table is created by a migration, and n8n applies migrations on startup — so a
checkout that carries the migration paired with a database last used by a
branch that predates it is the normal situation, not a missing feature. The old
wording pointed at the code when the answer was the database.
Now the three cases are distinguished by looking at both the migration files in
the checkout and the migrations table in the database: the migration is absent
from this checkout, or it is present but unapplied here (with the fix — start
n8n once from this worktree), or it is recorded as applied while the table is
gone, which means something is wrong with the database rather than the setup.
`pnpm seed:account` builds n8n-core and runs the seed in one command, so the
build prerequisite stops being something you have to know.
Tokens now resolve from the environment or from a dotenv file this repo already
uses for local secrets — packages/cli/.env, packages/@n8n/instance-ai/.env, or a
root .env, in that order, with the shell always winning. Only the two token
variables are taken from those files, so nothing else in them leaks into the
process. A missing token is named with the credential it is for, where to
generate it, and every place it can be set, each marked exists or create it. The
run report says which source each token came from and how long it is, never its
value.
Also puts it where an agent will actually look: a section in the root AGENTS.md
saying what the seed contains, when to reach for it, and how it differs from
seedInstance.mjs. An empty instance is a bad place to test anything that reasons
about a user's work, and until now nothing in the docs an agent loads by default
said a seed existed.
Replaces the SQL seed with one script that writes six kinds of data in
dependency order: data tables, credentials, workflows, execution history,
Instance AI threads, and the activity log. Ten workflows now, two of them
reading the data tables.
SQL could not carry three of those. `execution_data.data` is flatted-encoded
rather than JSON, so a hand-written payload is unreadable. Data-table rows
live in a `data_table_user_<id>` table that n8n creates at runtime, so the DDL
has to be issued before the rows exist. And credential secrets are encrypted
with the instance key.
On that last point: the key can be pinned across a team, and n8n's ciphertext
is portable when it is — the salt travels inside the OpenSSL envelope, so any
instance with the same key string decrypts it. But a key shared next to its
ciphertext gives no confidentiality; committing an encrypted token beside the
key that opens it is a slower way of committing the token. And pinning is a
hard cutover, since n8n refuses to start when the config file and the env var
disagree. So no key is pinned: the script reads the key this machine already
has, takes the tokens from the environment, and encrypts locally. The script
is what gets shared. Token lengths are reported, never values.
Running the phases in one pass is what keeps the parts honest. Activity
entries cite the execution ids that were actually inserted, threads reference
workflows that exist, and the data-table workflows point at tables that were
really created — none of which survives being hand-maintained across six
tables.
The estate stays deliberately biased: every workflow uses the Anthropic chat
model and touches Linear, and for every job where n8n offers a choice exactly
one node is used everywhere. The Data Table node joins that table with no
alternative to pass over, since there is none. Still no Code node anywhere.
Verified against copies of a real instance database. All 68 nodes resolve to
node types that exist in this tree at the version given. Every execution
decodes through flatted with the run data, per-node statuses and failing node
intact, and both failures stop at the node that threw. Data-table rows read
back with the column types n8n's own DDL would have produced. All 12 thread
messages parse to the shape the agent memory writes. Nine activity entries
cite an execution and none of them dangles; no entry is left unscoped. Three
consecutive runs hold the counts steady with no duplicated credential blocks,
and a build without the activity table skips that phase with a printed note
rather than failing.
Eight small workflows written straight into the database, so a fresh account
has a plausible recent history rather than an empty list. Every workflow uses
the Anthropic chat model and touches Linear, and where n8n offers more than
one node for a job exactly one was chosen and used everywhere — If over
Switch, Filter over If-plus-NoOp, Loop Over Items over a Code loop, and no
Code node at all. The header records every choice and what it was chosen over,
because the consistency is the point: adding a workflow means following that
table rather than picking again.
SQL rather than the public API for two reasons the API cannot cover.
`activity_event` is written from the event bus and has no endpoint, so a
history can only be inserted. And the API stamps timestamps at request time,
which makes every workflow look a second old — the estate is spread over a
fortnight instead, which is what makes an activity feed worth reading.
Credentials are the thing SQL cannot do: their secrets are encrypted with the
instance key. Nodes are seeded unattached, with an optional trailing section
that links whichever Anthropic and Linear credentials it finds. That section
guards on EXISTS — without it a missing credential makes the sub-select NULL
and REPLACE would blank the whole nodes column.
Nothing lands active, no Linear team is chosen, and the one HTTP Request node
posts to example.invalid, so seeding cannot wake or call anything.
The activity section is last and self-contained: on a build without those
migrations its four statements fail with "no such table" and everything above
is already committed.
Verified by running it against copies of a real instance database: 8 workflows
and 8 sharing rows land, every nodes/connections blob round-trips through
JSON, all 54 nodes resolve to node types that exist in this tree at the
version given, the credential-linking section attaches ids without breaking
the JSON, the EXISTS guard leaves nodes intact when no credential is present,
repeated runs stay at 8 workflows and 24 activity entries, and a build with
the table seeds 24 entries with no unscoped rows.
Adds the `activity` tool. `list` reads further back than the window the agent
is handed, or filtered to one kind of event or one resource; `expand` opens a
single entry and returns everything else the log knows about the same
resource. That is the read the deferred `(resourceType, resourceId, id)` index
now exists for, added in its own migration rather than up front.
Expanding never fetches the live resource. `workflows`, `executions` and
`credentials` already do that, with their own permission checks — so an
expansion names the call to make instead, and the entry carries the id to make
it with. A missing id comes back as `notFound` rather than an error: entries
are pruned, so a stale id the agent read from an earlier block is an ordinary
outcome, not a failure.
Closes a scoping hole this feature opened. The window read had no project
filter, so a conversation not bound to a project would have been shown
activity from every project on the instance, including ones the user cannot
open. Reads are now bounded by the conversation's project, or by the user's
accessible projects when there is none, and an entry with no project is
excluded outright — there is no way to prove it is in scope. An expand outside
that scope is indistinguishable from a pruned id.
Keeping deletions visible under that rule needed the writer to change too. A
`workflow-deleted` event fires after the sharing rows that would name its
project are gone, so the project is recalled from the log's own earlier
entries for the same resource; credential updates and deletes do the same.
Without it the entries most worth surfacing would have been silently dropped
by the new filter.
The tool is always loaded when the log is enabled, and absent entirely when it
is not. The activity block names it, so deferring it would price every expand
at search_tools plus load_tool.
Reads the activity log, collapses it, and attaches it to the user's message as
a `<recent-activity>` block — the same per-turn context protocol as
`<editor-context>` and `<current-date-time>`, so `cleanStoredUserMessage`
strips it from the visible message on reload.
Never the system prompt: that is one prompt-cache entry shared by every thread
on the instance, which is why `projectId` is a presence flag there and why the
clock already lives on the per-turn input.
Collapsing folds repeated runs of one workflow into a counted entry wherever
they sit in the window, not only when adjacent — two schedules on different
intervals interleave, so adjacency-only folding leaves both expanded. A cap on
how many workflows may contribute runs keeps schedules from crowding out the
edits the user actually made.
A thread gets the window once and deltas after that, tracked by a high-water
mark in thread metadata. Re-sending the window would leave two overlapping
blocks in the conversation with the older one's counts already wrong — "ran
3x, all succeeded" above "ran 40x, 2 failed" for the same workflow is worse
than no second block. A delta says it is one, so a short list is not read as
the whole story.
The block is unit-verified, not run-verified: nothing here has yet been
observed reaching a live model. That is what the eval arms are for.
Adds an append-only `activity_event` table fed from the event bus, so the
instance agent can be handed what the user has been doing instead of spending
a turn discovering it. Entries are pointers — category, verb, and the ids
needed to fetch the full record — not copies of the resource.
Two decisions worth flagging:
`resourceId` carries no foreign key, unlike every other reference in the
schema. An entry has to outlive what it describes; a workflow deletion is the
entry most worth keeping, and a cascade would erase it exactly when it became
interesting. `workflow_review_activity.workflowId` made and undid this same
mistake. Neither vocabulary column gets a CHECK, for the reason that table
dropped its own: a feed covers more kinds with every feature, and each
widening is another SQLite table recreation.
The relay registers in `BaseCommand` rather than the instance-ai module.
`workflow-post-execute` is emitted wherever the execution ran, which in
scaling mode is the worker, and that module loads on main only — registering
there would silently lose every execution entry under scaling.
Off by default behind `N8N_INSTANCE_AI_ACTIVITY_LOG_ENABLED`; the flag is
read once at init, so a disabled instance carries no listeners. Pruning runs
on the leader main only, by age with an entry-count backstop.
Postgres schema docs are not regenerated — that leg needs Docker, which is
unavailable here. The sqlite docs are current.
Three additions to the recipe card and the agent guide, all aimed at the moment a
developer stops running the existing cases and starts building their own.
The env inventory, grouped by what breaks without it. Required-or-nothing-runs is
the useful grouping: the module is off by default, the judges run on a different
key from the builder, and run-debug has to be set on the backend or every context
claim degrades to incomplete. Every var listed was checked against the code that
reads it, and the two tuning knobs carry their real defaults with the note that no
case ever reaches them.
The declarable surface, from WORKFLOW_TEST_CASE_KEYS rather than from memory, since
the schema is strict and an unknown key fails the load instead of being ignored.
Split by what it does — establishing state before the graded turn, versus grading —
because that is the decision an author is actually making.
What the harness cannot do, which is the half that saves the most time. Nine gaps
with their consequence and ticket where one exists: no window control so volume is
unaskable, binary grading with no partial credit, no cross-thread context, prior
runs cost real executions, and presence rather than attention — we can prove a fact
was in the window but not that the model used it, so "had it and ignored it" is
inferred by crossing verdicts rather than observed.
Two entry points for engineers who want to test a context-management idea against
this suite, next to the code so they stay true rather than in a doc that drifts.
CONTEXT-EVALS.md is the recipe card: a one-pager built around an "if you're
testing X, run Y, watch Z" table. It leads with the thing most likely to be got
wrong — that measurement is three numbers, not one. Someone who runs the suite,
reads a single aggregate pass rate and concludes nothing changed has mismeasured
their own work: the red baseline is where an improvement shows up, the guards are
where damage shows up, and cost is where a win hides when quality is already at
ceiling.
AGENTS.md is the fuller guide: what each capability does and why, what each case
family is FOR, the authoring rules, and links out to the Linear project, the PR
and the Notion team page. It also states which deficits can produce a red case at
all, so nobody spends a week authoring cases over context the agent already has.
Both say to branch off this branch rather than master, and that the two failing CI
checks are expected consequences of one large PR and are not to be worked around.
Every command in the recipe card was run before publishing. That caught one: the
case filter matches comma-separated substrings, not globs, so the pattern first
written for the red family selected zero cases and would have looked like a
passing suite.
Six prior runs, four of them failed on the same node, and the most recent one
succeeded. The user asks why the workflow has been flaky all week, so the answer
is only visible by inspecting more than the newest record.
Written expecting a failure, on the theory that the agent samples the head of the
execution list. The theory was wrong: 6/6 in both iterations. It lists all six
executions, then debugs two of the non-latest error runs, and never settles for
the successful newest one. Five tool calls.
Kept, with the description rewritten to say what it actually measures. Multi-record
aggregation is a capability the agent already has, so this is a regression guard
rather than a baseline: a context change that surfaces only the latest run, or
compresses history away, shows up here as a drop from 6/6. At quality ceiling the
second axis is spend, and the tool-call and token totals recorded here are what a
cheaper approach has to beat without losing the pattern.
The assertion stays precise about the mechanism. `failedNode` cannot appear if only
the latest execution was debugged, because the latest execution did not fail.
Two cases that can only be answered by reading a prior run, using the pre-turn
execution step. Both verified against a live instance.
The first: last night's run failed on the HTTP node with a rate limit, and the
user says only "it broke again". Nothing about the failure is in the request, so
asking the user which node broke or guessing from the workflow's shape both fail.
The second is the inverse and the more interesting half — the run SUCCEEDED but
the user asserts it failed. Accepting that premise means editing a healthy
workflow, which is worse than doing nothing.
They are deliberately paired. An agent that always reports success, or always
reports failure, passes one and fails the other; only reading the record passes
both.
Both context claims are anchored at turn end, because the agent learns what
happened by calling the executions tool during the turn.
Two things the first live run taught, now written into the case:
Seeding appends a ` [seed <8hex>]` uniquifier to workflow names, so the name a
case authors never matches the name on the instance. Every prior run failed at
seed time until the resolver stripped it. The loud refusal earned its keep — the
error printed both names side by side and the cause was obvious immediately.
Hints steer the failure's nature, not its literal response body. The first
version asserted a string from the JSON body the hints asked for; the mock
returned an equivalent rate-limit error worded differently, and the assertion
failed on a run that was otherwise perfect. It now asserts `failedNode`, a field
of the executions debug payload, which cannot come from the workflow definition —
so it still proves the record was read, without depending on mock wording.
Execution history was unreachable as a context surface. The harness only ever
executed a workflow after the build, so a turn could not ask about "the last
run" — there was no last run. A case can now declare `seed.priorRuns`, naming
seeded workflows to execute before the live turn, which leaves real execution
records in history for the agent to look up.
No new endpoint: this reuses the mocked-execution path the post-build scenarios
already use, so the only new code is orchestration.
Two decisions worth keeping:
A failed prior run is usually the point, not a fault. A case establishes that
last night's run died on the HTTP node, then says only "it broke again", and the
honest answer requires reading the record. So a red run is logged and returned
rather than thrown, and only genuine transient aborts are retried — re-running a
legitimately red workflow would waste minutes and could flip the state the case
is trying to establish.
Names resolve to ids through the input-order guarantee restoreThread already
documents, and a length mismatch refuses rather than guesses: running the wrong
workflow would seed the wrong history silently, which is worse than not running.
Unknown names are refused at parse time, where the message reads as a typo
instead of an infrastructure fault.
The name check lives on the case-level refine rather than the seed arm. A refine
on a discriminated-union member turns it into a ZodEffects, which is not a valid
member and collapses the inferred type across every consumer of the union.
Two stale-fact cases asserted that the agent's context records the live instance
state, and named only the observation block and the system prompt as places to
look. Both parts were wrong.
The tier list omitted the raw message window, which is where the fact actually
lands. Since compression does not run at default thresholds the observation block
is always empty, so the claim could only pass if compression happened to fire —
and it contradicted the rule that a fact counts when present in any of the three
tiers. The judge said as much while failing it: it found the workflow described
as an unpublished draft in the window, then failed the claim because the claim
told it not to look there.
The anchor was wrong too. The agent learns the live state by checking the
instance during the turn, so this is a retrieval claim rather than a retention
one; at the moment the request arrives the information does not exist yet. Both
now grade at turn end.
Also rewords the restore expectation on the node-removal case. It required the
agent to wait for the user to ask before putting back a step it had found
missing, which penalises noticing and fixing. What the case cares about is that
the user is told: restoring on its own initiative is fine, the step silently
reappearing as though it had been there all along is not.
The harness puts the agent a seeded history last targeted at the front of the
restored ids, because the first ref is the one an agent-anchored case is graded
and executed against, and seed-array order is an authoring artifact. The
justification is that the server binds a thread to the agent its history last
targeted.
Under a session boundary that premise is false. The history is restored into the
separate seed thread, so the binding lands there and the live thread continues
nothing — promoting on that basis asserts a link the live turn never had. The
promotion is now skipped when a boundary is active, leaving seed-declaration
order, which is at least an explicit authoring choice.
Not refused at the schema level: agents are instance-scoped and crossing the
boundary is the documented intent. What does not cross is the binding. The
harness warns instead, and the warning says what the code actually does — with
more than one seeded agent the author decides which is graded by listing it
first — rather than claiming the live turn continues none, which would overstate
what is enforced.
The ordering moves into a pure function next to activeSeedAgentId so it can be
tested directly; a review found the guard was reachable by no test, and
reverting it left the whole suite green. It now fails one.
Grading context at the moment the request arrived is right for asking whether a
fact survived from earlier: the agent restates facts as it works, so the end
state would let a request manufacture its own evidence. It is wrong for asking
whether the agent went and fetched something. Tool calls land on step two and
later while that snapshot is step one, so a retrieval claim graded there fails
every time regardless of whether retrieval worked — the anchor excludes the very
thing the claim is about.
Both claim kinds now carry an optional anchor, defaulting to the arrival moment,
so existing cases are unchanged. Claims are grouped by anchor and judged one
call per group, each with its own context block, rather than handing the judge
both and asking it to infer which moment a claim meant. Verdicts are returned in
the order the case wrote them.
The judge prompt no longer states a timing of its own; the block says which
moment it holds, in its opening section and in every heading, and the prompt
explains how to read an absence at each. Getting this wrong was silent, which is
the argument for naming it in the case rather than inferring it from wording.
Found by re-running the cross-workflow case, where one grader passed on "the
agent called workflows(action='get') on two workflow IDs" while the other failed
with "the raw message window contains no tool calls or results retrieving any
existing workflow's node details". Both were right. Its two retrieval claims are
anchored at turn end here, which also means its earlier classification flip from
context-ignored to retrieval-gap was an instrumentation artifact — and those two
point at opposite conclusions about where the bottleneck is.
Also drops the cutover-date assertion from the write-step probe. A date is
rewordable, the agent keeps it in prose while regenerating the ISO string, so
the literal measured phrasing rather than recall. The upsert key, which survives
verbatim, still carries one.
Four of the new tests fail when the anchor is ignored, confirmed by
reintroducing that.
Three more cases stating one preference in a prior session and probing for it
in a new one, taking the family to the four the ticket scopes: a routing
convention (gateway host), a naming convention (workflow name prefix), and a
standing review instruction (tag).
All three run 2/2 built at 33%, and every verdict lands where the design says:
the preference is absent from context, the outcome misses it, and the process
expectation passes because the agent behaves honestly without it. Each asserts
a verbatim-stable literal rather than a date or quantity, since exact matching
cannot see paraphrase.
They classify as retrieval-gap in all six iterations, as does the first case in
the family. That is the cell worth having: the rest of the suite has been
landing in context-ignored, where the fact was already in context and a memory
system would not move the failure. Here the fact is genuinely unreachable, the
agent does the best available thing, and the request still cannot be satisfied
— so these are the failures cross-session memory would actually convert.
Two authoring notes carried into the case bodies. The preference is chosen so
nothing in the graded request could produce the asserted value by coincidence,
which is why a pass can only come from the preference crossing the boundary.
And the review case deliberately does not ask for the workflow to go live: that
would pit a standing instruction against an explicit in-session one and make
the correct behaviour genuinely ambiguous.
The extractor bounds each tool payload at 4,000 characters so one large tool
result cannot crowd the interesting content out of the judge's attention. The
deterministic assertion path shared that bounded output while documenting itself
as searching untruncated context, so a value the model demonstrably received —
sitting past the cap inside a fetched workflow, execution or table schema —
was reported absent. Those payloads routinely exceed the cap, so this hit
exactly the retrieved content the check exists to verify.
The cap is now a `capPayloads` option: on for the judge, whose attention is the
scarce resource, off for assertions, which have no attention budget.
Also in this commit, both found by review of the same range:
- The memory-expectations prompt still told the judge its snapshot was the end
of the thread ("final state", "on its last step") alongside the newer text
saying it was taken as the request arrived. The block it actually receives is
headed "as the probe arrived", so it was reading correct content under
contradictory instructions. One stale comment in the extractor claimed the
same thing.
- A case declaring a session boundary runs a second thread for its seeded
history. That id never reached the build result, so cleanup deleted only the
live thread and every such iteration leaked the backend state the deletion
exists to reclaim. Both are deleted now, deduped rather than trusting the two
ids to differ, since a double delete would 404 and report the cleanup unclean.
Each fix carries a test confirmed by reintroducing the defect and watching it
fail: 3 for the cap, 2 for the leak. Adds direct coverage of the at-probe
snapshot selection, which had only been exercised indirectly.
The session-boundary arm generated a second thread id but never created the
thread, so every boundary case died at seeding with a 404: `restore-thread`
asserts access to an existing thread rather than creating one. The single-thread
path never hit this because the live thread is created by `ensureThread` before
the restore runs; the seeded thread had no equivalent.
Create it in the same project, so its artifacts stay visible from the graded
thread — that is the half of the boundary that is supposed to carry over.
Only an end-to-end run could catch this. The unit tests cover the flag, the
transcript labelling and the schema, none of which touch thread lifecycle.
Also adds the first boundary case, as a deliberate red baseline. In a prior
session the user states a standing preference; the graded turn runs in a new
session and asks for a workflow without restating it. Verified against a live
instance: the preference does not cross, the agent builds email alerts instead,
and the judge attributes it correctly — "the agent had no knowledge of the
previous session's standing preference". The process expectation passes, because
the agent asks where alerts should go rather than inventing a destination, which
is the honest behaviour when it has no memory to draw on.
Drops an assertion that was measuring the wrong thing. It checked whether a
seeded workflow's name reached the agent's context, and read as a control on
artifacts crossing the boundary — but reaching context requires the agent to call
the workflows tool, and nothing in this request asks it to. It could only ever
fail, and its failure said nothing about artifact scope. Reachability needs a case
whose request requires consulting siblings.
A seed restored prior messages into the same thread the graded turn ran in, so
"long-term memory" was really earlier turns of one conversation. That conflates
window eviction, compression and genuine cross-session recall, and nothing could
tell them apart.
`seed.sessionBoundary` restores the messages into their own thread and runs the
graded turn in a fresh one. Artifacts cross the boundary and the conversation does
not: seeded workflows, data tables and agents are instance-scoped so the agent's
tools still find them, while the messages are thread-scoped so nothing about the
prior session is recoverable from the conversation. That is the split the arm
exists to create.
Uses calls that already exist — restore takes the thread id, and a thread is
created implicitly — so there is no product change and no new endpoint.
Seeded turns from a prior session are labelled as such in the graded transcript.
Same-thread seeded turns stay deliberately unlabelled, because there it genuinely
is one conversation; across a boundary the judge would otherwise read them as
earlier turns here and penalise the agent for not remembering something it never
received. That trap is why the flag reaches the transcript rather than staying in
the harness.
Refused on a `replay` seed, which reconstructs one real thread and where a
boundary is incoherent — the strict arm rejects it instead of dropping the flag.
Worth stating plainly: on today's product every memory tier is thread-scoped, so a
boundary case expecting recall fails 100%. That is a deliberate red baseline rather
than a regression, and the guidance says to write expectations around the honest
fallback of asking rather than guessing.
The comparison report priced two arms against each other but said nothing about
whether the context layer delivered. Cost alone cannot tell a better context
system from a cheaper one that fetches nothing, which is the comparison the report
exists to support.
Read the per-iteration context/build classification the harness now persists, and
print a per-arm tally under each arm's cost summary, ordered so it reads as a
diagnosis: never retrieved, present but unused, context used, passed without it.
The last cell is there deliberately. A working approach moves iterations out of
`never retrieved`; it must not move them into `passed without it`, which is the
build succeeding while the context lacked the fact — something other than the
context system carried it, and crediting the system there would flatter it.
The field is optional, and absent renders nothing rather than an empty tally: a
results file recorded before the harness wrote the classification genuinely has
none, and "not measured" must not read as "measured, found nothing". A cell name
the report has not been taught to order is still printed rather than dropped, so a
future classification cannot go silently missing.
Each recall probe graded its planted facts only through the judge. Where a fact
has a literal form, a substring check answers the same question without a model
in the path, so add one per fact to three of the six cases.
Applied selectively, following the rule the last run taught: assert tokens that
survive verbatim, and leave anything a model will reword to the judge. Channel
names and table identifiers travel unchanged, so `#payments-eu`,
`#payments-weekly`, `orders_clean`, `orders_raw`, `sync_failures` and `EMEA` are
good claims. The two cases whose facts are purely numeric or temporal — a 14-day
cutoff, an 07:00 schedule, a 2% versus 20% threshold — get none, because an
assertion on those would report "not retained" every time the agent said the same
thing in different words.
Each competing-fact case asserts BOTH values, so the pair also covers
discrimination: the wanted value present is only half the claim when a
near-identical one sits beside it.
The date assertion on the deep case keeps its deliberately literal form, with the
note rewritten to say so. That fact was measured surviving as prose while the ISO
string was regenerated, so a failure there means the wording travelled and the
value did not — which is worth recording rather than tuning away.
Context claims were graded against the end-of-thread state, which let a probe
manufacture its own evidence. The agent restates facts while it works, and
whatever it restates lands back in the window the grader reads — so asking it to
apply a rule makes it narrate the rule, and the narration was then found and
scored as retention. That is why every depth-parameterised recall case sat at
100% even with compression firing on each turn.
Capture the context as the model held it at the start of its final run, before it
produced anything in the turn being graded, and grade both the judged
expectations and the deterministic assertions against that. The end state is
still captured, but only to tell "never had it" apart from "re-derived it while
answering" — a distinction that is invisible when you grade the end state alone.
A deterministic assertion that finds its value only in the end state now fails
and says which happened.
The snapshot is anchored structurally, on the first step of the last run with
steps, rather than by matching the probe text: a follow-up turn is paraphrased by
the user proxy, so the sent message need not match the authored one. Output from
earlier turns remains in the snapshot, which is correct — a fact that travelled
across turns was retained. What is excluded is the response being graded.
Verified against a live run, which immediately showed the two graders disagreeing
on one fact: the judged expectation passed on "March 2026 cutoff filter" while
the assertion on the exact ISO value reported it re-derived. Both were right —
the fact survived in prose and the literal string did not. Exact matching cannot
see paraphrase, so the authoring guidance now says to assert tokens that survive
verbatim (identifiers, keys, channel names) and leave rewordable values to the
judge, and to read a disagreement between the two as a finding rather than a bug.
A context verdict and a build verdict sat side by side for a human to cross. That
cross is where the actionable signal is, and four situations collapse into the
same pass rate:
context present + build right -> working
context absent + build right -> passed without it (a context system did nothing)
context present + build wrong -> had it and ignored it
context absent + build wrong -> never retrieved it
Only the last is a retrieval problem. The third is a prompting or attention
problem, and building retrieval to fix it would be wasted work — so reporting
both as "failed" hides which project to start.
Derive the classification per iteration from verdicts already persisted, and
surface it in `eval-results.json` and on each case card in the HTML report. Pure
arithmetic over existing data: no judge, no new eval run, and earlier runs can be
reclassified without re-running them.
Both axes take the strictest reading — context counts as present only if every
graded context claim passed, and the build as correct only if every graded build
claim passed. Cases carry several of each, and averaging would turn the
interesting cells into a shrug; the point is to name a situation, and "some
context arrived and some of the build was right" names nothing actionable. The
alternative, pairing each context claim to the build claim it backs, needs
case-authoring support that does not exist yet.
`incomplete` verdicts are excluded from both axes. They mean "not graded", so
counting them either way would manufacture a finding out of a missing capture,
and a case with no gradable claim on both axes reports unclassified rather than
working — otherwise a case with no context claim at all would read as evidence
that the context layer did its job.
Grading whether a value reached the model was LLM-judged, which puts a model on
the load-bearing path for the claims where a wrong verdict misleads most: if the
judge says a fact was absent, the conclusion is "the context system failed", and
that is an expensive thing to get wrong.
For a concrete value — a date, a channel, a column name, a parameter key — the
question is a substring search over data the harness already captures. Add
`contextAssertions`, checked without an LLM: it cannot hallucinate, costs nothing,
and needs no rubric. `mustAppear: false` covers the inverse, a stale value that
should have been dropped.
It also searches the untruncated context. The judge's view is capped so the part
worth grading stays in its attention, which means a value living in an elided
region reads as absent; this check has no such limit.
Matching is case-insensitive and callers are told to assert atomic values rather
than formatted phrases, because the same value is serialised with different
spacing and quoting depending on which context tier carries it.
Judged `memoryExpectations` stay for claims a string search cannot express — that
a retrieved sibling was the right *kind* of artifact, or that a value is present
but altered. Verdicts from both share `kind: "memory"`, so the report and results
file group them together, and both degrade to `incomplete` on the same missing
capture.
Nothing measured whether a new build fits the instance it lands on, so retrieval
over sibling workflows was unverifiable.
Seeds ten workflows sharing one deliberately legible house style — names prefixed
`[Ops] <Domain> — <Action>`, every schedule at 06:00, every outbound fetch with
retryOnFail and maxTries 3, every failure routed to a Slack node named
`Notify Ops` on #ops-alerts. The live turn asks for one more sync of the same
kind and states none of those conventions, so the only way to match them is to
read the siblings off the instance. The seeded conversation is a two-line
greeting carrying no convention detail, so nothing is recoverable from the
conversation itself.
The siblings are generated from one template rather than hand-written: a
convention that drifted across ten examples would be a broken case rather than a
finding, so identical structure is a correctness property here.
Two memory expectations grade retrieval rather than the build — that sibling node
detail actually reached the context, and that the retrieved precedent was a
same-kind sync rather than whatever was listed first. Naming is graded on the
prefix only, because restore appends ` [seed <hex>]` to every seeded workflow
name; the other three conventions are structural and unaffected.
Context-gathering tools return their findings as tool results, so a retrieval
system's whole output arrives that way. The context extractor rendered tool
interactions as a bare `[tool-result: name]` placeholder and dropped the payload,
which would have made the judge report retrieved facts as absent — reporting a
retrieval system as having delivered nothing precisely when it worked.
Render tool-call arguments and tool-result payloads in full, bounded per segment
so one large result cannot crowd out every other fetched fact, and tell the judge
that retrieved content counts exactly as content the user typed.
The window is now middle-truncated rather than head-truncated. Presence claims
point at both ends: a fact established early sits at the head, content just
fetched sits at the tail, so cutting either end would let a truncation read as
absence. The elision is labelled and the judge is told to call such evidence
inconclusive instead of concluding the fact was missing. Its budget doubles,
since the window is where fetched context lands.
Test fixtures use the real AI SDK message shape (`toolName` with `input`/`output`
parts) rather than pre-built segments, so they exercise the actual parser the
harness runs against instead of a shape the product never produces.
Two findings from running the new memory grading against a live instance, both
non-obvious enough to cost the next author real time.
Compression does not run by default. The Observer fires past 30K message tokens
and the Reflector past 40K observation tokens, and no case in the suite reaches
either — every observational-memory task logs `skipped`, so nothing is ever
evicted. Records the local override values observed to work, and notes that the
Tier-2 sliding window is not tunable at all: the env var the docs name for it is
not bound to anything.
Eviction is also not sufficient for a fact to be lost. The agent restates facts
while it works, and whatever it restates lands back in the raw message window the
judge reads — so a probe that asks the agent to narrate the rules it is applying
partly manufactures its own evidence. A fact planted twenty messages back, with
compression firing every turn, still passed for that reason. Depth measured in
messages is therefore a much weaker lever than it looks.
Also documents two stage-direction placement rules. A turn mixing user content
with a direction is read by the proxy as instruction and its content is never
sent, which surfaces as an expectation failing against a turn the agent never
received. And the opening turn reaches the builder verbatim, so a direction there
leaks proxy instructions into the agent's context.
Two deep needle cases bundled a whole-script sequencing direction into the same
turn as real user content. The user proxy read those turns as instruction and
never sent the refinement they carried, so expectations asserting that
refinement graded against a turn the agent was never given — a case defect
reported as a product miss.
Observed on a local run: four authored live turns, three delivered, the mixed one
silently absent from the transcript. The judge correctly reported that the
refinement "was never requested anywhere in the conversation".
Drop the direction rather than relocate it. There is nowhere safe to put it: the
opening turn is sent to the agent verbatim (`openingMessage =
conversation[0].text`, with no stage-direction stripping), so parking it there
would leak proxy instructions into the agent's context, and any other
content-bearing turn risks being swallowed the same way. It is also unnecessary —
the proxy delivers the remaining turns in order without it.
Final-turn directions are left alone. Those carry no content ahead of the
bracket and are delivered correctly today.
Pass/fail is the wrong primary metric for a context or memory change. An agent
with weaker memory can compensate by re-reading instance state — which for some
cases is the desired behaviour — so two approaches can tie on pass rate while
differing sharply in what they spent. That difference only shows up in tokens,
and the harness reported none.
Sum cache-aware token totals and a tool-call count per iteration into
`eval-results.json`. Nothing new needed collecting: run debug is already captured
per build and already reaches the run result, so this is extraction plus emission.
Cache reads are kept separate from uncached input rather than folded together.
They are an order of magnitude cheaper, and cache hit rate is usually the term a
context change moves most, so combining them would hide the effect.
`build-cost-report` gains a third cost source that prices these counts locally,
preferred over the existing LangSmith thread join. A two-arm quality-per-token
comparison now reads straight from two result files with no credentials, no
network and no trace-retention window; the join remains the fallback for runs
recorded before these fields existed. Rates default to Sonnet-class list prices
and are overridable per token class.
Tool calls count every non-narration step rather than only `kind: 'tool-call'`.
Several tools render as their own step kind and never fall through — a
`create-tasks` call becomes `plan`, an `ask-user` call becomes `ask-user` — so
the narrower filter dropped orchestration calls most multi-step conversations
make. This matches `seededTurnCounters`, which produces the per-turn count
reported in the same artifact, so the two numbers cannot disagree. Seeded turns
are excluded: restored fixture is not work the evaluated run did.
The new fields are omitted rather than emitted all-null when a run captured
nothing, since this file is a cross-repo contract and the report already treats
an all-null array as absent.
Fact coverage is skewed toward "don't lose a fact", and nearly all of it sits at
ceiling. Only two cases test the opposite failure — over-trusting a remembered
fact that is no longer true — and one of those is at 100%, so it guards rather
than measures. A persistent memory store makes that failure mode more likely,
not less, which leaves the risk side of any memory change essentially unmeasured.
Add four cases on the same "seed asserts X, live instance says not-X" template:
a credential deleted since the conversation referenced it, a data-table column
renamed, a workflow no longer published, and a node removed outside the
conversation.
Each distinguishes noticing and disclosing the divergence from silently
proceeding on the remembered state. Disclosure is graded as a process
expectation, since it is about what the agent told the user; the reconciliation
itself is a memory expectation, so a miss is attributable to memory-vs-state
reconciliation rather than to the build.
The node-removed case is deliberately built so the final workflow is identical
whether or not the agent disclosed, leaving process and memory as the only
discriminators.
Two harness limits shaped these and are recorded in the case descriptions:
restore always recreates workflows unpublished, so the inverse divergence is not
authorable; and seeded data-table names get a unique suffix that seeded history
is not rewritten to match, so the cases never grade the table name.
Scoped to a `memory` dataset, so they stay out of `full` until calibrated
against a real build.
Outcome-graded cases cannot distinguish "remembered the fact" from "recovered
without it": an agent can build correctly having forgotten a fact, by
re-deriving it or asking again. Measuring recall needs facts planted at known
depths and a terse probe that resolves only via fact k.
Only one case in the suite has that shape, with a single shallow needle. Add six
parameterised over depth (1, ~5, ~20 messages back) and distractors (with and
without a competing similar fact), each carrying one `memoryExpectation` per
needle so a miss is attributable to recall rather than to the build.
Every needle value lives only in the inline seed, never in the authored
conversation. The user proxy is shown the authored script but seeds its
transcript from the opening turn alone, so it is structurally incapable of
leaking a needle into the probe. The director notes also forbid it from auditing
or correcting the agent — a push-back would repair the very recall failure the
case exists to observe.
In the four deeper cases the fact-dependent step is deferred on the opening
turn, so the agent cannot answer the probe by reading back the workflow it
already built.
All six are within-thread. The cross-session variant is deliberately omitted:
inline seeds restore prior messages into the same thread and replay
reconstructs one thread from a trace, so neither crosses a session boundary.
Scoped to a `memory` dataset, so they stay out of `full` until calibrated
against a real build.
Eval cases had two grading surfaces: `processExpectations`, judged on the build
conversation, and `outcomeExpectations`, judged on the built workflow. Neither
observes the memory subsystem, so a memory comparison could only be measured
end-to-end — a score delta was not attributable to retrieval rather than to a
luckier build.
Add a third kind, `memoryExpectations`, graded against context state the harness
already captured and discarded: run debug is stashed on every build, and
`parseSystemPromptForDisplay` / `parseUsageSummary` already parse it. No product
change is needed.
It gets its own judge call with its own rubric, whose only inputs are the final
compressed observation block, the raw message window the model received, and the
final system prompt. The conversation judge is handed the full transcript, which
would let it satisfy "the agent still knew X" from the turn where X was first
said — the confound this kind exists to remove. The raw window is not the
transcript: evicted messages are absent from it, which is why it can be trusted
as a record of what the agent still had.
A fact counts as retained if it appears in any of the three tiers. The Observer
only compresses past a token threshold, so on a short thread the observation
block is legitimately empty while every fact sits in the raw window; grading on
the block alone would fail every memory expectation for a structural reason on
exactly the short threads the suite is mostly made of.
Verdicts carry `kind: "memory"` on every path — judged, ungraded and
judge-error — since the report badge and `eval-results.json` filter on it.
`stashBuildExpectations` previously early-returned once any expectation was
ungraded. With three kinds a case can legitimately have ungraded memory
expectations and gradable outcome ones at once, so the verdict sets are now
merged; the early return would have dropped the latter and shrunk the pass-rate
denominator. `stashRunDebug` now runs just before it at all three call sites, so
the memory judge can read the capture.
