mirror of
https://github.com/n8n-io/n8n.git
synced 2026-10-23 04:50:43 +02:00
30b4d7946c
Three pieces so the surfaces built earlier can actually be judged. The `instance-awareness` skill teaches the ladder rather than the tools: ask about intent, look up state; read cheapest-first and stop at the first rung that answers the question; treat an absence in the node-usage histogram as evidence rather than as a gap. It also says where each surface stops — the log records that a save happened and the node count changed, never which nodes, so anything about contents is a workflow read. Four eval cases on a new `activity` tier, which also carries `memory` so they run under both. Two are authored from real reviewed traces rather than invented: lang-tracer observation 3870 and its siblings, where the agent restarted work it had just finished and re-elicited requirements it already had. Their A/B discriminator is a probe-anchored context assertion — with the feed off nothing carries the prior workflow into the turn, so it can only arrive via a tool call, which is turn-end and not probe. That distinction is the measurement. Every case is scored on behaviour, never on narration. A probe that asks the agent to describe its own context manufactures its own evidence, which is the restatement confound recorded on this branch. So the checks ask whether it re-elicited a known requirement, whether it named the broken workflow, whether it picked the house node — not whether it can recite the block. The fourth case is the negative control, and the one that should stop a rollout: loud recent activity plus an unrelated request. It deliberately does not assert that the recent workflow is absent from context — it will be there, the feed put it there. What is scored is whether the agent acted on it. Having a fact and using it are different, and that difference is the whole point. scripts/ab-activity-eval.sh drives both arms. They differ by a flag on the instance under test, not by a runner argument, so each arm needs its own start — the flag is read once when the relay registers. The script says what to read and in what order, with context-ignored first. Also opens the activity preamble up. It forbade narrating the feed but never granted permission to act on it, which left the agent with a list of work it was implicitly told to ignore. It now says this is work you can pick up, that a vague opener usually means the most recent thing, and points at the skill. Two honest limits recorded in the README. `node-usage` is not gated by the flag, so the preference case measures a capability both arms have and a difference there is noise. And that case has no historical precedent: a sweep of 16,262 traces found no conversation where a user stated a node preference their own workflows already showed. |
||
|---|---|---|
| .. | ||
| ai-gateway-prefers-covered-scraping-node.json | ||
| ai-gateway-prefers-supported-service-node.json | ||
| ai-gateway-respects-named-web-search-tool.json | ||
| applies-the-house-naming-prefix-across-a-session-boundary.json | ||
| carries-a-standing-preference-across-a-session-boundary.json | ||
| checks-the-record-before-accepting-a-false-premise.json | ||
| credential-setup-anthropic-browser.json | ||
| diagnoses-the-last-run-instead-of-asking.json | ||
| discloses-deleted-credential-on-next-edit.json | ||
| discloses-node-removed-outside-conversation.json | ||
| discloses-renamed-data-table-column.json | ||
| discloses-workflow-no-longer-active.json | ||
| does-not-drag-recent-activity-into-an-unrelated-ask.json | ||
| finds-the-failure-pattern-across-several-runs.json | ||
| follows-the-house-chat-model-without-being-told.json | ||
| honors-missing-credential-field-correction.json | ||
| index.ts | ||
| matches-instance-conventions-when-adding-a-sibling-sync.json | ||
| picks-up-the-workflow-from-a-previous-session.json | ||
| recalls-agreed-reminder-rules-from-immediately-prior-turn.json | ||
| recalls-alert-channel-when-two-channels-were-agreed.json | ||
| recalls-paging-threshold-among-two-similar-percentages.json | ||
| recalls-report-source-table-among-two-similar-tables.json | ||
| recalls-retry-and-region-rules-set-several-turns-earlier.json | ||
| recalls-write-step-rules-locked-in-at-thread-start.json | ||
| routes-through-the-house-gateway-across-a-session-boundary.json | ||
| surfaces-the-failed-run-on-an-open-opener.json | ||
| tags-work-for-review-across-a-session-boundary.json | ||