mirror of
https://github.com/n8n-io/n8n.git
synced 2026-10-22 20:41:50 +02:00
30b4d7946c
Three pieces so the surfaces built earlier can actually be judged. The `instance-awareness` skill teaches the ladder rather than the tools: ask about intent, look up state; read cheapest-first and stop at the first rung that answers the question; treat an absence in the node-usage histogram as evidence rather than as a gap. It also says where each surface stops — the log records that a save happened and the node count changed, never which nodes, so anything about contents is a workflow read. Four eval cases on a new `activity` tier, which also carries `memory` so they run under both. Two are authored from real reviewed traces rather than invented: lang-tracer observation 3870 and its siblings, where the agent restarted work it had just finished and re-elicited requirements it already had. Their A/B discriminator is a probe-anchored context assertion — with the feed off nothing carries the prior workflow into the turn, so it can only arrive via a tool call, which is turn-end and not probe. That distinction is the measurement. Every case is scored on behaviour, never on narration. A probe that asks the agent to describe its own context manufactures its own evidence, which is the restatement confound recorded on this branch. So the checks ask whether it re-elicited a known requirement, whether it named the broken workflow, whether it picked the house node — not whether it can recite the block. The fourth case is the negative control, and the one that should stop a rollout: loud recent activity plus an unrelated request. It deliberately does not assert that the recent workflow is absent from context — it will be there, the feed put it there. What is scored is whether the agent acted on it. Having a fact and using it are different, and that difference is the whole point. scripts/ab-activity-eval.sh drives both arms. They differ by a flag on the instance under test, not by a runner argument, so each arm needs its own start — the flag is read once when the relay registers. The script says what to read and in what order, with context-ignored first. Also opens the activity preamble up. It forbade narrating the feed but never granted permission to act on it, which left the agent with a list of work it was implicitly told to ignore. It now says this is work you can pick up, that a vague opener usually means the most recent thing, and points at the skill. Two honest limits recorded in the README. `node-usage` is not gated by the flag, so the preference case measures a capability both arms have and a difference there is noise. And that case has no historical precedent: a sweep of 16,262 traces found no conversation where a user stated a node preference their own workflows already showed. |
||
|---|---|---|
| .. | ||
| agents | ||
| ai-node-sdk | ||
| ai-utilities | ||
| ai-workflow-builder.ee | ||
| api-types | ||
| backend-common | ||
| backend-network | ||
| backend-test-utils | ||
| benchmark | ||
| blob-storage | ||
| chat-hub | ||
| cli | ||
| client-oauth2 | ||
| codemirror-lang | ||
| codemirror-lang-html | ||
| codemirror-lang-sql | ||
| computer-use | ||
| config | ||
| constants | ||
| crdt | ||
| create-node | ||
| db | ||
| decorators | ||
| di | ||
| engine | ||
| errors | ||
| eslint-config | ||
| eslint-plugin-community-nodes | ||
| expression-runtime | ||
| extension-sdk | ||
| imap | ||
| instance-ai | ||
| json-schema-to-zod | ||
| local-gateway | ||
| mcp-apps | ||
| mcp-browser | ||
| mcp-browser-extension | ||
| node-cli | ||
| node-engine-compatibility | ||
| nodes-langchain | ||
| permissions | ||
| scan-community-package | ||
| scheduler | ||
| stylelint-config | ||
| syslog-client | ||
| task-runner | ||
| task-runner-python | ||
| telemetry | ||
| tournament | ||
| typeorm | ||
| typescript-config | ||
| utils | ||
| vitest-config | ||
| workflow-sdk | ||