mirror of
https://github.com/n8n-io/n8n.git
synced 2026-10-19 16:55:02 +02:00
30b4d7946c
Three pieces so the surfaces built earlier can actually be judged. The `instance-awareness` skill teaches the ladder rather than the tools: ask about intent, look up state; read cheapest-first and stop at the first rung that answers the question; treat an absence in the node-usage histogram as evidence rather than as a gap. It also says where each surface stops — the log records that a save happened and the node count changed, never which nodes, so anything about contents is a workflow read. Four eval cases on a new `activity` tier, which also carries `memory` so they run under both. Two are authored from real reviewed traces rather than invented: lang-tracer observation 3870 and its siblings, where the agent restarted work it had just finished and re-elicited requirements it already had. Their A/B discriminator is a probe-anchored context assertion — with the feed off nothing carries the prior workflow into the turn, so it can only arrive via a tool call, which is turn-end and not probe. That distinction is the measurement. Every case is scored on behaviour, never on narration. A probe that asks the agent to describe its own context manufactures its own evidence, which is the restatement confound recorded on this branch. So the checks ask whether it re-elicited a known requirement, whether it named the broken workflow, whether it picked the house node — not whether it can recite the block. The fourth case is the negative control, and the one that should stop a rollout: loud recent activity plus an unrelated request. It deliberately does not assert that the recent workflow is absent from context — it will be there, the feed put it there. What is scored is whether the agent acted on it. Having a fact and using it are different, and that difference is the whole point. scripts/ab-activity-eval.sh drives both arms. They differ by a flag on the instance under test, not by a runner argument, so each arm needs its own start — the flag is read once when the relay registers. The script says what to read and in what order, with context-ignored first. Also opens the activity preamble up. It forbade narrating the feed but never granted permission to act on it, which left the agent with a list of work it was implicitly told to ignore. It now says this is work you can pick up, that a vague opener usually means the most recent thing, and points at the skill. Two honest limits recorded in the README. `node-usage` is not gated by the flag, so the preference case measures a capability both arms have and a difference there is noise. And that case has no historical precedent: a sweep of 16,262 traces found no conversation where a user stated a node preference their own workflows already showed. |
||
|---|---|---|
| .. | ||
| backend-module | ||
| dev-metrics | ||
| instance-seeding | ||
| licenses | ||
| mutation-health | ||
| ab-activity-eval.sh | ||
| agent-setup.mjs | ||
| block-npm-install.js | ||
| build-n8n.mjs | ||
| check-boundaries.mjs | ||
| check-workspace-deps.mjs | ||
| check-workspace-private-deps.mjs | ||
| cloud-session.mjs | ||
| codespace-env.mjs | ||
| codespace-env.test.mjs | ||
| dev-command-removed-notice.mjs | ||
| dev-up.mjs | ||
| dockerize-n8n.mjs | ||
| ensure-zx.mjs | ||
| format.mjs | ||
| generate-emoji-data.mjs | ||
| generate-lucide-icon-data.mjs | ||
| grind.mjs | ||
| inspect-activity-log.mjs | ||
| nathan.mjs | ||
| os-normalize.mjs | ||
| prepare.mjs | ||
| reset.mjs | ||
| scan-n8n-image.mjs | ||
| smoke-n8n-image.mjs | ||
| sync-agent-skill-links.mjs | ||
| turbo-critical-path.mjs | ||