diff --git a/packages/@n8n/instance-ai/evaluations/README.md b/packages/@n8n/instance-ai/evaluations/README.md index 92a068d60ab..b440d93e581 100644 --- a/packages/@n8n/instance-ai/evaluations/README.md +++ b/packages/@n8n/instance-ai/evaluations/README.md @@ -242,8 +242,11 @@ A test case can declare optional natural-language assertions, split by what they - **`processExpectations: string[]`** — about *how the build went* (clarifications asked, push-back, ordering). Judged from the **conversation transcript** (plus the workflow and conversation metrics). They require a transcript, so they are **skipped in prebuilt/MCP runs**. e.g. `"Before building, the agent asked which Slack channel to use."` - **`outcomeExpectations: string[]`** — about the **resulting workflow**. Judged from the **workflow JSON**, so they **also run in prebuilt/MCP runs** (which have no transcript). e.g. `"The final workflow splits the records envelope before posting."` +- **`memoryExpectations: string[]`** — about the agent's **context state**: what survived compression, and what retrieval put in front of the model. Judged by a **separate judge** (`build-expectations/memory-verifier.ts`) whose only input is the captured run debug: the final compressed observation block, the raw message window the model actually received, and the final system prompt. A fact counts as retained if it is present in **any** of those three tiers. They need a real instance-ai thread, so they are **skipped in prebuilt/MCP runs**. e.g. `"The observation block still records the alert channel agreed early in the thread, not the channel mentioned later in passing."` -Both are graded by the same Sonnet judge (`build-expectations/verifier.ts`) and **count as units in the pass rate**: evaluated expectations fold into the per-case and headline pass@k/pass^k alongside execution scenarios. They don't flip an individual scenario's pass/fail (each is its own unit), and a judge `incomplete` verdict is excluded from the count. A full build judges the union of both fields against the transcript; a prebuilt build judges only `outcomeExpectations` against the workflow. A case may omit `executionScenarios` entirely — a **build-only** case — and is then graded by these expectations plus the always-on workflow checks. +`memoryExpectations` deliberately do **not** see the full conversation transcript or the built workflow. The raw message window is not the transcript: messages evicted from the window do not appear in it, which is precisely why it can be trusted as a record of what the agent *still had*. Grading against the transcript instead would let the judge satisfy "the agent still knew X" from the turn where X was first said even after the fact had fallen out of context — the confound this kind exists to remove. Keeping the inputs disjoint from the build is what makes a miss attributable to **recall**, so write them as claims about what the *context* contains, not about what the agent *did*. Their verdicts are tagged `kind: "memory"` in `eval-results.json` and badged `memory` in the HTML report. The Observer only compresses once a thread crosses its token threshold, so on a short thread the observation block is legitimately empty while every fact is still in the raw window — the judge is told this explicitly, and an absent observation block is **never** on its own a reason to fail. That matters: grading on the observation block alone would fail every memory expectation for a structural reason on exactly the short threads the suite is mostly made of. Which tier held the fact is reported in the verdict, since that is the interesting detail when comparing two memory approaches. + +All three are graded by a Sonnet judge and **count as units in the pass rate**: evaluated expectations fold into the per-case and headline pass@k/pass^k alongside execution scenarios. They don't flip an individual scenario's pass/fail (each is its own unit), and a judge `incomplete` verdict is excluded from the count. A full build judges the union of both fields against the transcript; a prebuilt build judges only `outcomeExpectations` against the workflow. A case may omit `executionScenarios` entirely — a **build-only** case — and is then graded by these expectations plus the always-on workflow checks. Use them for things the binary checks and `successCriteria` don't cover: @@ -710,7 +713,7 @@ The corpus lives in **LangTracer** — suite `baseline` is what CI runs. Author } ``` -`conversation` (≥1 turn, first must be `user`), plus `complexity` and `tags`, are required. `executionScenarios`, `description`, `triggerType`, `messageBudget`, `processExpectations`, `outcomeExpectations`, `credentials`, and `datasets` (default `["full"]`) are optional — but **a case must declare at least one `executionScenario`, or one process/outcome expectation** (a case that asserts nothing is rejected at load). A _build-only_ case omits `executionScenarios` and is graded by its `processExpectations`/`outcomeExpectations` plus the always-on workflow checks: the workflow is still built, only the mock-execution `successCriteria` pass is skipped. A turn’s `text` may be a string or an array of strings joined with newlines — handy for long stage directions. +`conversation` (≥1 turn, first must be `user`), plus `complexity` and `tags`, are required. `executionScenarios`, `description`, `triggerType`, `messageBudget`, `processExpectations`, `outcomeExpectations`, `memoryExpectations`, `credentials`, and `datasets` (default `["full"]`) are optional — but **a case must declare at least one `executionScenario`, or one process/outcome/memory expectation** (a case that asserts nothing is rejected at load). A _build-only_ case omits `executionScenarios` and is graded by its `processExpectations`/`outcomeExpectations` plus the always-on workflow checks: the workflow is still built, only the mock-execution `successCriteria` pass is skipped. A turn’s `text` may be a string or an array of strings joined with newlines — handy for long stage directions. **One case = one LangSmith split**, named from the case slug (the LangTracer case name; for disk-loaded files, the filename without `.json`). Pick a slug you're happy to also use as a `--filter` target. diff --git a/packages/@n8n/instance-ai/evaluations/__tests__/build-orchestrator.test.ts b/packages/@n8n/instance-ai/evaluations/__tests__/build-orchestrator.test.ts index 8117fa85010..8d4eb3af2c3 100644 --- a/packages/@n8n/instance-ai/evaluations/__tests__/build-orchestrator.test.ts +++ b/packages/@n8n/instance-ai/evaluations/__tests__/build-orchestrator.test.ts @@ -1,3 +1,4 @@ +import { verifyMemoryExpectations } from '../build-expectations/memory-verifier'; import { verifyBuildExpectations } from '../build-expectations/verifier'; import type { CliArgs } from '../cli/args'; import type { N8nClient } from '../clients/n8n-client'; @@ -67,6 +68,18 @@ vi.mock('../build-expectations/verifier', async (importOriginal) => { }; }); +vi.mock('../build-expectations/memory-verifier', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + verifyMemoryExpectations: vi + .fn() + .mockResolvedValue([ + { expectation: 'the channel survived', pass: true, reason: 'present', kind: 'memory' }, + ]), + }; +}); + const silentLogger: EvalLogger = { info: () => {}, verbose: () => {}, @@ -151,6 +164,12 @@ function makeDeps( }; } +/** A transcript carrying real agent output — required for process/memory + * expectations to be judged rather than recorded as ungraded. */ +const transcriptWithOutput = [ + { userMessage: 'build it', steps: [{ kind: 'agent-text' as const, text: 'Done.' }] }, +]; + /** The eviction handler runs on a microtask after getOrBuild resolves. */ async function settleMicrotasks(): Promise { await Promise.resolve(); @@ -368,6 +387,81 @@ describe('createBuildOrchestrator', () => { expect(verdicts?.[0].reason).toContain('no agent output'); }); + it('merges memory verdicts alongside conversation verdicts on a full build', async () => { + vi.stubGlobal('fetch', vi.fn().mockResolvedValue({ ok: true })); + const tracedBuild = vi + .fn() + .mockResolvedValue(okBuild({ threadId: 'thread-9', transcript: transcriptWithOutput })); + const deps = makeDeps([makeLane(1, tracedBuild)], { + testCaseByFileSlug: new Map([ + [ + 'case-a', + baseCase({ + outcomeExpectations: ['sends a digest'], + memoryExpectations: ['the channel survived'], + }), + ], + ]), + }); + const orchestrator = createBuildOrchestrator(deps); + + await orchestrator.getOrBuild(0, 'case-a'); + const verdicts = await deps.buildExpectationsByKey.get('0:case-a'); + + // Both judges ran and BOTH sets survived the merge. The old early-return shape + // could only ever record one, which would silently shrink the denominator. + expect(vi.mocked(verifyBuildExpectations)).toHaveBeenCalledTimes(1); + expect(vi.mocked(verifyMemoryExpectations)).toHaveBeenCalledTimes(1); + expect(verdicts?.map((v) => [v.expectation, v.kind])).toEqual([ + ['sends a digest', undefined], + ['the channel survived', 'memory'], + ]); + }); + + it('reads the run debug stashed for this build when judging memory expectations', async () => { + vi.stubGlobal('fetch', vi.fn().mockResolvedValue({ ok: true })); + const tracedBuild = vi + .fn() + .mockResolvedValue(okBuild({ threadId: 'thread-9', transcript: transcriptWithOutput })); + const deps = makeDeps([makeLane(1, tracedBuild)], { + testCaseByFileSlug: new Map([ + ['case-a', baseCase({ memoryExpectations: ['the channel survived'] })], + ]), + }); + const orchestrator = createBuildOrchestrator(deps); + + await orchestrator.getOrBuild(0, 'case-a'); + await deps.buildExpectationsByKey.get('0:case-a'); + + // stashRunDebug must run BEFORE stashBuildExpectations, or the judge would be + // handed undefined and report "no context state" on every case. + expect(deps.runDebugByThreadId.has('thread-9')).toBe(true); + expect(vi.mocked(verifyMemoryExpectations)).toHaveBeenCalledWith(['the channel survived'], []); + }); + + it('keeps memory verdicts tagged when the memory judge rejects', async () => { + vi.stubGlobal('fetch', vi.fn().mockResolvedValue({ ok: true })); + vi.mocked(verifyMemoryExpectations).mockRejectedValueOnce(new Error('judge exploded')); + const tracedBuild = vi + .fn() + .mockResolvedValue(okBuild({ threadId: 'thread-9', transcript: transcriptWithOutput })); + const deps = makeDeps([makeLane(1, tracedBuild)], { + testCaseByFileSlug: new Map([ + ['case-a', baseCase({ memoryExpectations: ['the channel survived'] })], + ]), + }); + const orchestrator = createBuildOrchestrator(deps); + + await orchestrator.getOrBuild(0, 'case-a'); + const verdicts = await deps.buildExpectationsByKey.get('0:case-a'); + + // An untagged verdict here would be reported as a conversation miss. + expect(verdicts).toHaveLength(1); + expect(verdicts?.[0].kind).toBe('memory'); + expect(verdicts?.[0].incomplete).toBe(true); + expect(verdicts?.[0].reason).toContain('memory judge error'); + }); + it('serves prebuilt workflows by fetching them, never invoking the builder', async () => { const tracedBuild = vi.fn().mockResolvedValue(okBuild()); const lane = makeLane(1, tracedBuild); diff --git a/packages/@n8n/instance-ai/evaluations/__tests__/data-workflows-schema.test.ts b/packages/@n8n/instance-ai/evaluations/__tests__/data-workflows-schema.test.ts index f8c69c96f02..a4e3b9935f8 100644 --- a/packages/@n8n/instance-ai/evaluations/__tests__/data-workflows-schema.test.ts +++ b/packages/@n8n/instance-ai/evaluations/__tests__/data-workflows-schema.test.ts @@ -69,7 +69,7 @@ describe('EvalTestCaseSchema', () => { it('rejects 0 execution scenarios AND 0 expectations (a case must assert something)', () => { expect(() => EvalTestCaseSchema.parse({ ...validFixture(), executionScenarios: [] })).toThrow( - /at least one executionScenario, or a process\/outcome expectation/, + /at least one executionScenario, or a process\/outcome\/memory expectation/, ); }); @@ -733,7 +733,7 @@ describe('EvalTestCaseSchema · artifact grading via outcome expectations', () = it('rejects a case with no scenario and no process/outcome expectation', () => { const { executionScenarios: _omit, ...rest } = validFixture(); expect(() => EvalTestCaseSchema.parse(rest)).toThrow( - /needs at least one executionScenario, or a process\/outcome expectation/, + /needs at least one executionScenario, or a process\/outcome\/memory expectation/, ); }); diff --git a/packages/@n8n/instance-ai/evaluations/__tests__/memory-context.test.ts b/packages/@n8n/instance-ai/evaluations/__tests__/memory-context.test.ts new file mode 100644 index 00000000000..41abb822a33 --- /dev/null +++ b/packages/@n8n/instance-ai/evaluations/__tests__/memory-context.test.ts @@ -0,0 +1,163 @@ +import type { InstanceAiRunDebugResponse, InstanceAiRunDebugStep } from '@n8n/api-types'; + +import { buildMemoryContextBlock, summarizeMemoryContext } from '../harness/memory-context'; + +/** + * These exercise the REAL `parseSystemPromptForDisplay` from `@n8n/api-types` — the + * observation block is extracted by that parser, and mocking it would make these + * tests pass against a shape the product never produces. Fixtures therefore embed + * observations the way the agent actually does: an `` element inside + * the system prompt. + */ +function step(system: string, stepNumber = 1, messages?: unknown[]): InstanceAiRunDebugStep { + return { stepNumber, input: messages ? { system, messages } : { system } }; +} + +function run( + runId: string, + startedAt: number, + steps: InstanceAiRunDebugStep[], +): InstanceAiRunDebugResponse { + return { threadId: 'thread-1', runId, startedAt, steps, workflowCode: [] }; +} + +describe('summarizeMemoryContext', () => { + it('returns undefined when no run debug was captured', () => { + expect(summarizeMemoryContext(undefined)).toBeUndefined(); + expect(summarizeMemoryContext([])).toBeUndefined(); + }); + + it('returns undefined when runs carry no steps at all', () => { + expect(summarizeMemoryContext([run('r1', 1, [])])).toBeUndefined(); + }); + + it('extracts the observation block out of the system prompt', () => { + const summary = summarizeMemoryContext([ + run('r1', 1, [ + step('You are a builder.\nChannel is #billing-alerts'), + ]), + ]); + expect(summary?.observations).toBe('Channel is #billing-alerts'); + expect(summary?.observationStepCount).toBe(1); + }); + + it('keeps the observation block out of the reported system prompt', () => { + // The two are rendered as separate sections, so leaking observations into the + // system-prompt section would show the judge the same text twice. + const summary = summarizeMemoryContext([ + run('r1', 1, [ + step('You are a builder.\nChannel is #billing-alerts'), + ]), + ]); + expect(summary?.finalSystemPrompt).toContain('You are a builder.'); + expect(summary?.finalSystemPrompt).not.toContain('#billing-alerts'); + expect(summary?.finalSystemPrompt).not.toContain(''); + }); + + it('reports null observations when compression never ran', () => { + const summary = summarizeMemoryContext([run('r1', 1, [step('You are a builder.')])]); + expect(summary?.observations).toBeNull(); + expect(summary?.observationStepCount).toBe(0); + }); + + it('takes the LAST observation block, ordering runs by startedAt', () => { + // Out of order on purpose: the newest compressed state is what a memory + // expectation asserts about, so input order must not decide the winner. + const summary = summarizeMemoryContext([ + run('late', 200, [step('sys\nfinal state')]), + run('early', 100, [step('sys\nearlier state')]), + ]); + expect(summary?.observations).toBe('final state'); + expect(summary?.observationStepCount).toBe(2); + expect(summary?.runCount).toBe(2); + expect(summary?.stepCount).toBe(2); + }); + + it('takes the last observation block across steps within a run', () => { + const summary = summarizeMemoryContext([ + run('r1', 1, [ + step('sys\nfirst', 1), + step('sys\nsecond', 2), + ]), + ]); + expect(summary?.observations).toBe('second'); + }); + + it('retains the last observation block when a later step has none', () => { + // Compression having run is a fact about the thread; a subsequent + // uncompressed step must not erase it. + const summary = summarizeMemoryContext([ + run('r1', 1, [step('sys\nkept', 1), step('sys only', 2)]), + ]); + expect(summary?.observations).toBe('kept'); + expect(summary?.observationStepCount).toBe(1); + expect(summary?.stepCount).toBe(2); + }); + + it('does not let a trailing unparseable system prompt blank out the last real one', () => { + const summary = summarizeMemoryContext([ + run('r1', 1, [step('real prompt', 1), { stepNumber: 2, input: {} }]), + ]); + expect(summary?.finalSystemPrompt).toBe('real prompt'); + expect(summary?.stepCount).toBe(2); + }); +}); + +describe('buildMemoryContextBlock', () => { + it('tells the judge an absent observation block is not itself a failure', () => { + // The Observer only compresses past a token threshold, so most threads have no + // observation block while still holding every fact in the raw window. Reading + // the empty block as a failure would red every memory expectation structurally. + const summary = summarizeMemoryContext([run('r1', 1, [step('sys')])]); + const block = buildMemoryContextBlock(summary!); + expect(block).toContain('the Observer never compressed this thread'); + expect(block).toContain('NOT itself a failure'); + expect(block).toContain('"compressionRan": false'); + }); + + it('reports compressionRan when the Observer did compress', () => { + const summary = summarizeMemoryContext([ + run('r1', 1, [step('sys\nkept')]), + ]); + expect(buildMemoryContextBlock(summary!)).toContain('"compressionRan": true'); + }); + + it('includes the raw message window so an uncompressed thread is still gradable', () => { + const summary = summarizeMemoryContext([ + run('r1', 1, [step('sys', 1, [{ role: 'user', content: 'alert channel is #payments-eu' }])]), + ]); + expect(summary?.finalMessageWindow).toContain('#payments-eu'); + const block = buildMemoryContextBlock(summary!); + expect(block).toContain('Raw message window'); + expect(block).toContain('#payments-eu'); + }); + + it('truncates an oversized message window', () => { + const summary = summarizeMemoryContext([ + run('r1', 1, [step('sys', 1, [{ role: 'user', content: 'y'.repeat(60_000) }])]), + ]); + const block = buildMemoryContextBlock(summary!); + expect(block).toContain('[…truncated]'); + expect(block.length).toBeLessThan(60_000); + }); + + it('includes the observation block, the system prompt and the counters', () => { + const summary = summarizeMemoryContext([ + run('r1', 1, [step('You are a builder.\nChannel is #ops')]), + ]); + const block = buildMemoryContextBlock(summary!); + expect(block).toContain('Channel is #ops'); + expect(block).toContain('You are a builder.'); + expect(block).toContain('"stepsCarryingObservations": 1'); + }); + + it('truncates an oversized system prompt so the observation block stays visible', () => { + const summary = summarizeMemoryContext([ + run('r1', 1, [step(`${'x'.repeat(60_000)}\nneedle`)]), + ]); + const block = buildMemoryContextBlock(summary!); + expect(block).toContain('[…truncated]'); + expect(block).toContain('needle'); + expect(block.length).toBeLessThan(50_000); + }); +}); diff --git a/packages/@n8n/instance-ai/evaluations/__tests__/memory-verifier.test.ts b/packages/@n8n/instance-ai/evaluations/__tests__/memory-verifier.test.ts new file mode 100644 index 00000000000..3a8adc6b8e5 --- /dev/null +++ b/packages/@n8n/instance-ai/evaluations/__tests__/memory-verifier.test.ts @@ -0,0 +1,138 @@ +import type { InstanceAiRunDebugResponse } from '@n8n/api-types'; +import type { Mock, MockedFunction } from 'vitest'; +import { vi } from 'vitest'; + +// Only the LLM boundary is mocked. The context extraction under test runs for real, +// including the observation-block parser from @n8n/api-types. +vi.mock('../../src/utils/eval-agents', () => ({ + createEvalAgent: vi.fn(), + EPHEMERAL_CACHE: {}, + SONNET_MODEL: 'test-sonnet-model', +})); + +import { createEvalAgent } from '../../src/utils/eval-agents'; +import { verifyMemoryExpectations } from '../build-expectations/memory-verifier'; + +const mockCreateEvalAgent = createEvalAgent as MockedFunction; + +type JudgeResponse = { + structuredOutput?: { results: Array<{ index: number; pass: boolean; reason: string }> }; +}; +type GenerateFn = (messages: unknown, opts: unknown) => Promise; +type GenerateMock = Mock; + +function mockJudge(generate: GenerateMock): void { + const structuredOutput = vi.fn().mockReturnValue({ generate }); + mockCreateEvalAgent.mockReturnValue({ structuredOutput } as unknown as ReturnType< + typeof createEvalAgent + >); +} + +function runDebug(system: string): InstanceAiRunDebugResponse[] { + return [ + { + threadId: 'thread-1', + runId: 'run-1', + startedAt: 1, + steps: [{ stepNumber: 1, input: { system } }], + workflowCode: [], + }, + ]; +} + +describe('verifyMemoryExpectations', () => { + beforeEach(() => { + vi.clearAllMocks(); + vi.spyOn(console, 'warn').mockImplementation(() => undefined); + }); + + it('returns empty and never calls the judge when there are no memory expectations', async () => { + const results = await verifyMemoryExpectations([], runDebug('sys')); + expect(results).toEqual([]); + expect(mockCreateEvalAgent).not.toHaveBeenCalled(); + }); + + it('records incomplete, memory-tagged verdicts when no run debug was captured', async () => { + const results = await verifyMemoryExpectations(['m1'], undefined); + expect(results).toHaveLength(1); + expect(results[0]).toMatchObject({ + expectation: 'm1', + pass: false, + incomplete: true, + kind: 'memory', + }); + expect(results[0]?.reason).toContain('no run debug'); + // A missing capture is infrastructure, not a product miss — never judged. + expect(mockCreateEvalAgent).not.toHaveBeenCalled(); + }); + + it('tags judged verdicts as memory so they are identifiable in the results file', async () => { + const generate: GenerateMock = vi.fn().mockResolvedValue({ + structuredOutput: { results: [{ index: 0, pass: true, reason: 'still present' }] }, + }); + mockJudge(generate); + + const results = await verifyMemoryExpectations( + ['the channel survived compression'], + runDebug('sys\nChannel is #ops'), + ); + + expect(results).toEqual([ + { + expectation: 'the channel survived compression', + pass: true, + reason: 'still present', + kind: 'memory', + }, + ]); + }); + + it('grades against the context state and never the transcript', async () => { + const generate: GenerateMock = vi.fn().mockResolvedValue({ + structuredOutput: { results: [{ index: 0, pass: true, reason: 'ok' }] }, + }); + mockJudge(generate); + + await verifyMemoryExpectations( + ['the channel survived compression'], + runDebug('You are a builder.\nChannel is #ops'), + ); + + const sent = JSON.stringify(generate.mock.calls[0]?.[0]); + expect(sent).toContain('Channel is #ops'); + expect(sent).toContain('Compressed observation block'); + // The confound this expectation kind exists to remove: if the transcript were + // in scope the judge could satisfy recall from the turn the fact was first said. + expect(sent).not.toContain('Conversation transcript'); + }); + + it('uses its own judge rubric, not the conversation one', async () => { + const generate: GenerateMock = vi.fn().mockResolvedValue({ + structuredOutput: { results: [{ index: 0, pass: false, reason: 'gone' }] }, + }); + mockJudge(generate); + + await verifyMemoryExpectations(['m1'], runDebug('sys')); + + const [agentName, options] = mockCreateEvalAgent.mock.calls[0] ?? []; + expect(agentName).toBe('eval-memory-expectations-verifier'); + expect(options?.instructions).toContain('memory subsystem'); + }); + + it('tags verdicts as memory even when the judge dies', async () => { + const generate: GenerateMock = vi.fn().mockRejectedValue(new Error('judge exploded')); + mockJudge(generate); + + const results = await verifyMemoryExpectations(['m1'], runDebug('sys')); + + expect(results).toEqual([ + { + expectation: 'm1', + pass: false, + reason: 'judge produced no result', + incomplete: true, + kind: 'memory', + }, + ]); + }); +}); diff --git a/packages/@n8n/instance-ai/evaluations/__tests__/select-expectations.test.ts b/packages/@n8n/instance-ai/evaluations/__tests__/select-expectations.test.ts index 35685c070f4..d15e9aa0d03 100644 --- a/packages/@n8n/instance-ai/evaluations/__tests__/select-expectations.test.ts +++ b/packages/@n8n/instance-ai/evaluations/__tests__/select-expectations.test.ts @@ -24,8 +24,13 @@ const realTranscript: TranscriptTurn[] = [ const noOutputTranscript: TranscriptTurn[] = [{ userMessage: 'build it', steps: [] }]; function testCase( - over: Partial> = {}, -): Pick { + over: Partial< + Pick + > = {}, +): Pick< + WorkflowTestCase, + 'processExpectations' | 'outcomeExpectations' | 'memoryExpectations' | 'conversation' +> { return { conversation, ...over }; } @@ -166,4 +171,102 @@ describe('selectAuthorExpectations', () => { expect(unjudged).toEqual([]); expect(warnings).toEqual([]); }); + describe('memoryExpectations', () => { + it('routes memory expectations to the context-state judge on a full build', () => { + const { logger, warnings } = makeLogger(); + const { expectations, memoryExpectations, unjudged } = selectAuthorExpectations({ + testCase: testCase({ + processExpectations: ['p1'], + outcomeExpectations: ['o1'], + memoryExpectations: ['m1', 'm2'], + }), + transcript: realTranscript, + buildSucceeded: true, + isPrebuilt: false, + logger, + }); + // Kept OUT of the conversation judge's list: it receives the transcript, which + // would let it satisfy a recall claim from the turn the fact was first said. + expect(expectations).toEqual(['p1', 'o1']); + expect(memoryExpectations).toEqual(['m1', 'm2']); + expect(unjudged).toEqual([]); + expect(warnings).toEqual([]); + }); + + it('still judges memory expectations when a full build lost its transcript', () => { + // Event capture failing does not mean no thread ran, so the run debug is + // probably there. Only the process expectations are lost to a capture bug. + const { logger } = makeLogger(); + const { expectations, memoryExpectations, unjudged } = selectAuthorExpectations({ + testCase: testCase({ + processExpectations: ['p1'], + outcomeExpectations: ['o1'], + memoryExpectations: ['m1'], + }), + transcript: noOutputTranscript, + buildSucceeded: true, + isPrebuilt: false, + logger, + }); + expect(expectations).toEqual(['o1']); + expect(memoryExpectations).toEqual(['m1']); + expect(unjudged).toEqual([]); + }); + + it('records memory expectations as ungraded for a prebuilt build, keeping outcome ones gradable', () => { + const { logger, warnings } = makeLogger(); + const { expectations, memoryExpectations, unjudged } = selectAuthorExpectations({ + testCase: testCase({ outcomeExpectations: ['o1'], memoryExpectations: ['m1'] }), + transcript: undefined, + buildSucceeded: true, + isPrebuilt: true, + logger, + }); + // The outcome expectation must survive: a prebuilt run can still grade the + // workflow, and dropping it would silently shrink the denominator. + expect(expectations).toEqual(['o1']); + expect(memoryExpectations).toEqual([]); + expect(unjudged).toHaveLength(1); + expect(unjudged[0]).toMatchObject({ + expectation: 'm1', + pass: false, + incomplete: true, + kind: 'memory', + }); + expect(unjudged[0]?.reason).toContain('no context state'); + expect(warnings).toHaveLength(1); + }); + + it('marks memory expectations ungraded when the build produced no agent output', () => { + const { logger } = makeLogger(); + const { expectations, memoryExpectations, unjudged } = selectAuthorExpectations({ + testCase: testCase({ processExpectations: ['p1'], memoryExpectations: ['m1'] }), + transcript: noOutputTranscript, + buildSucceeded: false, + isPrebuilt: false, + logger, + }); + expect(expectations).toEqual([]); + expect(memoryExpectations).toEqual([]); + // Both kinds recorded, and only the memory one tagged, so the report can tell + // a recall miss from a conversation miss even on the failed-build path. + expect(unjudged).toHaveLength(2); + expect(unjudged.find((v) => v.expectation === 'p1')?.kind).toBeUndefined(); + expect(unjudged.find((v) => v.expectation === 'm1')?.kind).toBe('memory'); + expect(unjudged.every((v) => v.incomplete)).toBe(true); + }); + + it('leaves memory expectations empty when the case declares none', () => { + const { logger } = makeLogger(); + const { memoryExpectations, unjudged } = selectAuthorExpectations({ + testCase: testCase({ outcomeExpectations: ['o1'] }), + transcript: realTranscript, + buildSucceeded: true, + isPrebuilt: false, + logger, + }); + expect(memoryExpectations).toEqual([]); + expect(unjudged).toEqual([]); + }); + }); }); diff --git a/packages/@n8n/instance-ai/evaluations/build-expectations/assertion-judge.ts b/packages/@n8n/instance-ai/evaluations/build-expectations/assertion-judge.ts index 92a46e1d2a9..b277c531536 100644 --- a/packages/@n8n/instance-ai/evaluations/build-expectations/assertion-judge.ts +++ b/packages/@n8n/instance-ai/evaluations/build-expectations/assertion-judge.ts @@ -38,21 +38,33 @@ export function buildAssertionsBlock(assertions: string[]): string { ].join('\n'); } +/** Lets a caller judge against a different rubric while reusing the retry, timeout + * and index-mapping behavior below. Omitted → the conversation/workflow rubric. */ +export interface JudgeExpectationsOptions { + agentName?: string; + instructions?: string; + /** Prefix on this judge's warnings, so a failing rubric is identifiable in logs. */ + logLabel?: string; +} + /** * Shared judge core: sends caller-built messages to the expectations judge and maps - * the structured verdicts back onto `assertions` by index. Used by both the - * workflow/conversation judge (`verifyBuildExpectations`) and static artifact handlers - * (`runAssertionJudge`) so retry/timeout/parsing behavior stays in one place. + * the structured verdicts back onto `assertions` by index. Used by the + * workflow/conversation judge (`verifyBuildExpectations`), the context-state judge + * (`verifyMemoryExpectations`) and static artifact handlers (`runAssertionJudge`) so + * retry/timeout/parsing behavior stays in one place. */ export async function judgeExpectations( messages: Message[], assertions: string[], + options: JudgeExpectationsOptions = {}, ): Promise { if (assertions.length === 0) return []; + const logLabel = options.logLabel ?? 'expectations'; for (let attempt = 1; attempt <= MAX_VERIFY_ATTEMPTS; attempt++) { - const agent = createEvalAgent('eval-build-expectations-verifier', { - instructions: BUILD_EXPECTATIONS_VERIFY_PROMPT, + const agent = createEvalAgent(options.agentName ?? 'eval-build-expectations-verifier', { + instructions: options.instructions ?? BUILD_EXPECTATIONS_VERIFY_PROMPT, cache: true, model: JUDGE_MODEL, }).structuredOutput(expectationResultSchema); @@ -70,7 +82,7 @@ export async function judgeExpectations( result = await agent.generate(messages, { abortSignal: abortController.signal }); } catch (error: unknown) { const msg = error instanceof Error ? error.message : String(error); - console.warn(`[expectations] attempt ${attempt}/${MAX_VERIFY_ATTEMPTS} failed: ${msg}`); + console.warn(`[${logLabel}] attempt ${attempt}/${MAX_VERIFY_ATTEMPTS} failed: ${msg}`); continue; } finally { clearTimeout(timer); @@ -100,11 +112,11 @@ export async function judgeExpectations( } console.warn( - `[expectations] attempt ${attempt}/${MAX_VERIFY_ATTEMPTS} produced no parseable results`, + `[${logLabel}] attempt ${attempt}/${MAX_VERIFY_ATTEMPTS} produced no parseable results`, ); } - console.warn(`[expectations] exhausted ${MAX_VERIFY_ATTEMPTS} attempts, returning all-fail`); + console.warn(`[${logLabel}] exhausted ${MAX_VERIFY_ATTEMPTS} attempts, returning all-fail`); return allFailVerdicts(assertions, 'judge produced no result'); } diff --git a/packages/@n8n/instance-ai/evaluations/build-expectations/collect.ts b/packages/@n8n/instance-ai/evaluations/build-expectations/collect.ts index 351164487ea..5cddb58e90f 100644 --- a/packages/@n8n/instance-ai/evaluations/build-expectations/collect.ts +++ b/packages/@n8n/instance-ai/evaluations/build-expectations/collect.ts @@ -1,7 +1,24 @@ -import type { WorkflowTestCase } from '../types'; +import type { BuildExpectationResult, WorkflowTestCase } from '../types'; /** - * Author expectations to judge for a full build: process turns first, then outcome. + * Tag verdicts as context-state graded. + * + * Applied on every path that can produce a memory verdict — judged, ungraded, and + * judge-error — so `kind` is a reliable filter in the report and in + * `eval-results.json` rather than something only the happy path sets. + */ +export function asMemoryVerdicts(verdicts: BuildExpectationResult[]): BuildExpectationResult[] { + return verdicts.map((verdict) => ({ ...verdict, kind: 'memory' as const })); +} + +/** + * Author expectations judged by the CONVERSATION/WORKFLOW judge: process turns + * first, then outcome. + * + * `memoryExpectations` are deliberately excluded — they go to their own judge with + * the context state as its only input (see `verifyMemoryExpectations`). Use + * `collectAllExpectations` when you need every authored expectation regardless of + * which judge owns it. * * The original single `buildExpectations` order isn't preserved across the split, * but verdicts are matched back by expectation string (not index), so ordering @@ -12,3 +29,19 @@ export function collectExpectations( ): string[] { return [...(testCase.processExpectations ?? []), ...(testCase.outcomeExpectations ?? [])]; } + +/** + * Every authored expectation across all three kinds. + * + * Used for the accounting paths — a build that produced nothing records all of them + * as ungraded, so the case keeps a stable unit count whichever judge would have + * owned each one. + */ +export function collectAllExpectations( + testCase: Pick< + WorkflowTestCase, + 'processExpectations' | 'outcomeExpectations' | 'memoryExpectations' + >, +): string[] { + return [...collectExpectations(testCase), ...(testCase.memoryExpectations ?? [])]; +} diff --git a/packages/@n8n/instance-ai/evaluations/build-expectations/memory-verifier.ts b/packages/@n8n/instance-ai/evaluations/build-expectations/memory-verifier.ts new file mode 100644 index 00000000000..676cb0d7896 --- /dev/null +++ b/packages/@n8n/instance-ai/evaluations/build-expectations/memory-verifier.ts @@ -0,0 +1,61 @@ +import type { Message } from '@n8n/agents'; +import type { InstanceAiRunDebugResponse } from '@n8n/api-types'; + +import { allFailVerdicts, buildAssertionsBlock, judgeExpectations } from './assertion-judge'; +import { asMemoryVerdicts } from './collect'; +import { EPHEMERAL_CACHE } from '../../src/utils/eval-agents'; +import { buildMemoryContextBlock, summarizeMemoryContext } from '../harness/memory-context'; +import { MEMORY_EXPECTATIONS_VERIFY_PROMPT } from '../system-prompts/memory-expectations-verify'; +import type { BuildExpectationResult } from '../types'; + +/** Recorded when the run debug never arrived, so the reason reaches the report + * instead of the expectations reading as product failures. */ +const NO_RUN_DEBUG_REASON = + 'not judged — no run debug was captured for this thread, so there was no context state to grade'; + +/** + * Judge author-written expectations about the agent's CONTEXT STATE. + * + * Runs as its own judge call against its own rubric, deliberately separate from + * `verifyBuildExpectations`. The conversation judge receives the full transcript, + * which would let it satisfy "the agent still knew X" from the turn where X was + * first said — exactly the confound this expectation kind exists to remove. Keeping + * the inputs disjoint is what makes a miss attributable to recall rather than to the + * build. + * + * Returns `incomplete` verdicts (not failures) when there is no context state to + * grade, so a missing capture reads as "no verdict" and stays out of the pass rate. + */ +export async function verifyMemoryExpectations( + expectations: string[], + runDebug: InstanceAiRunDebugResponse[] | undefined, +): Promise { + if (expectations.length === 0) return []; + + const summary = summarizeMemoryContext(runDebug); + if (!summary) return asMemoryVerdicts(allFailVerdicts(expectations, NO_RUN_DEBUG_REASON)); + + const messages: Message[] = [ + { + role: 'user', + content: [ + // The context block is stable for this build — one cache breakpoint, matching + // how the conversation judge caches its workflow/artifact context. + { + type: 'text', + text: buildMemoryContextBlock(summary), + providerOptions: EPHEMERAL_CACHE, + }, + { type: 'text', text: buildAssertionsBlock(expectations) }, + ], + }, + ]; + + return asMemoryVerdicts( + await judgeExpectations(messages, expectations, { + agentName: 'eval-memory-expectations-verifier', + instructions: MEMORY_EXPECTATIONS_VERIFY_PROMPT, + logLabel: 'memory-expectations', + }), + ); +} diff --git a/packages/@n8n/instance-ai/evaluations/build-expectations/select.ts b/packages/@n8n/instance-ai/evaluations/build-expectations/select.ts index bfd399f7c0f..7ab9f8d04c0 100644 --- a/packages/@n8n/instance-ai/evaluations/build-expectations/select.ts +++ b/packages/@n8n/instance-ai/evaluations/build-expectations/select.ts @@ -1,5 +1,5 @@ import { allFailVerdicts } from './assertion-judge'; -import { collectExpectations } from './collect'; +import { asMemoryVerdicts, collectAllExpectations, collectExpectations } from './collect'; import type { EvalLogger } from '../harness/logger'; import type { BuildExpectationResult, TranscriptTurn, WorkflowTestCase } from '../types'; import { conversationUserTurnsAsText } from '../utils/conversation-text'; @@ -8,10 +8,14 @@ import { conversationUserTurnsAsText } from '../utils/conversation-text'; const NO_AGENT_OUTPUT_REASON = 'not judged — the build produced no agent output, so there was nothing to grade'; +/** Prebuilt/MCP builds run no instance-ai thread, so no context state is captured. */ +const NO_CONTEXT_STATE_REASON = + 'not judged — this run built no instance-ai thread, so no context state was captured to grade'; + export interface SelectAuthorExpectationsArgs { testCase: Pick< WorkflowTestCase, - 'processExpectations' | 'outcomeExpectations' | 'conversation' | 'seed' + 'processExpectations' | 'outcomeExpectations' | 'memoryExpectations' | 'conversation' | 'seed' >; /** Captured build transcript, if any. Empty/absent for prebuilt/MCP builds. */ transcript: TranscriptTurn[] | undefined; @@ -28,10 +32,11 @@ export interface SelectAuthorExpectationsArgs { * feed the judge. * * - Full build with a transcript → judge the process + outcome union against the - * real transcript. + * real transcript, and hand `memoryExpectations` to the context-state judge. * - No transcript but the build succeeded → judge only `outcomeExpectations` * against the workflow, with the authored conversation as request context. - * This is the prebuilt/MCP path. + * This is the prebuilt/MCP path; it runs no thread, so memory expectations are + * recorded ungraded rather than judged against an absent context state. * - Build failed with no transcript → judge nothing. * * "No transcript" means no AGENT OUTPUT, not an empty array. A run that dies @@ -48,6 +53,9 @@ export interface SelectAuthorExpectationsArgs { */ export function selectAuthorExpectations(args: SelectAuthorExpectationsArgs): { expectations: string[]; + /** Expectations for the context-state judge, which grades them against the + * captured run debug rather than the transcript. */ + memoryExpectations: string[]; transcript: TranscriptTurn[]; /** Expectations deliberately left ungraded, already shaped as `incomplete` * verdicts. Recording them keeps the case's unit count stable across runs (so @@ -77,7 +85,7 @@ export function selectAuthorExpectations(args: SelectAuthorExpectationsArgs): { // A failed build that produced nothing at all: record every expectation as // ungraded rather than handing the judge an empty conversation to describe. if (!hasAgentOutput && !buildSucceeded) { - const authored = collectExpectations(testCase); + const authored = collectAllExpectations(testCase); if (authored.length > 0) { logger.warn( ` Build produced no agent output — leaving all ${String(authored.length)} expectation(s) ungraded; judging them would score an empty transcript`, @@ -85,8 +93,14 @@ export function selectAuthorExpectations(args: SelectAuthorExpectationsArgs): { } return { expectations: [], + memoryExpectations: [], transcript, - unjudged: allFailVerdicts(authored, NO_AGENT_OUTPUT_REASON), + unjudged: [ + ...allFailVerdicts(collectExpectations(testCase), NO_AGENT_OUTPUT_REASON), + ...asMemoryVerdicts( + allFailVerdicts(testCase.memoryExpectations ?? [], NO_AGENT_OUTPUT_REASON), + ), + ], }; } @@ -94,5 +108,24 @@ export function selectAuthorExpectations(args: SelectAuthorExpectationsArgs): { ? collectExpectations(testCase) : (testCase.outcomeExpectations ?? []); - return { expectations, transcript, unjudged: [] }; + // Memory expectations need a real instance-ai thread to have produced run debug. + // Gated on `isPrebuilt`, NOT on the transcript: prebuilt/MCP runs genuinely have + // no thread, whereas a full build whose event capture failed still ran one, so its + // run debug is very likely present. Sending those through means a capture bug costs + // us the process expectations but not the memory ones; if the run debug really is + // missing, the judge records that specific reason itself. + const authoredMemory = testCase.memoryExpectations ?? []; + if (authoredMemory.length > 0 && isPrebuilt) { + logger.warn( + ` Prebuilt/MCP build — leaving ${String(authoredMemory.length)} memory expectation(s) ungraded; this run ran no instance-ai thread`, + ); + return { + expectations, + memoryExpectations: [], + transcript, + unjudged: asMemoryVerdicts(allFailVerdicts(authoredMemory, NO_CONTEXT_STATE_REASON)), + }; + } + + return { expectations, memoryExpectations: authoredMemory, transcript, unjudged: [] }; } diff --git a/packages/@n8n/instance-ai/evaluations/cli/mcp-builder.ts b/packages/@n8n/instance-ai/evaluations/cli/mcp-builder.ts index 14f2f4e3829..fbfa4771bc1 100644 --- a/packages/@n8n/instance-ai/evaluations/cli/mcp-builder.ts +++ b/packages/@n8n/instance-ai/evaluations/cli/mcp-builder.ts @@ -185,6 +185,10 @@ export const MCP_BUILD_KEY_SUPPORT: Record< // transcript-less MCP builds there); declaring them needs no build-side setup. processExpectations: 'supported', outcomeExpectations: 'supported', + // Same shape as processExpectations: judged by the harness after the build and + // skipped there for MCP builds (which run no instance-ai thread, so capture no + // context state), and declaring them needs no build-side setup. + memoryExpectations: 'supported', // Forbidden legacy key — the schema rejects it at load, so it can never // reach this check; classified only to keep the map schema-complete. buildExpectations: 'supported', diff --git a/packages/@n8n/instance-ai/evaluations/harness/memory-context.ts b/packages/@n8n/instance-ai/evaluations/harness/memory-context.ts new file mode 100644 index 00000000000..ef8cd0e93fb --- /dev/null +++ b/packages/@n8n/instance-ai/evaluations/harness/memory-context.ts @@ -0,0 +1,159 @@ +import type { InstanceAiRunDebugResponse } from '@n8n/api-types'; +import { parseMessageBlocks, parseSystemPromptForDisplay } from '@n8n/api-types'; + +/** Cap on the rendered context block. The final system prompt carries the whole + * baked knowledge base, which dwarfs the observation block and would push the + * part we actually grade out of the judge's attention. */ +const MAX_SYSTEM_PROMPT_CHARS = 40_000; + +/** Cap on the rendered message window. Tool results dominate the message log, so an + * uncapped window would swamp both other sections. */ +const MAX_MESSAGE_WINDOW_CHARS = 40_000; + +export interface MemoryContextSummary { + /** Last non-empty compressed observation block seen across the thread's runs. */ + observations: string | null; + /** System prompt the model saw on the final captured step. */ + finalSystemPrompt: string; + /** + * Raw message window the model was handed on the final captured step. + * + * Load-bearing, not extra colour. The Observer only compresses past its token + * threshold, so a thread that stays under it has NO observation block while every + * fact is still sitting in this window — the model genuinely still has them. + * Grading such a thread on the observation block alone fails every memory + * expectation for a structural reason rather than a recall one, which would make + * the whole kind useless on exactly the short threads the suite is made of. + */ + finalMessageWindow: string; + runCount: number; + stepCount: number; + /** How many steps carried an observation block — 0 means compression never ran. */ + observationStepCount: number; +} + +function segmentsToText(content: string, segments: unknown): string { + if (!Array.isArray(segments) || segments.length === 0) return content; + const parts: string[] = []; + for (const segment of segments) { + if (typeof segment !== 'object' || segment === null) continue; + const seg = segment as { type?: unknown; text?: unknown; name?: unknown }; + if ((seg.type === 'text' || seg.type === 'reasoning') && typeof seg.text === 'string') { + parts.push(seg.text); + } else if (typeof seg.name === 'string') { + parts.push(`[${String(seg.type)}: ${seg.name}]`); + } + } + return parts.length > 0 ? parts.join('\n') : content; +} + +/** + * Reduce a thread's captured run debug to the context state worth grading. + * + * Walks every step in start order so the *last* observation block wins — that is + * the compressed state the agent was carrying at the end of the thread, which is + * what a memory expectation asserts about. Steps without an observation block are + * counted, not skipped: `observationStepCount === 0` is itself the finding that + * compression never ran, and the judge is told so explicitly rather than being + * handed an empty section it might read as a failed assertion. + */ +export function summarizeMemoryContext( + runDebug: InstanceAiRunDebugResponse[] | undefined, +): MemoryContextSummary | undefined { + if (!runDebug || runDebug.length === 0) return undefined; + + let observations: string | null = null; + let finalSystemPrompt = ''; + let finalMessageWindow = ''; + let stepCount = 0; + let observationStepCount = 0; + + const runs = [...runDebug].sort((a, b) => a.startedAt - b.startedAt); + for (const run of runs) { + for (const step of run.steps ?? []) { + stepCount++; + const parsed = parseSystemPromptForDisplay(step.input?.system); + if (parsed.observations) { + observations = parsed.observations; + observationStepCount++; + } + const systemText = parsed.systemBlocks + .map((block) => segmentsToText(block.content, block.segments)) + .filter((text) => text.trim().length > 0) + .join('\n\n'); + // Keep the newest non-empty prompt: a trailing step whose system prompt + // failed to parse must not blank out the one we'd otherwise report. + if (systemText.trim().length > 0) finalSystemPrompt = systemText; + + const windowText = parseMessageBlocks(step.input?.messages) + .map((block) => `### ${block.role}\n${segmentsToText(block.content, block.segments)}`) + .filter((text) => text.trim().length > 0) + .join('\n\n'); + if (windowText.trim().length > 0) finalMessageWindow = windowText; + } + } + + if (stepCount === 0) return undefined; + + return { + observations, + finalSystemPrompt, + finalMessageWindow, + runCount: runs.length, + stepCount, + observationStepCount, + }; +} + +function truncate(text: string, limit: number, fallback: string): string { + if (text.length === 0) return fallback; + return text.length > limit ? `${text.slice(0, limit)}\n\n[…truncated]` : text; +} + +/** + * Render the context state as the judge's sole grading input. + * + * Carries what the model was actually handed on its final step — the compressed + * observation block, the system prompt, and the raw message window — and + * deliberately NOT the full conversation transcript or the built workflow. The + * distinction matters: the window is what the agent still had, whereas the + * transcript also contains turns long since evicted from it. Grading against the + * transcript would let the judge satisfy "the agent still knew X" from the turn + * where X was first said even after the fact fell out of context, which is the + * confound this expectation kind exists to remove; grading against the window is + * simply the truth about what was retained. + */ +export function buildMemoryContextBlock(summary: MemoryContextSummary): string { + // Named so the judge is told which tier held the fact, without being invited to + // treat an absent observation block as an automatic failure. + const observations = + summary.observations ?? + '(none — the Observer never compressed this thread, so no facts were moved into observational memory. This is NOT itself a failure: on a thread this short the facts should still be present in the raw message window below, which is what the model actually received.)'; + + return [ + '## Compressed observation block (final state)', + '', + observations, + '', + '## Raw message window (what the model received on its last step)', + '', + truncate(summary.finalMessageWindow, MAX_MESSAGE_WINDOW_CHARS, '(no message window captured)'), + '', + '## Final system prompt (what the model saw on the last step)', + '', + truncate(summary.finalSystemPrompt, MAX_SYSTEM_PROMPT_CHARS, '(no system prompt captured)'), + '', + '## Capture summary (ground truth — do not recount)', + '', + `\`\`\`json\n${JSON.stringify( + { + runs: summary.runCount, + steps: summary.stepCount, + stepsCarryingObservations: summary.observationStepCount, + compressionRan: summary.observationStepCount > 0, + }, + null, + 2, + )}\n\`\`\``, + ].join('\n'); +} diff --git a/packages/@n8n/instance-ai/evaluations/harness/schema.ts b/packages/@n8n/instance-ai/evaluations/harness/schema.ts index ba02b990d4a..ebe39159cec 100644 --- a/packages/@n8n/instance-ai/evaluations/harness/schema.ts +++ b/packages/@n8n/instance-ai/evaluations/harness/schema.ts @@ -137,6 +137,12 @@ const evalTestCaseObjectSchema = z * and from the rendered agent/config-eval context when the build produced one, so they also * cover artifact existence/absence/content. Also run in prebuilt/MCP runs. Counted as units. */ outcomeExpectations: z.array(z.string().min(1)).optional(), + /** Optional NL assertions about the agent's CONTEXT STATE — what survived compression, and + * what retrieval put in front of the model. LLM-judged from the captured run debug (the + * compressed observation block plus the final system prompt), so a miss is attributable to + * recall rather than to the build. Needs run debug, so skipped in prebuilt/MCP runs. + * Counted as units. */ + memoryExpectations: z.array(z.string().min(1)).optional(), /** * Removed in favour of the process/outcome split. Declared as a forbidden key (rather * than dropped from the shape) so a legacy fixture fails loudly with a migration hint, @@ -267,16 +273,18 @@ export const EvalTestCaseSchema = evalTestCaseObjectSchema // // A case needs at least one gradable unit. Execution scenarios grade the built workflow; // process/outcome expectations grade the conversation, the workflow, and any non-workflow - // artifact (agent, config-eval) rendered into the judge context. + // artifact (agent, config-eval) rendered into the judge context; memory expectations grade + // the captured context state. if ( (c.executionScenarios?.length ?? 0) === 0 && (c.processExpectations?.length ?? 0) === 0 && - (c.outcomeExpectations?.length ?? 0) === 0 + (c.outcomeExpectations?.length ?? 0) === 0 && + (c.memoryExpectations?.length ?? 0) === 0 ) { ctx.addIssue({ code: z.ZodIssueCode.custom, message: - 'a case needs at least one executionScenario, or a process/outcome expectation to grade it', + 'a case needs at least one executionScenario, or a process/outcome/memory expectation to grade it', }); } }); diff --git a/packages/@n8n/instance-ai/evaluations/report/workflow-report.ts b/packages/@n8n/instance-ai/evaluations/report/workflow-report.ts index 7465c0fa1ab..6358ac623e7 100644 --- a/packages/@n8n/instance-ai/evaluations/report/workflow-report.ts +++ b/packages/@n8n/instance-ai/evaluations/report/workflow-report.ts @@ -1268,7 +1268,10 @@ function renderBuildExpectations(results: BuildExpectationResult[] | undefined): const judgment = r.reason ? `
${escapeHtml(r.reason)}
` : ''; - return `
  • ${icon}
    ${escapeHtml(r.expectation)}
    ${judgment}
    • `; + // Memory verdicts are graded against the context state, not the conversation — + // tagged so a recall miss is not read as a build miss. + const kindTag = r.kind === 'memory' ? 'memory' : ''; + return `
  • ${icon}
    ${kindTag}${escapeHtml(r.expectation)}
    ${judgment}
    • `; }); } diff --git a/packages/@n8n/instance-ai/evaluations/run/build-orchestrator.ts b/packages/@n8n/instance-ai/evaluations/run/build-orchestrator.ts index f0bcf972578..3251561ff26 100644 --- a/packages/@n8n/instance-ai/evaluations/run/build-orchestrator.ts +++ b/packages/@n8n/instance-ai/evaluations/run/build-orchestrator.ts @@ -13,6 +13,8 @@ import { sleep } from '@n8n/utils/sleep'; import type { LaneAllocator } from './lane-allocator'; import { provisionCaseBuildUser, type LaneUserPool } from './lane-users'; +import { asMemoryVerdicts } from '../build-expectations/collect'; +import { verifyMemoryExpectations } from '../build-expectations/memory-verifier'; import { selectAuthorExpectations } from '../build-expectations/select'; import { allFailVerdicts, verifyBuildExpectations } from '../build-expectations/verifier'; import type { CliArgs } from '../cli/args'; @@ -401,6 +403,8 @@ export function createBuildOrchestrator(deps: BuildOrchestratorDeps): BuildOrche // Full builds judge process + outcome against the real transcript; prebuilt/MCP // builds (no transcript) judge only outcome expectations against the workflow, // with the authored conversation as request context — mirroring the direct loop. + // `memoryExpectations` take a second, separate judge call against the captured + // context state, so requires stashRunDebug to have run for this build first. function stashBuildExpectations( key: string, fileSlug: string, @@ -438,7 +442,7 @@ export function createBuildOrchestrator(deps: BuildOrchestratorDeps): BuildOrche ); }) : undefined; - const { expectations, transcript, unjudged } = selectAuthorExpectations({ + const { expectations, memoryExpectations, transcript, unjudged } = selectAuthorExpectations({ testCase, transcript: build.transcript, buildSucceeded: build.success, @@ -461,41 +465,67 @@ export function createBuildOrchestrator(deps: BuildOrchestratorDeps): BuildOrche verdicts: BuildExpectationResult[] | Promise, ): Promise => injected ? [...(await verdicts), ...(await injected)] : await verdicts; - // Recorded as incomplete rather than dropped, so the case keeps its unit - // count and the report says why they weren't graded. - if (unjudged.length > 0) { - buildExpectationsByKey.set(key, withInjected(attribute(unjudged))); - return; - } - if (expectations.length === 0) { + if (unjudged.length === 0 && expectations.length === 0 && memoryExpectations.length === 0) { if (injected) buildExpectationsByKey.set(key, injected); return; } + const judgeConversation = async (): Promise => + await (async () => + await verifyBuildExpectations(expectations, { + transcript, + workflowJson: build.workflowJsons[0], + metrics: build.conversationMetrics, + // Rendered non-workflow artifacts (agent AND config-eval), sectioned + // with "(no produced)" fallbacks, so outcome expectations can + // judge artifact existence, absence and content — parity with the + // retired direct loop, which always threaded resolveArtifactContext. + artifactContext: await resolveArtifactContext({ + artifactRefs: build.artifactRefs ?? [], + client, + logger, + }), + }))() + .catch((error: unknown) => + allFailVerdicts( + expectations, + `judge error: ${error instanceof Error ? error.message : String(error)}`, + ), + ) + .then(attribute); + // Reads the run debug stashed for this build — hence stashRunDebug runs first. + // Graded on its own inputs (observation block + final system prompt) and never + // the transcript, so a miss is attributable to recall, not to the build. + const judgeMemory = async (): Promise => + await verifyMemoryExpectations( + memoryExpectations, + build.threadId ? await runDebugByThreadId.get(build.threadId) : undefined, + ) + // Tagged here too: `verifyMemoryExpectations` can reject before its own + // guards run (createEvalAgent throws synchronously, outside the judge's + // try), and an untagged verdict would be reported as a conversation miss. + .catch((error: unknown) => + asMemoryVerdicts( + allFailVerdicts( + memoryExpectations, + `memory judge error: ${error instanceof Error ? error.message : String(error)}`, + ), + ), + ) + .then(attribute); + // Every kind is merged rather than early-returned: a case can carry ungraded + // expectations (recorded as incomplete, so its unit count stays stable) AND + // gradable ones at the same time, and dropping either would silently change + // the denominator. buildExpectationsByKey.set( key, withInjected( - (async () => - await verifyBuildExpectations(expectations, { - transcript, - workflowJson: build.workflowJsons[0], - metrics: build.conversationMetrics, - // Rendered non-workflow artifacts (agent AND config-eval), sectioned - // with "(no produced)" fallbacks, so outcome expectations can - // judge artifact existence, absence and content — parity with the - // retired direct loop, which always threaded resolveArtifactContext. - artifactContext: await resolveArtifactContext({ - artifactRefs: build.artifactRefs ?? [], - client, - logger, - }), - }))() - .catch((error: unknown) => - allFailVerdicts( - expectations, - `judge error: ${error instanceof Error ? error.message : String(error)}`, - ), - ) - .then(attribute), + (async () => { + const [conversation, memory] = await Promise.all([ + expectations.length > 0 ? judgeConversation() : Promise.resolve([]), + memoryExpectations.length > 0 ? judgeMemory() : Promise.resolve([]), + ]); + return [...attribute(unjudged), ...conversation, ...memory]; + })(), ), ); } @@ -550,8 +580,8 @@ export function createBuildOrchestrator(deps: BuildOrchestratorDeps): BuildOrche stashTranscript(build); // isPrebuilt=true: MCP builds have no build transcript, so only // outcome expectations are judged (against the workflow), like prebuilt. - stashBuildExpectations(key, fileSlug, lane.runner.client, build, true); stashRunDebug(lane.runner.client, build); + stashBuildExpectations(key, fileSlug, lane.runner.client, build, true); if (build.success && !build.workflowChecks) { build.workflowChecks = await runWorkflowChecks({ workflow: build.workflowJsons[0], @@ -578,8 +608,8 @@ export function createBuildOrchestrator(deps: BuildOrchestratorDeps): BuildOrche const buildDurationMs = Date.now() - start; buildDurations.set(key, buildDurationMs); stashTranscript(build); - stashBuildExpectations(key, fileSlug, lane.runner.client, build, true); stashRunDebug(lane.runner.client, build); + stashBuildExpectations(key, fileSlug, lane.runner.client, build, true); if (build.success && !build.workflowChecks) { // No transcript in prebuilt mode, but the authored conversation still // carries the user's request — feed it so prompt-aware checks (e.g. @@ -647,8 +677,8 @@ export function createBuildOrchestrator(deps: BuildOrchestratorDeps): BuildOrche buildDurations.set(key, buildDurationMs); stashTranscript(build); stashAgentContext(key, lane.runner.client, build); - stashBuildExpectations(key, fileSlug, lane.runner.client, build, false); stashRunDebug(lane.runner.client, build); + stashBuildExpectations(key, fileSlug, lane.runner.client, build, false); logger.info( `[lane ${String(lane.laneNum)}] built ${fileSlug} (iteration ${String(iteration)}) thread=${build.threadId ?? 'none'} success=${String(build.success)}`, ); diff --git a/packages/@n8n/instance-ai/evaluations/system-prompts/memory-expectations-verify.ts b/packages/@n8n/instance-ai/evaluations/system-prompts/memory-expectations-verify.ts new file mode 100644 index 00000000000..6a6e0bf1184 --- /dev/null +++ b/packages/@n8n/instance-ai/evaluations/system-prompts/memory-expectations-verify.ts @@ -0,0 +1,46 @@ +export const MEMORY_EXPECTATIONS_VERIFY_PROMPT = `You are an expert evaluator for the memory subsystem of n8n's AI workflow builder. The builder agent held a multi-turn conversation, during which its context was managed by two background processes: an **Observer** that compresses older messages into a dense "observation block", and a **Reflector** that condenses that block once it grows too large. Your job is to judge author-written expectations about the agent's **context state** — what its memory actually retained, and what was placed in front of the model. + +## What makes this judgement different + +You are deliberately NOT given the conversation transcript, and NOT given the built workflow. You receive only the agent's context state — what was actually in front of the model. + +This is intentional. The question is never "did the agent ever know X" — it obviously did at the moment the user said it. The question is "was X still in the agent's context at the end". If you cannot find something in the material below, the honest verdict is that it was not retained, even if it seems like something the agent must have known. Do not infer content that is not present, and do not credit the agent for facts you assume were said earlier. + +Equally, do not fail a fact you *can* find. Your job is to report what the context contains, in either direction. + +## What you receive + +1. **Compressed observation block (final state)**: the last non-empty observation block the Observer produced — the compressed memory the agent carried at the end. The Observer only runs once a thread crosses a token threshold, so on a short thread this is legitimately empty. +2. **Raw message window**: the actual messages the model received on its last step. On an uncompressed thread this is where the facts live. Note what this is *not*: it is not the full conversation. Messages evicted from the window do not appear here — that is the point, and it is why you can trust it as a record of what the agent still had. +3. **Final system prompt**: what the model saw on its last step. This includes retrieved and baked context — instructions, knowledge base, and any facts retrieval placed there. It may be truncated; if an expectation depends on a region that is clearly truncated, say so rather than guessing. +4. **Capture summary**: deterministic counters (runs, steps, how many steps carried an observation block, whether compression ran). Treat these as authoritative and do not recount. +5. **Expectations**: a numbered list (indices start at 0). Judge each one. + +**A fact counts as retained if it is present in ANY of those three places.** The observation block, the raw window and the system prompt are three tiers of one context, and the model reads all of them. An empty observation block is never by itself a reason to fail an expectation — check the window before concluding a fact was lost. Where it matters, name the tier you found the fact in, since which tier held it is the interesting detail when comparing memory approaches. + +## How to judge + +- Judge **each expectation independently** and literally. Read exactly what it asserts. +- **Survival claims** ("fact X survived compression", "the channel agreed in turn 1 is still present") → look for the fact across all three sections. Present and correct anywhere → pass, naming where. Absent from all three → fail. Present but **altered** (a different value, a different channel, a different threshold) → fail, and say what it became. +- **Discrimination claims** ("retrieval returned the credential fact, not the schedule fact") → both halves must hold. The wanted fact present AND the competing fact not crowding it out. Quote both findings. When two similar values are both present, judge which one the context actually associates with the thing the expectation names — two values coexisting is not automatically a failure. +- **Absence claims** ("the stale value is no longer present") → pass only when it genuinely is not there, in any of the three sections. +- When compression never ran, say so in your reason — it is diagnostic, and materially different from "compression dropped it" — but decide pass/fail from the raw window, not from the empty observation block. +- **Quote the evidence.** Point at the phrase in the observation block or system prompt that decides the verdict. Avoid "likely", "presumably", "the agent would have". +- An expectation about something that simply is not there is a **fail**, never "n/a". +- Keep each \`reason\` to **one sentence**. + +## Output format + +Return an object with a \`results\` array — exactly one entry per expectation, using the 0-based \`index\` from the numbered list: + +\`\`\`json +{ + "results": [ + { "index": 0, "pass": true, "reason": "The observation block still records the target channel as #billing-alerts, matching what was agreed early in the thread." }, + { "index": 1, "pass": false, "reason": "The observation block retains the 30-second wait but no longer mentions the retry limit, so that fact did not survive compression." } + ] +} +\`\`\` + +Return a verdict for every numbered expectation. +`; diff --git a/packages/@n8n/instance-ai/evaluations/types.ts b/packages/@n8n/instance-ai/evaluations/types.ts index 6082bfda352..ffa7857e585 100644 --- a/packages/@n8n/instance-ai/evaluations/types.ts +++ b/packages/@n8n/instance-ai/evaluations/types.ts @@ -245,6 +245,12 @@ export interface WorkflowTestCase { * absence and content ("an agent was created and no workflow", "the agent instructions mention * escalating refunds"). Also run in prebuilt/MCP runs. Counted toward the pass rate. */ outcomeExpectations?: string[]; + /** Optional NL assertions about the agent's CONTEXT STATE — what survived compression, and what + * retrieval put in front of the model. LLM-judged from the captured run debug (the compressed + * observation block plus the final system prompt) rather than from the transcript or the + * workflow, so a miss is attributable to recall rather than to the build. Requires run debug, + * so skipped in prebuilt/MCP runs. Counted toward the pass rate. */ + memoryExpectations?: string[]; /** * Credentials visible to this case's build. Created for real before the build * and pinned as the thread's entire credential view — cases without this @@ -308,6 +314,12 @@ export interface BuildExpectationResult { /** Who owns a failed expectation. Stamped where the verdicts are attached to * a row (the only place that also knows whether the build died on infra). */ attribution?: EvalAttribution; + /** Set on verdicts graded against the CONTEXT STATE rather than the conversation + * or the workflow. Present so a memory miss is identifiable in the report and in + * `eval-results.json` without matching expectation strings back to the case file — + * the whole point of the kind being separate. Conversation-judged verdicts are + * left untagged, so absence means "process or outcome". */ + kind?: 'memory'; } export interface WorkflowTestCaseResult {