feat(core): Check exact context values deterministically instead of by judge (no-changelog)

Grading whether a value reached the model was LLM-judged, which puts a model on
the load-bearing path for the claims where a wrong verdict misleads most: if the
judge says a fact was absent, the conclusion is "the context system failed", and
that is an expensive thing to get wrong.

For a concrete value — a date, a channel, a column name, a parameter key — the
question is a substring search over data the harness already captures. Add
`contextAssertions`, checked without an LLM: it cannot hallucinate, costs nothing,
and needs no rubric. `mustAppear: false` covers the inverse, a stale value that
should have been dropped.

It also searches the untruncated context. The judge's view is capped so the part
worth grading stays in its attention, which means a value living in an elided
region reads as absent; this check has no such limit.

Matching is case-insensitive and callers are told to assert atomic values rather
than formatted phrases, because the same value is serialised with different
spacing and quoting depending on which context tier carries it.

Judged `memoryExpectations` stay for claims a string search cannot express — that
a retrieved sibling was the right *kind* of artifact, or that a value is present
but altered. Verdicts from both share `kind: "memory"`, so the report and results
file group them together, and both degrade to `incomplete` on the same missing
capture.

packages/@n8n/instance-ai/evaluations/README.md

@@ -283,6 +283,39 @@ A test case can declare optional natural-language assertions, split by what they
 
 `memoryExpectations` deliberately do **not** see the full conversation transcript or the built workflow. The raw message window is not the transcript: messages evicted from the window do not appear in it, which is precisely why it can be trusted as a record of what the agent *still had*. Grading against the transcript instead would let the judge satisfy "the agent still knew X" from the turn where X was first said even after the fact had fallen out of context — the confound this kind exists to remove. Keeping the inputs disjoint from the build is what makes a miss attributable to **recall**, so write them as claims about what the *context* contains, not about what the agent *did*. Their verdicts are tagged `kind: "memory"` in `eval-results.json` and badged `memory` in the HTML report. The Observer only compresses once a thread crosses its token threshold, so on a short thread the observation block is legitimately empty while every fact is still in the raw window — the judge is told this explicitly, and an absent observation block is **never** on its own a reason to fail. That matters: grading on the observation block alone would fail every memory expectation for a structural reason on exactly the short threads the suite is mostly made of. Which tier held the fact is reported in the verdict, since that is the interesting detail when comparing two memory approaches.
 
+#### `contextAssertions` — the deterministic alternative
+
+For a **concrete value**, prefer this over a `memoryExpectation`. It asks the same
+question — did this reach the model? — by substring search over the captured context,
+with no LLM involved:
+
+```json
+"contextAssertions": [
+  { "text": "#ops-alerts", "note": "the house alert channel, from a sibling workflow" },
+  { "text": "triggerAtHour" },
+  { "text": "email_address", "mustAppear": false, "note": "the renamed column, which should have been dropped" }
+]
+```
+
+`mustAppear` defaults to true; set it false for a stale value that should be gone.
+Verdicts are tagged `kind: "memory"` like the judged ones and name the tier the value
+was found in.
+
+Why prefer it: it cannot hallucinate, costs nothing, needs no rubric, and it searches
+the **untruncated** context — the judge's view is capped so the interesting part stays
+in its attention, but this check has no such limit. That keeps the judge off the
+load-bearing path for exactly the claims where a wrong verdict would be most
+misleading.
+
+Two rules. Assert **atomic** values (`triggerAtHour`, `#ops-alerts`, `2026-03-01`),
+not formatted phrases (`triggerAtHour: 6`) — the same value is serialised with
+different spacing and quoting depending on which tier carries it. And matching is
+case-insensitive, because casing drifts as content is re-rendered through tool
+payloads and a casing difference is never the finding.
+
+Keep `memoryExpectations` for claims a string search cannot express — "the retrieved
+sibling was the same *kind* of workflow", "the value is present but altered".
+
 #### Writing a memory expectation that can actually fail
 
 Two things make a memory expectation harder to falsify than it looks. Both were measured, not theorised.

packages/@n8n/instance-ai/evaluations/__tests__/context-assertions.test.ts

@@ -0,0 +1,124 @@
+import type { InstanceAiRunDebugResponse } from '@n8n/api-types';
+
+import { checkContextAssertions } from '../harness/context-assertions';
+
+/** Real capture shape: system prompt string plus SDK message parts. */
+function debug(system: string, messages?: unknown[]): InstanceAiRunDebugResponse[] {
+	return [
+		{
+			threadId: 't1',
+			runId: 'r1',
+			startedAt: 1,
+			steps: [{ stepNumber: 0, input: messages ? { system, messages } : { system } }],
+			workflowCode: [],
+		},
+	];
+}
+
+describe('checkContextAssertions', () => {
+	it('returns nothing when a case declares none', () => {
+		expect(checkContextAssertions(undefined, debug('sys'))).toEqual([]);
+		expect(checkContextAssertions([], debug('sys'))).toEqual([]);
+	});
+
+	it('passes when the value is present, naming the tier it was found in', () => {
+		const [v] = checkContextAssertions(
+			[{ text: '#ops-alerts' }],
+			debug('sys', [{ role: 'user', content: 'post failures to #ops-alerts please' }]),
+		);
+		expect(v.pass).toBe(true);
+		expect(v.kind).toBe('memory');
+		expect(v.reason).toContain('message window');
+	});
+
+	it('fails when the value never reached the context', () => {
+		const [v] = checkContextAssertions(
+			[{ text: 'maxTries' }],
+			debug('sys', [{ role: 'user', content: 'just fetch it' }]),
+		);
+		expect(v.pass).toBe(false);
+		expect(v.reason).toContain('not present');
+	});
+
+	it('finds values that arrived inside a tool result', () => {
+		// The whole point: fetched context arrives as a tool payload, not as prose.
+		const [v] = checkContextAssertions(
+			[{ text: 'triggerAtHour' }],
+			debug('sys', [
+				{
+					role: 'assistant',
+					content: [
+						{
+							type: 'tool-result',
+							toolName: 'workflows',
+							output: { nodes: [{ parameters: { rule: { triggerAtHour: 6 } } }] },
+						},
+					],
+				},
+			]),
+		);
+		expect(v.pass).toBe(true);
+	});
+
+	it('searches the system prompt and the observation block too', () => {
+		const sys = 'You are a builder.\n<observations>channel is #ops-alerts</observations>';
+		const [inObs] = checkContextAssertions([{ text: '#ops-alerts' }], debug(sys));
+		expect(inObs.pass).toBe(true);
+		expect(inObs.reason).toContain('observation block');
+
+		const [inSys] = checkContextAssertions([{ text: 'You are a builder' }], debug(sys));
+		expect(inSys.pass).toBe(true);
+		expect(inSys.reason).toContain('system prompt');
+	});
+
+	it('supports must-NOT-appear for stale values that should have been dropped', () => {
+		const stale = debug('sys', [{ role: 'user', content: 'the old column was email_address' }]);
+		const [present] = checkContextAssertions([{ text: 'email_address', mustAppear: false }], stale);
+		expect(present.pass).toBe(false);
+		expect(present.reason).toContain('still present');
+
+		const [absent] = checkContextAssertions(
+			[{ text: 'email_address', mustAppear: false }],
+			debug('sys', [{ role: 'user', content: 'the column is subscriber_email' }]),
+		);
+		expect(absent.pass).toBe(true);
+		expect(absent.reason).toContain('correctly absent');
+	});
+
+	it('matches case-insensitively, since casing drifts as content is re-rendered', () => {
+		const [v] = checkContextAssertions(
+			[{ text: 'notify ops' }],
+			debug('sys', [{ role: 'user', content: 'the alert node is called Notify Ops' }]),
+		);
+		expect(v.pass).toBe(true);
+	});
+
+	it('searches past the judge truncation cap', () => {
+		// The judge's view is capped so the interesting part stays in its attention;
+		// this check has no such limit, so a value in an elided region is still found.
+		const buried = [
+			{ role: 'user', content: 'x'.repeat(200_000) },
+			{ role: 'user', content: 'BURIED_NEEDLE' },
+			{ role: 'user', content: 'y'.repeat(200_000) },
+		];
+		const [v] = checkContextAssertions([{ text: 'BURIED_NEEDLE' }], debug('sys', buried));
+		expect(v.pass).toBe(true);
+	});
+
+	it('records incomplete, not failed, when no run debug was captured', () => {
+		const [v] = checkContextAssertions([{ text: 'anything' }], undefined);
+		expect(v.pass).toBe(false);
+		expect(v.incomplete).toBe(true);
+		expect(v.kind).toBe('memory');
+		expect(v.reason).toContain('no run debug');
+	});
+
+	it('includes the author note in the verdict label', () => {
+		const [v] = checkContextAssertions(
+			[{ text: '2026-03-01', note: 'the cutover agreed at thread start' }],
+			debug('sys'),
+		);
+		expect(v.expectation).toContain('2026-03-01');
+		expect(v.expectation).toContain('the cutover agreed at thread start');
+	});
+});

packages/@n8n/instance-ai/evaluations/build-expectations/select.ts

@@ -1,7 +1,12 @@
 import { allFailVerdicts } from './assertion-judge';
 import { asMemoryVerdicts, collectAllExpectations, collectExpectations } from './collect';
 import type { EvalLogger } from '../harness/logger';
-import type { BuildExpectationResult, TranscriptTurn, WorkflowTestCase } from '../types';
+import type {
+	BuildExpectationResult,
+	ContextAssertion,
+	TranscriptTurn,
+	WorkflowTestCase,
+} from '../types';
 import { conversationUserTurnsAsText } from '../utils/conversation-text';
 
 /** Recorded on expectations we refuse to judge, so the reason reaches the report. */
@@ -15,7 +20,12 @@ const NO_CONTEXT_STATE_REASON =
 export interface SelectAuthorExpectationsArgs {
 	testCase: Pick<
 		WorkflowTestCase,
-		'processExpectations' | 'outcomeExpectations' | 'memoryExpectations' | 'conversation' | 'seed'
+		| 'processExpectations'
+		| 'outcomeExpectations'
+		| 'memoryExpectations'
+		| 'contextAssertions'
+		| 'conversation'
+		| 'seed'
 	>;
 	/** Captured build transcript, if any. Empty/absent for prebuilt/MCP builds. */
 	transcript: TranscriptTurn[] | undefined;
@@ -56,6 +66,9 @@ export function selectAuthorExpectations(args: SelectAuthorExpectationsArgs): {
 	/** Expectations for the context-state judge, which grades them against the
 	 *  captured run debug rather than the transcript. */
 	memoryExpectations: string[];
+	/** Deterministic context claims, checked by substring search rather than judged.
+	 *  Gated exactly like memoryExpectations: both need captured run debug. */
+	contextAssertions: ContextAssertion[];
 	transcript: TranscriptTurn[];
 	/** Expectations deliberately left ungraded, already shaped as `incomplete`
 	 *  verdicts. Recording them keeps the case's unit count stable across runs (so
@@ -94,11 +107,18 @@ export function selectAuthorExpectations(args: SelectAuthorExpectationsArgs): {
 		return {
 			expectations: [],
 			memoryExpectations: [],
+			contextAssertions: [],
 			transcript,
 			unjudged: [
 				...allFailVerdicts(collectExpectations(testCase), NO_AGENT_OUTPUT_REASON),
 				...asMemoryVerdicts(
-					allFailVerdicts(testCase.memoryExpectations ?? [], NO_AGENT_OUTPUT_REASON),
+					allFailVerdicts(
+						[
+							...(testCase.memoryExpectations ?? []),
+							...contextAssertionLabels(testCase.contextAssertions),
+						],
+						NO_AGENT_OUTPUT_REASON,
+					),
 				),
 			],
 		};
@@ -115,17 +135,35 @@ export function selectAuthorExpectations(args: SelectAuthorExpectationsArgs): {
 	// us the process expectations but not the memory ones; if the run debug really is
 	// missing, the judge records that specific reason itself.
 	const authoredMemory = testCase.memoryExpectations ?? [];
-	if (authoredMemory.length > 0 && isPrebuilt) {
+	const authoredAssertions = testCase.contextAssertions ?? [];
+	if ((authoredMemory.length > 0 || authoredAssertions.length > 0) && isPrebuilt) {
+		const ungraded = [...authoredMemory, ...contextAssertionLabels(authoredAssertions)];
 		logger.warn(
-			`  Prebuilt/MCP build — leaving ${String(authoredMemory.length)} memory expectation(s) ungraded; this run ran no instance-ai thread`,
+			`  Prebuilt/MCP build — leaving ${String(ungraded.length)} context check(s) ungraded; this run ran no instance-ai thread`,
 		);
 		return {
 			expectations,
 			memoryExpectations: [],
+			contextAssertions: [],
 			transcript,
-			unjudged: asMemoryVerdicts(allFailVerdicts(authoredMemory, NO_CONTEXT_STATE_REASON)),
+			unjudged: asMemoryVerdicts(allFailVerdicts(ungraded, NO_CONTEXT_STATE_REASON)),
 		};
 	}
 
-	return { expectations, memoryExpectations: authoredMemory, transcript, unjudged: [] };
+	return {
+		expectations,
+		memoryExpectations: authoredMemory,
+		contextAssertions: authoredAssertions,
+		transcript,
+		unjudged: [],
+	};
+}
+
+/** Verdict labels for assertions, matching what `checkContextAssertions` produces, so
+ *  an ungraded assertion is named the same way on every path. */
+function contextAssertionLabels(assertions: ContextAssertion[] | undefined): string[] {
+	return (assertions ?? []).map(
+		(a) =>
+			`context ${a.mustAppear === false ? 'excludes' : 'contains'} "${a.text}"${a.note ? ` (${a.note})` : ''}`,
+	);
 }

packages/@n8n/instance-ai/evaluations/cli/mcp-builder.ts

@@ -189,6 +189,10 @@ export const MCP_BUILD_KEY_SUPPORT: Record<
 	// skipped there for MCP builds (which run no instance-ai thread, so capture no
 	// context state), and declaring them needs no build-side setup.
 	memoryExpectations: 'supported',
+	// Deterministic post-build check over captured context; like the expectation
+	// kinds it needs no build-side setup, and it degrades to `incomplete` when a run
+	// captured no context state (as MCP builds never do).
+	contextAssertions: 'supported',
 	// Forbidden legacy key — the schema rejects it at load, so it can never
 	// reach this check; classified only to keep the map schema-complete.
 	buildExpectations: 'supported',

packages/@n8n/instance-ai/evaluations/harness/context-assertions.ts

@@ -0,0 +1,81 @@
+import type { InstanceAiRunDebugResponse } from '@n8n/api-types';
+
+import { summarizeMemoryContext } from './memory-context';
+import type { BuildExpectationResult, ContextAssertion } from '../types';
+
+/** Recorded when there was no captured context to search, so a missing capture reads
+ *  as "not checked" rather than as the value being absent. */
+const NO_CONTEXT_REASON =
+	'not checked — no run debug was captured for this thread, so there was no context state to search';
+
+/**
+ * Deterministically check whether exact values reached the agent's context.
+ *
+ * Deliberately not LLM-graded. For a concrete value — a date, a channel, a column
+ * name, a parameter key — "does this string appear in what the model was sent" is a
+ * substring search over data already captured, so it cannot hallucinate, costs
+ * nothing, and needs no rubric. That keeps the judge off the load-bearing path for
+ * the claims that matter most, leaving it for genuinely fuzzy ones ("the retrieved
+ * sibling was the right *kind* of workflow").
+ *
+ * Two deliberate choices:
+ * - Searches the **untruncated** context. The judge's view is capped so the
+ *   interesting part stays in its attention; this check has no such constraint, so a
+ *   value living in an elided region is still found rather than reported missing.
+ * - Matching is case-insensitive. Casing drifts freely as content is re-rendered
+ *   through tool payloads, and a casing difference is never the finding.
+ *
+ * Assert **atomic** values (`triggerAtHour`, `#ops-alerts`, `2026-03-01`), not
+ * formatted phrases (`triggerAtHour: 6`) — the same value is serialised with
+ * different spacing and quoting depending on which tier it arrives in.
+ */
+export function checkContextAssertions(
+	assertions: ContextAssertion[] | undefined,
+	runDebug: InstanceAiRunDebugResponse[] | undefined,
+): BuildExpectationResult[] {
+	if (!assertions || assertions.length === 0) return [];
+
+	const describe = (a: ContextAssertion) =>
+		`context ${a.mustAppear === false ? 'excludes' : 'contains'} "${a.text}"${a.note ? ` (${a.note})` : ''}`;
+
+	const summary = summarizeMemoryContext(runDebug);
+	if (!summary) {
+		return assertions.map((a) => ({
+			expectation: describe(a),
+			pass: false,
+			reason: NO_CONTEXT_REASON,
+			incomplete: true,
+			kind: 'memory' as const,
+		}));
+	}
+
+	// One haystack across all three tiers: the question is whether the model had the
+	// value at all, not which tier carried it. Untruncated on purpose.
+	const tiers: Array<[string, string]> = [
+		['observation block', summary.observations ?? ''],
+		['message window', summary.finalMessageWindow],
+		['system prompt', summary.finalSystemPrompt],
+	];
+
+	return assertions.map((assertion) => {
+		const needle = assertion.text.toLowerCase();
+		const found = tiers
+			.filter(([, text]) => text.toLowerCase().includes(needle))
+			.map(([tier]) => tier);
+		const mustAppear = assertion.mustAppear !== false;
+		const pass = mustAppear ? found.length > 0 : found.length === 0;
+		const where = found.length > 0 ? found.join(', ') : 'none';
+		return {
+			expectation: describe(assertion),
+			pass,
+			reason: mustAppear
+				? found.length > 0
+					? `found in: ${where} (exact match, case-insensitive)`
+					: 'not present in the observation block, message window or system prompt'
+				: found.length === 0
+					? 'correctly absent from all three context tiers'
+					: `still present in: ${where}`,
+			kind: 'memory' as const,
+		};
+	});
+}

packages/@n8n/instance-ai/evaluations/harness/schema.ts

@@ -143,6 +143,27 @@ const evalTestCaseObjectSchema = z
 		 *  recall rather than to the build. Needs run debug, so skipped in prebuilt/MCP runs.
 		 *  Counted as units. */
 		memoryExpectations: z.array(z.string().min(1)).optional(),
+		/** Exact values that must (or must not) appear in the agent's captured context.
+		 *  Checked deterministically by substring search — no LLM, so it cannot
+		 *  hallucinate and needs no rubric. Use for concrete values (a date, a channel,
+		 *  a column name, a parameter key); leave genuinely fuzzy claims to
+		 *  `memoryExpectations`. Assert ATOMIC values, not formatted phrases: the same
+		 *  value is serialised with different spacing depending on which tier carries
+		 *  it. Counted as units. */
+		contextAssertions: z
+			.array(
+				z
+					.object({
+						text: z.string().min(1),
+						/** Omit or true → must appear. False → must NOT appear (a stale value
+						 *  that should have been dropped). */
+						mustAppear: z.boolean().optional(),
+						/** Shown in the verdict when the raw string is not self-explanatory. */
+						note: z.string().min(1).optional(),
+					})
+					.strict(),
+			)
+			.optional(),
 		/**
 		 * Removed in favour of the process/outcome split. Declared as a forbidden key (rather
 		 * than dropped from the shape) so a legacy fixture fails loudly with a migration hint,
@@ -279,12 +300,13 @@ export const EvalTestCaseSchema = evalTestCaseObjectSchema
 			(c.executionScenarios?.length ?? 0) === 0 &&
 			(c.processExpectations?.length ?? 0) === 0 &&
 			(c.outcomeExpectations?.length ?? 0) === 0 &&
-			(c.memoryExpectations?.length ?? 0) === 0
+			(c.memoryExpectations?.length ?? 0) === 0 &&
+			(c.contextAssertions?.length ?? 0) === 0
 		) {
 			ctx.addIssue({
 				code: z.ZodIssueCode.custom,
 				message:
-					'a case needs at least one executionScenario, or a process/outcome/memory expectation to grade it',
+					'a case needs at least one executionScenario, or a process/outcome/memory expectation, or a context assertion to grade it',
 			});
 		}
 	});

packages/@n8n/instance-ai/evaluations/run/build-orchestrator.ts

@@ -43,6 +43,7 @@ import {
 } from '../harness/build-workflow';
 import { captureThreadRunDebug } from '../harness/capture-run-debug';
 import { effectiveTimeoutMs, runWorkflowChecks } from '../harness/cleanup';
+import { checkContextAssertions } from '../harness/context-assertions';
 import {
 	credentialSetupExpectationTexts,
 	runCredentialSetupChecks,
@@ -442,13 +443,14 @@ export function createBuildOrchestrator(deps: BuildOrchestratorDeps): BuildOrche
 					);
 				})
 			: undefined;
-		const { expectations, memoryExpectations, transcript, unjudged } = selectAuthorExpectations({
-			testCase,
-			transcript: build.transcript,
-			buildSucceeded: build.success,
-			isPrebuilt,
-			logger,
-		});
+		const { expectations, memoryExpectations, contextAssertions, transcript, unjudged } =
+			selectAuthorExpectations({
+				testCase,
+				transcript: build.transcript,
+				buildSucceeded: build.success,
+				isPrebuilt,
+				logger,
+			});
 		// Attributed here, where we still know WHY the build ended: a build that
 		// died on infra produced nothing to judge, so its expectations are unowned
 		// rather than the agent's miss. Both readers of this map (the row outputs
@@ -465,7 +467,12 @@ export function createBuildOrchestrator(deps: BuildOrchestratorDeps): BuildOrche
 			verdicts: BuildExpectationResult[] | Promise<BuildExpectationResult[]>,
 		): Promise<BuildExpectationResult[]> =>
 			injected ? [...(await verdicts), ...(await injected)] : await verdicts;
-		if (unjudged.length === 0 && expectations.length === 0 && memoryExpectations.length === 0) {
+		if (
+			unjudged.length === 0 &&
+			expectations.length === 0 &&
+			memoryExpectations.length === 0 &&
+			contextAssertions.length === 0
+		) {
 			if (injected) buildExpectationsByKey.set(key, injected);
 			return;
 		}
@@ -495,6 +502,11 @@ export function createBuildOrchestrator(deps: BuildOrchestratorDeps): BuildOrche
 		// Reads the run debug stashed for this build — hence stashRunDebug runs first.
 		// Graded on its own inputs (observation block + final system prompt) and never
 		// the transcript, so a miss is attributable to recall, not to the build.
+		const checkAssertions = async (): Promise<BuildExpectationResult[]> =>
+			checkContextAssertions(
+				contextAssertions,
+				build.threadId ? await runDebugByThreadId.get(build.threadId) : undefined,
+			);
 		const judgeMemory = async (): Promise<BuildExpectationResult[]> =>
 			await verifyMemoryExpectations(
 				memoryExpectations,
@@ -520,11 +532,12 @@ export function createBuildOrchestrator(deps: BuildOrchestratorDeps): BuildOrche
 			key,
 			withInjected(
 				(async () => {
-					const [conversation, memory] = await Promise.all([
+					const [conversation, memory, asserted] = await Promise.all([
 						expectations.length > 0 ? judgeConversation() : Promise.resolve([]),
 						memoryExpectations.length > 0 ? judgeMemory() : Promise.resolve([]),
+						contextAssertions.length > 0 ? checkAssertions() : Promise.resolve([]),
 					]);
-					return [...attribute(unjudged), ...conversation, ...memory];
+					return [...attribute(unjudged), ...conversation, ...memory, ...attribute(asserted)];
 				})(),
 			),
 		);

packages/@n8n/instance-ai/evaluations/types.ts

@@ -251,6 +251,10 @@ export interface WorkflowTestCase {
 	 *  workflow, so a miss is attributable to recall rather than to the build. Requires run debug,
 	 *  so skipped in prebuilt/MCP runs. Counted toward the pass rate. */
 	memoryExpectations?: string[];
+	/** Exact values that must (or must not) appear in the agent's captured context,
+	 *  checked by deterministic substring search rather than by the judge. Counted
+	 *  toward the pass rate. See `harness/context-assertions.ts`. */
+	contextAssertions?: ContextAssertion[];
 	/**
 	 * Credentials visible to this case's build. Created for real before the build
 	 * and pinned as the thread's entire credential view — cases without this
@@ -303,6 +307,14 @@ export interface ExecutionScenarioResult {
 	incomplete?: boolean;
 }
 
+/** One deterministic claim about the agent's captured context. */
+export interface ContextAssertion {
+	text: string;
+	/** Omit or true → must appear. False → must NOT appear. */
+	mustAppear?: boolean;
+	note?: string;
+}
+
 /** Verdict for one author-written build expectation. Scored as a unit in the
  *  pass rate alongside execution scenarios. */
 export interface BuildExpectationResult {