github-mirror/n8n
1
0
Fork 0
mirror of https://github.com/n8n-io/n8n.git synced 2026-10-17 05:29:44 +02:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity Graph

Merge branch 'master' into cstuncsik/hackmation-sample-data-pinner

Browse Source
This commit is contained in:
Csaba Tuncsik 2026-05-14 05:59:04 +02:00 committed by GitHub
commit 43ca24e0f1
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
185 changed files with 18527 additions and 822 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
.claude/plugins/n8n/skills/setup-experiment/SKILL.md Normal file
View File

@ -0,0 +1,14 @@
---
name: setup-experiment
description: >-
Scaffolds new `packages/frontend/editor-ui` experiments using the n8n
experiment standard. Use when creating a new editor-ui experiment, adding a
PostHog experiment scaffold, or setting up experiment files for a new
experiment.
---
# Setup experiment
Use this skill when scaffolding a new `packages/frontend/editor-ui` experiment.
Follow [reference.md](reference.md) for the canonical workflow, scaffold rules, and output reminders.

140
.claude/plugins/n8n/skills/setup-experiment/reference.md Normal file
View File

@ -0,0 +1,140 @@
# Setup experiment reference
Use this workflow when scaffolding a new experiment under `packages/frontend/editor-ui/src/experiments/`.
## Ask or infer
1. Experiment folder name, for example `setupPanelV2`.
2. Experiment constant name, for example `SETUP_PANEL_V2_EXPERIMENT`.
3. Variant shape:
- simple `control/variant`, or
- multi-variant.
4. Public API shape:
- default to a Pinia store,
- only use a composable when the experiment is genuinely tiny and stateless.
5. Whether persisted UI state is needed.
6. Whether optional `components/`, `data/`, `workflows/`, or host-surface wiring are in scope.
## Always do
1. Start in `packages/frontend/editor-ui/src/app/constants/experiments.ts`.
2. Add the experiment constant.
3. Add the experiment name to `EXPERIMENTS_TO_TRACK`.
4. Create `packages/frontend/editor-ui/src/experiments/<name>/`.
5. Pick one public API path:
- Store-backed by default: add a `STORES.EXPERIMENT_*` entry in `packages/frontend/@n8n/stores/src/constants.ts`, then create `stores/<name>.store.ts` and `stores/<name>.store.test.ts`.
- Tiny stateless composable only by exception: skip the `STORES.EXPERIMENT_*` entry and create a single composable such as `use<Name>Experiment.ts` with an adjacent test when needed.
6. If the experiment needs persisted UI state or a shared mutable API, use the store path, not the composable path.
## Enrollment standard
- Use `isEnabled` for PostHog enrollment only.
- For simple experiments, use `posthogStore.isVariantEnabled(EXPERIMENT.name, EXPERIMENT.variant)`.
- For multi-variant experiments, expose `currentVariant` and derive booleans from it.
- Keep display state separate with `shouldShow*`, `can*`, `hasDismissed*`, or `hasInteracted*`.
Do not mix enrollment with:
- `userIsTrialing`,
- cloud-only checks,
- selected-app checks,
- workflow-count checks,
- dismissal state.
If the user wants those conditions, treat them as local display logic, not enrollment logic.
## Persistence standard
Use `packages/frontend/editor-ui/src/app/composables/useStorage.ts` for experiment-owned persisted UI state.
Use it for:
- dismissed state,
- interaction state,
- last-seen state.
Do not use raw `localStorage` for new experiment-owned state unless the experiment must touch an existing non-experiment app key.
## Telemetry standard
- Keep `"User is part of experiment"` tracking centralized.
- Scaffold only experiment-specific `track*` helpers such as `trackViewed`, `trackClicked`, `trackDismissed`, or `trackCompleted`.
## Store scaffold
Default to this store shape and adapt names as needed:
```ts
import { useTelemetry } from '@/app/composables/useTelemetry';
import { useStorage } from '@/app/composables/useStorage';
import { SETUP_PANEL_V2_EXPERIMENT } from '@/app/constants';
import { usePostHog } from '@/app/stores/posthog.store';
import { STORES } from '@n8n/stores';
import { defineStore } from 'pinia';
import { computed } from 'vue';
const DISMISSED_KEY = 'N8N_SETUP_PANEL_V2_DISMISSED';
export const useSetupPanelV2Store = defineStore(STORES.EXPERIMENT_SETUP_PANEL_V2, () => {
const posthogStore = usePostHog();
const telemetry = useTelemetry();
const dismissed = useStorage(DISMISSED_KEY);
const isEnabled = computed(() =>
posthogStore.isVariantEnabled(
SETUP_PANEL_V2_EXPERIMENT.name,
SETUP_PANEL_V2_EXPERIMENT.variant,
),
);
const isDismissed = computed(() => dismissed.value === 'true');
const shouldShowCallout = computed(() => isEnabled.value && !isDismissed.value);
function dismiss() {
dismissed.value = 'true';
}
function trackViewed() {
telemetry.track('Setup panel v2 viewed');
}
return {
isEnabled,
isDismissed,
shouldShowCallout,
dismiss,
trackViewed,
};
});
```
## Test scaffold
Default to one public API test next to the store. Use `stores/<name>.store.test.ts` as the standard location. Mirror the local style from:
- `packages/frontend/editor-ui/src/experiments/credentialsAppSelection/stores/credentialsAppSelection.store.test.ts`
- `packages/frontend/editor-ui/src/experiments/resourceCenter/stores/__tests__/resourceCenter.store.test.ts`
Use `__tests__/` only when the surrounding experiment already follows that layout or when the local package pattern makes colocating the test awkward.
The default assertions should cover:
1. enrollment logic,
2. persisted state when present,
3. experiment-specific telemetry helpers when present.
## Do not do by default
- Do not wire routes, layouts, sidebars, or modals automatically.
- Do not add empty `components/`, `data/`, or `workflows/` folders unless requested.
- Do not add product gating on top of PostHog enrollment unless the user explicitly wants local display logic.
- Do not scaffold participation tracking inside the experiment.
## Output reminder
At the end of the scaffold, explicitly tell the user what is still manual, for example:
- host-surface wiring,
- i18n keys,
- route guards,
- modal registration,
- extra workflow payload files.

110
.code-health-baseline.json
View File

@ -1,7 +1,7 @@
{
"version": 1,
"generated": "2026-05-12T09:37:31.489Z",
"totalViolations": 82,
"generated": "2026-05-13T10:23:52.870Z",
"totalViolations": 87,
"violations": {
"packages/@n8n/ai-workflow-builder.ee/package.json": [
{
@ -52,13 +52,25 @@
"packages/@n8n/nodes-langchain/package.json": [
{
"rule": "catalog-violations",
"line": 292,
"line": 266,
"message": "@n8n/typeorm@0.3.20-16 should use \"catalog:\" (exists in pnpm-workspace.yaml)",
"hash": "a7c1bca1c439"
},
{
"rule": "catalog-violations",
"line": 267,
"message": "mysql2@3.17.0 should use \"catalog:\" (exists in pnpm-workspace.yaml)",
"hash": "2d7d1b715d0c"
},
{
"rule": "catalog-violations",
"line": 293,
"message": "openai@^6.34.0 should use \"catalog:\" (exists in pnpm-workspace.yaml)",
"hash": "3c1f53f0afe3"
},
{
"rule": "catalog-violations",
"line": 299,
"line": 300,
"message": "tmp-promise appears in 4 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "88d67e2ef747"
},
@ -70,25 +82,25 @@
},
{
"rule": "catalog-violations",
"line": 274,
"line": 275,
"message": "cheerio appears in 2 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "8cd029bb871e"
},
{
"rule": "catalog-violations",
"line": 284,
"line": 285,
"message": "jsdom appears in 4 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "26f20ebea4b1"
},
{
"rule": "catalog-violations",
"line": 289,
"line": 290,
"message": "mongodb appears in 2 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "46cb48884e22"
},
{
"rule": "catalog-violations",
"line": 293,
"line": 294,
"message": "pdf-parse appears in 2 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "0c7d44a9c2e4"
}
@ -220,49 +232,49 @@
"packages/@n8n/instance-ai/package.json": [
{
"rule": "catalog-violations",
"line": 80,
"line": 81,
"message": "@ai-sdk/anthropic appears in 3 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "5b2153508e47"
},
{
"rule": "catalog-violations",
"line": 86,
"line": 87,
"message": "@types/psl appears in 2 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "56dabb51b433"
},
{
"rule": "catalog-violations",
"line": 56,
"line": 57,
"message": "@mozilla/readability appears in 5 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "8fa6b9a8fc91"
},
{
"rule": "catalog-violations",
"line": 64,
"line": 65,
"message": "csv-parse appears in 3 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "8f082fc2e8b6"
},
{
"rule": "catalog-violations",
"line": 71,
"line": 72,
"message": "turndown appears in 3 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "9a9d97065952"
},
{
"rule": "catalog-violations",
"line": 87,
"line": 88,
"message": "@types/turndown appears in 3 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "12e346c47b39"
},
{
"rule": "catalog-violations",
"line": 50,
"line": 51,
"message": "@joplin/turndown-plugin-gfm appears in 2 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "a3cf1504b5c2"
},
{
"rule": "catalog-violations",
"line": 68,
"line": 69,
"message": "pdf-parse appears in 2 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "283fa9114c03"
}
@ -290,61 +302,61 @@
"packages/nodes-base/package.json": [
{
"rule": "catalog-violations",
"line": 911,
"line": 913,
"message": "change-case appears in 5 packages with 3 different versions — add to pnpm-workspace.yaml catalog",
"hash": "2d1fab7a5b05"
},
{
"rule": "catalog-violations",
"line": 961,
"line": 963,
"message": "semver appears in 4 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "2daf37aa14e4"
},
{
"rule": "catalog-violations",
"line": 966,
"line": 968,
"message": "tmp-promise appears in 4 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "3f93c404ae9c"
},
{
"rule": "catalog-violations",
"line": 900,
"line": 902,
"message": "@mozilla/readability appears in 5 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "ca4ac788adc6"
},
{
"rule": "catalog-violations",
"line": 912,
"line": 914,
"message": "cheerio appears in 2 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "1a1b5bbc50c9"
},
{
"rule": "catalog-violations",
"line": 915,
"line": 917,
"message": "csv-parse appears in 3 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "781db4a1e068"
},
{
"rule": "catalog-violations",
"line": 917,
"line": 919,
"message": "eventsource appears in 2 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "9795e6c6d9e9"
},
{
"rule": "catalog-violations",
"line": 930,
"line": 932,
"message": "jsdom appears in 4 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "02341f2b5e3e"
},
{
"rule": "catalog-violations",
"line": 941,
"line": 943,
"message": "mongodb appears in 2 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "f688907d087a"
},
{
"rule": "catalog-violations",
"line": 892,
"line": 894,
"message": "eslint-plugin-n8n-nodes-base appears in 2 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "ac254baa61f9"
}
@ -384,19 +396,19 @@
},
{
"rule": "catalog-violations",
"line": 90,
"line": 91,
"message": "prettier appears in 3 packages with 3 different versions — add to pnpm-workspace.yaml catalog",
"hash": "9e9c7ec09a0b"
},
{
"rule": "catalog-violations",
"line": 92,
"line": 93,
"message": "semver appears in 4 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "d8c606e42c92"
},
{
"rule": "catalog-violations",
"line": 77,
"line": 78,
"message": "esprima-next appears in 3 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "62156c2613b2"
}
@ -491,6 +503,14 @@
"hash": "733c3960022e"
}
],
"packages/@n8n/computer-use/package.json": [
{
"rule": "catalog-violations",
"line": 47,
"message": "eventsource appears in 2 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "f50c1eee2ed6"
}
],
"packages/workflow/package.json": [
{
"rule": "catalog-violations",
@ -511,14 +531,6 @@
"hash": "b660317b5f6f"
}
],
"packages/@n8n/computer-use/package.json": [
{
"rule": "catalog-violations",
"line": 47,
"message": "eventsource appears in 2 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "f50c1eee2ed6"
}
],
"packages/@n8n/eslint-plugin-community-nodes/package.json": [
{
"rule": "catalog-violations",
@ -540,6 +552,30 @@
"message": "stylelint appears in 2 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "955f3fe044c7"
}
],
"packages/@n8n/db/src/migrations/common/1783000000000-CreateAgentTables.ts": [
{
"rule": "migration-timestamp",
"line": 1,
"message": "1783000000000-CreateAgentTables.ts is prefixed with a future timestamp (1783000000000, ~50d ahead of now). Migration prefixes must be the exact Date.now() value at the time of creation — AI agents commonly fabricate future timestamps.",
"hash": "d81428d567aa"
}
],
"packages/@n8n/db/src/migrations/common/1783000000001-CreateAgentExecutionTables.ts": [
{
"rule": "migration-timestamp",
"line": 1,
"message": "1783000000001-CreateAgentExecutionTables.ts is prefixed with a future timestamp (1783000000001, ~50d ahead of now). Migration prefixes must be the exact Date.now() value at the time of creation — AI agents commonly fabricate future timestamps.",
"hash": "e33a8a794126"
}
],
"packages/@n8n/db/src/migrations/common/1784000000000-CreateAgentObservationTables.ts": [
{
"rule": "migration-timestamp",
"line": 1,
"message": "1784000000000-CreateAgentObservationTables.ts is prefixed with a future timestamp (1784000000000, ~61d ahead of now). Migration prefixes must be the exact Date.now() value at the time of creation — AI agents commonly fabricate future timestamps.",
"hash": "4bae5ef065ad"
}
]
}
}

1
.github/workflows/ci-pr-quality.yml vendored
View File

@ -134,6 +134,7 @@ jobs:
pnpm-workspace.yaml
.code-health-baseline.json
packages/testing/code-health/**
.github/workflows/**
check-static-analysis:
name: Static Analysis

22
packages/@n8n/api-types/src/index.ts
View File

@ -221,6 +221,7 @@ export {
type DataTableListSortBy,
dateTimeSchema,
dataTableColumnNameSchema,
dataTableIdSchema,
} from './schemas/data-table.schema';
export type {
@ -439,6 +440,27 @@ export {
type StartTestRunPayload,
} from './schemas/evaluations.schema';
export {
EVAL_COLLECTIONS_FLAG,
evalCollectionVersionEntrySchema,
createEvaluationCollectionSchema,
CreateEvaluationCollectionDto,
updateEvaluationCollectionSchema,
UpdateEvaluationCollectionDto,
addRunToCollectionSchema,
AddRunToCollectionDto,
type EvalCollectionVersionEntry,
type CreateEvaluationCollectionPayload,
type UpdateEvaluationCollectionPayload,
type AddRunToCollectionPayload,
type EvalCollectionRunStatus,
type EvaluationCollectionRecord,
type EvaluationCollectionRunSummary,
type EvaluationCollectionDetail,
type EvalVersionEntry,
type EvalVersionsResponse,
} from './schemas/eval-collections.schema';
export { ALLOWED_DOMAINS, isAllowedDomain } from './utils/allowed-domains';
export type { PublishTimelineEvent } from './schemas/workflow-publish-timeline.schema';

5
packages/@n8n/api-types/src/schemas/data-table.schema.ts
View File

@ -5,7 +5,10 @@ import type { ListDataTableQueryDto } from '../dto';
export const insertRowReturnType = z.union([z.literal('all'), z.literal('count'), z.literal('id')]);
export const dataTableNameSchema = z.string().trim().min(1).max(128);
export const dataTableIdSchema = z.string().max(36);
export const dataTableIdSchema = z
.string()
.max(36)
.regex(/^[a-zA-Z0-9]+$/);
// Postgres does not allow leading numbers or -
export const DATA_TABLE_COLUMN_REGEX = /^[a-zA-Z][a-zA-Z0-9_]*$/;

112
packages/@n8n/api-types/src/schemas/eval-collections.schema.ts Normal file
View File

@ -0,0 +1,112 @@
import { z } from 'zod';
import { Z } from '../zod-class';
// Mirrors the `TestRunStatus` union in `@n8n/db`. Duplicated here so
// `@n8n/api-types` does not need to depend on the db package.
export type EvalCollectionRunStatus = 'new' | 'running' | 'completed' | 'error' | 'cancelled';
// PostHog rollout flag id gating the eval-collections feature surface. All
// new endpoints + frontend entry points consult this; flag-off cohort sees
// the legacy single-run flow unchanged. Single source of truth shared
// between FE and BE so the two cannot drift. Matches the spec verbatim —
// the existing `080_eval_parallel_execution` flag follows a numeric-prefix
// convention but the spec called for the literal id here.
export const EVAL_COLLECTIONS_FLAG = 'eval_collections';
// Per-version entry on a create-collection request. Either reference an
// existing test run (`existingTestRunId`) to reuse it, or omit it and the
// service will schedule a fresh run pinned to `workflowVersionId`. A null
// `workflowVersionId` means "current draft" — the service snapshots a new
// `WorkflowHistory` row at run start so the run stays comparable.
export const evalCollectionVersionEntrySchema = z.object({
workflowVersionId: z.string().min(1).nullable(),
existingTestRunId: z.string().min(1).optional(),
label: z.string().min(1).max(128).optional(),
});
export type EvalCollectionVersionEntry = z.infer<typeof evalCollectionVersionEntrySchema>;
const createEvaluationCollectionShape = {
name: z.string().min(1).max(128),
description: z.string().nullable().optional(),
evaluationConfigId: z.string().min(1),
versions: z.array(evalCollectionVersionEntrySchema).min(1),
concurrency: z.number().int().min(1).max(10).optional(),
};
export const createEvaluationCollectionSchema = z.object(createEvaluationCollectionShape);
export type CreateEvaluationCollectionPayload = z.infer<typeof createEvaluationCollectionSchema>;
export class CreateEvaluationCollectionDto extends Z.class(createEvaluationCollectionShape) {}
const updateEvaluationCollectionShape = {
name: z.string().min(1).max(128).optional(),
description: z.string().nullable().optional(),
};
export const updateEvaluationCollectionSchema = z.object(updateEvaluationCollectionShape);
export type UpdateEvaluationCollectionPayload = z.infer<typeof updateEvaluationCollectionSchema>;
export class UpdateEvaluationCollectionDto extends Z.class(updateEvaluationCollectionShape) {}
const addRunToCollectionShape = {
testRunId: z.string().min(1),
};
export const addRunToCollectionSchema = z.object(addRunToCollectionShape);
export type AddRunToCollectionPayload = z.infer<typeof addRunToCollectionSchema>;
export class AddRunToCollectionDto extends Z.class(addRunToCollectionShape) {}
// Response shapes. These are not validated input — they describe what the
// server returns. Keeping them as plain TypeScript types (not zod schemas)
// avoids forcing the server to round-trip its own outputs through zod.
export type EvaluationCollectionRecord = {
id: string;
name: string;
description: string | null;
workflowId: string;
evaluationConfigId: string;
createdById: string | null;
createdAt: string;
updatedAt: string;
runCount: number;
};
export type EvaluationCollectionRunSummary = {
testRunId: string;
workflowVersionId: string | null;
status: EvalCollectionRunStatus;
runAt: string | null;
completedAt: string | null;
avgScore: number | null;
metrics: Record<string, number> | null;
};
export type EvaluationCollectionDetail = EvaluationCollectionRecord & {
runs: EvaluationCollectionRunSummary[];
};
// Versions-picker endpoint response. The setup wizard renders one row per
// item; `lastRun = null` means "no run yet · will run new" in the UI.
export type EvalVersionEntry = {
workflowVersionId: string | null;
label: string;
sourceLabel: string;
isCurrent: boolean;
lastRun: {
testRunId: string;
runAt: string;
status: EvalCollectionRunStatus;
avgScore: number | null;
isBest: boolean;
isCritical: boolean;
} | null;
};
export type EvalVersionsResponse = {
evaluationConfigId: string;
versions: EvalVersionEntry[];
};

58
packages/@n8n/db/src/entities/evaluation-collection.ee.ts Normal file
View File

@ -0,0 +1,58 @@
import { Column, Entity, Index, PrimaryColumn } from '@n8n/typeorm';
import type { IDataObject } from 'n8n-workflow';
import { JsonColumn, WithTimestamps } from './abstract-entity';
/**
* A named, user-curated group of {@link TestRun}s sharing one
* {@link EvaluationConfig} (= one dataset + one set of metrics + one workflow),
* typically spread across multiple workflow versions. The collection is the
* unit of comparison in the eval compare view.
*
* Invariants enforced at the service layer:
* - Every run in a collection has the same `evaluationConfigId`.
* - Every run is on the same workflow as the collection.
* - Runs typically differ by `workflowVersionId`.
*
* **TypeORM relations:** the `workflowId` and `evaluationConfigId` columns
* are FK columns *without* `@ManyToOne` decorators. Service code resolves
* the related entities through their own repositories. Declaring the
* relations on this entity would extend the TypeORM entity-relation type
* graph enough to trip TS2589 ("type instantiation excessively deep") in
* unrelated repositories that transitively reach `TestRun` →
* `EvaluationCollection` through `WorkflowEntity.testRuns` — e.g.
* `FolderRepository.upsert()` in `source-control-import.service.ee.ts`.
* The FK constraints are preserved by the migration's raw SQL; the
* decorators are purely TypeORM-side metadata.
*/
@Entity()
export class EvaluationCollection extends WithTimestamps {
@PrimaryColumn({ type: 'varchar', length: 36 })
id: string;
@Column('varchar', { length: 128 })
name: string;
@Column('text', { nullable: true })
description: string | null;
@Index()
@Column('varchar', { length: 36 })
workflowId: string;
@Index()
@Column('varchar', { length: 36 })
evaluationConfigId: string;
@Column({ type: 'uuid', nullable: true })
createdById: string | null;
/**
* Cached AI-generated insights for this collection. Lazily populated when
* a user opens the compare view; re-generated when runs change. Shape
* mirrors the `AiInsightsResponse` type produced by the insights agent —
* stored loose here so PR 2 can iterate on the schema without a migration.
*/
@JsonColumn({ nullable: true })
insightsCache: IDataObject | null;
}

3
packages/@n8n/db/src/entities/index.ts
View File

@ -11,6 +11,7 @@ import {
} from './credential-dependency-entity';
import { CredentialsEntity } from './credentials-entity';
import { DeploymentKey } from './deployment-key';
import { EvaluationCollection } from './evaluation-collection.ee';
import { EvaluationConfig } from './evaluation-config.ee';
import { ExecutionAnnotation } from './execution-annotation.ee';
import { ExecutionData } from './execution-data';
@ -63,6 +64,7 @@ export {
CredentialDependency,
type CredentialDependencyType,
DeploymentKey,
EvaluationCollection,
EvaluationConfig,
Folder,
Project,
@ -109,6 +111,7 @@ export const entities = {
CredentialsEntity,
CredentialDependency,
DeploymentKey,
EvaluationCollection,
EvaluationConfig,
Folder,
Project,

26
packages/@n8n/db/src/entities/test-run.ee.ts
View File

@ -1,4 +1,4 @@
import { Column, Entity, OneToMany, ManyToOne } from '@n8n/typeorm';
import { Column, Entity, Index, OneToMany, ManyToOne } from '@n8n/typeorm';
import type { IDataObject } from 'n8n-workflow';
import { DateTimeColumn, JsonColumn, WithTimestampsAndStringId } from './abstract-entity';
@ -63,6 +63,30 @@ export class TestRun extends WithTimestampsAndStringId {
@Column('boolean', { default: false })
cancelRequested: boolean;
@Column('varchar', { length: 36, nullable: true })
workflowVersionId: string | null;
// FK column only — no `@ManyToOne` decorator. Service code queries via
// `evaluationConfigId` and resolves the config through its own repository
// when needed. Declaring the relation on `TestRun` would extend the
// TypeORM entity-relation type graph enough to trip TS2589
// ("type instantiation excessively deep") in unrelated repositories that
// transitively reach `TestRun` through `WorkflowEntity.testRuns` — e.g.
// `FolderRepository.upsert()` in `source-control-import.service.ee.ts`.
// The FK constraint is preserved by the migration's raw SQL; the
// decorator is purely TypeORM-side metadata.
@Index()
@Column('varchar', { length: 36, nullable: true })
evaluationConfigId: string | null;
@JsonColumn({ nullable: true })
evaluationConfigSnapshot: IDataObject | null;
// FK column only — see `evaluationConfigId` comment above for rationale.
@Index()
@Column('varchar', { length: 36, nullable: true })
collectionId: string | null;
/**
* Calculated property to determine the final result of the test run
* depending on the statuses of test case executions

85
packages/@n8n/db/src/migrations/common/1778496086558-CreateEvaluationCollection.ts Normal file
View File

@ -0,0 +1,85 @@
import type { MigrationContext, ReversibleMigration } from '../migration-types';
const EVALUATION_COLLECTION_TABLE = 'evaluation_collection';
const TEST_RUN_TABLE = 'test_run';
const EVALUATION_CONFIG_TABLE = 'evaluation_config';
const WORKFLOW_TABLE = 'workflow_entity';
const USER_TABLE = 'user';
const FK_TEST_RUN_COLLECTION = 'FK_test_run_collection_id';
export class CreateEvaluationCollection1778496086558 implements ReversibleMigration {
async up({
schemaBuilder: { createTable, addColumns, addForeignKey, createIndex, column },
queryRunner,
isSqlite,
}: MigrationContext) {
await createTable(EVALUATION_COLLECTION_TABLE)
.withColumns(
column('id').varchar(36).primary.notNull,
column('name').varchar(128).notNull,
column('description').text,
column('workflowId').varchar(36).notNull,
column('evaluationConfigId').varchar(36).notNull,
column('createdById').uuid,
column('insightsCache').json,
)
.withForeignKey('workflowId', {
tableName: WORKFLOW_TABLE,
columnName: 'id',
onDelete: 'CASCADE',
})
.withForeignKey('evaluationConfigId', {
tableName: EVALUATION_CONFIG_TABLE,
columnName: 'id',
onDelete: 'CASCADE',
})
.withForeignKey('createdById', {
tableName: USER_TABLE,
columnName: 'id',
onDelete: 'SET NULL',
})
.withIndexOn('workflowId')
.withIndexOn('evaluationConfigId').withTimestamps;
// On SQLite, `addColumns` and `addForeignKey` both trigger a full
// table rebuild via TypeORM's cached `Table` metadata. The prior
// migration `1778100002000-AddEvaluationConfigColumnsToTestRun`
// added the `evaluationConfigSnapshot` column on SQLite via raw
// `ALTER TABLE ADD COLUMN`, which TypeORM doesn't observe — so its
// cached metadata is stale by the time this migration runs. Without
// this refresh, the next rebuild silently drops the column and
// every subsequent `TestRun` insert fails with
// "SQLITE_ERROR: table test_run has no column named
// evaluationConfigSnapshot". `queryRunner.getTable()` reloads from
// PRAGMA table_info so the rebuild reads the actual current schema.
if (isSqlite) {
await queryRunner.getTable(TEST_RUN_TABLE);
}
await addColumns(TEST_RUN_TABLE, [column('collectionId').varchar(36)]);
await addForeignKey(
TEST_RUN_TABLE,
'collectionId',
[EVALUATION_COLLECTION_TABLE, 'id'],
FK_TEST_RUN_COLLECTION,
'SET NULL',
);
await createIndex(TEST_RUN_TABLE, ['collectionId']);
}
async down({
schemaBuilder: { dropColumns, dropTable, dropForeignKey, dropIndex },
}: MigrationContext) {
await dropIndex(TEST_RUN_TABLE, ['collectionId']);
await dropForeignKey(
TEST_RUN_TABLE,
'collectionId',
[EVALUATION_COLLECTION_TABLE, 'id'],
FK_TEST_RUN_COLLECTION,
);
await dropColumns(TEST_RUN_TABLE, ['collectionId']);
await dropTable(EVALUATION_COLLECTION_TABLE);
}
}

2
packages/@n8n/db/src/migrations/postgresdb/index.ts
View File

@ -172,6 +172,7 @@ import { AddExecutionDeduplicationKey1778000000000 } from '../common/17780000000
import { CreateEvaluationConfig1778100000000 } from '../common/1778100000000-CreateEvaluationConfig';
import { AddWorkflowVersionToTestRun1778100001000 } from '../common/1778100001000-AddWorkflowVersionToTestRun';
import { AddEvaluationConfigColumnsToTestRun1778100002000 } from '../common/1778100002000-AddEvaluationConfigColumnsToTestRun';
import { CreateEvaluationCollection1778496086558 } from '../common/1778496086558-CreateEvaluationCollection';
import { CreateAgentTables1783000000000 } from '../common/1783000000000-CreateAgentTables';
import { CreateAgentExecutionTables1783000000001 } from '../common/1783000000001-CreateAgentExecutionTables';
import { CreateAgentObservationTables1784000000000 } from '../common/1784000000000-CreateAgentObservationTables';
@ -352,6 +353,7 @@ export const postgresMigrations: Migration[] = [
CreateEvaluationConfig1778100000000,
AddWorkflowVersionToTestRun1778100001000,
AddEvaluationConfigColumnsToTestRun1778100002000,
CreateEvaluationCollection1778496086558,
CreateAgentTables1783000000000,
CreateAgentExecutionTables1783000000001,
CreateAgentObservationTables1784000000000,

2
packages/@n8n/db/src/migrations/sqlite/index.ts
View File

@ -165,6 +165,7 @@ import { AddExecutionDeduplicationKey1778000000000 } from '../common/17780000000
import { CreateEvaluationConfig1778100000000 } from '../common/1778100000000-CreateEvaluationConfig';
import { AddWorkflowVersionToTestRun1778100001000 } from '../common/1778100001000-AddWorkflowVersionToTestRun';
import { AddEvaluationConfigColumnsToTestRun1778100002000 } from '../common/1778100002000-AddEvaluationConfigColumnsToTestRun';
import { CreateEvaluationCollection1778496086558 } from '../common/1778496086558-CreateEvaluationCollection';
import { CreateAgentTables1783000000000 } from '../common/1783000000000-CreateAgentTables';
import { CreateAgentExecutionTables1783000000001 } from '../common/1783000000001-CreateAgentExecutionTables';
import { CreateAgentObservationTables1784000000000 } from '../common/1784000000000-CreateAgentObservationTables';
@ -338,6 +339,7 @@ const sqliteMigrations: Migration[] = [
CreateEvaluationConfig1778100000000,
AddWorkflowVersionToTestRun1778100001000,
AddEvaluationConfigColumnsToTestRun1778100002000,
CreateEvaluationCollection1778496086558,
CreateAgentTables1783000000000,
CreateAgentExecutionTables1783000000001,
CreateAgentObservationTables1784000000000,

237
packages/@n8n/db/src/repositories/__tests__/evaluation-collection.repository.test.ts Normal file
View File

@ -0,0 +1,237 @@
import { Container } from '@n8n/di';
import { EvaluationCollection } from '../../entities/evaluation-collection.ee';
import { TestRun } from '../../entities/test-run.ee';
import { mockEntityManager } from '../../utils/test-utils/mock-entity-manager';
import { EvaluationCollectionRepository } from '../evaluation-collection.repository';
describe('EvaluationCollectionRepository', () => {
const entityManager = mockEntityManager(EvaluationCollection);
const repo = Container.get(EvaluationCollectionRepository);
beforeEach(() => {
jest.resetAllMocks();
});
describe('createCollection', () => {
it('persists the collection with insightsCache initialised to null', async () => {
(entityManager.create as jest.Mock).mockImplementation(
(_target: unknown, entityLike: unknown) => entityLike as EvaluationCollection,
);
entityManager.save.mockImplementationOnce(async (_target, entity) => entity);
await repo.createCollection({
id: 'col-1',
name: 'C',
description: null,
workflowId: 'wf-1',
evaluationConfigId: 'cfg-1',
createdById: 'user-1',
});
const savedEntity = entityManager.save.mock.calls[0]?.[1];
expect(savedEntity).toMatchObject({
id: 'col-1',
name: 'C',
description: null,
workflowId: 'wf-1',
evaluationConfigId: 'cfg-1',
createdById: 'user-1',
insightsCache: null,
});
});
});
describe('findByIdAndWorkflowId', () => {
it('scopes the lookup to the route workflowId so foreign-workflow ids 404 the same as missing ids', async () => {
entityManager.findOne.mockResolvedValueOnce(null);
const result = await repo.findByIdAndWorkflowId('col-x', 'wf-1');
expect(result).toBeNull();
const callArgs = entityManager.findOne.mock.calls[0];
expect(callArgs?.[1]).toEqual({ where: { id: 'col-x', workflowId: 'wf-1' } });
});
});
describe('listByWorkflowId', () => {
it('joins runCount onto each collection with one aggregate query', async () => {
const collections = [
{ id: 'col-a', workflowId: 'wf-1', createdAt: new Date() },
{ id: 'col-b', workflowId: 'wf-1', createdAt: new Date() },
] as EvaluationCollection[];
entityManager.find.mockResolvedValueOnce(collections);
const qb = {
select: jest.fn().mockReturnThis(),
addSelect: jest.fn().mockReturnThis(),
where: jest.fn().mockReturnThis(),
groupBy: jest.fn().mockReturnThis(),
getRawMany: jest.fn().mockResolvedValueOnce([
{ collectionId: 'col-a', count: '3' },
{ collectionId: 'col-b', count: '0' },
]),
};
entityManager.createQueryBuilder.mockReturnValueOnce(qb as never);
const result = await repo.listByWorkflowId('wf-1');
expect(entityManager.createQueryBuilder).toHaveBeenCalledTimes(1);
expect(result.map((c) => c.runCount)).toEqual([3, 0]);
});
it('returns an empty array without an aggregate query when no collections exist', async () => {
entityManager.find.mockResolvedValueOnce([]);
const result = await repo.listByWorkflowId('wf-1');
expect(result).toEqual([]);
expect(entityManager.createQueryBuilder).not.toHaveBeenCalled();
});
});
describe('addRunsToCollection', () => {
it('updates TestRun.collectionId in bulk by id', async () => {
entityManager.update.mockResolvedValueOnce({ affected: 2, generatedMaps: [], raw: [] });
await repo.addRunsToCollection('col-1', ['tr-a', 'tr-b']);
const callArgs = entityManager.update.mock.calls[0];
expect(callArgs?.[0]).toBe(TestRun);
expect(callArgs?.[2]).toEqual({ collectionId: 'col-1' });
});
it('is a no-op when given an empty id list', async () => {
await repo.addRunsToCollection('col-1', []);
expect(entityManager.update).not.toHaveBeenCalled();
});
});
describe('removeRunFromCollection', () => {
it('only unlinks runs that are actually in the named collection', async () => {
entityManager.update.mockResolvedValueOnce({ affected: 1, generatedMaps: [], raw: [] });
const affected = await repo.removeRunFromCollection('col-1', 'tr-a');
expect(affected).toBe(1);
const callArgs = entityManager.update.mock.calls[0];
expect(callArgs?.[1]).toEqual({ id: 'tr-a', collectionId: 'col-1' });
expect(callArgs?.[2]).toEqual({ collectionId: null });
});
it('returns 0 when the run is not part of the collection', async () => {
entityManager.update.mockResolvedValueOnce({ affected: 0, generatedMaps: [], raw: [] });
expect(await repo.removeRunFromCollection('col-1', 'tr-missing')).toBe(0);
});
});
describe('deleteByIdAndWorkflowId', () => {
it('counts runs before delete so callers can report runs_unlinked for telemetry', async () => {
entityManager.findOne.mockResolvedValueOnce({ id: 'col-1' } as EvaluationCollection);
entityManager.count.mockResolvedValueOnce(4);
entityManager.delete.mockResolvedValueOnce({ affected: 1, raw: [] });
const result = await repo.deleteByIdAndWorkflowId('col-1', 'wf-1');
expect(result).toEqual({ deleted: true, runsUnlinked: 4 });
});
it('returns { deleted: false } without touching delete when the collection does not exist', async () => {
entityManager.findOne.mockResolvedValueOnce(null);
const result = await repo.deleteByIdAndWorkflowId('col-missing', 'wf-1');
expect(result).toEqual({ deleted: false, runsUnlinked: 0 });
expect(entityManager.delete).not.toHaveBeenCalled();
});
});
describe('updateInsightsCache', () => {
it('updates the insightsCache JSON column on the row', async () => {
entityManager.update.mockResolvedValueOnce({ affected: 1, generatedMaps: [], raw: [] });
await repo.updateInsightsCache('col-1', { winner: { versionLabel: 'A' } } as never);
const callArgs = entityManager.update.mock.calls[0];
expect(callArgs?.[1]).toBe('col-1');
expect(callArgs?.[2]).toEqual({ insightsCache: { winner: { versionLabel: 'A' } } });
});
it('accepts null to clear the cache', async () => {
entityManager.update.mockResolvedValueOnce({ affected: 1, generatedMaps: [], raw: [] });
await repo.updateInsightsCache('col-1', null);
const callArgs = entityManager.update.mock.calls[0];
expect(callArgs?.[2]).toEqual({ insightsCache: null });
});
});
describe('updateMeta', () => {
it('returns null without writing when the collection does not exist', async () => {
entityManager.findOne.mockResolvedValueOnce(null);
const result = await repo.updateMeta('col-x', 'wf-1', { name: 'New' });
expect(result).toBeNull();
expect(entityManager.save).not.toHaveBeenCalled();
});
it('merges name + description onto the existing entity', async () => {
const existing = {
id: 'col-1',
name: 'Old',
description: 'old desc',
workflowId: 'wf-1',
} as EvaluationCollection;
entityManager.findOne.mockResolvedValueOnce(existing);
(entityManager.create as jest.Mock).mockImplementation(
(_target: unknown, entityLike: unknown) => entityLike as EvaluationCollection,
);
entityManager.save.mockImplementationOnce(async (_target, entity) => entity);
await repo.updateMeta('col-1', 'wf-1', { name: 'New', description: null });
const savedEntity = entityManager.save.mock.calls[0]?.[1];
expect(savedEntity).toMatchObject({
id: 'col-1',
name: 'New',
description: null,
});
});
it('preserves description when only name is provided', async () => {
const existing = {
id: 'col-1',
name: 'Old',
description: 'old desc',
workflowId: 'wf-1',
} as EvaluationCollection;
entityManager.findOne.mockResolvedValueOnce(existing);
entityManager.save.mockImplementationOnce(async (_target, entity) => entity);
await repo.updateMeta('col-1', 'wf-1', { name: 'New' });
const savedEntity = entityManager.save.mock.calls[0]?.[1] as EvaluationCollection;
expect(savedEntity.name).toBe('New');
expect(savedEntity.description).toBe('old desc');
});
it('preserves existing fields when payload props are undefined', async () => {
const existing = {
id: 'col-1',
name: 'Old',
description: 'old desc',
workflowId: 'wf-1',
} as EvaluationCollection;
entityManager.findOne.mockResolvedValueOnce(existing);
entityManager.save.mockImplementationOnce(async (_target, entity) => entity);
await repo.updateMeta('col-1', 'wf-1', { name: undefined, description: undefined });
const savedEntity = entityManager.save.mock.calls[0]?.[1] as EvaluationCollection;
expect(savedEntity.name).toBe('Old');
expect(savedEntity.description).toBe('old desc');
});
});
});

156
packages/@n8n/db/src/repositories/evaluation-collection.repository.ts Normal file
View File

@ -0,0 +1,156 @@
import { Service } from '@n8n/di';
import { DataSource, In, Repository } from '@n8n/typeorm';
import type { IDataObject } from 'n8n-workflow';
import { EvaluationCollection } from '../entities/evaluation-collection.ee';
import { TestRun } from '../entities/test-run.ee';
export type EvaluationCollectionListItem = {
id: EvaluationCollection['id'];
name: EvaluationCollection['name'];
description: EvaluationCollection['description'];
workflowId: EvaluationCollection['workflowId'];
evaluationConfigId: EvaluationCollection['evaluationConfigId'];
createdById: EvaluationCollection['createdById'];
insightsCache: EvaluationCollection['insightsCache'];
createdAt: EvaluationCollection['createdAt'];
updatedAt: EvaluationCollection['updatedAt'];
runCount: number;
};
@Service()
export class EvaluationCollectionRepository extends Repository<EvaluationCollection> {
constructor(dataSource: DataSource) {
super(EvaluationCollection, dataSource.manager);
}
async createCollection(input: {
id: string;
name: string;
description: string | null;
workflowId: string;
evaluationConfigId: string;
createdById: string | null;
}): Promise<EvaluationCollection> {
const entity = this.create({
id: input.id,
name: input.name,
description: input.description,
workflowId: input.workflowId,
evaluationConfigId: input.evaluationConfigId,
createdById: input.createdById,
insightsCache: null,
});
return await this.save(entity);
}
async findByIdAndWorkflowId(
id: string,
workflowId: string,
): Promise<EvaluationCollection | null> {
return await this.findOne({ where: { id, workflowId } });
}
/**
* Returns collections for a workflow with `runCount` populated. The list
* page uses this — it does not need the full run rows, just the count and
* the collection's own fields.
*/
async listByWorkflowId(workflowId: string): Promise<EvaluationCollectionListItem[]> {
const collections = await this.find({
where: { workflowId },
order: { createdAt: 'DESC' },
});
if (collections.length === 0) return [];
const runCounts = await this.manager
.createQueryBuilder(TestRun, 'tr')
.select('tr.collectionId', 'collectionId')
.addSelect('COUNT(tr.id)', 'count')
.where('tr.collectionId IN (:...ids)', { ids: collections.map((c) => c.id) })
.groupBy('tr.collectionId')
.getRawMany<{ collectionId: string; count: string }>();
const countByCollectionId = new Map(
runCounts.map(({ collectionId, count }) => [collectionId, Number(count)]),
);
return collections.map((collection) => ({
...collection,
runCount: countByCollectionId.get(collection.id) ?? 0,
}));
}
/**
* Returns the collection plus its runs ordered by createdAt ascending so
* the compare view can render versions left-to-right by run order.
*/
async getDetailByIdAndWorkflowId(
id: string,
workflowId: string,
): Promise<{ collection: EvaluationCollection; runs: TestRun[] } | null> {
const collection = await this.findOne({ where: { id, workflowId } });
if (!collection) return null;
const runs = await this.manager.find(TestRun, {
where: { collectionId: id },
order: { createdAt: 'ASC' },
});
return { collection, runs };
}
async addRunsToCollection(collectionId: string, testRunIds: string[]): Promise<void> {
if (testRunIds.length === 0) return;
await this.manager.update(TestRun, { id: In(testRunIds) }, { collectionId });
}
async removeRunFromCollection(collectionId: string, testRunId: string): Promise<number> {
const result = await this.manager.update(
TestRun,
{ id: testRunId, collectionId },
{ collectionId: null },
);
return result.affected ?? 0;
}
/**
* Returns how many runs were unlinked (set to `collectionId = null`) by
* the delete. Callers feed this into the `runs_unlinked` telemetry field.
*
* The FK on `test_run.collectionId` is `SET NULL`, so we'd technically
* get the unlink for free — but counting runs first lets the caller
* surface the number in the response and telemetry without an extra
* round-trip.
*/
async deleteByIdAndWorkflowId(
id: string,
workflowId: string,
): Promise<{ deleted: boolean; runsUnlinked: number }> {
const collection = await this.findOne({ where: { id, workflowId } });
if (!collection) return { deleted: false, runsUnlinked: 0 };
const runsUnlinked = await this.manager.count(TestRun, { where: { collectionId: id } });
await this.delete({ id });
return { deleted: true, runsUnlinked };
}
async updateInsightsCache(id: string, insights: IDataObject | null): Promise<void> {
await this.update(id, { insightsCache: insights });
}
async updateMeta(
id: string,
workflowId: string,
payload: { name?: string; description?: string | null },
): Promise<EvaluationCollection | null> {
const existing = await this.findByIdAndWorkflowId(id, workflowId);
if (!existing) return null;
if (payload.name !== undefined) existing.name = payload.name;
if (payload.description !== undefined) existing.description = payload.description;
return await this.save(existing);
}
}

4
packages/@n8n/db/src/repositories/index.ts
View File

@ -13,6 +13,10 @@ export {
type DeploymentKeySortDirection,
type ListDeploymentKeysOptions,
} from './deployment-key.repository';
export {
EvaluationCollectionRepository,
type EvaluationCollectionListItem,
} from './evaluation-collection.repository';
export { EvaluationConfigRepository } from './evaluation-config.repository';
export { ExecutionAnnotationRepository } from './execution-annotation.repository';
export { ExecutionDataRepository } from './execution-data.repository';

14
packages/@n8n/db/src/repositories/test-run.repository.ee.ts
View File

@ -22,12 +22,24 @@ export class TestRunRepository extends Repository<TestRun> {
super(TestRun, dataSource.manager);
}
async createTestRun(workflowId: string): Promise<TestRun> {
async createTestRun(
workflowId: string,
attrs?: {
collectionId?: string | null;
workflowVersionId?: string | null;
evaluationConfigId?: string | null;
evaluationConfigSnapshot?: IDataObject | null;
},
): Promise<TestRun> {
const testRun = this.create({
status: 'new',
workflow: {
id: workflowId,
},
collectionId: attrs?.collectionId ?? null,
workflowVersionId: attrs?.workflowVersionId ?? null,
evaluationConfigId: attrs?.evaluationConfigId ?? null,
evaluationConfigSnapshot: attrs?.evaluationConfigSnapshot ?? null,
});
return await this.save(testRun);

139
packages/@n8n/instance-ai/evaluations/binaryChecks/checks/http-generic-auth-type-matches-prompt.test.ts Normal file
View File

@ -0,0 +1,139 @@
import { httpGenericAuthTypeMatchesPrompt } from './http-generic-auth-type-matches-prompt';
import type { WorkflowResponse } from '../../clients/n8n-client';
function workflowWithHttpRequest(parameters: Record<string, unknown>): WorkflowResponse {
return {
id: 'wf-1',
name: 'HTTP auth test',
active: false,
nodes: [
{
name: 'Manual Trigger',
type: 'n8n-nodes-base.manualTrigger',
parameters: {},
},
{
name: 'HTTP Request',
type: 'n8n-nodes-base.httpRequest',
parameters,
},
],
connections: {
'Manual Trigger': {
main: [[{ node: 'HTTP Request', type: 'main', index: 0 }]],
},
},
};
}
describe('httpGenericAuthTypeMatchesPrompt', () => {
it('fails when prompt asks for Bearer but builder picks httpHeaderAuth', async () => {
const workflow = workflowWithHttpRequest({
authentication: 'genericCredentialType',
genericAuthType: 'httpHeaderAuth',
});
const result = await httpGenericAuthTypeMatchesPrompt.run(workflow, {
prompt: 'Call the GitHub API using a Bearer token',
});
expect(result.pass).toBe(false);
expect(result.comment).toContain('HTTP Request');
expect(result.comment).toContain('httpBearerAuth');
});
it('fails when prompt mentions Authorization: Bearer header explicitly', async () => {
const workflow = workflowWithHttpRequest({
authentication: 'genericCredentialType',
genericAuthType: 'httpHeaderAuth',
});
const result = await httpGenericAuthTypeMatchesPrompt.run(workflow, {
prompt: 'Send a request with Authorization: Bearer <token>',
});
expect(result.pass).toBe(false);
});
it('passes when builder picks httpBearerAuth for a Bearer prompt', async () => {
const workflow = workflowWithHttpRequest({
authentication: 'genericCredentialType',
genericAuthType: 'httpBearerAuth',
});
const result = await httpGenericAuthTypeMatchesPrompt.run(workflow, {
prompt: 'Call the GitHub API using a Bearer token',
});
expect(result).toEqual({ pass: true });
});
it('passes when prompt specifies a custom header name and builder picks httpHeaderAuth', async () => {
const workflow = workflowWithHttpRequest({
authentication: 'genericCredentialType',
genericAuthType: 'httpHeaderAuth',
});
const result = await httpGenericAuthTypeMatchesPrompt.run(workflow, {
prompt: 'Call the Algolia API with an X-API-Key header',
});
expect(result).toEqual({ pass: true });
});
it('passes when the prompt has no auth signal at all', async () => {
const workflow = workflowWithHttpRequest({
authentication: 'genericCredentialType',
genericAuthType: 'httpHeaderAuth',
});
const result = await httpGenericAuthTypeMatchesPrompt.run(workflow, {
prompt: 'Fetch data from a public endpoint',
});
expect(result).toEqual({ pass: true });
});
it('ignores HTTP Request nodes that do not use generic credential type', async () => {
const workflow = workflowWithHttpRequest({
authentication: 'predefinedCredentialType',
nodeCredentialType: 'githubApi',
});
const result = await httpGenericAuthTypeMatchesPrompt.run(workflow, {
prompt: 'Call the GitHub API using a Bearer token',
});
expect(result).toEqual({ pass: true });
});
it('ignores non-HTTP-Request nodes', async () => {
const workflow: WorkflowResponse = {
id: 'wf-2',
name: 'Slack only',
active: false,
nodes: [
{
name: 'Manual Trigger',
type: 'n8n-nodes-base.manualTrigger',
parameters: {},
},
{
name: 'Slack',
type: 'n8n-nodes-base.slack',
parameters: {
authentication: 'genericCredentialType',
genericAuthType: 'httpHeaderAuth',
},
},
],
connections: {},
};
const result = await httpGenericAuthTypeMatchesPrompt.run(workflow, {
prompt: 'Send a Bearer-authenticated message',
});
expect(result).toEqual({ pass: true });
});
});

68
packages/@n8n/instance-ai/evaluations/binaryChecks/checks/http-generic-auth-type-matches-prompt.ts Normal file
View File

@ -0,0 +1,68 @@
import type { WorkflowNodeResponse } from '../../clients/n8n-client';
import type { BinaryCheck } from '../types';
const HTTP_REQUEST_TYPE = 'n8n-nodes-base.httpRequest';
const BEARER_PROMPT_PATTERNS = [
/\bbearer\s+(?:token|auth(?:entication)?)\b/i,
/\bauthorization\s*:\s*bearer\b/i,
/\b(?:use|with|via|using)\s+(?:a\s+|my\s+)?bearer\b/i,
];
const CUSTOM_HEADER_NAME_PATTERNS = [
/\bx-[a-z][a-z0-9-]*\b/i,
/\bheader\s+(?:named|called)\s+["'`]?[a-z][\w-]*["'`]?/i,
/\bcustom\s+header\b/i,
/\bheader\s+name\b/i,
/\bapikey\s+header\b/i,
];
function isHttpRequestWithGenericAuth(node: WorkflowNodeResponse): boolean {
return (
node.type === HTTP_REQUEST_TYPE && node.parameters?.authentication === 'genericCredentialType'
);
}
function getGenericAuthType(node: WorkflowNodeResponse): string {
const value = node.parameters?.genericAuthType;
return typeof value === 'string' ? value : '';
}
function promptAsksForBearer(prompt: string): boolean {
return BEARER_PROMPT_PATTERNS.some((p) => p.test(prompt));
}
function promptSpecifiesCustomHeaderName(prompt: string): boolean {
return CUSTOM_HEADER_NAME_PATTERNS.some((p) => p.test(prompt));
}
export const httpGenericAuthTypeMatchesPrompt: BinaryCheck = {
name: 'http_generic_auth_type_matches_prompt',
description:
'HTTP Request nodes pick a genericAuthType consistent with the prompt — prefer httpBearerAuth for Authorization: Bearer flows; reserve httpHeaderAuth for custom header names',
kind: 'deterministic',
run(workflow, ctx) {
const issues: string[] = [];
for (const node of workflow.nodes ?? []) {
if (!isHttpRequestWithGenericAuth(node)) continue;
const genericAuthType = getGenericAuthType(node);
if (genericAuthType !== 'httpHeaderAuth') continue;
const bearerIntent = promptAsksForBearer(ctx.prompt);
const customHeaderIntent = promptSpecifiesCustomHeaderName(ctx.prompt);
if (bearerIntent && !customHeaderIntent) {
issues.push(
`"${node.name}" uses httpHeaderAuth but the prompt asks for Bearer auth — should be httpBearerAuth`,
);
}
}
return {
pass: issues.length === 0,
...(issues.length > 0 ? { comment: issues.join('; ') } : {}),
};
},
};

2
packages/@n8n/instance-ai/evaluations/binaryChecks/checks/index.ts
View File

@ -14,6 +14,7 @@ import { handlesMultipleItems } from './handles-multiple-items';
import { hasNodes } from './has-nodes';
import { hasStartNode } from './has-start-node';
import { hasTrigger } from './has-trigger';
import { httpGenericAuthTypeMatchesPrompt } from './http-generic-auth-type-matches-prompt';
import { inboundTriggerAuthDefaults } from './inbound-trigger-auth-defaults';
import { memoryProperlyConnected } from './memory-properly-connected';
import { memorySessionKeyExpression } from './memory-session-key-expression';
@ -50,6 +51,7 @@ export const DETERMINISTIC_CHECKS: BinaryCheck[] = [
toolsHaveParameters,
noUnreachableNodes,
inboundTriggerAuthDefaults,
httpGenericAuthTypeMatchesPrompt,
validNodeConfig,
];

274
packages/@n8n/instance-ai/evaluations/cli/compare-pairwise.ts
View File

@ -17,6 +17,9 @@ import { jsonParse } from 'n8n-workflow';
import { promises as fs } from 'node:fs';
import path from 'node:path';
import { scrubSecretsInText } from '../../src/utils/scrub-secrets';
import { redactSecrets } from '../harness/redact';
// ---------------------------------------------------------------------------
// Shared shape after normalization
// ---------------------------------------------------------------------------
@ -50,6 +53,8 @@ export interface BuilderRecord {
toolCallErrors?: number;
/** Total tool calls observed, used as the error-rate denominator. */
toolCallsTotal?: number;
/** Raw tool-call trace. IA-only — populated by `loadInstanceAiRun`. */
toolCalls?: IAToolCallTrace[];
}
interface BuilderSummary {
@ -190,6 +195,7 @@ async function loadInstanceAiRun(dir: string): Promise<BuilderRun> {
submitCalls: tcs.filter((tc) => tc.toolName === 'submit-workflow').length,
toolCallErrors: tcs.filter(isErroredIAToolCall).length,
toolCallsTotal: tcs.length,
toolCalls: tcs,
};
});
@ -634,7 +640,7 @@ function renderBuilderColumn(label: string, record: BuilderRecord | undefined):
}
const errorBlock = record.errorMessage
? `<div class="error">${escapeHtml(record.errorMessage)}</div>`
? `<div class="error">${escapeHtml(scrubSecretsInText(record.errorMessage))}</div>`
: '';
const idLine = record.exampleId
@ -654,12 +660,110 @@ function renderBuilderColumn(label: string, record: BuilderRecord | undefined):
</div>`;
}
function renderRow(row: ComparisonRow, index: number): string {
interface SideLabels {
ee: string;
ia: string;
}
function summarizeToolCallArgs(toolName: string, args: unknown): string {
if (!args || typeof args !== 'object') return '';
// Redact secret-shaped keys (password / token / api_key / …) and scrub the
// extracted free-form strings for known credential patterns. The same
// summarized text is used as a data-attribute on the row, so any leaked
// secret would persist in the HTML and be searchable via filter input.
const a = redactSecrets(args) as Record<string, unknown>;
const str = (v: unknown): string => (typeof v === 'string' ? scrubSecretsInText(v) : '');
const trunc = (s: string, n = 160): string => (s.length > n ? s.slice(0, n) + '…' : s);
switch (toolName) {
case 'mastra_workspace_execute_command':
return trunc(str(a.command));
case 'mastra_workspace_edit_file':
return trunc(str(a.path));
case 'mastra_workspace_write_file': {
const p = str(a.path);
const len = typeof a.content === 'string' ? a.content.length : 0;
return `${p}${len ? ` (${len} chars)` : ''}`;
}
case 'mastra_workspace_read_file': {
const p = str(a.path);
const off = typeof a.offset === 'number' ? a.offset : undefined;
const lim = typeof a.limit === 'number' ? a.limit : undefined;
const range = off !== undefined || lim !== undefined ? ` @${off ?? 0}+${lim ?? '∞'}` : '';
return trunc(`${p}${range}`);
}
case 'mastra_workspace_grep':
return trunc(`${str(a.pattern)}${a.path ? ` in ${str(a.path)}` : ''}`);
case 'mastra_workspace_lsp_inspect': {
const line = typeof a.line === 'number' ? String(a.line) : '?';
return trunc(`${str(a.path)}:${line} ${str(a.match)}`);
}
case 'mastra_workspace_mkdir':
return trunc(str(a.path));
case 'submit-workflow':
return trunc(`${str(a.name)} ${str(a.filePath)}`);
case 'verify-built-workflow':
return trunc(str(a.workflowId) || str(a.workItemId));
case 'credentials':
case 'data-tables':
case 'nodes':
case 'workflows':
return trunc(str(a.action));
default:
return trunc(scrubSecretsInText(JSON.stringify(a)), 120);
}
}
function renderToolCallRows(traces: IAToolCallTrace[]): string {
return traces
.map((tc) => {
const errored = isErroredIAToolCall(tc);
const elapsed = tc.elapsedMs !== undefined ? formatDuration(tc.elapsedMs) : '—';
const cls = errored ? 'tc-err' : '';
const argSummary = summarizeToolCallArgs(tc.toolName, tc.args);
const argCell = argSummary ? `<code class="tc-args">${escapeHtml(argSummary)}</code>` : '';
const detail = errored && tc.error ? escapeHtml(scrubSecretsInText(tc.error)) : '';
return `<tr class="${cls}"><td>${tc.step}</td><td>${escapeHtml(tc.toolName)}</td><td>${argCell}</td><td>${elapsed}</td><td>${detail}</td></tr>`;
})
.join('');
}
function renderToolCallsColumn(label: string, record: BuilderRecord | undefined): string {
if (!record) {
return `<div class="tc-col missing"><div class="tc-col-label">${escapeHtml(label)}</div><div class="missing-msg">No record.</div></div>`;
}
const traces = record.toolCalls ?? [];
if (traces.length === 0) {
return `<div class="tc-col"><div class="tc-col-label">${escapeHtml(label)} <span class="tc-summary">(0 calls)</span></div><div class="missing-msg">No tool calls captured.</div></div>`;
}
const errCount = traces.filter(isErroredIAToolCall).length;
const errBadge = errCount > 0 ? ` · <span class="tc-err-badge">${errCount} err</span>` : '';
return `<div class="tc-col">
<div class="tc-col-label">${escapeHtml(label)} <span class="tc-summary">(${traces.length} call${traces.length === 1 ? '' : 's'}${errBadge})</span></div>
<table class="tool-calls">
<thead><tr><th>#</th><th>Tool</th><th>Args</th><th>Time</th><th>Error</th></tr></thead>
<tbody>${renderToolCallRows(traces)}</tbody>
</table>
</div>`;
}
function renderToolCallsSection(row: ComparisonRow, labels: SideLabels): string {
const anyTraces = (row.ee?.toolCalls?.length ?? 0) > 0 || (row.ia?.toolCalls?.length ?? 0) > 0;
if (!anyTraces) return '';
return `<details class="tool-calls-section">
<summary>Tool calls</summary>
<div class="tc-grid">
${renderToolCallsColumn(labels.ee, row.ee)}
${renderToolCallsColumn(labels.ia, row.ia)}
</div>
</details>`;
}
function renderRow(row: ComparisonRow, index: number, labels: SideLabels): string {
const verdictLabel: Record<ComparisonRow['verdict'], string> = {
'both-pass': 'BOTH PASS',
'both-fail': 'BOTH FAIL',
'ee-only': 'CODE ONLY',
'ia-only': 'IA ONLY',
'ee-only': `${labels.ee.toUpperCase()} ONLY`,
'ia-only': `${labels.ia.toUpperCase()} ONLY`,
neither: '—',
};
const verdictCls: Record<ComparisonRow['verdict'], string> = {
@ -690,14 +794,33 @@ function renderRow(row: ComparisonRow, index: number): string {
// Heavy content (workflow previews + judge tables) is wrapped in a <template>
// so the n8n-demo web component is NOT instantiated until the user expands
// the row. The lazy loader script in the document head does the swap.
return `<details class="row ${verdictCls[row.verdict]}" id="row-${index}">
const usedTools = new Set<string>();
const toolArgsCorpus: string[] = [];
for (const tc of row.ee?.toolCalls ?? []) {
usedTools.add(tc.toolName);
const summary = summarizeToolCallArgs(tc.toolName, tc.args);
if (summary) toolArgsCorpus.push(summary);
}
for (const tc of row.ia?.toolCalls ?? []) {
usedTools.add(tc.toolName);
const summary = summarizeToolCallArgs(tc.toolName, tc.args);
if (summary) toolArgsCorpus.push(summary);
}
const dataAttrs = [
`data-verdict="${row.verdict}"`,
`data-prompt="${escapeAttr(row.prompt.toLowerCase())}"`,
`data-id="${escapeAttr(idText.toLowerCase())}"`,
`data-tools="${escapeAttr([...usedTools].join(' '))}"`,
`data-tool-args="${escapeAttr(toolArgsCorpus.join(' ␟ ').toLowerCase())}"`,
].join(' ');
return `<details class="row ${verdictCls[row.verdict]}" id="row-${index}" ${dataAttrs}>
<summary>
<span class="verdict">${verdictLabel[row.verdict]}</span>
${idChip}
<span class="prompt-preview">${escapeHtml(promptPreview)}</span>
<span class="builder-chips">
${builderChip('Code', eeHead)}
${builderChip('IA', iaHead)}
${builderChip(labels.ee, eeHead)}
${builderChip(labels.ia, iaHead)}
</span>
</summary>
<div class="body">
@ -712,12 +835,13 @@ function renderRow(row: ComparisonRow, index: number): string {
<div class="lazy-slot" data-loaded="false">
<template>
<div class="builder-grid">
${renderBuilderColumn('Code Builder', row.ee)}
${renderBuilderColumn('instance-ai', row.ia)}
${renderBuilderColumn(labels.ee, row.ee)}
${renderBuilderColumn(labels.ia, row.ia)}
</div>
</template>
<div class="lazy-placeholder">Click to load workflow previews and judge details…</div>
</div>
${renderToolCallsSection(row, labels)}
</div>
</details>`;
}
@ -770,7 +894,7 @@ function renderMetricsNote(): string {
</aside>`;
}
function renderVerdictTotals(rows: ComparisonRow[]): string {
function renderVerdictTotals(rows: ComparisonRow[], labels: SideLabels): string {
const counts: Record<ComparisonRow['verdict'], number> = {
'both-pass': 0,
'both-fail': 0,
@ -786,18 +910,23 @@ function renderVerdictTotals(rows: ComparisonRow[]): string {
return `<div class="verdict-grid">
${card('Both pass', counts['both-pass'], 'verdict-both-pass')}
${card('Code Builder only passes', counts['ee-only'], 'verdict-ee-only')}
${card('IA only passes', counts['ia-only'], 'verdict-ia-only')}
${card(`${labels.ee} only passes`, counts['ee-only'], 'verdict-ee-only')}
${card(`${labels.ia} only passes`, counts['ia-only'], 'verdict-ia-only')}
${card('Both fail', counts['both-fail'], 'verdict-both-fail')}
</div>`;
}
function renderDocument(ee: BuilderRun, ia: BuilderRun, rows: ComparisonRow[]): string {
function renderDocument(
ee: BuilderRun,
ia: BuilderRun,
rows: ComparisonRow[],
labels: SideLabels,
): string {
return `<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>Pairwise Eval Comparison — Code Builder vs Instance AI</title>
<title>Pairwise Eval Comparison — ${escapeHtml(labels.ee)} vs ${escapeHtml(labels.ia)}</title>
<script defer src="https://cdn.jsdelivr.net/npm/@webcomponents/webcomponentsjs@2.0.0/webcomponents-loader.js"></script>
<script defer src="https://www.unpkg.com/lit@2.0.0-rc.2/polyfill-support.js"></script>
<script type="module" src="https://cdn.jsdelivr.net/npm/@n8n_io/n8n-demo-component/n8n-demo.bundled.js"></script>
@ -939,25 +1068,102 @@ function renderDocument(ee: BuilderRun, ia: BuilderRun, rows: ComparisonRow[]):
table.judges tr:last-child td { border-bottom: none; }
table.judges td.judge-pass { color: var(--pass); font-weight: 600; }
table.judges td.judge-fail { color: var(--fail); font-weight: 600; }
.tool-calls-section { margin-top: 14px; border-top: 1px solid var(--border); padding-top: 10px; }
.tool-calls-section > summary { cursor: pointer; font-size: 13px; font-weight: 600; color: var(--fg); padding: 4px 0; user-select: none; }
.tool-calls-section > summary:hover { color: var(--accent, #58a6ff); }
.tc-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 12px; margin-top: 10px; }
.tc-col { background: var(--card); border: 1px solid var(--border); border-radius: 6px; padding: 10px; min-width: 0; }
.tc-col.missing { background: var(--subtle); }
.tc-col-label { font-weight: 600; font-size: 12px; margin-bottom: 6px; }
.tc-summary { color: var(--muted); font-weight: 400; }
.tc-err-badge { color: var(--fail); font-weight: 600; }
table.tool-calls { width: 100%; border-collapse: collapse; font-size: 11px; }
table.tool-calls th, table.tool-calls td { padding: 4px 6px; text-align: left; border-bottom: 1px solid var(--border); vertical-align: top; }
table.tool-calls th { color: var(--muted); font-weight: 500; font-size: 10px; text-transform: uppercase; letter-spacing: 0.04em; }
table.tool-calls td:nth-child(1) { color: var(--muted); width: 28px; font-variant-numeric: tabular-nums; }
table.tool-calls td:nth-child(2) { white-space: nowrap; }
table.tool-calls td:nth-child(3) { color: var(--muted); }
table.tool-calls td:nth-child(4) { color: var(--muted); width: 60px; font-variant-numeric: tabular-nums; white-space: nowrap; }
table.tool-calls tr:last-child td { border-bottom: none; }
table.tool-calls tr.tc-err td { color: var(--fail); }
table.tool-calls tr.tc-err td:nth-child(5) { white-space: pre-wrap; word-break: break-word; font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 10px; }
code.tc-args { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 10px; color: var(--fg); white-space: pre-wrap; word-break: break-all; display: inline-block; max-height: 60px; overflow-y: auto; }
.filter-bar { display: flex; gap: 10px; align-items: center; padding: 12px 16px; background: var(--card); border: 1px solid var(--border); border-radius: 6px; margin-bottom: 14px; flex-wrap: wrap; position: sticky; top: 0; z-index: 10; }
.filter-bar #filter-text { flex: 1 1 280px; min-width: 240px; padding: 6px 10px; background: var(--bg); color: var(--fg); border: 1px solid var(--border); border-radius: 4px; font-size: 13px; }
.filter-bar #filter-text:focus { outline: none; border-color: var(--accent, #58a6ff); }
.filter-chip { display: inline-flex; align-items: center; gap: 5px; padding: 4px 10px; background: var(--subtle); border: 1px solid var(--border); border-radius: 16px; font-size: 12px; cursor: pointer; user-select: none; }
.filter-chip input { cursor: pointer; }
.filter-count { font-size: 12px; color: var(--muted); font-variant-numeric: tabular-nums; }
.filter-reset { padding: 5px 12px; background: var(--subtle); color: var(--fg); border: 1px solid var(--border); border-radius: 4px; font-size: 12px; cursor: pointer; }
.filter-reset:hover { background: var(--card); }
</style>
</head>
<body>
<header class="top">
<h1>Pairwise Eval Comparison — Code Builder vs Instance AI</h1>
<div class="subhead">${rows.length} prompt${rows.length === 1 ? '' : 's'} compared. Rows are ordered: Code-only wins, IA-only wins, both fail, both pass.</div>
<h1>Pairwise Eval Comparison — ${escapeHtml(labels.ee)} vs ${escapeHtml(labels.ia)}</h1>
<div class="subhead">${rows.length} prompt${rows.length === 1 ? '' : 's'} compared. Rows are ordered: ${escapeHtml(labels.ee)}-only wins, ${escapeHtml(labels.ia)}-only wins, both fail, both pass.</div>
</header>
<main>
<section class="summary-row">
${renderSummaryCard('Code Builder', ee.summary, ee.records.length, ee.records)}
${renderSummaryCard('instance-ai', ia.summary, ia.records.length, ia.records)}
${renderSummaryCard(ee.summary.label || labels.ee, ee.summary, ee.records.length, ee.records)}
${renderSummaryCard(ia.summary.label || labels.ia, ia.summary, ia.records.length, ia.records)}
</section>
${renderVerdictTotals(rows)}
${renderVerdictTotals(rows, labels)}
${renderMetricsNote()}
<section class="filter-bar">
<input type="search" id="filter-text" placeholder="Filter by prompt, example id, tool name, or tool args (e.g. paths, commands)…" autocomplete="off" />
<label class="filter-chip"><input type="checkbox" data-verdict="both-pass" checked /> Both pass</label>
<label class="filter-chip"><input type="checkbox" data-verdict="ee-only" checked /> ${escapeHtml(labels.ee)} only</label>
<label class="filter-chip"><input type="checkbox" data-verdict="ia-only" checked /> ${escapeHtml(labels.ia)} only</label>
<label class="filter-chip"><input type="checkbox" data-verdict="both-fail" checked /> Both fail</label>
<span class="filter-count" id="filter-count">${rows.length} / ${rows.length}</span>
<button type="button" id="filter-reset" class="filter-reset">Reset</button>
</section>
<section class="rows">
${rows.map((r, i) => renderRow(r, i)).join('\n')}
${rows.map((r, i) => renderRow(r, i, labels)).join('\n')}
</section>
</main>
<script>
// Filter bar — narrows visible rows by free-text query AND verdict checkboxes.
(function() {
const input = document.getElementById('filter-text');
const checks = Array.from(document.querySelectorAll('.filter-bar input[type=checkbox]'));
const rows = Array.from(document.querySelectorAll('details.row'));
const counter = document.getElementById('filter-count');
const resetBtn = document.getElementById('filter-reset');
const total = rows.length;
function apply() {
const q = (input.value || '').trim().toLowerCase();
const verdicts = new Set(checks.filter((c) => c.checked).map((c) => c.dataset.verdict));
let visible = 0;
for (const r of rows) {
const v = r.dataset.verdict || '';
const prompt = r.dataset.prompt || '';
const id = r.dataset.id || '';
const tools = r.dataset.tools || '';
const toolArgs = r.dataset.toolArgs || '';
const passesVerdict = verdicts.has(v);
const passesText =
!q ||
prompt.includes(q) ||
id.includes(q) ||
tools.toLowerCase().includes(q) ||
toolArgs.includes(q);
const show = passesVerdict && passesText;
r.style.display = show ? '' : 'none';
if (show) visible++;
}
counter.textContent = visible + ' / ' + total;
}
input.addEventListener('input', apply);
for (const c of checks) c.addEventListener('change', apply);
resetBtn.addEventListener('click', () => {
input.value = '';
for (const c of checks) c.checked = true;
apply();
});
})();
// Lazy-load heavy preview content (n8n-demo + judge tables) on first expand.
// Each row contains <template> with the workflow previews inside a
// .lazy-slot[data-loaded="false"] div. On the first toggle-open we move the
@ -991,6 +1197,9 @@ interface CliArgs {
eeDir: string;
iaDir: string;
out: string;
eeAsIa: boolean;
eeLabel?: string;
iaLabel?: string;
}
function parseArgs(argv: string[]): CliArgs {
@ -1000,21 +1209,34 @@ function parseArgs(argv: string[]): CliArgs {
const value = argv[idx + 1];
return value && !value.startsWith('--') ? value : undefined;
};
const has = (flag: string): boolean => argv.includes(flag);
const eeDir = get('--ee-dir');
const iaDir = get('--ia-dir');
if (!eeDir || !iaDir) {
throw new Error(
'Usage: tsx evaluations/cli/compare-pairwise.ts --ee-dir <path> --ia-dir <path> [--out <path>]',
'Usage: tsx evaluations/cli/compare-pairwise.ts --ee-dir <path> --ia-dir <path> [--out <path>] [--ee-as-ia] [--ee-label <name>] [--ia-label <name>]',
);
}
const defaultOut = path.join(path.dirname(path.resolve(iaDir)), 'comparison.html');
const out = path.resolve(get('--out') ?? defaultOut);
return { eeDir: path.resolve(eeDir), iaDir: path.resolve(iaDir), out };
return {
eeDir: path.resolve(eeDir),
iaDir: path.resolve(iaDir),
out,
eeAsIa: has('--ee-as-ia'),
eeLabel: get('--ee-label'),
iaLabel: get('--ia-label'),
};
}
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
const [ee, ia] = await Promise.all([loadEERun(args.eeDir), loadInstanceAiRun(args.iaDir)]);
const [ee, ia] = await Promise.all([
args.eeAsIa ? loadInstanceAiRun(args.eeDir) : loadEERun(args.eeDir),
loadInstanceAiRun(args.iaDir),
]);
if (args.eeLabel) ee.summary.label = args.eeLabel;
if (args.iaLabel) ia.summary.label = args.iaLabel;
console.log(
`EE records: ${ee.records.length} (pass rate ${pct(ee.summary.totals.primaryPassRate)})`,
@ -1027,7 +1249,11 @@ async function main(): Promise<void> {
const matched = rows.filter((r) => r.ee && r.ia).length;
console.log(`Joined ${rows.length} prompts (${matched} matched on both sides)`);
const html = renderDocument(ee, ia, rows);
const labels: SideLabels = {
ee: args.eeLabel ?? (args.eeAsIa ? 'A' : 'Code'),
ia: args.iaLabel ?? (args.eeAsIa ? 'B' : 'IA'),
};
const html = renderDocument(ee, ia, rows, labels);
await fs.writeFile(args.out, html, 'utf8');
console.log(`Wrote comparison report to ${args.out}`);
}

4
packages/@n8n/instance-ai/evaluations/data/subagent/http-bearer-auth.json Normal file
View File

@ -0,0 +1,4 @@
{
"id": "http-bearer-auth",
"prompt": "Build a workflow that runs manually, calls GET https://api.example.com/v1/items with a Bearer token, and writes the response to a file. Configure all nodes completely and don't ask me for credentials — I'll set them up later."
}

12
packages/@n8n/instance-ai/src/agent/__tests__/system-prompt.test.ts
View File

@ -1,6 +1,18 @@
import { getSystemPrompt } from '../system-prompt';
describe('getSystemPrompt', () => {
describe('first visible turn guidance', () => {
it('instructs the agent to send a concise sentence before the first tool call', () => {
const prompt = getSystemPrompt({});
expect(prompt).toContain('before your first tool call');
expect(prompt).toContain('write one short sentence');
expect(prompt).toContain("Keep it tied to the user's goal, not the tool name");
expect(prompt).toContain('Never let an empty assistant message');
expect(prompt).toContain('[Calling tools: ...]');
});
});
describe('license hints', () => {
it('includes License Limitations section when hints are provided', () => {
const prompt = getSystemPrompt({

2
packages/@n8n/instance-ai/src/agent/system-prompt.ts
View File

@ -166,6 +166,8 @@ Examples: search "credential" for the credentials tool, search "file" for filesy
- Be concise. Ask for clarification when intent is ambiguous.
- No emojis unless the user explicitly requests them.
- At the beginning of a normal user-visible turn, before your first tool call, write one short sentence explaining what you are about to do or what decision you need. Keep it tied to the user's goal, not the tool name. For system-generated background or checkpoint follow-up turns, follow the follow-up instructions.
- Never let an empty assistant message or a \`[Calling tools: ...]\` placeholder be the first visible response.
- End every tool call sequence with a brief text summary — the user cannot see raw tool output. Do not end your turn silently after tool calls. Exception: after spawning a background agent (\`build-workflow-with-agent\`, \`plan\`, \`create-tasks\`, \`delegate\`, \`research-with-agent\`, \`manage-data-tables-with-agent\`) the task card replaces your reply — do not write text.
## Safety

1
packages/@n8n/instance-ai/src/index.ts
View File

@ -91,6 +91,7 @@ export type {
BackgroundTaskStatusSnapshot,
ConfirmationData,
PendingConfirmation,
RunStateTimeoutDetails,
StartedRunState,
SuspendedRunState,
} from './runtime/run-state-registry';

6
packages/@n8n/instance-ai/src/runtime/__tests__/run-state-registry.test.ts
View File

@ -1088,6 +1088,12 @@ describe('RunStateRegistry', () => {
const result = registry.sweepTimedOut(policy, 30_000);
expect(result.activeThreadIds).toEqual(['thread-old']);
expect(result.activeTimeouts['thread-old']).toMatchObject({
reason: 'idle_timeout',
surface: 'active-run',
timeoutMs: 30_000,
idleMs: 30_000,
});
});
it('identifies suspended runs older than maxAgeMs', () => {

13
packages/@n8n/instance-ai/src/runtime/resumable-stream-executor.ts
View File

@ -60,6 +60,12 @@ export interface ExecuteResumableStreamOptions {
control: ResumableStreamControl;
initialMastraRunId?: string;
llmStepTraceHooks?: LlmStepTraceHooks;
/**
* Optional side-channel observer for every mapped stream event. Used for
* cross-cutting telemetry (e.g. template usage). Errors are swallowed so
* the observer can never break stream consumption.
*/
onStreamEvent?: (event: InstanceAiEvent) => void;
}
export type TraceStatus = 'completed' | 'cancelled' | 'suspended' | 'errored';
@ -1927,6 +1933,13 @@ export async function executeResumableStream(
);
if (event) {
workSummaryAccumulator.observe(event);
if (options.onStreamEvent) {
try {
options.onStreamEvent(event);
} catch {
// Side-channel observers must never break stream consumption.
}
}
let shouldPublishEvent = true;
if (event.type === 'confirmation-request') {

32
packages/@n8n/instance-ai/src/runtime/run-state-registry.ts
View File

@ -2,7 +2,11 @@ import type { InstanceAiThreadStatusResponse } from '@n8n/api-types';
import { nanoid } from 'nanoid';
import type { InstanceAiTraceContext } from '../types';
import type { InstanceAiLivenessPolicy } from './liveness-policy';
import type {
InstanceAiLivenessPolicy,
InstanceAiLivenessSurface,
InstanceAiLivenessTimeoutReason,
} from './liveness-policy';
export interface ActiveRunState {
runId: string;
@ -72,6 +76,14 @@ export interface BackgroundTaskStatusSnapshot {
threadId: string;
}
export interface RunStateTimeoutDetails {
reason: InstanceAiLivenessTimeoutReason;
surface: InstanceAiLivenessSurface;
timeoutMs: number;
elapsedMs: number;
idleMs: number;
}
export interface StartRunOptions<TUser> {
threadId: string;
user: TUser;
@ -372,8 +384,12 @@ export class RunStateRegistry<TUser = unknown> {
activeThreadIds: string[];
suspendedThreadIds: string[];
confirmationRequestIds: string[];
activeTimeouts: Record<string, RunStateTimeoutDetails>;
suspendedTimeouts: Record<string, RunStateTimeoutDetails>;
confirmationTimeouts: Record<string, RunStateTimeoutDetails>;
} {
const activeThreadIds: string[] = [];
const activeTimeouts: Record<string, RunStateTimeoutDetails> = {};
for (const [threadId, run] of this.activeRuns) {
if (this.hasPendingConfirmationForThread(threadId)) continue;
@ -387,10 +403,12 @@ export class RunStateRegistry<TUser = unknown> {
});
if (decision.action === 'timeout') {
activeThreadIds.push(threadId);
activeTimeouts[threadId] = decision;
}
}
const suspendedThreadIds: string[] = [];
const suspendedTimeouts: Record<string, RunStateTimeoutDetails> = {};
for (const [threadId, run] of this.suspendedRuns) {
const startedAt = run.startedAt ?? run.createdAt;
const lastActivityAt = run.lastActivityAt ?? run.createdAt;
@ -402,9 +420,11 @@ export class RunStateRegistry<TUser = unknown> {
});
if (decision.action === 'timeout') {
suspendedThreadIds.push(threadId);
suspendedTimeouts[threadId] = decision;
}
}
const confirmationRequestIds: string[] = [];
const confirmationTimeouts: Record<string, RunStateTimeoutDetails> = {};
for (const [reqId, pending] of this.pendingConfirmations) {
const startedAt = pending.startedAt ?? pending.createdAt;
const lastActivityAt = pending.lastActivityAt ?? pending.createdAt;
@ -416,9 +436,17 @@ export class RunStateRegistry<TUser = unknown> {
});
if (decision.action === 'timeout') {
confirmationRequestIds.push(reqId);
confirmationTimeouts[reqId] = decision;
}
}
return { activeThreadIds, suspendedThreadIds, confirmationRequestIds };
return {
activeThreadIds,
suspendedThreadIds,
confirmationRequestIds,
activeTimeouts,
suspendedTimeouts,
confirmationTimeouts,
};
}
/**

7
packages/@n8n/instance-ai/src/stream/consume-with-hitl.ts
View File

@ -1,4 +1,5 @@
import type { Agent } from '@mastra/core/agent';
import type { InstanceAiEvent } from '@n8n/api-types';
import type { InstanceAiEventBus } from '../event-bus/event-bus.interface';
import type { Logger } from '../logger';
@ -30,6 +31,11 @@ export interface ConsumeWithHitlOptions {
maxSteps?: number;
/** Additional options to preserve when resuming a suspended stream. */
resumeOptions?: Record<string, unknown>;
/**
* Optional side-channel observer for every mapped stream event. Used for
* cross-cutting telemetry; errors are swallowed by the executor.
*/
onStreamEvent?: (event: InstanceAiEvent) => void;
}
export interface ConsumeWithHitlResult {
@ -83,6 +89,7 @@ export async function consumeStreamWithHitl(
: {}),
},
llmStepTraceHooks: options.llmStepTraceHooks,
onStreamEvent: options.onStreamEvent,
});
return { text: result.text ?? options.stream.text, workSummary: result.workSummary };

37
packages/@n8n/instance-ai/src/tools/__tests__/credentials.tool.test.ts
View File

@ -799,6 +799,43 @@ describe('credentials tool', () => {
});
});
it('should return missing_credentials error when credentials is undefined', async () => {
const context = createMockContext();
const suspendFn = jest.fn();
const tool = createCredentialsTool(context);
const result = await tool.execute!(
{ action: 'setup', credentialFlow: { stage: 'finalize' } } as unknown as Parameters<
NonNullable<typeof tool.execute>
>[0],
suspendCtx(suspendFn),
);
expect(result).toEqual({
error: 'missing_credentials',
message: expect.stringContaining('credentials'),
});
expect(suspendFn).not.toHaveBeenCalled();
expect(context.credentialService.list).not.toHaveBeenCalled();
});
it('should return missing_credentials error when credentials is an empty array', async () => {
const context = createMockContext();
const suspendFn = jest.fn();
const tool = createCredentialsTool(context);
const result = await tool.execute!(
{ action: 'setup', credentials: [] },
suspendCtx(suspendFn),
);
expect(result).toEqual({
error: 'missing_credentials',
message: expect.stringContaining('credentials'),
});
expect(suspendFn).not.toHaveBeenCalled();
});
it('should default reason when not provided in credential requests', async () => {
const context = createMockContext();
(context.credentialService.list as jest.Mock).mockResolvedValue([]);

8
packages/@n8n/instance-ai/src/tools/credentials.tool.ts
View File

@ -341,6 +341,14 @@ async function handleSetup(
const suspend = ctx?.agent?.suspend as ((payload: unknown) => Promise<void>) | undefined;
const isFinalize = input.credentialFlow?.stage === 'finalize';
if (!input.credentials || input.credentials.length === 0) {
return {
error: 'missing_credentials',
message:
'The `credentials` array is required for the setup action. Pass an array of { credentialType, reason?, suggestedName? } entries describing each credential to set up.',
};
}
// State 1: First call — look up existing credentials per type and suspend.
// Scope the lookup to `projectId` when provided so the candidates match what
// the workflow being built can actually use.

13
packages/@n8n/instance-ai/src/tools/orchestration/build-workflow-agent.prompt.ts
View File

@ -393,6 +393,19 @@ n8n normalizes column names to snake_case (e.g., \`dayName\` → \`day_name\`).
execute_command: grep -i "gmail" ${workspaceRoot}/node-types/index.txt
\`\`\`
d. **Look for similar workflow examples** in \`${workspaceRoot}/examples/\` — a curated set of real n8n workflows in SDK form. Grep the index, then read the closest match for structural inspiration:
\`\`\`
execute_command: grep -i "<keyword>" ${workspaceRoot}/examples/index.txt
execute_command: cat ${workspaceRoot}/examples/<file>.ts
\`\`\`
Each line in \`examples/index.txt\` is \`filename | name | nodes | tags | source-id\`. Use the example as a reference for **structure** (which credential type each node uses, how nodes are wired, where sub-nodes attach to an agent, where sticky notes go) — not as a verbatim copy. The user's request will rarely match an example one-to-one.
The \`examples/\` directory is **read-only reference**. Never edit files there; \`src/\` and \`chunks/\` are your scratch.
Examples use \`newCredential('Name', 'id')\` for clarity. When you copy a pattern into \`src/workflow.ts\`, replace those calls with raw \`{ id, name }\` from \`credentials(action="list")\` per the rules above.
If grep returns nothing, build from scratch. **Do not fabricate examples that do not exist.**
3. **Get node schemas**: Call \`nodes(action="type-definition")\` with ALL the node IDs you need in a single call (up to 5). For nodes with discriminators (from search results), include the \`resource\` and \`operation\` fields. **Read the definitions carefully** — they contain exact parameter names, types, required fields, valid enum values, credential types, displayOptions conditions, and \`@builderHint\` annotations with critical configuration guidance.
**Important**: Only call \`nodes(action="type-definition")\` for nodes you will actually use in the workflow. Do not speculatively fetch definitions "just in case". If a definition returns empty or an error, do not retry — proceed with the information from \`nodes(action="search")\` results instead.

35
packages/@n8n/instance-ai/src/tools/orchestration/build-workflow-agent.tool.ts
View File

@ -57,6 +57,13 @@ import {
import type { BuilderWorkspace } from '../../workspace/builder-sandbox-factory';
import { readFileViaSandbox } from '../../workspace/sandbox-fs';
import { getWorkspaceRoot } from '../../workspace/sandbox-setup';
import {
attachTemplateTelemetrySession,
createTemplateTelemetrySession,
createTypedToolObserver,
detachTemplateTelemetrySession,
type TemplateTelemetrySession,
} from '../../workspace/template-telemetry';
import { createCredentialsTool, type CredentialAction } from '../credentials.tool';
import { buildCredentialSnapshot, type CredentialEntry } from '../workflows/resolve-credentials';
import { createIdentityEnforcedSubmitWorkflowTool } from '../workflows/submit-workflow-identity';
@ -1068,6 +1075,8 @@ export async function startBuildWorkflowAgentTask(
await withTraceContextActor(traceContext, async () => {
let builderWs: BuilderWorkspace | undefined;
let activeBuilderSession: BuilderSandboxSession | undefined = reusedBuilderSession;
let telemetrySession: TemplateTelemetrySession | undefined;
let telemetryWorkspace: BuilderWorkspace['workspace'] | undefined;
const submitAttempts = new Map<string, SubmitWorkflowAttempt>();
// Append-only history so a later failed submit for the main path
// cannot mask an earlier successful submit during post-error recovery.
@ -1098,6 +1107,21 @@ export async function startBuildWorkflowAgentTask(
root = await getWorkspaceRoot(workspace);
}
// Bind a template-telemetry session to the workspace so any grep on
// examples/index.txt or cat on examples/*.ts gets observed via runInSandbox.
telemetrySession = createTemplateTelemetrySession({
context,
threadId: context.threadId,
runId: context.runId,
workItemId,
userRequestExcerpt: input.task,
});
telemetryWorkspace = workspace;
attachTemplateTelemetrySession(workspace, telemetrySession);
// Captures typed `mastra_workspace_grep` / `mastra_workspace_read_file`
// calls that never reach `runInSandbox` (shell-channel only).
const templateToolObserver = createTypedToolObserver(telemetrySession);
prompt = createSandboxBuilderAgentPrompt(root);
if (!activeBuilderSession && builderWs) {
activeBuilderSession = context.builderSandboxSessionRegistry?.create({
@ -1294,6 +1318,7 @@ export async function startBuildWorkflowAgentTask(
llmStepTraceHooks,
maxSteps: MAX_STEPS.BUILDER,
resumeOptions,
onStreamEvent: templateToolObserver,
});
});
@ -1594,6 +1619,16 @@ export async function startBuildWorkflowAgentTask(
await promoteMainWorkflow(domainContext, context.logger, fallbackMainWorkflowId);
return { text: toolFinalText };
} finally {
if (telemetrySession && telemetryWorkspace) {
try {
telemetrySession.flush();
detachTemplateTelemetrySession(telemetryWorkspace);
} catch (error) {
context.logger.warn('build-workflow-agent: failed to flush template telemetry', {
error: error instanceof Error ? error.message : String(error),
});
}
}
clearFilesystemMutationGuard?.();
if (activeBuilderSession && context.builderSandboxSessionRegistry) {
await context.builderSandboxSessionRegistry.release(activeBuilderSession.sessionId, {

34
packages/@n8n/instance-ai/src/utils/scrub-secrets.ts Normal file
View File

@ -0,0 +1,34 @@
/**
* Replace common credential patterns in free-form text with `[REDACTED]`.
*
* Used before persisting user-supplied text (telemetry excerpts, eval report
* HTML) where keys/tokens accidentally pasted into prompts or command lines
* could otherwise leak downstream.
*
* Conservative by design: matches well-known prefixed tokens and explicit
* `key=value` pairs only. We don't attempt to redact arbitrary long opaque
* strings — false positives on file paths, IDs, or base64 payloads would
* make the output unreadable.
*/
const SECRET_VALUE_PATTERNS: readonly RegExp[] = [
// Authorization-header substrings: `Bearer <token>`, `Basic <token>`, `Token <token>`
/\b(?:Bearer|Basic|Token)\s+[A-Za-z0-9._~+/=-]{12,}/gi,
// OpenAI / Anthropic API keys
/\bsk-(?:ant-|proj-)?[A-Za-z0-9_-]{16,}/g,
// Slack tokens (xoxb, xoxp, xoxa, xoxr, xoxs, xoxo)
/\bxox[abprso]-[A-Za-z0-9-]{10,}/g,
// GitHub tokens (ghp, ghs, gho, ghr, ghu)
/\bgh[psoru]_[A-Za-z0-9]{20,}/g,
// AWS access key id
/\bAKIA[0-9A-Z]{16}\b/g,
// Generic `password=...` / `api_key=...` / `secret=...` style assignments
/\b(?:password|passwd|secret|api[_-]?key|access[_-]?token|auth[_-]?token)\s*[:=]\s*\S+/gi,
];
export function scrubSecretsInText(input: string): string {
let out = input;
for (const pattern of SECRET_VALUE_PATTERNS) {
out = out.replace(pattern, '[REDACTED]');
}
return out;
}

21
packages/@n8n/instance-ai/src/workspace/__tests__/builder-sandbox-factory.test.ts
View File

@ -118,6 +118,7 @@ jest.mock('../sandbox-setup', () => ({
formatNodeCatalogLine: jest.fn((x: { name?: string }) => x.name ?? ''),
getWorkspaceRoot: jest.fn(async () => await Promise.resolve('/home/daytona/workspace')),
setupSandboxWorkspace: jest.fn(async () => await Promise.resolve()),
writeCuratedExamples: jest.fn(async () => await Promise.resolve()),
PACKAGE_JSON: '{}',
TSCONFIG_JSON: '{}',
BUILD_MJS: '',
@ -231,6 +232,21 @@ describe('BuilderSandboxFactory createDaytona snapshot branching', () => {
expect(ensureSnapshotSpy).toHaveBeenCalledWith(expect.anything(), 'proxy');
});
it('writes curated examples into the new Daytona sandbox', async () => {
// eslint-disable-next-line @typescript-eslint/no-require-imports, @typescript-eslint/consistent-type-imports
const sandboxSetup = require('../sandbox-setup') as typeof import('../sandbox-setup');
(sandboxSetup.writeCuratedExamples as jest.Mock).mockClear();
const config = makeDaytonaConfig();
const snapshotManager = new SnapshotManager('node:20', NOOP_LOGGER, '1.123.0');
jest.spyOn(snapshotManager, 'ensureSnapshot').mockResolvedValue('n8n/instance-ai:1.123.0');
const factory = new BuilderSandboxFactory(config, snapshotManager, NOOP_LOGGER);
await factory.create('builder-1', makeContext());
expect(sandboxSetup.writeCuratedExamples).toHaveBeenCalledTimes(1);
});
});
describe('BuilderSandboxFactory createDaytona error reporting', () => {
@ -382,11 +398,16 @@ describe('BuilderSandboxFactory.createN8nSandbox cleanup on failure', () => {
});
it('returns a cleanup handle that destroys the sandbox when create succeeds', async () => {
// eslint-disable-next-line @typescript-eslint/no-require-imports, @typescript-eslint/consistent-type-imports
const sandboxSetup = require('../sandbox-setup') as typeof import('../sandbox-setup');
(sandboxSetup.writeCuratedExamples as jest.Mock).mockClear();
const factory = new BuilderSandboxFactory(makeN8nSandboxConfig(), undefined);
const bw = await factory.create('b-3', makeContext());
expect(capturedSandboxes).toHaveLength(1);
expect(capturedSandboxes[0].destroy).not.toHaveBeenCalled();
expect(sandboxSetup.writeCuratedExamples).toHaveBeenCalledTimes(1);
await bw.cleanup();

24
packages/@n8n/instance-ai/src/workspace/__tests__/sandbox-setup.test.ts
View File

@ -171,6 +171,30 @@ describe('setupSandboxWorkspace', () => {
);
});
it('writes the curated examples bundle into examples/', async () => {
const runInSandbox: RunInSandboxMock = jest.fn<
Promise<{ exitCode: number; stdout: string; stderr: string }>,
[Workspace, string, string?]
>();
runInSandbox.mockResolvedValue({ exitCode: 0, stdout: '', stderr: '' });
const readFileViaSandbox: ReadFileViaSandboxMock = jest.fn<
Promise<string | null>,
[Workspace, string]
>();
readFileViaSandbox.mockResolvedValue(null);
const setupSandboxWorkspace = loadSetupSandboxWorkspaceWithFsMocks(
runInSandbox,
readFileViaSandbox,
);
const writeFile = jest.fn<Promise<void>, [string, string, { recursive: true }]>(async () => {});
await setupSandboxWorkspace(createLocalWorkspace(writeFile), createSetupContext());
const writtenPaths = writeFile.mock.calls.map(([path]) => path);
expect(writtenPaths).toContain('/sandbox/examples/index.txt');
expect(writtenPaths.some((p) => /^\/sandbox\/examples\/.+\.ts$/.test(p))).toBe(true);
});
it('does not write the initialized marker when npm install fails', async () => {
const runInSandbox: RunInSandboxMock = jest.fn<
Promise<{ exitCode: number; stdout: string; stderr: string }>,

16
packages/@n8n/instance-ai/src/workspace/builder-sandbox-factory.ts
View File

@ -3,7 +3,8 @@
*
* Creates an ephemeral sandbox + workspace per builder invocation.
* - Daytona mode: creates from pre-warmed Image (config + deps baked in),
* then writes the node-types catalog post-creation via filesystem API.
* then writes the node-types catalog and curated examples post-creation
* via filesystem API.
* - Local mode: per-builder subdirectory with full setup (development only)
*/
@ -27,7 +28,12 @@ import {
import { runInSandbox, writeFileViaSandbox } from './sandbox-fs';
import type { SnapshotManager } from './snapshot-manager';
import type { InstanceAiContext } from '../types';
import { formatNodeCatalogLine, getWorkspaceRoot, setupSandboxWorkspace } from './sandbox-setup';
import {
formatNodeCatalogLine,
getWorkspaceRoot,
setupSandboxWorkspace,
writeCuratedExamples,
} from './sandbox-setup';
const NOOP_LOGGER: Logger = {
info: () => {},
@ -266,6 +272,10 @@ export class BuilderSandboxFactory {
await writeFileViaSandbox(workspace, `${root}/node-types/index.txt`, catalog);
}
// Curated examples — also too large to bake into the image, written
// post-creation. Without this the builder sees an empty examples/ dir.
await writeCuratedExamples(workspace, this.logger);
await this.linkWorkspaceSdkIfEnabled(workspace, root);
return {
@ -320,6 +330,8 @@ export class BuilderSandboxFactory {
await writeFileViaSandbox(workspace, `${root}/node-types/index.txt`, catalog);
}
await writeCuratedExamples(workspace, this.logger);
await this.linkWorkspaceSdkIfEnabled(workspace, root);
return {

32
packages/@n8n/instance-ai/src/workspace/sandbox-fs.ts
View File

@ -11,10 +11,16 @@
import type { Workspace } from '@mastra/core/workspace';
import { getTemplateTelemetrySession } from './template-telemetry';
/**
* Execute a shell command in the sandbox and wait for completion.
* Tries `executeCommand` first (auto-generated by MastraSandbox when processes
* are provided), falls back to `processes.spawn` + wait.
*
* If a TemplateTelemetrySession is bound to the workspace via
* `attachTemplateTelemetrySession`, the command + stdout get observed for
* template-usage events. Failures in the observer never break the command.
*/
export async function runInSandbox(
workspace: Workspace,
@ -24,18 +30,28 @@ export async function runInSandbox(
const sandbox = workspace.sandbox;
if (!sandbox) throw new Error('Workspace has no sandbox');
let result: { exitCode: number; stdout: string; stderr: string };
if (sandbox.executeCommand) {
const result = await sandbox.executeCommand(command, [], { cwd });
return { exitCode: result.exitCode, stdout: result.stdout, stderr: result.stderr };
}
if (sandbox.processes) {
const r = await sandbox.executeCommand(command, [], { cwd });
result = { exitCode: r.exitCode, stdout: r.stdout, stderr: r.stderr };
} else if (sandbox.processes) {
const handle = await sandbox.processes.spawn(command, { cwd });
const result = await handle.wait();
return { exitCode: result.exitCode, stdout: result.stdout, stderr: result.stderr };
const r = await handle.wait();
result = { exitCode: r.exitCode, stdout: r.stdout, stderr: r.stderr };
} else {
throw new Error('Sandbox has neither executeCommand nor processes available');
}
throw new Error('Sandbox has neither executeCommand nor processes available');
const session = getTemplateTelemetrySession(workspace);
if (session) {
try {
session.observe(command, result.stdout);
} catch {
// Telemetry must never fail a command. Swallow.
}
}
return result;
}
/**

32
packages/@n8n/instance-ai/src/workspace/sandbox-setup.ts
View File

@ -22,8 +22,10 @@
*/
import type { Workspace } from '@mastra/core/workspace';
import { getExampleFiles } from '@n8n/workflow-sdk/examples-loader';
import { createRequire } from 'node:module';
import type { Logger } from '../logger';
import type { InstanceAiContext, SearchableNodeDescription } from '../types';
import { isLinkWorkspaceSdkEnabled } from './pack-workspace-sdk';
import { runInSandbox, readFileViaSandbox, escapeSingleQuotes } from './sandbox-fs';
@ -315,6 +317,35 @@ export async function getWorkspaceRoot(workspace: Workspace): Promise<string> {
return root;
}
/**
* Write the curated workflow examples bundle into `${root}/examples/`.
*
* Used by `setupSandboxWorkspace` (local provider) and by the Daytona /
* n8n-sandbox factory paths, which skip the full setup but still need the
* curated reference material the builder agent greps against.
*
* No-op when the loader returns an empty bundle (e.g. running against a
* workspace where the manifest hasn't been fetched).
*/
export async function writeCuratedExamples(workspace: Workspace, logger?: Logger): Promise<void> {
const start = Date.now();
const { files: exampleFiles, indexTxt } = getExampleFiles();
if (exampleFiles.length === 0) return;
const root = await getWorkspaceRoot(workspace);
const fileMap = new Map<string, string>();
fileMap.set('examples/index.txt', indexTxt);
for (const example of exampleFiles) {
fileMap.set(`examples/${example.filename}`, example.content);
}
await writeWorkspaceFiles(workspace, root, fileMap);
logger?.debug('[sandbox-setup] prepared curated examples', {
count: exampleFiles.length,
durationMs: Date.now() - start,
});
}
/**
* Initialize the sandbox workspace for the workflow builder agent.
* Idempotent — skips if already initialized (checks marker file).
@ -372,6 +403,7 @@ export async function setupSandboxWorkspace(
// ── Write workspace files ──────────────────────────────────────────────
await writeWorkspaceFiles(workspace, root, files);
await writeCuratedExamples(workspace, context.logger);
// npm install (must run after package.json is in place)
const npmResult = await runInSandbox(workspace, 'npm install --ignore-scripts', root);

417
packages/@n8n/instance-ai/src/workspace/template-telemetry.test.ts Normal file
View File

@ -0,0 +1,417 @@
import type { InstanceAiEvent } from '@n8n/api-types';
import {
createTemplateTelemetrySession,
createTypedToolObserver,
extractGrepQuery,
type TelemetrySessionOptions,
} from './template-telemetry';
import type { OrchestrationContext } from '../types';
function makeOpts(): {
opts: TelemetrySessionOptions;
calls: Array<{ name: string; props: Record<string, unknown> }>;
} {
const calls: Array<{ name: string; props: Record<string, unknown> }> = [];
const context = {
trackTelemetry: (name: string, props: Record<string, unknown>) => {
calls.push({ name, props });
},
} as unknown as OrchestrationContext;
return {
opts: {
context,
threadId: 'thread-1',
runId: 'run-42',
workItemId: 'wi_abc12345',
userRequestExcerpt: 'summarise emails to slack daily',
},
calls,
};
}
describe('createTemplateTelemetrySession', () => {
it('emits a search event when grep targets examples/index.txt', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
session.observe(
'grep -i "slack" /home/sandbox/workspace/examples/index.txt',
'slack-daily-summary.ts | Daily Slack ...\nother.ts | ...\n',
);
const search = calls.find((c) => c.name === 'Builder template search');
expect(search).toBeDefined();
expect(search!.props.query).toBe('slack');
expect(search!.props.result_count).toBe(2);
expect(search!.props.run_id).toBe('run-42');
expect(search!.props.work_item_id).toBe('wi_abc12345');
});
it('does not emit search for grep against a different file', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
session.observe('grep -i "gmail" /workspace/node-types/index.txt', 'something\n');
expect(calls.find((c) => c.name === 'Builder template search')).toBeUndefined();
});
it('emits a read event when cat targets examples/<file>.ts', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
session.observe(
'cat /home/sandbox/workspace/examples/slack-daily-summary.ts',
'/* file content here */',
);
const read = calls.find((c) => c.name === 'Builder template read');
expect(read).toBeDefined();
expect(read!.props.template_filename).toBe('slack-daily-summary.ts');
expect(read!.props.bytes_read).toBe('/* file content here */'.length);
});
it('handles head/sed/less/more in addition to cat', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
session.observe('head -50 /workspace/examples/slack-daily-summary.ts', 'lines');
session.observe('sed -n 1,30p /workspace/examples/build-your-first-ai-agent-6270.ts', 'lines');
const reads = calls.filter((c) => c.name === 'Builder template read');
expect(reads.length).toBe(2);
});
it('does not emit a read for cat on a non-template file', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
session.observe('cat /workspace/src/workflow.ts', 'content');
expect(calls.find((c) => c.name === 'Builder template read')).toBeUndefined();
});
it('emits a session rollup on flush', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
session.observe('grep -i "slack" /workspace/examples/index.txt', 'a\nb\n');
session.observe('cat /workspace/examples/slack-daily-summary.ts', 'X');
session.observe('cat /workspace/examples/slack-daily-summary.ts', 'X');
session.flush();
const rollup = calls.find((c) => c.name === 'Builder template session');
expect(rollup).toBeDefined();
expect(rollup!.props.search_count).toBe(1);
expect(rollup!.props.read_count).toBe(2);
expect(rollup!.props.unique_templates_read).toBe(1);
expect(rollup!.props.user_request_excerpt).toBe('summarise emails to slack daily');
});
it('truncates user_request_excerpt to 120 chars', () => {
const long = 'a'.repeat(200);
const { opts, calls } = makeOpts();
opts.userRequestExcerpt = long;
const session = createTemplateTelemetrySession(opts);
session.flush();
const rollup = calls.find((c) => c.name === 'Builder template session');
expect(rollup!.props.user_request_excerpt).toHaveLength(120);
});
it('flush is idempotent and stops further events', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
session.flush();
session.observe('cat /workspace/examples/foo.ts', 'X');
session.flush();
expect(calls.filter((c) => c.name === 'Builder template session').length).toBe(1);
expect(calls.filter((c) => c.name === 'Builder template read').length).toBe(0);
});
it('sets user_request_excerpt to null when none provided', () => {
const { opts, calls } = makeOpts();
opts.userRequestExcerpt = undefined;
const session = createTemplateTelemetrySession(opts);
session.flush();
const rollup = calls.find((c) => c.name === 'Builder template session');
expect(rollup!.props.user_request_excerpt).toBeNull();
});
it('redacts API keys and bearer tokens from user_request_excerpt', () => {
const { opts, calls } = makeOpts();
opts.userRequestExcerpt =
'call openai with sk-proj-abcdef0123456789 and slack xoxb-1234567890-abcdef';
const session = createTemplateTelemetrySession(opts);
session.flush();
const rollup = calls.find((c) => c.name === 'Builder template session');
const excerpt = rollup!.props.user_request_excerpt as string;
expect(excerpt).not.toContain('sk-proj-abcdef0123456789');
expect(excerpt).not.toContain('xoxb-1234567890-abcdef');
expect(excerpt).toContain('[REDACTED]');
});
it('redacts password=... assignments from user_request_excerpt', () => {
const { opts, calls } = makeOpts();
opts.userRequestExcerpt = 'connect to db with password=hunter2 and api_key=secret123';
const session = createTemplateTelemetrySession(opts);
session.flush();
const rollup = calls.find((c) => c.name === 'Builder template session');
const excerpt = rollup!.props.user_request_excerpt as string;
expect(excerpt).not.toContain('hunter2');
expect(excerpt).not.toContain('secret123');
expect(excerpt).toContain('[REDACTED]');
});
it('leaves normal English prompts unchanged', () => {
const { opts, calls } = makeOpts();
opts.userRequestExcerpt = 'summarise emails to slack daily';
const session = createTemplateTelemetrySession(opts);
session.flush();
const rollup = calls.find((c) => c.name === 'Builder template session');
expect(rollup!.props.user_request_excerpt).toBe('summarise emails to slack daily');
});
});
describe('extractGrepQuery', () => {
it('extracts double-quoted patterns', () => {
expect(extractGrepQuery('grep -i "slack post" examples/index.txt')).toBe('slack post');
});
it('extracts single-quoted patterns', () => {
expect(extractGrepQuery("grep -i 'gmail' examples/index.txt")).toBe('gmail');
});
it('extracts bare patterns when unquoted', () => {
expect(extractGrepQuery('grep slack examples/index.txt')).toBe('slack');
expect(extractGrepQuery('grep -i slack examples/index.txt')).toBe('slack');
});
it('caps the query at 200 characters', () => {
const long = 'x'.repeat(300);
const result = extractGrepQuery(`grep -i "${long}" examples/index.txt`);
expect(result.length).toBe(200);
});
it('returns empty string when no pattern found', () => {
expect(extractGrepQuery('grep')).toBe('');
});
});
describe('observeTypedRead / observeTypedSearch', () => {
it('observeTypedRead emits a read event with filename and bytes', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
session.observeTypedRead('slack-daily-summary.ts', 1234);
const read = calls.find((c) => c.name === 'Builder template read');
expect(read).toBeDefined();
expect(read!.props.template_filename).toBe('slack-daily-summary.ts');
expect(read!.props.bytes_read).toBe(1234);
expect(read!.props.run_id).toBe('run-42');
expect(read!.props.work_item_id).toBe('wi_abc12345');
});
it('observeTypedSearch emits a search event with query and result count', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
session.observeTypedSearch('slack', 5);
const search = calls.find((c) => c.name === 'Builder template search');
expect(search).toBeDefined();
expect(search!.props.query).toBe('slack');
expect(search!.props.result_count).toBe(5);
});
it('typed observations contribute to the session rollup', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
session.observeTypedSearch('slack', 3);
session.observeTypedRead('slack-daily-summary.ts', 100);
session.observeTypedRead('slack-daily-summary.ts', 50);
session.flush();
const rollup = calls.find((c) => c.name === 'Builder template session');
expect(rollup!.props.search_count).toBe(1);
expect(rollup!.props.read_count).toBe(2);
expect(rollup!.props.unique_templates_read).toBe(1);
});
it('typed methods are no-ops after flush', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
session.flush();
session.observeTypedRead('foo.ts', 10);
session.observeTypedSearch('foo', 1);
expect(calls.filter((c) => c.name === 'Builder template read')).toHaveLength(0);
expect(calls.filter((c) => c.name === 'Builder template search')).toHaveLength(0);
});
it('truncates typed search query to the query cap', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
session.observeTypedSearch('x'.repeat(300), 1);
const search = calls.find((c) => c.name === 'Builder template search');
expect(typeof search!.props.query).toBe('string');
expect((search!.props.query as string).length).toBe(200);
});
it('redacts secrets from typed search query before emit', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
session.observeTypedSearch('Bearer abcdef0123456789ABCDEF and password=hunter2', 1);
const search = calls.find((c) => c.name === 'Builder template search');
const query = search!.props.query as string;
expect(query).not.toContain('abcdef0123456789ABCDEF');
expect(query).not.toContain('hunter2');
expect(query).toContain('[REDACTED]');
});
it('redacts secrets from grep-derived search query before emit', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
session.observe(
'grep -i "sk-proj-abcdef0123456789xyz" /workspace/examples/index.txt',
'nothing\n',
);
const search = calls.find((c) => c.name === 'Builder template search');
const query = search!.props.query as string;
expect(query).not.toContain('sk-proj-abcdef0123456789xyz');
expect(query).toContain('[REDACTED]');
});
});
describe('createTypedToolObserver', () => {
function toolCall(
toolCallId: string,
toolName: string,
args: Record<string, unknown>,
): InstanceAiEvent {
return {
type: 'tool-call',
runId: 'run-42',
agentId: 'agent-1',
payload: { toolCallId, toolName, args },
};
}
function toolResult(toolCallId: string, result: unknown): InstanceAiEvent {
return {
type: 'tool-result',
runId: 'run-42',
agentId: 'agent-1',
payload: { toolCallId, result },
};
}
function toolError(toolCallId: string, error: string): InstanceAiEvent {
return {
type: 'tool-error',
runId: 'run-42',
agentId: 'agent-1',
payload: { toolCallId, error },
};
}
it('emits typed read for mastra_workspace_read_file targeting examples/<slug>.ts', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
const observe = createTypedToolObserver(session);
observe(
toolCall('tc-1', 'mastra_workspace_read_file', {
path: '/workspace/examples/slack-daily-summary.ts',
}),
);
observe(
toolResult(
'tc-1',
'/workspace/examples/slack-daily-summary.ts (200 bytes)\nfile content here\n',
),
);
const read = calls.find((c) => c.name === 'Builder template read');
expect(read).toBeDefined();
expect(read!.props.template_filename).toBe('slack-daily-summary.ts');
expect(read!.props.bytes_read).toBeGreaterThan(0);
});
it('emits typed search for mastra_workspace_grep targeting examples/', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
const observe = createTypedToolObserver(session);
observe(toolCall('tc-2', 'mastra_workspace_grep', { pattern: 'slack', path: 'examples/' }));
observe(toolResult('tc-2', 'examples/a.ts:1:1: slack\nexamples/b.ts:5:1: slack\n'));
const search = calls.find((c) => c.name === 'Builder template search');
expect(search).toBeDefined();
expect(search!.props.query).toBe('slack');
expect(search!.props.result_count).toBe(2);
});
it('emits typed search for grep targeting examples/index.txt', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
const observe = createTypedToolObserver(session);
observe(
toolCall('tc-3', 'mastra_workspace_grep', { pattern: 'slack', path: 'examples/index.txt' }),
);
observe(toolResult('tc-3', 'slack-daily.ts | Daily Slack\nslack-onboard.ts | Onboard\n'));
const search = calls.find((c) => c.name === 'Builder template search');
expect(search!.props.result_count).toBe(2);
});
it('ignores read_file outside examples/', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
const observe = createTypedToolObserver(session);
observe(toolCall('tc-x', 'mastra_workspace_read_file', { path: '/workspace/src/workflow.ts' }));
observe(toolResult('tc-x', 'content'));
expect(calls.find((c) => c.name === 'Builder template read')).toBeUndefined();
});
it('ignores grep that does not target examples/', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
const observe = createTypedToolObserver(session);
observe(toolCall('tc-y', 'mastra_workspace_grep', { pattern: 'foo', path: 'src/' }));
observe(toolResult('tc-y', 'src/a.ts:1:1: foo\n'));
expect(calls.find((c) => c.name === 'Builder template search')).toBeUndefined();
});
it('ignores unrelated tools', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
const observe = createTypedToolObserver(session);
observe(
toolCall('tc-z', 'mastra_workspace_write_file', { path: 'examples/foo.ts', content: 'x' }),
);
observe(toolResult('tc-z', 'ok'));
expect(calls.find((c) => c.name === 'Builder template read')).toBeUndefined();
expect(calls.find((c) => c.name === 'Builder template search')).toBeUndefined();
});
it('does not emit on tool-error and clears pending state', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
const observe = createTypedToolObserver(session);
observe(toolCall('tc-e', 'mastra_workspace_read_file', { path: 'examples/foo.ts' }));
observe(toolError('tc-e', 'permission denied'));
expect(calls.find((c) => c.name === 'Builder template read')).toBeUndefined();
});
it('handles non-string results gracefully (no emission)', () => {
const { opts, calls } = makeOpts();
const session = createTemplateTelemetrySession(opts);
const observe = createTypedToolObserver(session);
observe(toolCall('tc-n', 'mastra_workspace_read_file', { path: 'examples/foo.ts' }));
observe(toolResult('tc-n', { not: 'a string' }));
expect(calls.find((c) => c.name === 'Builder template read')).toBeUndefined();
});
});

272
packages/@n8n/instance-ai/src/workspace/template-telemetry.ts Normal file
View File

@ -0,0 +1,272 @@
/**
* Template usage telemetry for the builder agent.
*
* Pattern-detects template-related shell commands (grep on `examples/index.txt`,
* cat/head/sed on `examples/*.ts`) and emits three event types via the existing
* `context.trackTelemetry?.(name, props)` channel:
*
* - `Builder template search` — fired per index grep, with the search query
* - `Builder template read` — fired per template file read, with the filename
* - `Builder template session`— fired once per builder run on flush(), with rollups
*
* Sessions are attached to a workspace via `attachTemplateTelemetrySession`
* (a per-Workspace WeakMap binding). `runInSandbox` looks the session up and
* calls `observe()` after each command — no caller-side threading required.
*/
import type { Workspace } from '@mastra/core/workspace';
import type { InstanceAiEvent } from '@n8n/api-types';
import type { OrchestrationContext } from '../types';
import { scrubSecretsInText } from '../utils/scrub-secrets';
const MAX_QUERY_LENGTH = 200;
const MAX_USER_REQUEST_LENGTH = 120;
const SEARCH_PATTERN = /\bgrep\b[^|]*\bexamples\/index\.txt\b/;
const READ_COMMAND_HEADS = ['cat', 'head', 'tail', 'sed', 'less', 'more'];
// Match `examples/<slug>.ts` anywhere in the path. `\b` boundaries reject
// `someotherexamples/...` while accepting `/abs/path/examples/foo.ts`.
const READ_FILE_PATTERN = /\bexamples\/([a-zA-Z0-9._-]+\.ts)\b/;
export interface TemplateTelemetrySession {
/** Inspect a command + its stdout; emits search/read events when patterns match. */
observe(command: string, stdout: string): void;
/** Emit a `Builder template read` event for a typed tool invocation. */
observeTypedRead(filename: string, bytesRead: number): void;
/** Emit a `Builder template search` event for a typed tool invocation. */
observeTypedSearch(query: string, resultCount: number): void;
/** Emit the session rollup event and clear state. Idempotent. */
flush(): void;
/** Return true when the session has not been flushed yet. */
isOpen(): boolean;
}
export interface TelemetrySessionOptions {
context: OrchestrationContext;
threadId?: string;
runId: string;
workItemId: string;
/** Optional NL request from the user; truncated to 120 chars. */
userRequestExcerpt?: string;
}
export function createTemplateTelemetrySession(
opts: TelemetrySessionOptions,
): TemplateTelemetrySession {
const baseProps = {
thread_id: opts.threadId,
run_id: opts.runId,
work_item_id: opts.workItemId,
};
let searchCount = 0;
let readCount = 0;
const templatesRead: string[] = [];
let open = true;
function emit(name: string, extra: Record<string, unknown>) {
opts.context.trackTelemetry?.(name, { ...baseProps, ...extra });
}
function observeTypedRead(filename: string, bytesRead: number): void {
if (!open) return;
readCount++;
templatesRead.push(filename);
emit('Builder template read', {
template_filename: filename,
bytes_read: bytesRead,
});
}
function observeTypedSearch(query: string, resultCount: number): void {
if (!open) return;
searchCount++;
emit('Builder template search', {
query: scrubAndTruncateQuery(query),
result_count: resultCount,
});
}
function observe(command: string, stdout: string): void {
if (!open) return;
// Search detection: any grep at examples/index.txt
if (SEARCH_PATTERN.test(command)) {
searchCount++;
emit('Builder template search', {
query: scrubAndTruncateQuery(extractGrepQuery(command)),
result_count: countResultLines(stdout),
});
}
// Read detection: cat/head/sed/etc. against examples/*.ts
const head = command.trim().split(/\s+/, 1)[0]?.split('/').pop() ?? '';
if (READ_COMMAND_HEADS.includes(head)) {
const match = command.match(READ_FILE_PATTERN);
if (match) {
const filename = match[1];
readCount++;
templatesRead.push(filename);
emit('Builder template read', {
template_filename: filename,
bytes_read: stdout.length,
});
}
}
}
function flush(): void {
if (!open) return;
open = false;
emit('Builder template session', {
search_count: searchCount,
read_count: readCount,
templates_read: templatesRead,
unique_templates_read: Array.from(new Set(templatesRead)).length,
user_request_excerpt: opts.userRequestExcerpt
? truncate(scrubSecretsInText(opts.userRequestExcerpt), MAX_USER_REQUEST_LENGTH)
: null,
});
}
return {
observe,
observeTypedRead,
observeTypedSearch,
flush,
isOpen: () => open,
};
}
/** Pull the user's grep query out of the command string. Best-effort; capped. */
export function extractGrepQuery(command: string): string {
// Match: grep [-flags] "pattern" path OR grep [-flags] 'pattern' path OR grep -i pattern path
// First try quoted forms, then a bare token
const quoted = command.match(/grep\b[^"']*["']([^"']*)["']/);
if (quoted) return truncate(quoted[1], MAX_QUERY_LENGTH);
const bare = command.match(/grep\b\s+(?:-[a-zA-Z]+\s+)*([^\s|]+)/);
if (bare) return truncate(bare[1], MAX_QUERY_LENGTH);
return '';
}
function countResultLines(stdout: string): number {
if (!stdout) return 0;
const trimmed = stdout.replace(/\n+$/, '');
if (trimmed.length === 0) return 0;
return trimmed.split('\n').length;
}
function truncate(s: string, max: number): string {
if (s.length <= max) return s;
return s.slice(0, max);
}
/**
* Scrub credential patterns from a grep query, then enforce the size cap.
* Scrub-before-truncate prevents a secret that straddles the cap from
* surviving in a half-form the regex no longer matches.
*/
function scrubAndTruncateQuery(query: string): string {
return truncate(scrubSecretsInText(query), MAX_QUERY_LENGTH);
}
// ---------------------------------------------------------------------------
// Workspace binding
// ---------------------------------------------------------------------------
const SESSIONS = new WeakMap<Workspace, TemplateTelemetrySession>();
/** Bind a session to a workspace so `runInSandbox` can pick it up automatically. */
export function attachTemplateTelemetrySession(
workspace: Workspace,
session: TemplateTelemetrySession,
): void {
SESSIONS.set(workspace, session);
}
/** Remove the session binding. Safe to call multiple times. */
export function detachTemplateTelemetrySession(workspace: Workspace): void {
SESSIONS.delete(workspace);
}
/** Internal: look up the session bound to a workspace, if any. */
export function getTemplateTelemetrySession(
workspace: Workspace,
): TemplateTelemetrySession | undefined {
return SESSIONS.get(workspace);
}
// ---------------------------------------------------------------------------
// Typed-tool observation (covers `mastra_workspace_grep` / `mastra_workspace_read_file`,
// which never reach `runInSandbox` and so wouldn't otherwise be captured.)
// ---------------------------------------------------------------------------
const TYPED_READ_TOOL = 'mastra_workspace_read_file';
const TYPED_GREP_TOOL = 'mastra_workspace_grep';
// Path either is `examples` itself or sits under it: `examples`, `examples/`,
// `examples/foo.ts`, `/abs/examples/`, etc. Rejects `someexamples`, `examples-x`.
const EXAMPLES_PATH_PATTERN = /(?:^|\/)examples(?:$|\/)/;
type PendingTypedCall = { kind: 'read'; filename: string } | { kind: 'search'; query: string };
/**
* Pair `tool-call` + `tool-result` events for typed Mastra workspace tools and
* forward template accesses to the session. Returned function is meant to be
* fed every event from the agent's stream (e.g. via consume-with-hitl's
* `onStreamEvent` hook). Maintains pending state per `toolCallId`.
*/
export function createTypedToolObserver(
session: TemplateTelemetrySession,
): (event: InstanceAiEvent) => void {
const pending = new Map<string, PendingTypedCall>();
return (event) => {
if (!session.isOpen()) return;
if (event.type === 'tool-call') {
const matched = matchTypedTemplateCall(event.payload.toolName, event.payload.args);
if (matched) pending.set(event.payload.toolCallId, matched);
return;
}
if (event.type === 'tool-result') {
const match = pending.get(event.payload.toolCallId);
if (!match) return;
pending.delete(event.payload.toolCallId);
const result = event.payload.result;
if (typeof result !== 'string') return;
if (match.kind === 'read') {
session.observeTypedRead(match.filename, result.length);
} else {
session.observeTypedSearch(match.query, countResultLines(result));
}
return;
}
if (event.type === 'tool-error') {
pending.delete(event.payload.toolCallId);
}
};
}
function matchTypedTemplateCall(
toolName: string,
args: Record<string, unknown>,
): PendingTypedCall | undefined {
if (toolName === TYPED_READ_TOOL) {
const path = typeof args.path === 'string' ? args.path : '';
const match = path.match(READ_FILE_PATTERN);
if (!match) return undefined;
return { kind: 'read', filename: match[1] };
}
if (toolName === TYPED_GREP_TOOL) {
const path = typeof args.path === 'string' ? args.path : '';
if (!EXAMPLES_PATH_PATTERN.test(path)) return undefined;
const pattern = typeof args.pattern === 'string' ? args.pattern : '';
return { kind: 'search', query: pattern };
}
return undefined;
}

8
packages/@n8n/workflow-sdk/.gitignore vendored
View File

@ -12,3 +12,11 @@ test-fixtures/real-workflows/*.generated.ts
scripts/*.js
scripts/*.js.map
scripts/*.d.ts
# Curation pipeline intermediates (committed: manifest.json, templates.zip, _calibration.json, _coverage-report.json)
examples/_raw/
examples/_failures.log
examples/_catalog-snapshot.json
# Workflow JSONs are extracted from templates.zip at runtime — only the zip is committed
examples/workflows/

95
packages/@n8n/workflow-sdk/docs/template-criteria.md Normal file
View File

@ -0,0 +1,95 @@
# Template helpfulness criteria
This rubric scores public n8n.io workflows for inclusion in `examples/manifest.json`,
the curated set the instance-ai builder agent grep over while building. The rubric —
not the resulting list — is the durable artifact: re-run `pnpm regenerate-examples`
against a fresh catalog to refresh picks. See `scripts/criteria.ts` for the
implementation; tune weights there.
## What "helpful" means
A template is helpful if it (a) reflects a real-world pattern the agent will be
asked to build, (b) is structurally clear enough to reason about, and (c) adds
something the rest of the set doesn't.
## Mechanical viability gate (drop on fail)
Hard filters applied before scoring. Anything failing here is dropped, regardless
of other signals.
| Filter | Rule | Stage |
|---|---|---|
| Published | `status == 'published'` | detail |
| Free | `purchaseUrl == null` AND (`price` is null/undefined OR `price <= 0`) | catalog |
| Size upper bound | `nodes.length <= 40` (list count is sparse) | catalog |
| Size full range | `workflow.nodes.length` in `[3, 40]` | detail |
| Verified author | `user.verified == true` | catalog |
| Has trigger | At least one node whose type identifies as a trigger | detail |
Verified-author and node-count filters drop ~30% of the catalog combined; the
trade-off is quality density over breadth.
## Scoring dimensions
Six dimensions sum to a relative total. Weights are numbers in `criteria.ts`; the
manifest entry records each dimension's score so picks stay reviewable.
| Dimension | Weight | Signal |
|---|---|---|
| **Real-world traction** | 20 | `log10(totalViews + 1)` + `log10(recentViews + 1)` |
| **Recency** | 20 | linear decay from 1.0 (≤90d since `updatedAt`) to 0.0 (≥2y) |
| **Pattern coverage (marginal)** | 35 | `1.0 / (1 + countInBucket(running_set))` — bucket = `(triggerType, primaryIntegration, hasAI, controlFlowKind)`; recomputed as each pick is accepted |
| **AI-agent relevance** | 0 | Folded into bucket key (`hasAI`); no extra weight. Diversity bucketing alone delivers ~50% AI representation. |
| **Structural clarity** | 15 | `+0.4` if median node has a non-default name (not `Edit Fields1`, `HTTP Request2`); `+0.3` if has ≥1 sticky note; `+0.3` if has ≥3 distinct node types |
| **Pedagogical density** | 5 | `min(1, distinctNodeTypes / nodeCount)` — favours patterns over repetition |
### Why coverage dominates (35)
Pattern coverage is the heaviest weight. The builder agent needs to grep
across many distinct shapes (trigger types, integration mixes, control flow
patterns), not 100 copies of "schedule → openAi → slack." Each pick's coverage
score recomputes after the running set updates, so popular-but-redundant
candidates lose ground.
### Why AI bias is 0
The catalog is already ~53% AI-rich. The bucket key includes `hasAI`, so
round-robin selection naturally gives AI workflows ~50% of slots. Adding an
explicit AI bonus on top would over-fit. If telemetry later shows the agent
under-uses AI templates, lift this weight.
### Why traction is `log10(views) + log10(recentViews)`
Views distribution is wildly skewed (median 1, max 780k). Log-scale compresses
the long tail into a usable range. Adding recent views (last 30d) catches
"currently popular" alongside "historically popular."
## Diversity bucket
`bucketKey(detail) = (triggerType, primaryIntegration, hasAI, controlFlowKind)`
- `triggerType` ∈ `{webhook, schedule, chatTrigger, formTrigger, manual, telegram, gmail, other}`
- `primaryIntegration` — vendor prefix of the most-frequent non-trigger node (e.g. `googleSheets`, `slack`, `openAi`, `telegram`)
- `hasAI` ∈ `{true, false}` — any `@n8n/n8n-nodes-langchain.*` node OR any `openAi`/`anthropic` chat model
- `controlFlowKind` ∈ `{linear, branching, loop, parallel}` — derived from connections: branching = ifElse/switch present, loop = splitInBatches present, parallel = ≥1 node with multiple downstream connections, else linear
Selection is a rescoring loop: each round picks the candidate with the highest
total after recomputing scores against the running set. The coverage term
(`1 / (1 + countInBucket)`) biases toward underrepresented buckets without
enforcing strict round-robin.
## Calibration
`pnpm criteria:calibrate` runs the rubric against `examples/_calibration.json`
(20–30 hand-tagged workflows) and reports:
- Spearman correlation between rubric rank and expert verdict
- Top disagreements with explanations
We tune weights until correlation ≥ 0.7 on the calibration set.
## Coverage check
`pnpm criteria:coverage` checks the candidate set against real builder prompts
in `packages/@n8n/instance-ai/evaluations/data/workflows/`. Coverage = % of eval
prompts where at least one template matches ≥2 keywords. Phase 1 ships at ≥70%.

245
packages/@n8n/workflow-sdk/examples/_calibration.json Normal file
View File

@ -0,0 +1,245 @@
{
"instructions": "For each entry, set verdict to \"helpful\" | \"borderline\" | \"not-helpful\" and add a one-line rationale. This becomes the calibration set.",
"expert_tagged": [
{
"id": 6270,
"rank": 1,
"verdict": "TBD",
"rationale": "",
"name": "Build Your First AI Agent",
"triggerType": "chatTrigger",
"hasAI": true,
"nodeCount": 7,
"score": 95,
"source": "https://n8n.io/workflows/6270"
},
{
"id": 8237,
"rank": 2,
"verdict": "TBD",
"rationale": "",
"name": "Personal Life Manager with Telegram, Google Services & Voice-Enabled AI",
"triggerType": "telegram",
"hasAI": true,
"nodeCount": 12,
"score": 90.95,
"source": "https://n8n.io/workflows/8237"
},
{
"id": 2753,
"rank": 3,
"verdict": "TBD",
"rationale": "",
"name": "RAG Chatbot for Company Documents using Google Drive and Gemini",
"triggerType": "other",
"hasAI": true,
"nodeCount": 12,
"score": 90.35,
"source": "https://n8n.io/workflows/2753"
},
{
"id": 4846,
"rank": 4,
"verdict": "TBD",
"rationale": "",
"name": "Generate AI Videos with Google Veo3, Save to Google Drive and Upload to YouTube",
"triggerType": "manual",
"hasAI": true,
"nodeCount": 10,
"score": 90.19,
"source": "https://n8n.io/workflows/4846"
},
{
"id": 4352,
"rank": 5,
"verdict": "TBD",
"rationale": "",
"name": "AI-Powered Multi-Social Media Post Automation: Google Trends & Perplexity AI ",
"triggerType": "schedule",
"hasAI": true,
"nodeCount": 12,
"score": 89.9,
"source": "https://n8n.io/workflows/4352"
},
{
"id": 5148,
"rank": 6,
"verdict": "TBD",
"rationale": "",
"name": "Local Chatbot with Retrieval Augmented Generation (RAG)",
"triggerType": "formTrigger",
"hasAI": true,
"nodeCount": 10,
"score": 89.7,
"source": "https://n8n.io/workflows/5148"
},
{
"id": 4966,
"rank": 7,
"verdict": "TBD",
"rationale": "",
"name": "Customer Support WhatsApp Bot with Google Docs Knowledge Base and Gemini AI",
"triggerType": "other",
"hasAI": true,
"nodeCount": 12,
"score": 89.7,
"source": "https://n8n.io/workflows/4966"
},
{
"id": 5626,
"rank": 8,
"verdict": "TBD",
"rationale": "",
"name": "Free AI Image Generator - n8n Automation Workflow with Gemini/ChatGPT",
"triggerType": "chatTrigger",
"hasAI": true,
"nodeCount": 10,
"score": 89.61,
"source": "https://n8n.io/workflows/5626"
},
{
"id": 5338,
"rank": 9,
"verdict": "TBD",
"rationale": "",
"name": "Generate AI Viral Videos with Seedance and Upload to TikTok, YouTube & Instagram",
"triggerType": "schedule",
"hasAI": true,
"nodeCount": 12,
"score": 89.35,
"source": "https://n8n.io/workflows/5338"
},
{
"id": 4827,
"rank": 10,
"verdict": "TBD",
"rationale": "",
"name": "AI-Powered WhatsApp Chatbot for Text, Voice, Images, and PDF with RAG",
"triggerType": "manual",
"hasAI": true,
"nodeCount": 18,
"score": 89.26,
"source": "https://n8n.io/workflows/4827"
},
{
"id": 5110,
"rank": 11,
"verdict": "TBD",
"rationale": "",
"name": "Create & Upload AI-Generated ASMR YouTube Shorts with Seedance, Fal AI, and GPT-4",
"triggerType": "schedule",
"hasAI": true,
"nodeCount": 13,
"score": 88.91,
"source": "https://n8n.io/workflows/5110"
},
{
"id": 5385,
"rank": 12,
"verdict": "TBD",
"rationale": "",
"name": "Lead Generation System: Google Maps to Email Scraper with Google Sheets Export",
"triggerType": "manual",
"hasAI": false,
"nodeCount": 11,
"score": 88.89,
"source": "https://n8n.io/workflows/5385"
},
{
"id": 5678,
"rank": 13,
"verdict": "TBD",
"rationale": "",
"name": "Automate Email Filtering & AI Summarization. 100% free & effective, works 7/24 ",
"triggerType": "gmail",
"hasAI": true,
"nodeCount": 7,
"score": 88.39,
"source": "https://n8n.io/workflows/5678"
},
{
"id": 2860,
"rank": 14,
"verdict": "TBD",
"rationale": "",
"name": "AI Automated HR Workflow for CV Analysis and Candidate Evaluation",
"triggerType": "formTrigger",
"hasAI": true,
"nodeCount": 12,
"score": 88.38,
"source": "https://n8n.io/workflows/2860"
},
{
"id": 5962,
"rank": 15,
"verdict": "TBD",
"rationale": "",
"name": "Track SEO Keyword Rankings with Bright Data MCP and GPT-4o AI Analysis",
"triggerType": "schedule",
"hasAI": true,
"nodeCount": 10,
"score": 88.29,
"source": "https://n8n.io/workflows/5962"
},
{
"id": 10000,
"rank": 16,
"verdict": "TBD",
"rationale": "",
"name": "Auto-Create TikTok Videos with VEED.io AI Avatars, ElevenLabs & GPT-4",
"triggerType": "telegram",
"hasAI": true,
"nodeCount": 12,
"score": 87.92,
"source": "https://n8n.io/workflows/10000"
},
{
"id": 3586,
"rank": 17,
"verdict": "TBD",
"rationale": "",
"name": "AI-Powered WhatsApp Chatbot 🤖📲 for Text, Voice, Images & PDFs with memory 🧠",
"triggerType": "other",
"hasAI": true,
"nodeCount": 13,
"score": 87.87,
"source": "https://n8n.io/workflows/3586"
},
{
"id": 13270,
"rank": 18,
"verdict": "TBD",
"rationale": "",
"name": "Use skills In n8n agent node",
"triggerType": "chatTrigger",
"hasAI": true,
"nodeCount": 12,
"score": 87.62,
"source": "https://n8n.io/workflows/13270"
},
{
"id": 4484,
"rank": 19,
"verdict": "TBD",
"rationale": "",
"name": "Build a Voice AI Chatbot with ElevenLabs and InfraNodus Knowledge Experts",
"triggerType": "webhook",
"hasAI": true,
"nodeCount": 8,
"score": 87.53,
"source": "https://n8n.io/workflows/4484"
},
{
"id": 2846,
"rank": 20,
"verdict": "TBD",
"rationale": "",
"name": "AI Voice Chatbot with ElevenLabs & OpenAI for Customer Service and Restaurants",
"triggerType": "manual",
"hasAI": true,
"nodeCount": 14,
"score": 87.44,
"source": "https://n8n.io/workflows/2846"
}
]
}

809
packages/@n8n/workflow-sdk/examples/_coverage-report.json Normal file
View File

@ -0,0 +1,809 @@
{
"generatedAt": "2026-05-07T13:26:57.543Z",
"total_prompts": 14,
"covered": 14,
"uncovered": 0,
"coverage": 1,
"target": 0.7,
"passed": true,
"results": [
{
"prompt_file": "airtable-split-to-slack.json",
"prompt": "Every hour, fetch all records from an Airtable table. Use the HTTP Request node to call GET https://api.airtable.com/v0/app123abc/Tasks with a Bearer token auth header — Airtable responds with a JSON ",
"keywords": [
"hour",
"records",
"airtable",
"table",
"app123abc",
"tasks",
"bearer",
"token",
"header",
"responds",
"json",
"object",
"shape",
"where",
"record",
"post",
"slack",
"channel",
"daily-tasks",
"containing",
"task",
"status",
"later",
"build",
"schedule",
"split"
],
"matched": true,
"top_match": {
"slug": "automate-3d-body-model-generation-from-images-using-sam-3d-g-11460",
"matches": 10,
"matched_keywords": [
"tasks",
"header",
"json",
"shape",
"where",
"post",
"channel",
"task",
"status",
"schedule"
]
}
},
{
"prompt_file": "contact-form-automation.json",
"prompt": "Create a workflow that handles contact form submissions via a webhook. It should send an auto-reply email to the person who submitted the form, notify my team on Telegram, and log each submission to G",
"keywords": [
"handles",
"contact",
"form",
"webhook",
"auto-reply",
"email",
"person",
"who",
"notify",
"team",
"telegram",
"log",
"google",
"sheets",
"documentid",
"1bximvs0xra5nfmdkvbdbzjgmuuqptlbs74ogve2upms",
"sheet",
"later",
"build",
"gmail",
"multi",
"action"
],
"matched": true,
"top_match": {
"slug": "lead-generation-system-google-maps-to-email-scraper-with-goo-5385",
"matches": 12,
"matched_keywords": [
"handles",
"contact",
"form",
"email",
"team",
"log",
"google",
"sheets",
"sheet",
"build",
"multi",
"action"
]
}
},
{
"prompt_file": "cross-team-linear-report.json",
"prompt": "Get all the Linear issues created in the last 2 weeks. Filter them for issues created for a different team than the one the creator is in. I have this team mapping to use: Alice (alice@company.com) be",
"keywords": [
"linear",
"issues",
"created",
"last",
"weeks",
"filter",
"different",
"team",
"than",
"creator",
"mapping",
"alice",
"company",
"belongs",
"both",
"frontend",
"bob",
"backend",
"carol",
"store",
"note",
"person",
"belong",
"multiple",
"teams",
"cross-team",
"issue",
"only",
"not",
"list",
"calculate",
"number",
"tickets",
"per",
"post",
"ordered",
"descending",
"slack",
"channel",
"called",
"cross-team-reports",
"later",
"build",
"schedule",
"processing"
],
"matched": true,
"top_match": {
"slug": "extract-invoice-data-from-email-to-google-sheets-using-gpt-4-4376",
"matches": 19,
"matched_keywords": [
"linear",
"issues",
"filter",
"different",
"team",
"company",
"note",
"multiple",
"teams",
"issue",
"only",
"not",
"list",
"calculate",
"number",
"per",
"channel",
"schedule",
"processing"
]
}
},
{
"prompt_file": "daily-slack-summary.json",
"prompt": "Every day, get the posts made in the past day on 3 different Slack channels: #general (C04GENERAL01), #engineering (C04ENGINEER1), and #product (C04PRODUCT01). Summarize them using AI, and post the su",
"keywords": [
"day",
"posts",
"made",
"past",
"different",
"slack",
"channels",
"general",
"c04general01",
"engineering",
"c04engineer1",
"product",
"c04product01",
"summarize",
"post",
"summary",
"daily-digest",
"c04dailydg01",
"later",
"build",
"schedule"
],
"matched": true,
"top_match": {
"slug": "ai-telegram-bot-agent-smart-assistant-content-summarizer-4457",
"matches": 9,
"matched_keywords": [
"day",
"made",
"different",
"product",
"summarize",
"post",
"summary",
"build",
"schedule"
]
}
},
{
"prompt_file": "form-to-hubspot.json",
"prompt": "Create a form that collects: name, email, company, and interest level (dropdown: starter, professional, enterprise). When submitted, create a new contact in HubSpot with firstname, lastname (split fro",
"keywords": [
"form",
"collects",
"email",
"company",
"interest",
"level",
"dropdown",
"starter",
"professional",
"enterprise",
"new",
"contact",
"hubspot",
"firstname",
"lastname",
"split",
"custom",
"property",
"confirmation",
"sendgrid",
"address",
"subject",
"reaching",
"body",
"mention",
"later",
"build",
"trigger",
"crm"
],
"matched": true,
"top_match": {
"slug": "scrape-google-maps-business-leads-with-apify-gpt-4-email-ext-10640",
"matches": 13,
"matched_keywords": [
"form",
"email",
"company",
"professional",
"contact",
"hubspot",
"split",
"custom",
"confirmation",
"address",
"build",
"trigger",
"crm"
]
}
},
{
"prompt_file": "github-notion-sync.json",
"prompt": "Every day, fetch all open GitHub issues from repository 'acme-corp/backend' that have the label 'bug'. For each issue, create a page in a Notion database (database ID: 'a1b2c3d4e5f6789012345678abcdef0",
"keywords": [
"day",
"open",
"github",
"issues",
"repository",
"acme-corp",
"backend",
"label",
"bug",
"issue",
"page",
"notion",
"database",
"a1b2c3d4e5f6789012345678abcdef01",
"properties",
"title",
"html",
"created",
"date",
"assignee",
"login",
"unassigned",
"status",
"directly",
"repos",
"labels",
"state",
"bearer",
"token",
"authorization",
"header",
"later",
"build",
"schedule",
"sync"
],
"matched": true,
"top_match": {
"slug": "extract-invoice-data-from-email-to-google-sheets-using-gpt-4-4376",
"matches": 11,
"matched_keywords": [
"day",
"open",
"issues",
"label",
"issue",
"page",
"database",
"date",
"labels",
"header",
"schedule"
]
}
},
{
"prompt_file": "linear-bq-leaderboard.json",
"prompt": "Every two weeks I want to check the amount of n8n usage and bug reporting that the team has done and produce a leaderboard that then gets posted to Slack (channel ID: D034WT7G4CW).\n\nHere are the users",
"keywords": [
"weeks",
"want",
"check",
"amount",
"n8n",
"usage",
"bug",
"reporting",
"team",
"produce",
"leaderboard",
"posted",
"slack",
"channel",
"d034wt7g4cw",
"here",
"users",
"david",
"roberts",
"arens",
"niklas",
"hatje",
"example",
"last",
"jonathan",
"clift",
"tickets",
"execs",
"hours",
"fabian",
"puehringer",
"tuukka",
"kantola",
"linear",
"created",
"manual",
"registered",
"accounts",
"ordered",
"number",
"desc",
"bugs",
"user",
"reported",
"query",
"issues",
"any",
"label",
"case-sensitive",
"matched",
"connect",
"bigquery",
"something",
"similar",
"following",
"settings",
"select",
"timestamp",
"start",
"cutoff",
"end",
"unnest",
"struct",
"string",
"exec",
"trunc",
"hour",
"instance",
"status",
"rudder",
"schema",
"finished",
"inner",
"join",
"cross",
"where",
"between",
"union",
"summary",
"count",
"distinct",
"instances",
"group",
"later",
"build",
"schedule"
],
"matched": true,
"top_match": {
"slug": "extract-invoice-data-from-email-to-google-sheets-using-gpt-4-4376",
"matches": 29,
"matched_keywords": [
"check",
"amount",
"n8n",
"reporting",
"team",
"channel",
"hours",
"linear",
"manual",
"accounts",
"number",
"desc",
"issues",
"any",
"label",
"connect",
"similar",
"settings",
"select",
"timestamp",
"end",
"struct",
"hour",
"instance",
"join",
"cross",
"count",
"group",
"schedule"
]
}
},
{
"prompt_file": "notification-router.json",
"prompt": "Create a workflow that receives webhook notifications with a JSON body containing 'level' (high, medium, or low), 'title', and 'message'. Route them based on level: high priority goes to Microsoft Tea",
"keywords": [
"receives",
"webhook",
"notifications",
"json",
"body",
"containing",
"level",
"high",
"medium",
"low",
"title",
"route",
"based",
"priority",
"goes",
"microsoft",
"teams",
"team",
"9b4c3a2f-1d8e-4f5b-a6c7-8e9f0b1d2c3a",
"channel",
"a1b2c3d4e5f6",
"thread",
"tacv2",
"slack",
"gmail",
"alerts",
"ourcompany",
"notification",
"include",
"payload",
"later",
"build",
"switch",
"routing"
],
"matched": true,
"top_match": {
"slug": "extract-invoice-data-from-email-to-google-sheets-using-gpt-4-4376",
"matches": 13,
"matched_keywords": [
"json",
"body",
"level",
"high",
"medium",
"low",
"based",
"microsoft",
"teams",
"team",
"channel",
"gmail",
"include"
]
}
},
{
"prompt_file": "rest-api-data-pipeline.json",
"prompt": "Fetch the latest posts from the JSONPlaceholder API (GET https://jsonplaceholder.typicode.com/posts). Filter out any posts where the title contains the word 'qui'. Then post a summary message to a Sla",
"keywords": [
"latest",
"posts",
"jsonplaceholder",
"typicode",
"filter",
"any",
"where",
"title",
"contains",
"word",
"qui",
"post",
"summary",
"slack",
"channel",
"called",
"api-digest",
"says",
"many",
"remain",
"lists",
"titles",
"later",
"build",
"transformation",
"schedule"
],
"matched": true,
"top_match": {
"slug": "scrape-linkedin-job-listings-for-hiring-signals-prospecting--3580",
"matches": 10,
"matched_keywords": [
"posts",
"filter",
"any",
"title",
"word",
"qui",
"post",
"lists",
"titles",
"build"
]
}
},
{
"prompt_file": "set-edit-fields-contract.json",
"prompt": "Every day, fetch one post from the JSONPlaceholder API (GET https://jsonplaceholder.typicode.com/posts/1). Then use an Edit Fields (Set) node, not a Code node, to add a field called caption from the p",
"keywords": [
"day",
"post",
"jsonplaceholder",
"typicode",
"posts",
"edit",
"not",
"code",
"called",
"caption",
"title",
"source",
"while",
"preserving",
"original",
"later",
"build",
"schedule",
"transformation"
],
"matched": true,
"top_match": {
"slug": "scrape-linkedin-job-listings-for-hiring-signals-prospecting--3580",
"matches": 8,
"matched_keywords": ["day", "post", "posts", "edit", "not", "code", "title", "build"]
}
},
{
"prompt_file": "telegram-chatbot-memory-session.json",
"prompt": "Build a Telegram chatbot workflow for a family assistant. It should receive Telegram messages, answer with an AI Agent using an OpenAI chat model, keep short-term conversation memory scoped separately",
"keywords": [
"build",
"telegram",
"chatbot",
"family",
"assistant",
"receive",
"answer",
"agent",
"openai",
"chat",
"model",
"keep",
"short-term",
"conversation",
"memory",
"scoped",
"separately",
"back",
"same",
"later",
"expressions"
],
"matched": true,
"top_match": {
"slug": "build-a-voice-ai-chatbot-with-elevenlabs-and-infranodus-know-4484",
"matches": 13,
"matched_keywords": [
"build",
"telegram",
"chatbot",
"receive",
"answer",
"agent",
"openai",
"chat",
"keep",
"conversation",
"memory",
"back",
"same"
]
}
},
{
"prompt_file": "weather-alert.json",
"prompt": "Every day at 8am, check the weather in Berlin using the OpenMeteo API and send me an email to david@thedavid.co.uk using the gmail node if it's going to rain",
"keywords": [
"day",
"8am",
"check",
"weather",
"berlin",
"openmeteo",
"email",
"david",
"thedavid",
"gmail",
"going",
"rain",
"build",
"schedule",
"conditional"
],
"matched": true,
"top_match": {
"slug": "extract-invoice-data-from-email-to-google-sheets-using-gpt-4-4376",
"matches": 7,
"matched_keywords": ["day", "check", "email", "gmail", "going", "rain", "schedule"]
}
},
{
"prompt_file": "weather-monitoring.json",
"prompt": "Every hour, check the current weather for London, New York, and Tokyo using the OpenWeatherMap API. Use 3 separate HTTP Request nodes, one per city. If any city has a temperature above 30°C, send a Te",
"keywords": [
"hour",
"check",
"current",
"weather",
"london",
"new",
"york",
"tokyo",
"openweathermap",
"separate",
"per",
"city",
"any",
"temperature",
"above",
"telegram",
"alert",
"chat",
"-1001234567890",
"listing",
"hot",
"cities",
"log",
"readings",
"airtable",
"table",
"base",
"appk2xgfgnoirl2gt",
"tbl8xk3np5mq7rs9w",
"columns",
"humidity",
"timestamp",
"later",
"build",
"schedule",
"conditional",
"multi"
],
"matched": true,
"top_match": {
"slug": "extract-invoice-data-from-email-to-google-sheets-using-gpt-4-4376",
"matches": 14,
"matched_keywords": [
"hour",
"check",
"current",
"new",
"per",
"city",
"any",
"chat",
"log",
"table",
"base",
"timestamp",
"schedule",
"multi"
]
}
},
{
"prompt_file": "workflow-data-table.json",
"prompt": "I want you to build a workflow that will read n8n workflow databases and extract certain information and then populate that information in a data table called 'workflows'.\n\nThe schema of the data tabl",
"keywords": [
"want",
"you",
"build",
"read",
"n8n",
"databases",
"extract",
"certain",
"information",
"populate",
"table",
"called",
"schema",
"follows",
"instanceid",
"workflowid",
"workflowname",
"tags",
"run",
"multiple",
"times",
"update",
"current",
"rows",
"rather",
"than",
"creating",
"dupes",
"instance",
"wonderman",
"users",
"cloud",
"later",
"schedule"
],
"matched": true,
"top_match": {
"slug": "extract-invoice-data-from-email-to-google-sheets-using-gpt-4-4376",
"matches": 15,
"matched_keywords": [
"you",
"read",
"n8n",
"extract",
"certain",
"information",
"populate",
"table",
"multiple",
"times",
"update",
"current",
"instance",
"cloud",
"schedule"
]
}
}
]
}

3491
packages/@n8n/workflow-sdk/examples/manifest.json Normal file
View File

File diff suppressed because one or more lines are too long

BIN
packages/@n8n/workflow-sdk/examples/templates.zip Normal file
View File

Binary file not shown.

5
packages/@n8n/workflow-sdk/jest.config.js
View File

@ -1,2 +1,5 @@
/** @type {import('jest').Config} */
module.exports = require('../../../jest.config');
module.exports = {
...require('../../../jest.config'),
globalSetup: '<rootDir>/scripts/jest-global-setup.ts',
};

19
packages/@n8n/workflow-sdk/package.json
View File

@ -37,6 +37,10 @@
"./prompts/node-selection": {
"types": "./dist/prompts/node-selection/index.d.ts",
"default": "./dist/prompts/node-selection/index.js"
},
"./examples-loader": {
"types": "./dist/examples-loader.d.ts",
"default": "./dist/examples-loader.js"
}
},
"scripts": {
@ -49,7 +53,6 @@
"lint": "eslint . --quiet",
"lint:fix": "eslint . --fix",
"watch": "tsc -p tsconfig.build.json --watch",
"pretest": "npx tsx scripts/extract-workflows.ts",
"extract-workflows": "npx tsx scripts/extract-workflows.ts",
"test": "jest",
"test:unit": "jest",
@ -57,8 +60,13 @@
"generate-types": "npx tsx src/generate-types/generate-types.ts",
"fetch-workflows": "npx tsx scripts/fetch-test-workflows.ts",
"create-workflows-zip": "npx tsx scripts/create-workflows-zip.ts",
"create-examples-zip": "npx tsx scripts/create-examples-zip.ts",
"json-to-code": "npx tsx src/cli/index.ts json-to-code",
"code-to-json": "npx tsx src/cli/index.ts code-to-json"
"code-to-json": "npx tsx src/cli/index.ts code-to-json",
"fetch-templates": "npx tsx scripts/fetch-templates.ts",
"regenerate-examples": "npx tsx scripts/regenerate-examples.ts",
"criteria:calibrate": "npx tsx scripts/calibration.ts",
"criteria:coverage": "npx tsx scripts/coverage.ts"
},
"main": "dist/index.js",
"module": "src/index.ts",
@ -79,11 +87,16 @@
],
"prompts/node-selection": [
"./dist/prompts/node-selection/index.d.ts"
],
"examples-loader": [
"./dist/examples-loader.d.ts"
]
}
},
"files": [
"dist/**/*"
"dist/**/*",
"examples/manifest.json",
"examples/templates.zip"
],
"devDependencies": {
"@n8n/eslint-plugin-community-nodes": "workspace:*",

179
packages/@n8n/workflow-sdk/scripts/calibration.ts Normal file
View File

@ -0,0 +1,179 @@
/**
* Calibration runner.
*
* Loads `examples/_calibration.json` (hand-tagged expert verdicts), runs the
* rubric over each entry, and reports:
* - Spearman rank correlation between rubric scores and expert verdicts
* - Top disagreements with explanations
*
* Usage: pnpm criteria:calibrate
*
* The calibration file is a manually maintained JSON. Tagging convention:
* { id: 1954, verdict: 'helpful' | 'borderline' | 'not-helpful', rationale: '...' }
* One entry per workflow id. 20-30 entries is the right size to anchor the
* rubric without overfitting.
*/
import * as fs from 'fs';
import * as path from 'path';
import { scoreDetailedTemplate, type ScoreResult } from './criteria';
import { fetchDetail, loadCachedCatalog, type CatalogEntry } from './fetch-templates';
const CALIBRATION_PATH = path.resolve(__dirname, '../examples/_calibration.json');
const VERDICT_TO_RANK: Record<string, number> = {
helpful: 2,
borderline: 1,
'not-helpful': 0,
};
interface CalibrationEntry {
id: number;
// 'TBD' is a placeholder for unlabelled entries; scoring skips them.
verdict: 'helpful' | 'borderline' | 'not-helpful' | 'TBD' | string;
rationale?: string;
}
interface CalibrationFile {
expert_tagged: CalibrationEntry[];
}
interface ScoredCalibrationEntry extends CalibrationEntry {
score: number;
breakdown: ScoreResult['breakdown'];
name: string;
}
function loadCalibration(): CalibrationFile {
if (!fs.existsSync(CALIBRATION_PATH)) {
throw new Error(
`Calibration file not found at ${CALIBRATION_PATH}. ` +
'Create it with shape { "expert_tagged": [{"id": N, "verdict": "helpful", "rationale": "..."}] }',
);
}
return JSON.parse(fs.readFileSync(CALIBRATION_PATH, 'utf-8')) as CalibrationFile;
}
/** Spearman rank correlation between two parallel arrays. Higher = stronger agreement. */
export function spearmanCorrelation(xs: number[], ys: number[]): number {
if (xs.length !== ys.length) throw new Error('arrays must have equal length');
if (xs.length < 2) return 0;
const xRanks = ranksWithTies(xs);
const yRanks = ranksWithTies(ys);
const n = xs.length;
const meanX = xRanks.reduce((a, b) => a + b, 0) / n;
const meanY = yRanks.reduce((a, b) => a + b, 0) / n;
let num = 0;
let denX = 0;
let denY = 0;
for (let i = 0; i < n; i++) {
const dx = xRanks[i] - meanX;
const dy = yRanks[i] - meanY;
num += dx * dy;
denX += dx * dx;
denY += dy * dy;
}
if (denX === 0 || denY === 0) return 0;
return num / Math.sqrt(denX * denY);
}
/** Average rank for ties (standard for Spearman). */
function ranksWithTies(values: number[]): number[] {
const indexed = values.map((v, i) => ({ v, i }));
indexed.sort((a, b) => a.v - b.v);
const ranks = new Array<number>(values.length);
let i = 0;
while (i < indexed.length) {
let j = i;
while (j + 1 < indexed.length && indexed[j + 1].v === indexed[i].v) j++;
const avgRank = (i + j) / 2 + 1;
for (let k = i; k <= j; k++) ranks[indexed[k].i] = avgRank;
i = j + 1;
}
return ranks;
}
async function scoreCalibration(
calib: CalibrationFile,
catalog: CatalogEntry[],
): Promise<ScoredCalibrationEntry[]> {
const byId = new Map(catalog.map((c) => [c.id, c]));
const out: ScoredCalibrationEntry[] = [];
for (const entry of calib.expert_tagged) {
if (!Object.hasOwn(VERDICT_TO_RANK, entry.verdict)) {
console.warn(` skipping ${entry.id}: verdict not yet labelled (got "${entry.verdict}")`);
continue;
}
const cat = byId.get(entry.id);
if (!cat) {
console.warn(` skipping ${entry.id}: not in cached catalog`);
continue;
}
const detail = await fetchDetail(entry.id);
if (!detail) {
console.warn(` skipping ${entry.id}: detail fetch failed`);
continue;
}
const result = scoreDetailedTemplate(cat, detail, []);
out.push({
...entry,
score: result.total,
breakdown: result.breakdown,
name: detail.data.attributes.name,
});
}
return out;
}
function printTopDisagreements(scored: ScoredCalibrationEntry[]): void {
const expertRanks = ranksWithTies(scored.map((s) => VERDICT_TO_RANK[s.verdict]));
const rubricRanks = ranksWithTies(scored.map((s) => s.score));
const disagreements = scored.map((s, i) => ({
entry: s,
expertRank: expertRanks[i],
rubricRank: rubricRanks[i],
gap: Math.abs(expertRanks[i] - rubricRanks[i]),
}));
disagreements.sort((a, b) => b.gap - a.gap);
console.log('\nTop disagreements (rubric rank vs expert rank):');
for (const d of disagreements.slice(0, 10)) {
const direction = d.rubricRank > d.expertRank ? 'overrated' : 'underrated';
console.log(
` ${direction.padEnd(10)} gap=${d.gap.toFixed(1).padStart(4)} | ` +
`rubric=${d.rubricRank.toFixed(0).padStart(3)} expert=${d.expertRank.toFixed(0).padStart(3)} | ` +
`verdict=${d.entry.verdict.padEnd(11)} | ${d.entry.id} ${d.entry.name.slice(0, 50)}`,
);
if (d.entry.rationale) console.log(` expert: "${d.entry.rationale}"`);
}
}
async function main() {
const calib = loadCalibration();
console.log(`Loaded ${calib.expert_tagged.length} calibration entries\n`);
const catalog = loadCachedCatalog();
const scored = await scoreCalibration(calib, catalog);
if (scored.length < 2) {
console.error('Not enough scored entries to compute correlation');
process.exit(1);
}
const expertScores = scored.map((s) => VERDICT_TO_RANK[s.verdict]);
const rubricScores = scored.map((s) => s.score);
const rho = spearmanCorrelation(expertScores, rubricScores);
console.log(`Calibration set size: ${scored.length}`);
console.log(`Spearman rank correlation: ${rho.toFixed(3)}`);
console.log(`Threshold (Phase 1 ships at): 0.700`);
console.log(`Status: ${rho >= 0.7 ? 'PASS' : 'TUNE WEIGHTS'}`);
printTopDisagreements(scored);
}
if (require.main === module) {
main().catch((error) => {
console.error(error);
process.exit(1);
});
}

64
packages/@n8n/workflow-sdk/scripts/coverage.test.ts Normal file
View File

@ -0,0 +1,64 @@
import { extractKeywords, matchPrompt } from './coverage';
describe('coverage helpers', () => {
describe('extractKeywords', () => {
it('lowercases, splits on whitespace, drops stopwords and short tokens', () => {
const out = extractKeywords('Create a workflow that posts to Slack');
expect(out).toContain('posts');
expect(out).toContain('slack');
expect(out).not.toContain('the');
expect(out).not.toContain('a');
expect(out).not.toContain('to');
});
it('mixes in tag tokens', () => {
const out = extractKeywords('handle webhook submissions', ['google-sheets', 'multi-action']);
expect(out).toContain('webhook');
expect(out).toContain('google');
expect(out).toContain('sheets');
expect(out).toContain('multi');
expect(out).toContain('action');
});
it('dedupes repeated tokens', () => {
const out = extractKeywords('slack slack slack notify slack');
expect(out.filter((t) => t === 'slack').length).toBe(1);
});
it('removes punctuation', () => {
const out = extractKeywords('Notify the team in #general about it!');
expect(out).toContain('notify');
expect(out).toContain('team');
expect(out).toContain('general');
});
});
describe('matchPrompt', () => {
const wf = {
id: 1,
slug: 'gmail-to-slack',
name: 'Gmail to Slack notifier',
description: 'Forwards new Gmail messages to a Slack channel',
nodes: ['n8n-nodes-base.gmailTrigger', 'n8n-nodes-base.slack'],
tags: ['trigger:gmail', 'integration:slack'],
triggerType: 'gmail',
hasAI: false,
};
it('counts substring matches across name + description + nodes + tags', () => {
const result = matchPrompt(['gmail', 'slack', 'notify'], wf);
expect(result.matches).toBe(2); // gmail, slack
expect(result.matchedKeywords).toEqual(expect.arrayContaining(['gmail', 'slack']));
});
it('returns 0 matches when no keywords overlap', () => {
const result = matchPrompt(['airtable', 'discord'], wf);
expect(result.matches).toBe(0);
});
it('matches case-insensitively', () => {
const result = matchPrompt(['GMAIL', 'SLACK'], wf);
expect(result.matches).toBe(2);
});
});
});

324
packages/@n8n/workflow-sdk/scripts/coverage.ts Normal file
View File

@ -0,0 +1,324 @@
/**
* Coverage check: do our curated templates cover the patterns real users ask for?
*
* Uses the existing instance-ai eval prompt corpus as a held-out test set.
* For each eval prompt, we extract a small keyword bag (filtered tokens from
* the prompt + the prompt's structural tags) and look for any manifest entry
* whose name+description+nodes+tags blob matches ≥2 keywords. Coverage = %
* of prompts with at least one match.
*
* This is intentionally a smoke test — keyword overlap will miss semantic
* matches and over-credit name overlap. The real signal lives in Phase 3
* telemetry on `Builder template read`. Coverage just stops us shipping a
* curated set with obvious gaps.
*
* Usage: pnpm criteria:coverage
*/
import * as fs from 'fs';
import * as path from 'path';
const MANIFEST_PATH = path.resolve(__dirname, '../examples/manifest.json');
const EVAL_DIR = path.resolve(__dirname, '../../instance-ai/evaluations/data/workflows');
const REPORT_PATH = path.resolve(__dirname, '../examples/_coverage-report.json');
const COVERAGE_TARGET = 0.7;
const MIN_KEYWORD_MATCHES = 3;
/**
* Words too generic to be useful matches. We want integration names, action
* verbs specific to a domain, and trigger types — not "send", "http", "node",
* "data", which appear in nearly every workflow.
*/
const STOPWORDS = new Set([
// Articles, conjunctions, prepositions
'a',
'an',
'and',
'are',
'as',
'at',
'be',
'but',
'by',
'for',
'from',
'has',
'have',
'i',
'if',
'in',
'is',
'it',
'its',
'me',
'my',
'of',
'on',
'or',
'so',
'the',
'their',
'them',
'this',
'that',
'to',
'was',
'we',
'were',
'will',
'with',
'into',
'over',
'under',
'they',
'one',
'two',
'three',
'been',
'being',
// Generic shape verbs
'create',
'use',
'used',
'using',
'should',
'can',
'configure',
'set',
'setup',
'send',
'sent',
'sends',
'sending',
'fetch',
'fetches',
'fetching',
'do',
'done',
'add',
'adds',
'adding',
'get',
'got',
'gets',
'getting',
'make',
'makes',
// Workflow-shape generic
'workflow',
'workflows',
'node',
'nodes',
'data',
'request',
'response',
'message',
'messages',
'item',
'items',
'value',
'values',
'field',
'fields',
'result',
'results',
'name',
'names',
'http',
'https',
'api',
'url',
'auth',
'options',
'parameters',
'credentials',
'credential',
'output',
'input',
'call',
'calls',
'com',
'org',
'net',
// Submission/contact tokens (too generic)
'submit',
'submitted',
'submission',
'submissions',
'submits',
'complete',
'completely',
'possible',
'help',
'please',
'thanks',
'don',
'all',
'every',
'each',
'when',
'then',
'how',
'what',
'about',
'via',
'out',
'ask',
// Single chars from contractions
't',
'm',
's',
're',
've',
'll',
'd',
]);
interface ManifestWorkflow {
id: number;
slug: string;
name: string;
description: string;
nodes: string[];
tags: string[];
triggerType: string;
hasAI: boolean;
}
interface ManifestFile {
workflows: ManifestWorkflow[];
}
interface EvalPrompt {
prompt: string;
complexity?: string;
tags?: string[];
triggerType?: string;
}
interface PromptCoverageResult {
prompt_file: string;
prompt: string;
keywords: string[];
matched: boolean;
top_match: { slug: string; matches: number; matched_keywords: string[] } | null;
}
export function extractKeywords(text: string, extraTags: string[] = []): string[] {
const tokens = text
.toLowerCase()
.replace(/[^a-z0-9\s-]/g, ' ')
.split(/\s+/)
.filter((t) => t.length >= 3 && !STOPWORDS.has(t));
const tagTokens = extraTags
.flatMap((t) => t.toLowerCase().split(/[-\s]+/))
.filter((t) => t.length >= 3 && !STOPWORDS.has(t));
const dedup = new Set<string>([...tokens, ...tagTokens]);
return Array.from(dedup);
}
export function matchPrompt(
keywords: string[],
workflow: ManifestWorkflow,
): { matches: number; matchedKeywords: string[] } {
const haystack = [
workflow.name,
workflow.description,
workflow.nodes.join(' '),
workflow.tags.join(' '),
workflow.triggerType,
]
.join(' ')
.toLowerCase();
const matched: string[] = [];
for (const k of keywords) {
if (haystack.includes(k.toLowerCase())) matched.push(k);
}
return { matches: matched.length, matchedKeywords: matched };
}
function loadEvalPrompts(): Array<{ filename: string; data: EvalPrompt }> {
if (!fs.existsSync(EVAL_DIR)) {
throw new Error(`Eval directory not found at ${EVAL_DIR}`);
}
const files = fs.readdirSync(EVAL_DIR).filter((f) => f.endsWith('.json'));
return files.map((f) => ({
filename: f,
data: JSON.parse(fs.readFileSync(path.join(EVAL_DIR, f), 'utf-8')) as EvalPrompt,
}));
}
function loadManifest(): ManifestFile {
if (!fs.existsSync(MANIFEST_PATH)) {
throw new Error(
`Manifest not found at ${MANIFEST_PATH}. Run \`pnpm regenerate-examples\` first.`,
);
}
return JSON.parse(fs.readFileSync(MANIFEST_PATH, 'utf-8')) as ManifestFile;
}
function main() {
const manifest = loadManifest();
const prompts = loadEvalPrompts();
const results: PromptCoverageResult[] = [];
for (const { filename, data } of prompts) {
const keywords = extractKeywords(data.prompt, data.tags ?? []);
let bestMatch: { slug: string; matches: number; matched_keywords: string[] } | null = null;
for (const wf of manifest.workflows) {
const m = matchPrompt(keywords, wf);
if (m.matches >= MIN_KEYWORD_MATCHES) {
if (!bestMatch || m.matches > bestMatch.matches) {
bestMatch = { slug: wf.slug, matches: m.matches, matched_keywords: m.matchedKeywords };
}
}
}
results.push({
prompt_file: filename,
prompt: data.prompt.slice(0, 200),
keywords,
matched: bestMatch !== null,
top_match: bestMatch,
});
}
const covered = results.filter((r) => r.matched).length;
const ratio = covered / results.length;
const report = {
generatedAt: new Date().toISOString(),
total_prompts: results.length,
covered,
uncovered: results.length - covered,
coverage: Number(ratio.toFixed(3)),
target: COVERAGE_TARGET,
passed: ratio >= COVERAGE_TARGET,
results,
};
fs.writeFileSync(REPORT_PATH, JSON.stringify(report, null, 2));
console.log(
`Coverage: ${covered}/${results.length} (${(ratio * 100).toFixed(0)}%) — target ${(COVERAGE_TARGET * 100).toFixed(0)}%`,
);
console.log(`Status: ${ratio >= COVERAGE_TARGET ? 'PASS' : 'TUNE'}`);
console.log(`Wrote ${path.relative(process.cwd(), REPORT_PATH)}\n`);
const uncovered = results.filter((r) => !r.matched);
if (uncovered.length > 0) {
console.log('Uncovered prompts:');
for (const r of uncovered) {
console.log(` - ${r.prompt_file}: "${r.prompt.slice(0, 80)}..."`);
}
}
}
if (require.main === module) {
try {
main();
} catch (error) {
console.error(error);
process.exit(1);
}
}

40
packages/@n8n/workflow-sdk/scripts/create-examples-zip.ts Normal file
View File

@ -0,0 +1,40 @@
/**
* Packages all workflow JSON files from `examples/workflows/` into a single
* `examples/templates.zip` for committing. The manifest stays as a separate
* committed file — it is the source of truth and not zipped.
*
* Usage: pnpm create-examples-zip
*/
import AdmZip from 'adm-zip';
import * as fs from 'fs';
import * as path from 'path';
const EXAMPLES_DIR = path.resolve(__dirname, '..', 'examples');
const WORKFLOWS_DIR = path.join(EXAMPLES_DIR, 'workflows');
const ZIP_PATH = path.join(EXAMPLES_DIR, 'templates.zip');
function createExamplesZip(): void {
if (!fs.existsSync(WORKFLOWS_DIR)) {
console.error(`Error: workflows dir not found at ${WORKFLOWS_DIR}`);
console.error('Run `pnpm regenerate-examples` first.');
process.exit(1);
}
const zip = new AdmZip();
const files = fs.readdirSync(WORKFLOWS_DIR).filter((f) => f.endsWith('.json'));
if (files.length === 0) {
console.error(`Error: no workflow JSON files found in ${WORKFLOWS_DIR}`);
process.exit(1);
}
for (const file of files) {
zip.addLocalFile(path.join(WORKFLOWS_DIR, file));
}
zip.writeZip(ZIP_PATH);
console.log(`Created: ${ZIP_PATH}`);
console.log(`Contents: ${files.length} workflows`);
}
createExamplesZip();

452
packages/@n8n/workflow-sdk/scripts/criteria.test.ts Normal file
View File

@ -0,0 +1,452 @@
import {
mechanicalGateCatalog,
mechanicalGateDetail,
tractionScore,
recencyScore,
coverageScore,
clarityScore,
densityScore,
bucketKey,
bucketKeyToString,
scoreCatalogEntry,
scoreDetailedTemplate,
hasAI,
WEIGHTS,
NODE_COUNT_MIN,
NODE_COUNT_MAX,
__test,
type BucketKey,
} from './criteria';
import type { CatalogEntry, DetailResponse } from './fetch-templates';
// ---------------------------------------------------------------------------
// Fixture builders
// ---------------------------------------------------------------------------
function makeCatalog(overrides: Partial<CatalogEntry> = {}): CatalogEntry {
return {
id: 1,
name: 'Test Workflow',
totalViews: 1000,
createdAt: new Date().toISOString(),
user: { username: 'someone', verified: true },
purchaseUrl: null,
nodes: Array.from({ length: 5 }, (_, i) => ({ name: `node-${i}` })),
...overrides,
};
}
function makeDetail(
opts: {
nodes?: Array<Record<string, unknown>>;
connections?: Record<string, unknown>;
views?: number;
recentViews?: number;
updatedAt?: string;
status?: string;
} = {},
): DetailResponse {
return {
data: {
id: 1,
attributes: {
name: 'Detail',
description: '',
workflow: {
nodes: opts.nodes ?? [
{ id: 't', name: 'Schedule Trigger', type: 'n8n-nodes-base.scheduleTrigger' },
{ id: 'a', name: 'My Slack Post', type: 'n8n-nodes-base.slack' },
{ id: 'b', name: 'Format Output', type: 'n8n-nodes-base.set' },
],
connections: opts.connections ?? {},
},
createdAt: '2024-01-01T00:00:00Z',
updatedAt: opts.updatedAt ?? new Date().toISOString(),
views: opts.views ?? 1000,
recentViews: opts.recentViews ?? 5,
hidden: false,
username: 'someone',
status: opts.status ?? 'published',
price: null,
difficulty: null,
readyToDemo: null,
},
},
meta: {},
};
}
// ---------------------------------------------------------------------------
// Mechanical gate
// ---------------------------------------------------------------------------
describe('mechanicalGateCatalog', () => {
it('passes a typical free verified workflow', () => {
expect(mechanicalGateCatalog(makeCatalog())).toEqual({ ok: true });
});
it('rejects paid via purchaseUrl', () => {
const r = mechanicalGateCatalog(makeCatalog({ purchaseUrl: 'https://example.com/buy' }));
expect(r.ok).toBe(false);
});
it('rejects paid via positive price', () => {
const r = mechanicalGateCatalog(makeCatalog({ price: 5 }));
expect(r.ok).toBe(false);
});
it('rejects unverified author', () => {
const r = mechanicalGateCatalog(makeCatalog({ user: { username: 'rando', verified: false } }));
expect(r.ok).toBe(false);
});
it('rejects oversized workflow at the catalog stage', () => {
const big = makeCatalog({
nodes: Array.from({ length: NODE_COUNT_MAX + 5 }, (_, i) => ({ name: `n${i}` })),
});
const r = mechanicalGateCatalog(big);
expect(r.ok).toBe(false);
});
it('does NOT enforce lower bound at catalog stage (list nodes are sparse)', () => {
const small = makeCatalog({ nodes: [{ name: 'x' }] });
expect(mechanicalGateCatalog(small)).toEqual({ ok: true });
});
});
describe('mechanicalGateDetail', () => {
it('passes a typical workflow with trigger', () => {
expect(mechanicalGateDetail(makeDetail())).toEqual({ ok: true });
});
it('rejects unpublished status', () => {
const r = mechanicalGateDetail(makeDetail({ status: 'draft' }));
expect(r.ok).toBe(false);
});
it('rejects too-small node count', () => {
const r = mechanicalGateDetail(
makeDetail({
nodes: Array.from({ length: NODE_COUNT_MIN - 1 }, (_, i) => ({
id: String(i),
type: 'n8n-nodes-base.set',
})),
}),
);
expect(r.ok).toBe(false);
});
it('rejects too-large node count', () => {
const r = mechanicalGateDetail(
makeDetail({
nodes: Array.from({ length: NODE_COUNT_MAX + 1 }, (_, i) => ({
id: String(i),
type: 'n8n-nodes-base.set',
})),
}),
);
expect(r.ok).toBe(false);
});
it('rejects when no trigger present', () => {
const r = mechanicalGateDetail(
makeDetail({
nodes: [
{ id: '1', type: 'n8n-nodes-base.set' },
{ id: '2', type: 'n8n-nodes-base.slack' },
{ id: '3', type: 'n8n-nodes-base.gmail' },
],
}),
);
expect(r.ok).toBe(false);
if (!r.ok) expect(r.reason).toContain('trigger');
});
});
// ---------------------------------------------------------------------------
// Per-dimension scorers
// ---------------------------------------------------------------------------
describe('tractionScore', () => {
it('is monotonic in views', () => {
expect(tractionScore(10, 0)).toBeLessThan(tractionScore(1000, 0));
expect(tractionScore(1000, 0)).toBeLessThan(tractionScore(100000, 0));
});
it('is bounded in [0, 1]', () => {
expect(tractionScore(0, 0)).toBe(0);
expect(tractionScore(10_000_000, 10_000_000)).toBeLessThanOrEqual(1);
});
});
describe('recencyScore', () => {
const now = new Date('2026-05-07T00:00:00Z').getTime();
it('is 1.0 for fresh workflows (≤90d)', () => {
const updatedAt = new Date(now - 30 * 24 * 3600 * 1000).toISOString();
expect(recencyScore(updatedAt, now)).toBe(1);
});
it('is 0 for workflows older than 2y', () => {
const updatedAt = new Date(now - 1000 * 24 * 3600 * 1000).toISOString();
expect(recencyScore(updatedAt, now)).toBe(0);
});
it('decays linearly between 90d and 2y', () => {
const oneYearAgo = new Date(now - 365 * 24 * 3600 * 1000).toISOString();
const score = recencyScore(oneYearAgo, now);
expect(score).toBeGreaterThan(0);
expect(score).toBeLessThan(1);
});
});
describe('coverageScore', () => {
const bucket: BucketKey = {
triggerType: 'webhook',
primaryIntegration: 'slack',
hasAI: false,
controlFlowKind: 'linear',
};
it('is 1.0 for an empty running set', () => {
expect(coverageScore(bucket, [])).toBe(1);
});
it('halves with each duplicate bucket', () => {
expect(coverageScore(bucket, [bucket])).toBeCloseTo(0.5);
expect(coverageScore(bucket, [bucket, bucket])).toBeCloseTo(1 / 3);
});
it('is unaffected by other buckets', () => {
const other: BucketKey = { ...bucket, triggerType: 'schedule' };
expect(coverageScore(bucket, [other, other, other])).toBe(1);
});
});
describe('clarityScore', () => {
it('rewards named nodes, sticky notes, and distinct types', () => {
const detail = makeDetail({
nodes: [
{ id: 'sticky', type: 'n8n-nodes-base.stickyNote', name: 'Doc' },
{ id: 't', type: 'n8n-nodes-base.scheduleTrigger', name: 'Daily 9am' },
{ id: 'a', type: 'n8n-nodes-base.slack', name: 'Post update' },
{ id: 'b', type: 'n8n-nodes-base.gmail', name: 'Send digest' },
],
});
// All real nodes named, sticky present, ≥3 distinct types → 0.4 + 0.3 + 0.3 = 1.0
expect(clarityScore(detail)).toBeCloseTo(1.0);
});
it('penalises default-named workflows', () => {
const detail = makeDetail({
nodes: [
{ id: 't', type: 'n8n-nodes-base.scheduleTrigger', name: 'Schedule Trigger' },
{ id: 's1', type: 'n8n-nodes-base.set', name: 'Edit Fields' },
{ id: 's2', type: 'n8n-nodes-base.set', name: 'Edit Fields1' },
],
});
// All default-named, no sticky, only 2 distinct types
expect(clarityScore(detail)).toBeLessThan(0.4);
});
});
describe('densityScore', () => {
it('is 1.0 when every node is a different type', () => {
const detail = makeDetail({
nodes: [
{ id: 't', type: 'n8n-nodes-base.scheduleTrigger' },
{ id: 'a', type: 'n8n-nodes-base.slack' },
{ id: 'b', type: 'n8n-nodes-base.gmail' },
],
});
expect(densityScore(detail)).toBe(1);
});
it('is low when many nodes share types', () => {
const detail = makeDetail({
nodes: [
{ id: 't', type: 'n8n-nodes-base.scheduleTrigger' },
{ id: '1', type: 'n8n-nodes-base.set' },
{ id: '2', type: 'n8n-nodes-base.set' },
{ id: '3', type: 'n8n-nodes-base.set' },
],
});
// 2 distinct / 4 nodes = 0.5
expect(densityScore(detail)).toBeCloseTo(0.5);
});
});
// ---------------------------------------------------------------------------
// Bucket key
// ---------------------------------------------------------------------------
describe('bucketKey', () => {
it('classifies a webhook → slack workflow without AI', () => {
const detail = makeDetail({
nodes: [
{ id: 't', type: 'n8n-nodes-base.webhook' },
{ id: 'a', type: 'n8n-nodes-base.slack' },
{ id: 'b', type: 'n8n-nodes-base.set' },
],
});
const key = bucketKey(detail);
expect(key.triggerType).toBe('webhook');
expect(key.primaryIntegration).toBe('slack');
expect(key.hasAI).toBe(false);
expect(key.controlFlowKind).toBe('linear');
});
it('classifies AI agent workflows', () => {
const detail = makeDetail({
nodes: [
{ id: 't', type: '@n8n/n8n-nodes-langchain.chatTrigger' },
{ id: 'm', type: '@n8n/n8n-nodes-langchain.lmChatOpenAi' },
{ id: 'a', type: '@n8n/n8n-nodes-langchain.agent' },
],
});
const key = bucketKey(detail);
expect(key.triggerType).toBe('chatTrigger');
expect(key.hasAI).toBe(true);
});
it('detects branching control flow', () => {
const detail = makeDetail({
nodes: [
{ id: 't', type: 'n8n-nodes-base.scheduleTrigger' },
{ id: 'i', type: 'n8n-nodes-base.if' },
{ id: 'a', type: 'n8n-nodes-base.slack' },
{ id: 'b', type: 'n8n-nodes-base.gmail' },
],
connections: {},
});
expect(bucketKey(detail).controlFlowKind).toBe('branching');
});
it('detects loop control flow', () => {
const detail = makeDetail({
nodes: [
{ id: 't', type: 'n8n-nodes-base.scheduleTrigger' },
{ id: 'l', type: 'n8n-nodes-base.splitInBatches' },
{ id: 'a', type: 'n8n-nodes-base.slack' },
],
});
expect(bucketKey(detail).controlFlowKind).toBe('loop');
});
it('detects parallel control flow from connections fan-out', () => {
const detail = makeDetail({
nodes: [
{ id: 't', type: 'n8n-nodes-base.scheduleTrigger', name: 'Trigger' },
{ id: 'a', type: 'n8n-nodes-base.slack', name: 'A' },
{ id: 'b', type: 'n8n-nodes-base.gmail', name: 'B' },
],
connections: {
Trigger: {
main: [
[
{ node: 'A', type: 'main', index: 0 },
{ node: 'B', type: 'main', index: 0 },
],
],
},
},
});
expect(bucketKey(detail).controlFlowKind).toBe('parallel');
});
});
describe('bucketKeyToString', () => {
it('produces stable keys', () => {
const a: BucketKey = {
triggerType: 'webhook',
primaryIntegration: 'slack',
hasAI: false,
controlFlowKind: 'linear',
};
expect(bucketKeyToString(a)).toBe('webhook|slack|noai|linear');
});
});
// ---------------------------------------------------------------------------
// Composition
// ---------------------------------------------------------------------------
describe('scoreCatalogEntry', () => {
it('uses traction + recency only (other dims zero)', () => {
const entry = makeCatalog({ totalViews: 10000, createdAt: new Date().toISOString() });
const result = scoreCatalogEntry(entry);
expect(result.breakdown.coverage).toBe(0);
expect(result.breakdown.clarity).toBe(0);
expect(result.breakdown.density).toBe(0);
expect(result.total).toBeGreaterThan(0);
});
});
describe('scoreDetailedTemplate', () => {
it('combines all weighted dimensions', () => {
const entry = makeCatalog();
const detail = makeDetail();
const result = scoreDetailedTemplate(entry, detail, []);
expect(result.breakdown.traction).toBeGreaterThan(0);
expect(result.breakdown.recency).toBeGreaterThan(0);
expect(result.breakdown.coverage).toBe(1); // empty running set
expect(result.total).toBeGreaterThan(0);
});
it('drops coverage as duplicates accumulate', () => {
const entry = makeCatalog();
const detail = makeDetail();
const bucket = bucketKey(detail);
const empty = scoreDetailedTemplate(entry, detail, []);
const oneDup = scoreDetailedTemplate(entry, detail, [bucket]);
expect(empty.total).toBeGreaterThan(oneDup.total);
});
it('aiAgent dimension contributes 0 to total because weight is 0', () => {
const entry = makeCatalog();
const aiDetail = makeDetail({
nodes: [
{ id: 't', type: '@n8n/n8n-nodes-langchain.chatTrigger' },
{ id: 'a', type: '@n8n/n8n-nodes-langchain.agent' },
{ id: 'm', type: '@n8n/n8n-nodes-langchain.memoryBufferWindow' },
{ id: 'l', type: '@n8n/n8n-nodes-langchain.lmChatOpenAi' },
],
});
const noAiDetail = makeDetail();
const aiResult = scoreDetailedTemplate(entry, aiDetail, []);
const noAiResult = scoreDetailedTemplate(entry, noAiDetail, []);
// AI signal is computed but the weight is 0 for now
expect(aiResult.breakdown.aiAgent).toBeGreaterThan(0);
expect(noAiResult.breakdown.aiAgent).toBe(0);
expect(WEIGHTS.aiAgent).toBe(0);
});
});
// ---------------------------------------------------------------------------
// Tagged internals (compact tests, low-risk)
// ---------------------------------------------------------------------------
describe('internal helpers', () => {
it('vendorOf strips type prefix', () => {
expect(__test.vendorOf('n8n-nodes-base.googleSheets')).toBe('googleSheets');
expect(__test.vendorOf('@n8n/n8n-nodes-langchain.openAi')).toBe('langchain');
});
it('isDefaultName flags repeated default-style names', () => {
expect(__test.isDefaultName({ name: 'Edit Fields', type: 'n8n-nodes-base.set' })).toBe(true);
expect(__test.isDefaultName({ name: 'Edit Fields1', type: 'n8n-nodes-base.set' })).toBe(true);
expect(__test.isDefaultName({ name: 'My Slack Post', type: 'n8n-nodes-base.slack' })).toBe(
false,
);
});
it('hasAI is true for any langchain-prefixed node', () => {
const detail = makeDetail({
nodes: [
{ id: 't', type: 'n8n-nodes-base.scheduleTrigger' },
{ id: 'a', type: '@n8n/n8n-nodes-langchain.agent' },
],
});
expect(hasAI(detail)).toBe(true);
});
});

398
packages/@n8n/workflow-sdk/scripts/criteria.ts Normal file
View File

@ -0,0 +1,398 @@
/**
* Helpfulness rubric for template curation.
*
* Tunable knobs are at the top of the file. Documentation lives in
* `docs/template-criteria.md`. The score breakdown is preserved per template
* so manifest entries are reviewable: anyone can see why a template ranked
* where it did.
*
* Two scoring stages:
* - `scoreCatalogEntry()` — runs against cheap list metadata, used to pick
* the top-K candidates worth fetching detail for.
* - `scoreDetailedTemplate()` — runs against full detail JSON, used to
* pick the final manifest set with full diversity bucketing.
*/
import type { CatalogEntry, DetailResponse } from './fetch-templates';
// ---------------------------------------------------------------------------
// Tunable weights
// ---------------------------------------------------------------------------
export const WEIGHTS = {
traction: 20,
recency: 20,
coverage: 35,
aiAgent: 0, // folded into bucket key; lift if telemetry shows under-use
clarity: 15,
density: 5,
} as const;
export const NODE_COUNT_MIN = 3;
export const NODE_COUNT_MAX = 40;
export const RECENCY_FRESH_DAYS = 90;
export const RECENCY_STALE_DAYS = 730; // 2 years
// ---------------------------------------------------------------------------
// Types
// ---------------------------------------------------------------------------
export type GateResult = { ok: true } | { ok: false; reason: string };
export interface RubricBreakdown {
traction: number;
recency: number;
coverage: number;
aiAgent: number;
clarity: number;
density: number;
}
export interface ScoreResult {
total: number;
breakdown: RubricBreakdown;
}
export type TriggerType =
| 'webhook'
| 'schedule'
| 'chatTrigger'
| 'formTrigger'
| 'manual'
| 'telegram'
| 'gmail'
| 'other';
export type ControlFlowKind = 'linear' | 'branching' | 'loop' | 'parallel';
export interface BucketKey {
triggerType: TriggerType;
primaryIntegration: string;
hasAI: boolean;
controlFlowKind: ControlFlowKind;
}
// ---------------------------------------------------------------------------
// Mechanical gate
// ---------------------------------------------------------------------------
/** Catalog-stage gate. List-only metadata. */
export function mechanicalGateCatalog(entry: CatalogEntry): GateResult {
if (entry.purchaseUrl !== null) return { ok: false, reason: 'paid (purchaseUrl set)' };
if (entry.price !== null && entry.price !== undefined && entry.price > 0) {
return { ok: false, reason: `paid (price=${entry.price})` };
}
if (!entry.user?.verified) return { ok: false, reason: 'unverified author' };
const listNodeCount = entry.nodes?.length ?? 0;
// List nodes is sparse; only enforce upper bound here. Lower bound rechecked at detail.
if (listNodeCount > NODE_COUNT_MAX) {
return { ok: false, reason: `too large (list reports ${listNodeCount} nodes)` };
}
return { ok: true };
}
/** Detail-stage gate. Real workflow JSON. */
export function mechanicalGateDetail(detail: DetailResponse): GateResult {
const attrs = detail.data.attributes;
if (attrs.status !== 'published') return { ok: false, reason: `status=${attrs.status}` };
const realNodeCount = attrs.workflow.nodes?.length ?? 0;
if (realNodeCount < NODE_COUNT_MIN || realNodeCount > NODE_COUNT_MAX) {
return {
ok: false,
reason: `nodeCount ${realNodeCount} outside [${NODE_COUNT_MIN}, ${NODE_COUNT_MAX}]`,
};
}
if (findTriggerNode(detail) === null) {
return { ok: false, reason: 'no trigger node' };
}
return { ok: true };
}
// ---------------------------------------------------------------------------
// Per-dimension scorers
// ---------------------------------------------------------------------------
/** Traction: log-scaled views; mixes total + recent so freshness matters too. */
export function tractionScore(totalViews: number, recentViews: number): number {
const total = Math.log10(Math.max(0, totalViews) + 1);
const recent = Math.log10(Math.max(0, recentViews) + 1);
// Normalise to ~[0, 1]: log10(1M) = 6, log10(1K) = 3 → cap divisor at 8
return Math.min(1, (total + recent) / 8);
}
/** Recency: 1.0 if updated within N days, 0.0 once stale, linear in between. */
export function recencyScore(updatedAt: string, now: number = Date.now()): number {
const ageMs = now - new Date(updatedAt).getTime();
const ageDays = ageMs / (24 * 3600 * 1000);
if (ageDays <= RECENCY_FRESH_DAYS) return 1;
if (ageDays >= RECENCY_STALE_DAYS) return 0;
return 1 - (ageDays - RECENCY_FRESH_DAYS) / (RECENCY_STALE_DAYS - RECENCY_FRESH_DAYS);
}
/** Marginal coverage: how rare is this template's bucket in the running set? */
export function coverageScore(bucket: BucketKey, runningSet: BucketKey[]): number {
const key = bucketKeyToString(bucket);
const count = runningSet.reduce(
(n, existing) => (bucketKeyToString(existing) === key ? n + 1 : n),
0,
);
return 1 / (1 + count);
}
/** AI-agent presence (currently weighted 0; signal still computed for telemetry). */
export function aiAgentScore(detail: DetailResponse): number {
if (!hasAI(detail)) return 0;
if (hasSubnodePattern(detail)) return 1;
return 0.6;
}
/** Structural clarity: signals of authoring care. */
export function clarityScore(detail: DetailResponse): number {
const nodes = detail.data.attributes.workflow.nodes ?? [];
let score = 0;
const realNodes = nodes.filter((n) => !isStickyNote(n));
const namedRatio =
realNodes.length === 0
? 0
: realNodes.filter((n) => !isDefaultName(n)).length / realNodes.length;
if (namedRatio >= 0.5) score += 0.4;
const stickyCount = nodes.filter(isStickyNote).length;
if (stickyCount >= 1) score += 0.3;
const distinctTypes = new Set(realNodes.map((n) => String(n.type ?? ''))).size;
if (distinctTypes >= 3) score += 0.3;
return score;
}
/** Pedagogical density: distinct node types relative to total. */
export function densityScore(detail: DetailResponse): number {
const nodes = detail.data.attributes.workflow.nodes ?? [];
const realNodes = nodes.filter((n) => !isStickyNote(n));
if (realNodes.length === 0) return 0;
const distinctTypes = new Set(realNodes.map((n) => String(n.type ?? ''))).size;
return Math.min(1, distinctTypes / realNodes.length);
}
// ---------------------------------------------------------------------------
// Public scorers (compose per-dimension scores into totals)
// ---------------------------------------------------------------------------
/** Catalog-stage score. Does not include coverage, clarity, density (need detail). */
export function scoreCatalogEntry(entry: CatalogEntry): ScoreResult {
const traction = tractionScore(entry.totalViews ?? 0, 0); // recentViews unavailable here
// List endpoint only returns createdAt; detail stage rescores with updatedAt.
const recency = recencyScore(entry.createdAt);
const breakdown: RubricBreakdown = {
traction,
recency,
coverage: 0,
aiAgent: 0,
clarity: 0,
density: 0,
};
const total = WEIGHTS.traction * traction + WEIGHTS.recency * recency;
return { total, breakdown };
}
/** Detail-stage score. Full rubric. `runningSet` lets coverage update as picks accumulate. */
export function scoreDetailedTemplate(
entry: CatalogEntry,
detail: DetailResponse,
runningSet: BucketKey[],
): ScoreResult {
const attrs = detail.data.attributes;
const traction = tractionScore(attrs.views ?? 0, attrs.recentViews ?? 0);
const recency = recencyScore(attrs.updatedAt);
const bucket = bucketKey(detail);
const coverage = coverageScore(bucket, runningSet);
const aiAgent = aiAgentScore(detail);
const clarity = clarityScore(detail);
const density = densityScore(detail);
const breakdown: RubricBreakdown = { traction, recency, coverage, aiAgent, clarity, density };
const total =
WEIGHTS.traction * traction +
WEIGHTS.recency * recency +
WEIGHTS.coverage * coverage +
WEIGHTS.aiAgent * aiAgent +
WEIGHTS.clarity * clarity +
WEIGHTS.density * density;
return { total, breakdown };
}
// ---------------------------------------------------------------------------
// Bucket key derivation
// ---------------------------------------------------------------------------
export function bucketKey(detail: DetailResponse): BucketKey {
return {
triggerType: classifyTrigger(detail),
primaryIntegration: classifyIntegration(detail),
hasAI: hasAI(detail),
controlFlowKind: classifyControlFlow(detail),
};
}
export function bucketKeyToString(b: BucketKey): string {
return `${b.triggerType}|${b.primaryIntegration}|${b.hasAI ? 'ai' : 'noai'}|${b.controlFlowKind}`;
}
function classifyTrigger(detail: DetailResponse): TriggerType {
const node = findTriggerNode(detail);
if (!node) return 'other';
const type = String(node.type ?? '');
if (type.includes('chatTrigger')) return 'chatTrigger';
if (type.includes('webhook')) return 'webhook';
if (type.includes('scheduleTrigger') || type.includes('cron')) return 'schedule';
if (type.includes('formTrigger')) return 'formTrigger';
if (type.includes('manualTrigger')) return 'manual';
if (type.includes('telegram')) return 'telegram';
if (type.includes('gmail')) return 'gmail';
return 'other';
}
function classifyIntegration(detail: DetailResponse): string {
const counts = new Map<string, number>();
for (const node of detail.data.attributes.workflow.nodes ?? []) {
const t = String(node.type ?? '');
if (!t) continue;
// Skip triggers and meta nodes from primary integration calc
if (isTriggerType(t) || isMetaType(t)) continue;
const integ = vendorOf(t);
counts.set(integ, (counts.get(integ) ?? 0) + 1);
}
if (counts.size === 0) return 'none';
const sorted = Array.from(counts.entries()).sort((a, b) => b[1] - a[1]);
return sorted[0][0];
}
function classifyControlFlow(detail: DetailResponse): ControlFlowKind {
const nodes = detail.data.attributes.workflow.nodes ?? [];
const types = nodes.map((n) => String(n.type ?? ''));
if (types.some((t) => t.includes('splitInBatches'))) return 'loop';
if (types.some((t) => t.includes('.if') || t.includes('.switch'))) return 'branching';
const connections = detail.data.attributes.workflow.connections ?? {};
for (const fromNode of Object.values(connections)) {
const main = (fromNode as { main?: unknown[][] }).main;
if (Array.isArray(main) && main.length > 1) return 'parallel';
if (Array.isArray(main) && main[0] && Array.isArray(main[0]) && main[0].length > 1) {
return 'parallel';
}
}
return 'linear';
}
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
function findTriggerNode(detail: DetailResponse): Record<string, unknown> | null {
for (const node of detail.data.attributes.workflow.nodes ?? []) {
const type = String(node.type ?? '');
if (isTriggerType(type)) return node;
}
return null;
}
function isTriggerType(type: string): boolean {
return (
type.includes('Trigger') ||
type.endsWith('.webhook') ||
type.endsWith('.cron') ||
type.endsWith('.manualTrigger')
);
}
function isMetaType(type: string): boolean {
return type.endsWith('.stickyNote') || type.endsWith('.noOp');
}
function isStickyNote(node: Record<string, unknown>): boolean {
return String(node.type ?? '').endsWith('.stickyNote');
}
/**
* Names n8n auto-assigns when a node is added. Numeric suffixes (`Edit Fields1`,
* `HTTP Request2`) are stripped before lookup. Curated; extend as new common
* defaults surface.
*/
const KNOWN_DEFAULT_NAMES = new Set([
'Edit Fields',
'HTTP Request',
'Schedule Trigger',
'Manual Trigger',
'Webhook',
'Form Trigger',
'Email Trigger (IMAP)',
'When chat message received',
'Sticky Note',
'AI Agent',
'Basic LLM Chain',
'OpenAI Chat Model',
'Simple Memory',
'Window Buffer Memory',
]);
/**
* Heuristic: is this node's name auto-generated by n8n rather than authored?
*
* Two flavours of default:
* - A single Title-Case word optionally followed by a digit: `Set`, `Code`,
* `Slack2`, `Gmail`. These are generated from the node's display name.
* - A known multi-word default in `KNOWN_DEFAULT_NAMES`.
*
* Custom names typically break out of these patterns by adding lowercase words
* (`Post daily update`), specific personalisation (`My Slack post`), or by
* being longer than the canonical default form.
*/
function isDefaultName(node: Record<string, unknown>): boolean {
const name = String(node.name ?? '').trim();
if (!name) return true;
const stripped = name.replace(/\d+$/, '').trim();
if (KNOWN_DEFAULT_NAMES.has(stripped)) return true;
// Single Title-Case word: 'Set', 'Slack', 'Gmail'
if (/^[A-Z][a-zA-Z]*$/.test(stripped)) return true;
return false;
}
function vendorOf(type: string): string {
// '@n8n/n8n-nodes-langchain.openAi' → 'langchain'
// 'n8n-nodes-base.googleSheets' → 'googleSheets'
if (type.startsWith('@n8n/n8n-nodes-langchain.')) return 'langchain';
const dot = type.lastIndexOf('.');
if (dot < 0) return type;
return type.slice(dot + 1);
}
export function hasAI(detail: DetailResponse): boolean {
for (const node of detail.data.attributes.workflow.nodes ?? []) {
const t = String(node.type ?? '');
if (t.startsWith('@n8n/n8n-nodes-langchain.')) return true;
if (t.includes('openAi') || t.includes('anthropic')) return true;
}
return false;
}
function hasSubnodePattern(detail: DetailResponse): boolean {
const types = (detail.data.attributes.workflow.nodes ?? []).map((n) => String(n.type ?? ''));
const hasAgent = types.some((t) => t.endsWith('.agent'));
const hasMemory = types.some((t) => t.includes('memory'));
const hasTool = types.some((t) => t.includes('tool'));
return hasAgent && (hasMemory || hasTool);
}
// ---------------------------------------------------------------------------
// Exports for tests
// ---------------------------------------------------------------------------
export const __test = {
classifyTrigger,
classifyIntegration,
classifyControlFlow,
findTriggerNode,
isDefaultName,
vendorOf,
};

55
packages/@n8n/workflow-sdk/scripts/extract-workflows.ts
View File

@ -1,8 +1,11 @@
/**
* Extract Test Workflows from Zip
* Extract Test Workflows from Zips
*
* Extracts workflow JSON files from the committed zip file for testing.
* This runs automatically via `pretest` hook before running tests.
* Extracts workflow JSON files from committed zip files for testing:
* - test-fixtures/real-workflows/public_published_templates.zip → test-fixtures/real-workflows/
* - examples/templates.zip → examples/workflows/ (used by examples-roundtrip test)
*
* Runs automatically via Jest's globalSetup (see scripts/jest-global-setup.ts).
*
* Usage:
* npx tsx scripts/extract-workflows.ts
@ -13,35 +16,51 @@ import * as path from 'path';
import AdmZip from 'adm-zip';
const FIXTURES_DIR = path.resolve(__dirname, '../test-fixtures/real-workflows');
const ZIP_FILE = path.join(FIXTURES_DIR, 'public_published_templates.zip');
const FIXTURES_ZIP = path.join(FIXTURES_DIR, 'public_published_templates.zip');
function main() {
if (!fs.existsSync(ZIP_FILE)) {
console.log('No zip file found, skipping extraction');
const EXAMPLES_DIR = path.resolve(__dirname, '../examples');
const EXAMPLES_ZIP = path.join(EXAMPLES_DIR, 'templates.zip');
const EXAMPLES_WORKFLOWS_DIR = path.join(EXAMPLES_DIR, 'workflows');
function extractZip(zipPath: string, outputDir: string, label: string) {
if (!fs.existsSync(zipPath)) {
console.log(`No ${label} zip found at ${zipPath}, skipping extraction`);
return;
}
// Check if extraction is needed by looking for any .json file
const existingFiles = fs.readdirSync(FIXTURES_DIR).filter((f) => f.endsWith('.json'));
fs.mkdirSync(outputDir, { recursive: true });
const existingFiles = fs.readdirSync(outputDir).filter((f) => f.endsWith('.json'));
if (existingFiles.length > 0) {
console.log(`Found ${existingFiles.length} workflow files, skipping extraction`);
console.log(`Found ${existingFiles.length} ${label} files, skipping extraction`);
return;
}
console.log('Extracting workflows from zip...');
const zip = new AdmZip(ZIP_FILE);
console.log(`Extracting ${label} from zip...`);
const zip = new AdmZip(zipPath);
const entries = zip.getEntries();
const resolvedOutputDir = path.resolve(outputDir);
let count = 0;
for (const entry of entries) {
if (entry.entryName.endsWith('.json')) {
const outputPath = path.join(FIXTURES_DIR, entry.entryName);
fs.writeFileSync(outputPath, entry.getData());
count++;
if (!entry.entryName.endsWith('.json')) continue;
const outputPath = path.resolve(outputDir, entry.entryName);
if (outputPath !== resolvedOutputDir && !outputPath.startsWith(resolvedOutputDir + path.sep)) {
console.warn(`Skipping out-of-bounds entry: ${entry.entryName}`);
continue;
}
fs.writeFileSync(outputPath, entry.getData());
count++;
}
console.log(`Extracted ${count} workflow files`);
console.log(`Extracted ${count} ${label} files`);
}
main();
export function extractAllWorkflows() {
extractZip(FIXTURES_ZIP, FIXTURES_DIR, 'test-fixtures workflows');
extractZip(EXAMPLES_ZIP, EXAMPLES_WORKFLOWS_DIR, 'examples workflows');
}
if (require.main === module) {
extractAllWorkflows();
}

230
packages/@n8n/workflow-sdk/scripts/fetch-templates.ts Normal file
View File

@ -0,0 +1,230 @@
/**
* Fetch n8n.io public template catalog for criteria-driven curation.
*
* Two passes:
* Pass 1 (list) – paginate the templates list endpoint, persist each page,
* concat into examples/_raw/_catalog.json. Cheap metadata only.
* Pass 2 (detail) – per-id fetch of the full workflow JSON. Called by
* regenerate-examples.ts only for candidates the rubric picks.
*
* Usage:
* pnpm fetch-templates # full catalog walk (pass 1)
* pnpm fetch-templates --rebuild # ignore cached pages, re-walk
*
* Both passes are resumable: existing files are skipped on rerun.
*/
import * as fs from 'fs';
import * as path from 'path';
const RAW_DIR = path.resolve(__dirname, '../examples/_raw');
const LIST_DIR = path.join(RAW_DIR, '_list');
const CATALOG_FILE = path.join(RAW_DIR, '_catalog.json');
const DETAIL_DIR = RAW_DIR;
// Search endpoint paginates correctly (api.n8n.io/api/templates/workflows ignores ?page=).
// Mirrors the working pattern in scripts/fetch-test-workflows.ts.
const LIST_URL = 'https://n8n.io/api/product-api/workflows/search';
const DETAIL_URL = 'https://api.n8n.io/api/workflows';
const ROWS_PER_PAGE = 200;
const RATE_LIMIT_MS = 200; // ~5 req/s
const MAX_RETRIES = 3;
const FETCH_TIMEOUT_MS = 30_000;
export interface CatalogEntry {
id: number;
name: string;
description?: string;
totalViews: number;
createdAt: string;
user: { username: string; verified: boolean };
price?: number | null;
purchaseUrl: string | null;
nodes: Array<{ name: string; group?: string; displayName?: string }>;
}
interface ListResponse {
totalWorkflows: number;
workflows: CatalogEntry[];
}
export interface DetailResponse {
data: {
id: number;
attributes: {
name: string;
description: string;
workflow: {
nodes: Array<Record<string, unknown>>;
connections: Record<string, unknown>;
meta?: Record<string, unknown>;
pinData?: Record<string, unknown>;
settings?: Record<string, unknown>;
};
workflowInfo?: {
nodeCount: number;
nodeTypes: Record<string, { count: number }>;
};
createdAt: string;
updatedAt: string;
views: number;
recentViews: number;
hidden: boolean;
username: string;
status: string;
price: number | null;
difficulty: string | null;
readyToDemo: boolean | null;
};
};
meta: Record<string, unknown>;
}
const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));
async function politeFetch<T>(url: string, attempt = 1): Promise<T | null> {
try {
const response = await fetch(url, { signal: AbortSignal.timeout(FETCH_TIMEOUT_MS) });
if (response.status === 429 || response.status >= 500) {
if (attempt > MAX_RETRIES) {
console.error(` Giving up on ${url} after ${MAX_RETRIES} retries (${response.status})`);
return null;
}
const backoff = 2 ** attempt * 1000;
console.log(
` ${response.status} from ${url}, backing off ${backoff}ms (attempt ${attempt})`,
);
await sleep(backoff);
return politeFetch<T>(url, attempt + 1);
}
if (!response.ok) {
console.error(` ${response.status} ${response.statusText}: ${url}`);
return null;
}
return (await response.json()) as T;
} catch (error) {
if (attempt > MAX_RETRIES) {
console.error(` Network error on ${url} after ${MAX_RETRIES} retries:`, error);
return null;
}
const backoff = 2 ** attempt * 1000;
console.log(` Network error on ${url}, backing off ${backoff}ms (attempt ${attempt})`);
await sleep(backoff);
return politeFetch<T>(url, attempt + 1);
}
}
function ensureDirs() {
for (const dir of [RAW_DIR, LIST_DIR]) {
if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
}
}
function pagePath(page: number) {
return path.join(LIST_DIR, `page-${String(page).padStart(4, '0')}.json`);
}
export async function fetchCatalog(opts: { rebuild?: boolean } = {}): Promise<CatalogEntry[]> {
ensureDirs();
console.log('Pass 1: walking template catalog...\n');
const allEntries: CatalogEntry[] = [];
let page = 1;
let totalExpected: number | undefined;
while (true) {
const cachePath = pagePath(page);
let response: ListResponse | null;
if (!opts.rebuild && fs.existsSync(cachePath)) {
response = JSON.parse(fs.readFileSync(cachePath, 'utf-8')) as ListResponse;
console.log(` Page ${page}: ${response.workflows.length} workflows (cached)`);
} else {
const url = `${LIST_URL}?page=${page}&rows=${ROWS_PER_PAGE}`;
response = await politeFetch<ListResponse>(url);
if (response === null) {
// Fail fast: writing a partial catalog would look valid on disk
// but silently drop later pages, masking the failure.
throw new Error(`Catalog walk failed at page ${page}; refusing to write partial catalog`);
}
fs.writeFileSync(cachePath, JSON.stringify(response, null, 2));
console.log(
` Page ${page}: ${response.workflows.length} workflows (fetched, total=${response.totalWorkflows})`,
);
await sleep(RATE_LIMIT_MS);
}
if (totalExpected === undefined) totalExpected = response.totalWorkflows;
if (response.workflows.length === 0) break;
allEntries.push(...response.workflows);
if (allEntries.length >= response.totalWorkflows) break;
page++;
}
const dedupedById = new Map<number, CatalogEntry>();
for (const entry of allEntries) dedupedById.set(entry.id, entry);
const catalog = Array.from(dedupedById.values());
fs.writeFileSync(CATALOG_FILE, JSON.stringify(catalog, null, 2));
console.log(
`\nWrote ${catalog.length} unique entries to ${path.relative(process.cwd(), CATALOG_FILE)} (server total=${totalExpected})\n`,
);
return catalog;
}
function detailPath(id: number) {
return path.join(DETAIL_DIR, `${id}.json`);
}
export async function fetchDetail(
id: number,
opts: { force?: boolean } = {},
): Promise<DetailResponse | null> {
ensureDirs();
const cachePath = detailPath(id);
if (!opts.force && fs.existsSync(cachePath)) {
try {
return JSON.parse(fs.readFileSync(cachePath, 'utf-8')) as DetailResponse;
} catch {
// fall through to refetch
}
}
const response = await politeFetch<DetailResponse>(`${DETAIL_URL}/${id}`);
if (response === null) return null;
fs.writeFileSync(cachePath, JSON.stringify(response, null, 2));
await sleep(RATE_LIMIT_MS);
return response;
}
export function loadCachedCatalog(): CatalogEntry[] {
if (!fs.existsSync(CATALOG_FILE)) {
throw new Error(`Catalog not found at ${CATALOG_FILE}. Run \`pnpm fetch-templates\` first.`);
}
return JSON.parse(fs.readFileSync(CATALOG_FILE, 'utf-8')) as CatalogEntry[];
}
export function loadCachedDetail(id: number): DetailResponse | null {
const p = detailPath(id);
if (!fs.existsSync(p)) return null;
try {
return JSON.parse(fs.readFileSync(p, 'utf-8')) as DetailResponse;
} catch {
return null;
}
}
async function main() {
const rebuild = process.argv.includes('--rebuild');
const catalog = await fetchCatalog({ rebuild });
console.log(`Done. Catalog has ${catalog.length} entries.`);
}
if (require.main === module) {
main().catch((error) => {
console.error(error);
process.exit(1);
});
}

10
packages/@n8n/workflow-sdk/scripts/jest-global-setup.ts Normal file
View File

@ -0,0 +1,10 @@
/**
* Jest globalSetup hook: extract committed zips into the expected on-disk
* layout before any test runs. We can't rely on npm/pnpm's pretest hook here
* because CI invokes `test:unit` (turbo task), not `test`.
*/
import { extractAllWorkflows } from './extract-workflows';
module.exports = () => {
extractAllWorkflows();
};

551
packages/@n8n/workflow-sdk/scripts/regenerate-examples.ts Normal file
View File

@ -0,0 +1,551 @@
/**
* Regenerate `examples/manifest.json`, `examples/workflows/*.json`, and
* `examples/templates.zip` from the cached public-template catalog.
*
* The goal is a small, diverse set of high-quality real workflows the builder
* agent can grep when constructing new ones. Diversity beats raw popularity:
* we'd rather have one good `webhook + Slack + AI + branching` example than
* ten of them.
*
* Stage 1 — Catalog gate + cheap score (`scoreCatalogEntry`)
* For every entry in the cached catalog, drop paid templates, unverified
* authors, and oversized workflows. Score survivors on `traction` (log-scaled
* views) and `recency` (linear decay from 90→730 days). By default the full
* survivor set is used; pass `--candidates=N` to cap the detail-fetch budget
* on a cold cache.
*
* Stage 2 — Detail fetch + full-rubric score (`scoreDetailedTemplate`)
* For each candidate, fetch its detail JSON (cached), re-gate on real node
* count + trigger presence, compute its bucket key
* `(triggerType, primaryIntegration, hasAI, controlFlowKind)`, and score on
* the full 6-dimension rubric (traction, recency, coverage, aiAgent, clarity,
* density). Weights live in `criteria.ts` — `coverage` is the dominant
* weight (35) because it drives bucket diversity.
*
* Stage 3 — Greedy round-robin pick
* Loop until `--target` (default 50) accepted: re-score every remaining
* candidate against the *current* `acceptedBuckets` (coverage decays once a
* bucket fills), pick the highest scorer, validate it round-trips through
* `generateWorkflowCode` + `emitInstanceAi`, accept on success. Validation
* failures are logged to `_failures.log` and the candidate is dropped.
*
* Stage 3b — Coverage patch for must-cover node types
* `MUST_COVER_NODE_TYPES` (Postgres + all langchain vector stores) must each
* appear somewhere in the manifest. For any missing type, first scan the
* 1000-candidate pool; if none has it, fall back to scanning the full
* catalog and fetching detail on demand for up to 25 ranked candidates.
* This is why the final count usually exceeds `--target`.
*
* Stage 4 — Write outputs
* Clear `examples/workflows/`, write one JSON per accepted candidate, write
* `manifest.json` sorted by score (with the per-dimension breakdown so picks
* are reviewable), write `_catalog-snapshot.json`, then pack the workflow
* JSONs into `examples/templates.zip`. Only the manifest and the zip are
* committed; the unpacked JSONs are gitignored and recreated on demand by
* `examples-loader`.
*
* Usage:
* pnpm regenerate-examples # default target 50
* pnpm regenerate-examples --target=100 # explicit target
* pnpm regenerate-examples --candidates=2000 # cap detail-fetch budget
*/
import AdmZip from 'adm-zip';
import * as fs from 'fs';
import * as path from 'path';
import { generateWorkflowCode, emitInstanceAi } from '../src/codegen';
import {
bucketKey,
bucketKeyToString,
mechanicalGateCatalog,
mechanicalGateDetail,
scoreCatalogEntry,
scoreDetailedTemplate,
type BucketKey,
type ScoreResult,
} from './criteria';
import {
fetchDetail,
loadCachedCatalog,
type CatalogEntry,
type DetailResponse,
} from './fetch-templates';
const EXAMPLES_DIR = path.resolve(__dirname, '../examples');
const WORKFLOWS_DIR = path.join(EXAMPLES_DIR, 'workflows');
const MANIFEST_PATH = path.join(EXAMPLES_DIR, 'manifest.json');
const ZIP_PATH = path.join(EXAMPLES_DIR, 'templates.zip');
const SNAPSHOT_PATH = path.join(EXAMPLES_DIR, '_catalog-snapshot.json');
const FAILURES_LOG = path.join(EXAMPLES_DIR, '_failures.log');
const DEFAULT_TARGET = 50;
// No cap by default — process the full catalog. Detail JSONs are cached on
// disk in `examples/_raw/`, so warm runs are cheap. Pass `--candidates=N` to
// bound the detail-fetch budget on a cold cache.
const DEFAULT_CANDIDATES = Infinity;
/**
* Popular catalog node types that must be represented in the manifest. After
* the main bucket-pick loop, any of these missing from the manifest gets a
* coverage patch: we force-include the highest-scoring eligible candidate
* containing that type. Extends the manifest beyond `target`.
*
* If a must-cover type's candidates didn't make the top-K catalog cohort, the
* patch step falls back to fetching detail for the type's highest-scoring
* catalog entries on demand.
*/
const MUST_COVER_NODE_TYPES = [
'n8n-nodes-base.postgres',
'@n8n/n8n-nodes-langchain.vectorStorePinecone',
'@n8n/n8n-nodes-langchain.vectorStoreSupabase',
'@n8n/n8n-nodes-langchain.vectorStoreQdrant',
'@n8n/n8n-nodes-langchain.vectorStoreInMemory',
'@n8n/n8n-nodes-langchain.vectorStorePGVector',
'@n8n/n8n-nodes-langchain.vectorStoreMongoDBAtlas',
'@n8n/n8n-nodes-langchain.vectorStoreWeaviate',
'@n8n/n8n-nodes-langchain.vectorStoreMilvus',
'@n8n/n8n-nodes-langchain.vectorStoreRedis',
] as const;
interface CliArgs {
target: number;
candidates: number;
}
interface ManifestEntry {
id: number;
slug: string;
name: string;
description: string;
nodes: string[];
tags: string[];
triggerType: string;
hasAI: boolean;
score: number;
scoreBreakdown: ScoreResult['breakdown'];
source: string;
author: string;
success: true;
}
interface SnapshotEntry {
id: number;
name: string;
score: number;
createdAt: string;
totalViews: number;
picked: boolean;
dropReason?: string;
}
function parseArgs(): CliArgs {
const args = { target: DEFAULT_TARGET, candidates: DEFAULT_CANDIDATES };
for (const a of process.argv.slice(2)) {
const [k, v] = a.split('=');
if (k === '--target') args.target = Number(v);
else if (k === '--candidates') args.candidates = Number(v);
}
return args;
}
function ensureDirs() {
for (const dir of [EXAMPLES_DIR, WORKFLOWS_DIR]) {
if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
}
}
function clearFailuresLog() {
if (fs.existsSync(FAILURES_LOG)) fs.unlinkSync(FAILURES_LOG);
}
function logFailure(id: number, name: string, reason: string) {
fs.appendFileSync(FAILURES_LOG, `${id} | ${name} | ${reason}\n`);
}
function makeSlug(id: number, name: string): string {
const base = name
.toLowerCase()
.replace(/[^a-z0-9]+/g, '-')
.replace(/^-|-$/g, '')
.slice(0, 60);
return `${base || 'workflow'}-${id}`;
}
function detailToWorkflowJson(detail: DetailResponse) {
const attrs = detail.data.attributes;
return {
id: `wf-${detail.data.id}`,
name: attrs.name,
nodes: attrs.workflow.nodes,
connections: attrs.workflow.connections,
settings: attrs.workflow.settings ?? {},
pinData: attrs.workflow.pinData ?? {},
};
}
function buildTags(detail: DetailResponse, key: BucketKey): string[] {
const tags: string[] = [`trigger:${key.triggerType}`];
if (key.hasAI) tags.push('ai');
if (key.primaryIntegration && key.primaryIntegration !== 'none') {
tags.push(`integration:${key.primaryIntegration}`);
}
return tags;
}
function uniqueNodeTypes(detail: DetailResponse): string[] {
const seen = new Set<string>();
for (const node of detail.data.attributes.workflow.nodes ?? []) {
const type = String(node.type ?? '');
if (type) seen.add(type);
}
return Array.from(seen);
}
function validateRoundtrip(detail: DetailResponse): { ok: true } | { ok: false; reason: string } {
const json = detailToWorkflowJson(detail);
let firstPass: string;
try {
firstPass = generateWorkflowCode(json);
} catch (error) {
return { ok: false, reason: `codegen threw: ${(error as Error).message}` };
}
if (!firstPass || firstPass.length === 0) {
return { ok: false, reason: 'codegen produced empty output' };
}
try {
const wrapped = emitInstanceAi(json);
if (!wrapped.includes("from '@n8n/workflow-sdk'")) {
return { ok: false, reason: 'emitInstanceAi produced output without SDK import' };
}
} catch (error) {
return { ok: false, reason: `emitInstanceAi threw: ${(error as Error).message}` };
}
return { ok: true };
}
interface ScoredCandidate {
entry: CatalogEntry;
detail: DetailResponse;
bucket: BucketKey;
bucketStr: string;
scoreAtPick: ScoreResult;
}
async function main() {
const args = parseArgs();
ensureDirs();
clearFailuresLog();
const candidatesLabel = Number.isFinite(args.candidates) ? args.candidates : 'all';
console.log(`Regenerating examples (target=${args.target}, candidates=${candidatesLabel})\n`);
// Stage 1: catalog-stage filter and score
const catalog = loadCachedCatalog();
console.log(`Loaded ${catalog.length} catalog entries`);
const snapshot: SnapshotEntry[] = [];
const survivors: Array<{ entry: CatalogEntry; score: number }> = [];
for (const entry of catalog) {
const gate = mechanicalGateCatalog(entry);
if (!gate.ok) {
snapshot.push({
id: entry.id,
name: entry.name,
score: 0,
createdAt: entry.createdAt,
totalViews: entry.totalViews,
picked: false,
dropReason: gate.reason,
});
continue;
}
const s = scoreCatalogEntry(entry);
survivors.push({ entry, score: s.total });
snapshot.push({
id: entry.id,
name: entry.name,
score: s.total,
createdAt: entry.createdAt,
totalViews: entry.totalViews,
picked: false,
});
}
survivors.sort((a, b) => b.score - a.score);
const topCandidates = survivors.slice(0, args.candidates);
console.log(
`Stage 1: ${survivors.length} catalog survivors → top ${topCandidates.length} for detail fetch`,
);
// Stage 2: fetch detail and full-rubric score
const scored: ScoredCandidate[] = [];
let detailFetched = 0;
let detailDropped = 0;
let detailFailed = 0;
for (let i = 0; i < topCandidates.length; i++) {
const { entry } = topCandidates[i];
if (i % 50 === 0) {
console.log(` detail fetch ${i}/${topCandidates.length}...`);
}
const detail = await fetchDetail(entry.id);
if (!detail) {
detailFailed++;
logFailure(entry.id, entry.name, 'detail fetch failed');
continue;
}
detailFetched++;
const gate = mechanicalGateDetail(detail);
if (!gate.ok) {
detailDropped++;
logFailure(entry.id, entry.name, `detail gate: ${gate.reason}`);
continue;
}
const bucket = bucketKey(detail);
// score with empty running set; we'll re-score during bucket pick
const s = scoreDetailedTemplate(entry, detail, []);
scored.push({
entry,
detail,
bucket,
bucketStr: bucketKeyToString(bucket),
scoreAtPick: s,
});
}
console.log(
`Stage 2: detail fetched=${detailFetched}, dropped=${detailDropped}, failed=${detailFailed}, scored=${scored.length}`,
);
// Stage 3: greedy round-robin pick by bucket count, recomputing coverage
const accepted: ScoredCandidate[] = [];
const acceptedBuckets: BucketKey[] = [];
while (accepted.length < args.target) {
let best: { idx: number; score: number; cand: ScoredCandidate } | null = null;
for (let i = 0; i < scored.length; i++) {
const cand = scored[i];
if (accepted.includes(cand)) continue;
const fresh = scoreDetailedTemplate(cand.entry, cand.detail, acceptedBuckets);
if (best === null || fresh.total > best.score) {
best = { idx: i, score: fresh.total, cand };
}
}
if (best === null) break;
// Validate before accepting
const valid = validateRoundtrip(best.cand.detail);
if (!valid.ok) {
scored.splice(best.idx, 1);
logFailure(best.cand.entry.id, best.cand.entry.name, `validation: ${valid.reason}`);
continue;
}
const fresh = scoreDetailedTemplate(best.cand.entry, best.cand.detail, acceptedBuckets);
best.cand.scoreAtPick = fresh;
accepted.push(best.cand);
acceptedBuckets.push(best.cand.bucket);
scored.splice(best.idx, 1);
}
console.log(`Stage 3: accepted ${accepted.length} workflows after validation`);
// Stage 3b: coverage patch — force-include must-cover node types missing from accepted
const acceptedTypes = new Set<string>();
for (const cand of accepted) {
for (const node of cand.detail.data.attributes.workflow.nodes ?? []) {
acceptedTypes.add(String(node.type ?? ''));
}
}
let patchedCount = 0;
for (const mustType of MUST_COVER_NODE_TYPES) {
if (acceptedTypes.has(mustType)) continue;
// First-tier: try scored candidates already in our pool.
const fromScored = scored
.map((cand) => {
const types = (cand.detail.data.attributes.workflow.nodes ?? []).map((n) =>
String(n.type ?? ''),
);
if (!types.includes(mustType)) return null;
const fresh = scoreDetailedTemplate(cand.entry, cand.detail, acceptedBuckets);
return { cand, score: fresh.total };
})
.filter((x): x is { cand: ScoredCandidate; score: number } => x !== null)
.sort((a, b) => b.score - a.score);
let added = false;
for (const { cand, score } of fromScored) {
const valid = validateRoundtrip(cand.detail);
if (!valid.ok) {
logFailure(cand.entry.id, cand.entry.name, `coverage-patch validation: ${valid.reason}`);
continue;
}
const fresh = scoreDetailedTemplate(cand.entry, cand.detail, acceptedBuckets);
cand.scoreAtPick = fresh;
accepted.push(cand);
acceptedBuckets.push(cand.bucket);
scored.splice(scored.indexOf(cand), 1);
for (const node of cand.detail.data.attributes.workflow.nodes ?? []) {
acceptedTypes.add(String(node.type ?? ''));
}
console.log(
` coverage patch (+1 for ${mustType}): id=${cand.entry.id} score=${score.toFixed(2)} ${cand.entry.name.slice(0, 60)}`,
);
patchedCount++;
added = true;
break;
}
if (added) continue;
// Second-tier: type wasn't in the top-K candidate pool. Scan the full
// catalog for entries whose sparse list contains the type, fetch detail
// on demand (cached after first hit), and accept the first that passes
// gate + roundtrip. Ranked by catalog-stage score so we try the
// strongest candidate first.
const catalogCandidates = catalog
.filter((entry) => (entry.nodes ?? []).some((n) => n.name === mustType))
.filter((entry) => mechanicalGateCatalog(entry).ok)
.map((entry) => ({ entry, score: scoreCatalogEntry(entry).total }))
.sort((a, b) => b.score - a.score)
.slice(0, 25); // bounded — don't go fetching the whole tail
for (const { entry } of catalogCandidates) {
const detail = await fetchDetail(entry.id);
if (!detail) continue;
if (!mechanicalGateDetail(detail).ok) continue;
// Catalog `entry.nodes` is a sparse list that can drift from the
// real workflow JSON; re-verify against the detail before accepting.
const detailHasType = (detail.data.attributes.workflow.nodes ?? []).some(
(n) => String(n.type ?? '') === mustType,
);
if (!detailHasType) continue;
const valid = validateRoundtrip(detail);
if (!valid.ok) {
logFailure(entry.id, entry.name, `coverage-patch fallback validation: ${valid.reason}`);
continue;
}
const bucket = bucketKey(detail);
const fresh = scoreDetailedTemplate(entry, detail, acceptedBuckets);
const cand: ScoredCandidate = {
entry,
detail,
bucket,
bucketStr: bucketKeyToString(bucket),
scoreAtPick: fresh,
};
accepted.push(cand);
acceptedBuckets.push(bucket);
for (const node of detail.data.attributes.workflow.nodes ?? []) {
acceptedTypes.add(String(node.type ?? ''));
}
console.log(
` coverage patch fallback (+1 for ${mustType}): id=${entry.id} score=${fresh.total.toFixed(2)} ${entry.name.slice(0, 60)}`,
);
patchedCount++;
break;
}
}
if (patchedCount > 0) {
console.log(`Stage 3b: coverage patch added ${patchedCount} workflows`);
}
console.log();
// Stage 4: write workflows + manifest
// Clear existing committed workflow files
for (const f of fs.readdirSync(WORKFLOWS_DIR)) {
if (f.endsWith('.json')) fs.unlinkSync(path.join(WORKFLOWS_DIR, f));
}
const manifestEntries: ManifestEntry[] = [];
const slugSet = new Set<string>();
for (const cand of accepted) {
const baseSlug = makeSlug(cand.entry.id, cand.entry.name);
let slug = baseSlug;
let suffix = 2;
while (slugSet.has(slug)) slug = `${baseSlug}-${suffix++}`;
slugSet.add(slug);
const wfJson = detailToWorkflowJson(cand.detail);
fs.writeFileSync(path.join(WORKFLOWS_DIR, `${slug}.json`), JSON.stringify(wfJson, null, 2));
const tags = buildTags(cand.detail, cand.bucket);
manifestEntries.push({
id: cand.entry.id,
slug,
name: cand.entry.name,
description: cand.detail.data.attributes.description ?? '',
nodes: uniqueNodeTypes(cand.detail),
tags,
triggerType: cand.bucket.triggerType,
hasAI: cand.bucket.hasAI,
score: Number(cand.scoreAtPick.total.toFixed(2)),
scoreBreakdown: {
traction: Number(cand.scoreAtPick.breakdown.traction.toFixed(3)),
recency: Number(cand.scoreAtPick.breakdown.recency.toFixed(3)),
coverage: Number(cand.scoreAtPick.breakdown.coverage.toFixed(3)),
aiAgent: Number(cand.scoreAtPick.breakdown.aiAgent.toFixed(3)),
clarity: Number(cand.scoreAtPick.breakdown.clarity.toFixed(3)),
density: Number(cand.scoreAtPick.breakdown.density.toFixed(3)),
},
source: `https://n8n.io/workflows/${cand.entry.id}`,
author: cand.detail.data.attributes.username || cand.entry.user.username || 'unknown',
success: true,
});
// Mark in snapshot
const snap = snapshot.find((s) => s.id === cand.entry.id);
if (snap) snap.picked = true;
}
manifestEntries.sort((a, b) => b.score - a.score);
fs.writeFileSync(
MANIFEST_PATH,
JSON.stringify(
{
generatedAt: new Date().toISOString(),
workflows: manifestEntries,
},
null,
2,
),
);
fs.writeFileSync(SNAPSHOT_PATH, JSON.stringify(snapshot, null, 2));
// Pack workflow JSONs into the committed zip so the unpacked dir can be gitignored.
const zip = new AdmZip();
for (const entry of manifestEntries) {
zip.addLocalFile(path.join(WORKFLOWS_DIR, `${entry.slug}.json`));
}
zip.writeZip(ZIP_PATH);
// Bucket distribution report
const bucketDistribution = new Map<string, number>();
for (const e of manifestEntries) {
const k = `${e.triggerType}|${e.hasAI ? 'ai' : 'noai'}`;
bucketDistribution.set(k, (bucketDistribution.get(k) ?? 0) + 1);
}
console.log('Bucket distribution (triggerType × hasAI):');
for (const [k, v] of Array.from(bucketDistribution.entries()).sort((a, b) => b[1] - a[1])) {
console.log(` ${v.toString().padStart(3)} | ${k}`);
}
console.log();
console.log(
`Wrote ${manifestEntries.length} entries to ${path.relative(process.cwd(), MANIFEST_PATH)}`,
);
console.log(`Wrote workflow JSONs to ${path.relative(process.cwd(), WORKFLOWS_DIR)}/`);
console.log(`Wrote zip to ${path.relative(process.cwd(), ZIP_PATH)}`);
console.log(`Catalog snapshot: ${path.relative(process.cwd(), SNAPSHOT_PATH)}`);
if (fs.existsSync(FAILURES_LOG)) {
const failuresCount = fs.readFileSync(FAILURES_LOG, 'utf-8').split('\n').filter(Boolean).length;
console.log(`Failures (${failuresCount}): ${path.relative(process.cwd(), FAILURES_LOG)}`);
}
}
main().catch((error) => {
console.error(error);
process.exit(1);
});

81
packages/@n8n/workflow-sdk/src/__tests__/examples-roundtrip.test.ts Normal file
View File

@ -0,0 +1,81 @@
/**
* Roundtrip test for the curated examples set.
*
* For every entry in `examples/manifest.json` (where success !== false), assert:
* - The source JSON loads
* - `emitInstanceAi()` produces non-empty output with the expected SDK import
* - `parseWorkflowCode()` can parse it back, and the parsed node count matches
*
* This is the only CI hook for the examples pipeline. It catches both codegen
* regressions and SDK drift in a single test, mirroring the pattern in
* src/codegen/codegen-roundtrip.test.ts.
*/
import * as fs from 'fs';
import * as path from 'path';
import { emitInstanceAi } from '../codegen/emit-instance-ai';
import { parseWorkflowCode } from '../codegen/parse-workflow-code';
import type { WorkflowJSON } from '../types/base';
const EXAMPLES_DIR = path.resolve(__dirname, '../../examples');
const MANIFEST_PATH = path.join(EXAMPLES_DIR, 'manifest.json');
const WORKFLOWS_DIR = path.join(EXAMPLES_DIR, 'workflows');
interface ManifestEntry {
id: number;
slug: string;
name: string;
success: boolean;
skip?: boolean;
skipReason?: string;
}
function loadManifest(): ManifestEntry[] {
if (!fs.existsSync(MANIFEST_PATH)) return [];
// eslint-disable-next-line n8n-local-rules/no-uncaught-json-parse -- Test fixture file
const raw = JSON.parse(fs.readFileSync(MANIFEST_PATH, 'utf-8')) as { workflows: ManifestEntry[] };
return raw.workflows ?? [];
}
function loadWorkflowJson(slug: string): WorkflowJSON {
const filePath = path.join(WORKFLOWS_DIR, `${slug}.json`);
// eslint-disable-next-line n8n-local-rules/no-uncaught-json-parse -- Test fixture file
return JSON.parse(fs.readFileSync(filePath, 'utf-8')) as WorkflowJSON;
}
const entries = loadManifest().filter((e) => e.success && !e.skip);
/**
* `parseWorkflowCode` does not accept ESM import declarations; emit-instance-ai
* adds them. Strip them (and any leading JSDoc header) before parsing.
*/
function stripHeader(code: string): string {
let body = code;
body = body.replace(/^\s*\/\*\*[\s\S]*?\*\/\s*/, '');
body = body.replace(/^import\s+\{[^}]*\}\s+from\s+'[^']+'\s*;\s*/m, '');
return body.trimStart();
}
// When the manifest is empty (run `pnpm regenerate-examples` to populate),
// `it.each([])` registers no tests, which is the desired behaviour.
describe('examples manifest roundtrip', () => {
it.each(entries.map((e) => [e.slug, e]))(
'%s: emitInstanceAi → parseWorkflowCode roundtrips',
(_slug, entry) => {
const json = loadWorkflowJson(entry.slug);
const code = emitInstanceAi(json);
expect(code.length).toBeGreaterThan(0);
expect(code).toContain("from '@n8n/workflow-sdk'");
expect(code).toContain('workflow(');
expect(code).toContain('export default');
const body = stripHeader(code);
const parsed = parseWorkflowCode(body);
const sourceNodeCount = json.nodes?.length ?? 0;
const parsedNodeCount = parsed.nodes?.length ?? 0;
// Some normalisation can shift the count by 1 (e.g. sticky note handling).
expect(Math.abs(parsedNodeCount - sourceNodeCount)).toBeLessThanOrEqual(1);
},
);
});

363
packages/@n8n/workflow-sdk/src/codegen/emit-instance-ai.test.ts Normal file
View File

@ -0,0 +1,363 @@
import * as sdk from '../index';
import {
emitInstanceAi,
buildImports,
hoistSharedCredentials,
SDK_FUNCTIONS,
} from './emit-instance-ai';
import type { WorkflowJSON } from '../types/base';
describe('emit-instance-ai', () => {
describe('emitInstanceAi', () => {
it('emits a single SDK import line above the workflow body', () => {
const wf: WorkflowJSON = {
id: 'simple',
name: 'Simple',
nodes: [
{
id: '1',
name: 'Manual',
type: 'n8n-nodes-base.manualTrigger',
typeVersion: 1,
position: [0, 0],
parameters: {},
},
],
connections: {},
};
const out = emitInstanceAi(wf);
expect(out).toMatch(/^import \{[^}]*\} from '@n8n\/workflow-sdk';/);
expect(out).toContain('workflow(');
expect(out).toContain('trigger(');
});
it('hoists credentials shared across multiple nodes', () => {
const wf: WorkflowJSON = {
id: 'shared',
name: 'Shared Credentials',
nodes: [
{
id: 't',
name: 'Manual',
type: 'n8n-nodes-base.manualTrigger',
typeVersion: 1,
position: [0, 0],
parameters: {},
},
{
id: 'a',
name: 'Slack A',
type: 'n8n-nodes-base.slack',
typeVersion: 2.2,
position: [100, 0],
parameters: { channel: '#a' },
credentials: { slackApi: { id: 'cred-1', name: 'Team Slack' } },
},
{
id: 'b',
name: 'Slack B',
type: 'n8n-nodes-base.slack',
typeVersion: 2.2,
position: [200, 0],
parameters: { channel: '#b' },
credentials: { slackApi: { id: 'cred-1', name: 'Team Slack' } },
},
],
connections: {
Manual: { main: [[{ node: 'Slack A', type: 'main', index: 0 }]] },
'Slack A': { main: [[{ node: 'Slack B', type: 'main', index: 0 }]] },
},
};
const out = emitInstanceAi(wf);
// Top-level const declared once
const declMatches = out.match(
/const teamSlackCred = newCredential\('Team Slack', 'cred-1'\);/g,
);
expect(declMatches).not.toBeNull();
expect(declMatches!.length).toBe(1);
// No leftover inline newCredential('Team Slack', 'cred-1') calls in nodes
const inlineMatches = out.match(/newCredential\('Team Slack', 'cred-1'\)/g) ?? [];
expect(inlineMatches.length).toBe(1); // only the const declaration
expect(out).toMatch(/slackApi: teamSlackCred\b/);
});
it('leaves single-use credentials inline', () => {
const wf: WorkflowJSON = {
id: 'single',
name: 'Single Cred',
nodes: [
{
id: 't',
name: 'Manual',
type: 'n8n-nodes-base.manualTrigger',
typeVersion: 1,
position: [0, 0],
parameters: {},
},
{
id: 's',
name: 'Slack',
type: 'n8n-nodes-base.slack',
typeVersion: 2.2,
position: [100, 0],
parameters: {},
credentials: { slackApi: { id: 'one-only', name: 'Solo' } },
},
],
connections: { Manual: { main: [[{ node: 'Slack', type: 'main', index: 0 }]] } },
};
const out = emitInstanceAi(wf);
expect(out).not.toMatch(/^const \w+Cred =/m);
expect(out).toContain("newCredential('Solo', 'one-only')");
});
it('treats different credential ids of the same name as distinct', () => {
const wf: WorkflowJSON = {
id: 'distinct',
name: 'Distinct',
nodes: [
{
id: 't',
name: 'Manual',
type: 'n8n-nodes-base.manualTrigger',
typeVersion: 1,
position: [0, 0],
parameters: {},
},
{
id: 'a',
name: 'Slack A',
type: 'n8n-nodes-base.slack',
typeVersion: 2.2,
position: [100, 0],
parameters: {},
credentials: { slackApi: { id: 'one', name: 'Team' } },
},
{
id: 'b',
name: 'Slack B',
type: 'n8n-nodes-base.slack',
typeVersion: 2.2,
position: [200, 0],
parameters: {},
credentials: { slackApi: { id: 'two', name: 'Team' } },
},
],
connections: {
Manual: { main: [[{ node: 'Slack A', type: 'main', index: 0 }]] },
'Slack A': { main: [[{ node: 'Slack B', type: 'main', index: 0 }]] },
},
};
const out = emitInstanceAi(wf);
// Both occur once → no hoist
expect(out).not.toMatch(/^const \w+Cred =/m);
expect(out).toContain("newCredential('Team', 'one')");
expect(out).toContain("newCredential('Team', 'two')");
});
it('prepends a JSDoc header when supplied', () => {
const wf: WorkflowJSON = {
id: 'h',
name: 'Header',
nodes: [
{
id: 't',
name: 'Manual',
type: 'n8n-nodes-base.manualTrigger',
typeVersion: 1,
position: [0, 0],
parameters: {},
},
],
connections: {},
};
const header = '/**\n * @template\n * @name Header\n */';
const out = emitInstanceAi(wf, { jsdocHeader: header });
expect(out.startsWith(header)).toBe(true);
});
});
describe('buildImports', () => {
it('includes only functions actually used in the body', () => {
const body = "const t = trigger({ ... }); const c = newCredential('A', '1');";
expect(buildImports(body)).toBe(
"import { trigger, newCredential } from '@n8n/workflow-sdk';",
);
});
it('returns empty string when no SDK functions appear', () => {
expect(buildImports('const x = 1;')).toBe('');
});
it('does not match identifier prefixes', () => {
// `mynewCredential(` should not include `newCredential` in imports
const body = 'const x = mynewCredential(); const y = trigger();';
expect(buildImports(body)).toBe("import { trigger } from '@n8n/workflow-sdk';");
});
});
describe('SDK_FUNCTIONS drift guard', () => {
// Every function exported from `@n8n/workflow-sdk` falls into one of two buckets:
// - SDK_FUNCTIONS → workflow-callable, must appear in generated import line
// - INTENTIONALLY_EXCLUDED → utility/codegen/validation/type guard, never appears
// inside an emitted workflow body
// When a new export is added, this test fails until it's categorised. That's the
// signal to decide whether `emit-instance-ai` should know about it.
const INTENTIONALLY_EXCLUDED = new Set<string>([
// Classes
'PluginRegistry',
'ValidationError',
'ValidationWarning',
// Expression helpers (workflows use `expr()` only; the rest are parser internals)
'createFromAIExpression',
'isExpression',
'nodeJson',
'parseExpression',
// Type guards (callable but never appear in workflow bodies)
'isDataTableType',
'isHttpRequestType',
'isIfNodeType',
'isMergeNodeType',
'isNodeChain',
'isNodeInstance',
'isSplitInBatchesType',
'isStickyNoteType',
'isSwitchNodeType',
'isWebhookType',
// Codegen + parse round-trip
'emitInstanceAi',
'generateWorkflowCode',
'parseWorkflowCode',
'parseWorkflowCodeToBuilder',
// Code-step helpers — appear as arguments to `.code()` callbacks, not as
// standalone calls in the emitted body
'runOnceForAllItems',
'runOnceForEachItem',
// Safe-access utilities (used inside .code() callbacks too)
'getProperty',
'hasProperty',
'isPlainObject',
// Validation
'validateNodeConfig',
'validateWorkflow',
'setSchemaBaseDirs',
// Pin-data + schema discovery
'discoverOutputSchemaForNode',
'discoverSchemasForNode',
'findSchemaForOperation',
'inferSchemasFromRunData',
'needsPinData',
'normalizePinData',
// Display-options matching
'matchesDisplayOptions',
// Plugin registration
'registerDefaultPlugins',
// Generate-types module (build-time type generation, never appears in workflows)
'buildDiscriminatorTree',
'extractDiscriminatorCombinations',
'extractOutputTypes',
'filterPropertiesForVersion',
'generateDiscriminatedUnion',
'generateDiscriminatorSchemaFile',
'generateIndexFile',
'generateJsonSchemaFromData',
'generateNodeJSDoc',
'generateNodeTypeFile',
'generatePropertyJSDoc',
'generatePropertyLine',
'generateResourceIndexSchemaFile',
'generateSchemaIndexFile',
'generateSchemaPropertyLine',
'generateSingleVersionSchemaFile',
'generateSingleVersionTypeFile',
'generateSplitVersionIndexSchemaFile',
'generateSubnodeUnionTypes',
'generateSubnodesFile',
'generateTypes',
'generateVersionIndexFile',
'getHighestVersion',
'getPackageName',
'getPropertiesForCombination',
'getSubnodeOutputType',
'groupNodesByOutputType',
'groupVersionsByProperties',
'hasDiscriminatorPattern',
'jsonSchemaToTypeScript',
'literalUnion',
'loadNodeTypes',
'mapPropertyToZodSchema',
'mapPropertyType',
'multiOptionsSchema',
'nodeNameToFileName',
'optionsWithExpression',
'orchestrateGeneration',
'planSplitVersionFiles',
'planSplitVersionSchemaFiles',
'propertyAppliesToVersion',
'versionToFileName',
'versionToTypeName',
]);
const sdkFunctionExports = new Set(
Object.entries(sdk)
.filter(([, v]) => typeof v === 'function')
.map(([k]) => k),
);
it('categorises every SDK function export', () => {
const categorised = new Set([...SDK_FUNCTIONS, ...INTENTIONALLY_EXCLUDED]);
const uncategorised = [...sdkFunctionExports].filter((n) => !categorised.has(n)).sort();
expect(uncategorised).toEqual([]);
});
it('has no stale entries in SDK_FUNCTIONS', () => {
const stale = SDK_FUNCTIONS.filter((n) => !sdkFunctionExports.has(n)).sort();
expect(stale).toEqual([]);
});
it('has no stale entries in INTENTIONALLY_EXCLUDED', () => {
const stale = [...INTENTIONALLY_EXCLUDED].filter((n) => !sdkFunctionExports.has(n)).sort();
expect(stale).toEqual([]);
});
it('does not double-list a function in both sets', () => {
const overlap = SDK_FUNCTIONS.filter((n) => INTENTIONALLY_EXCLUDED.has(n)).sort();
expect(overlap).toEqual([]);
});
});
describe('hoistSharedCredentials', () => {
it('is a no-op when there are no credentials', () => {
const body = 'const x = node({ ... });';
expect(hoistSharedCredentials(body)).toBe(body);
});
it('handles three or more shared occurrences', () => {
const call = "newCredential('Triple', 't-1')";
const body = `const a = node({ c: ${call} });\nconst b = node({ c: ${call} });\nconst c = node({ c: ${call} });`;
const out = hoistSharedCredentials(body);
expect(out).toMatch(/const tripleCred = newCredential\('Triple', 't-1'\);/);
const callMatches = out.match(/newCredential\('Triple', 't-1'\)/g) ?? [];
expect(callMatches.length).toBe(1); // only in const declaration
const refMatches = out.match(/c: tripleCred\b/g) ?? [];
expect(refMatches.length).toBe(3);
});
});
});

203
packages/@n8n/workflow-sdk/src/codegen/emit-instance-ai.ts Normal file
View File

@ -0,0 +1,203 @@
/**
* instance-ai-flavored codegen.
*
* Wraps `generateWorkflowCode()` to produce TypeScript that matches what the
* instance-ai builder agent writes in its sandbox: a top `import` line, shared
* credentials hoisted to top-level `const` declarations, and an optional JSDoc
* header pulled from manifest metadata.
*
* Why dedup credentials: real-world workflows often reference the same credential
* id from multiple nodes. The default codegen emits the same `newCredential('Name',
* 'id')` literal at every callsite, which is structurally correct but obscures the
* "these nodes share a credential" pattern. Hoisting makes it explicit.
*/
// Import from source module to avoid the codegen/index.ts barrel cycle.
import { generateCode } from './code-generator';
import { buildCompositeTree } from './composite-builder';
import { annotateGraph } from './graph-annotator';
import { buildSemanticGraph } from './semantic-graph';
import type { WorkflowJSON } from '../types/base';
function generateWorkflowCode(workflow: WorkflowJSON): string {
const graph = buildSemanticGraph(workflow);
annotateGraph(graph);
const tree = buildCompositeTree(graph);
return generateCode(tree, workflow, graph, {});
}
/**
* SDK functions a workflow file may call. Order matters only for tidiness.
*
* A drift test (`emit-instance-ai.test.ts`) asserts this list stays in sync with the
* function exports of `@n8n/workflow-sdk`. When you add or remove a function export,
* the test will fail until the new name is placed either here (if a workflow file
* may call it) or in the test's intentional-exclusion list (if it's a utility).
*/
export const SDK_FUNCTIONS = [
'workflow',
'trigger',
'node',
'sticky',
'placeholder',
'merge',
'ifElse',
'switchCase',
'newCredential',
'languageModel',
'memory',
'tool',
'outputParser',
'embedding',
'embeddings',
'vectorStore',
'retriever',
'documentLoader',
'textSplitter',
'fromAi',
'splitInBatches',
'nextBatch',
'expr',
] as const;
export interface EmitInstanceAiOptions {
/**
* Optional JSDoc header to prepend (already wrapped in a comment block).
* Typically built from the curation manifest entry.
*/
jsdocHeader?: string;
}
/**
* Generate a workflow .ts file matching the shape the instance-ai builder agent writes.
*
* Returns a single string containing: optional JSDoc header, single import line,
* hoisted shared credentials, and the workflow body.
*/
export function emitInstanceAi(workflow: WorkflowJSON, opts: EmitInstanceAiOptions = {}): string {
const body = generateWorkflowCode(workflow);
const dedupedBody = hoistSharedCredentials(body);
const importLine = buildImports(dedupedBody);
const sections: string[] = [];
if (opts.jsdocHeader) sections.push(opts.jsdocHeader.trim());
if (importLine) sections.push(importLine);
sections.push(dedupedBody);
return sections.join('\n\n') + '\n';
}
/**
* Build the single SDK import line by detecting which functions appear in the body.
* Looks for `name(` to avoid false positives from substring matches.
*/
export function buildImports(body: string): string {
const used: string[] = [];
for (const fn of SDK_FUNCTIONS) {
if (containsCall(body, fn)) used.push(fn);
}
if (used.length === 0) return '';
return `import { ${used.join(', ')} } from '@n8n/workflow-sdk';`;
}
function containsCall(body: string, fnName: string): boolean {
// `\bname\(` — word-boundary anchored to avoid matching `mynewCredential(`
const pattern = new RegExp(`\\b${fnName}\\(`);
return pattern.test(body);
}
/**
* Find every `newCredential('Name', 'id')` invocation, group by literal args,
* and for groups with ≥2 occurrences hoist a `const xCred = newCredential(...)`
* declaration above the workflow body and replace each occurrence with the const.
*
* Single-use credentials are left inline (current codegen behavior) — hoisting them
* adds noise without readability gain.
*/
export function hoistSharedCredentials(body: string): string {
// Two-arg form. Single-arg `newCredential('X')` is rare in real templates and
// already concise enough; deduping it provides little benefit.
const callPattern = /newCredential\('([^'\\]*)',\s*'([^'\\]*)'\)/g;
const groups = new Map<string, { match: string; name: string; id: string; count: number }>();
let m: RegExpExecArray | null;
while ((m = callPattern.exec(body)) !== null) {
const [match, name, id] = m;
const key = `${name}::${id}`;
const existing = groups.get(key);
if (existing) {
existing.count++;
} else {
groups.set(key, { match, name, id, count: 1 });
}
}
const usedNames = collectExistingIdentifiers(body);
const hoists: string[] = [];
let result = body;
// Hoist in stable order: by first-occurrence position in the body
const dedupable = Array.from(groups.values())
.filter((g) => g.count >= 2)
.sort((a, b) => body.indexOf(a.match) - body.indexOf(b.match));
for (const group of dedupable) {
const constName = uniqueConstName(group.name, usedNames);
hoists.push(`const ${constName} = ${group.match};`);
// Exact-string replace; the match is a literal newCredential(...) call
result = splitJoin(result, group.match, constName);
}
if (hoists.length === 0) return body;
return `${hoists.join('\n')}\n\n${result}`;
}
function splitJoin(haystack: string, needle: string, replacement: string): string {
return haystack.split(needle).join(replacement);
}
/**
* Build a deterministic, readable `const` name from a credential display name.
* `'OpenAI account'` → `openAiAccountCred`. Suffixes a number on collision.
*/
export function uniqueConstName(credName: string, used: Set<string>): string {
const sanitised = credName
.replace(/[^a-zA-Z0-9]+/g, ' ')
.trim()
.split(/\s+/)
.filter(Boolean);
let camel: string;
if (sanitised.length === 0) {
camel = 'shared';
} else {
camel =
sanitised[0].charAt(0).toLowerCase() +
sanitised[0].slice(1) +
sanitised
.slice(1)
.map((word) => word.charAt(0).toUpperCase() + word.slice(1).toLowerCase())
.join('');
}
const base = `${camel}Cred`;
let candidate = base;
let i = 1;
while (used.has(candidate)) {
i++;
candidate = `${base}${i}`;
}
used.add(candidate);
return candidate;
}
/**
* Collect identifier-shaped tokens already present in the body so newly hoisted
* `const` names don't shadow existing locals.
*/
function collectExistingIdentifiers(body: string): Set<string> {
const ids = new Set<string>();
const pattern = /\b([a-zA-Z_][a-zA-Z0-9_]*)\b/g;
let m: RegExpExecArray | null;
while ((m = pattern.exec(body)) !== null) ids.add(m[1]);
return ids;
}

1
packages/@n8n/workflow-sdk/src/codegen/index.ts
View File

@ -55,6 +55,7 @@ export { buildSemanticGraph } from './semantic-graph';
export { annotateGraph } from './graph-annotator';
export { buildCompositeTree } from './composite-builder';
export { generateCode } from './code-generator';
export { emitInstanceAi, type EmitInstanceAiOptions } from './emit-instance-ai';
export {
getOutputName,
getInputName,

50
packages/@n8n/workflow-sdk/src/examples-loader.test.ts Normal file
View File

@ -0,0 +1,50 @@
import { getExampleFiles, resetExampleFilesCache } from './examples-loader';
describe('examples-loader', () => {
beforeEach(() => resetExampleFilesCache());
it('loads manifest entries when present', () => {
const bundle = getExampleFiles();
// In CI without a manifest, both arrays will be empty — that's also valid.
// When a manifest exists (post `pnpm regenerate-examples`), assert structure.
if (bundle.files.length === 0) {
expect(bundle.indexTxt).toBe('');
return;
}
// Each generated file has the expected shape
for (const file of bundle.files) {
expect(file.filename).toMatch(/\.ts$/);
expect(file.content).toContain('@template');
expect(file.content).toContain("from '@n8n/workflow-sdk'");
expect(file.content).toContain('export default');
// Untrusted catalog description must not be embedded in JSDoc.
expect(file.content).not.toContain('@description');
}
// Index lines match the file count
const indexLines = bundle.indexTxt.trim().split('\n');
expect(indexLines.length).toBe(bundle.files.length);
// Each index line has the documented 5-field shape
for (const line of indexLines) {
const parts = line.split(' | ');
expect(parts.length).toBe(5);
expect(parts[0]).toMatch(/\.ts$/);
expect(parts[4]).toMatch(/^n8n:\d+/);
}
});
it('memoises across calls', () => {
const a = getExampleFiles();
const b = getExampleFiles();
expect(a).toBe(b);
});
it('resetExampleFilesCache forces a reload', () => {
const a = getExampleFiles();
resetExampleFilesCache();
const b = getExampleFiles();
expect(a).not.toBe(b);
});
});

144
packages/@n8n/workflow-sdk/src/examples-loader.ts Normal file
View File

@ -0,0 +1,144 @@
/**
* Runtime loader for the curated workflow examples.
*
* Reads `examples/manifest.json` + `examples/workflows/*.json`, runs each
* through `emitInstanceAi` with a JSDoc header pulled from the manifest entry,
* and returns the resulting `.ts` strings plus a flat grep-able `index.txt`.
*
* Used by the instance-ai sandbox-setup to populate `${workspaceRoot}/examples/`
* so the builder agent can grep the index and `cat` matching `.ts` files.
*
* Results are memoised — the manifest is committed and immutable per package
* version, so loading once per process is enough.
*/
import * as fs from 'fs';
import * as path from 'path';
import { emitInstanceAi } from './codegen/emit-instance-ai';
import { ensureExtracted } from './examples-zip';
import type { WorkflowJSON } from './types/base';
// Resolve relative to this file. At runtime this lives at <package>/dist/examples-loader.js,
// so `../examples` reaches <package>/examples/.
const EXAMPLES_DIR = path.resolve(__dirname, '..', 'examples');
const MANIFEST_PATH = path.join(EXAMPLES_DIR, 'manifest.json');
const WORKFLOWS_DIR = path.join(EXAMPLES_DIR, 'workflows');
const NODES_INLINE_LIMIT = 5;
const INDEX_NODE_SEPARATOR = ',';
export interface ExampleFile {
/** Filename relative to `examples/` (e.g. `slack-daily-summary.ts`). */
filename: string;
/** Full file content: optional JSDoc header, single SDK import, workflow body. */
content: string;
}
export interface ExampleFilesBundle {
/** One generated `.ts` string per manifest entry, in score-descending order. */
files: ExampleFile[];
/** Flat grep-able index, one line per template, sorted by score descending. */
indexTxt: string;
}
interface ManifestEntry {
id: number;
slug: string;
name: string;
description: string;
nodes: string[];
tags: string[];
triggerType: string;
hasAI: boolean;
score: number;
source: string;
author: string;
success: boolean;
skip?: boolean;
}
interface ManifestFile {
workflows: ManifestEntry[];
}
let cached: ExampleFilesBundle | null = null;
/**
* Load and prepare the curated examples for sandbox use. Memoised per process.
*
* Returns an empty bundle if the manifest does not exist (e.g. the consumer is
* running against an unfetched workspace). Sandbox-setup checks for an empty
* bundle and skips the write.
*/
export function getExampleFiles(): ExampleFilesBundle {
if (cached !== null) return cached;
cached = loadFromDisk();
return cached;
}
/** Reset the memoisation cache. Tests use this; production callers should not. */
export function resetExampleFilesCache(): void {
cached = null;
}
function loadFromDisk(): ExampleFilesBundle {
if (!fs.existsSync(MANIFEST_PATH)) return { files: [], indexTxt: '' };
ensureExtracted();
// eslint-disable-next-line n8n-local-rules/no-uncaught-json-parse -- Internal manifest file
const manifest = JSON.parse(fs.readFileSync(MANIFEST_PATH, 'utf-8')) as ManifestFile;
const entries = (manifest.workflows ?? [])
.filter((e) => e.success && !e.skip)
.sort((a, b) => b.score - a.score);
const files: ExampleFile[] = [];
const indexLines: string[] = [];
for (const entry of entries) {
const wfPath = path.join(WORKFLOWS_DIR, `${entry.slug}.json`);
if (!fs.existsSync(wfPath)) continue;
// eslint-disable-next-line n8n-local-rules/no-uncaught-json-parse -- Internal workflow fixture
const wf = JSON.parse(fs.readFileSync(wfPath, 'utf-8')) as WorkflowJSON;
const header = buildJsdocHeader(entry);
const code = emitInstanceAi(wf, { jsdocHeader: header });
files.push({ filename: `${entry.slug}.ts`, content: code });
indexLines.push(buildIndexLine(entry));
}
const indexTxt = indexLines.join('\n') + (indexLines.length > 0 ? '\n' : '');
return { files, indexTxt };
}
function buildJsdocHeader(entry: ManifestEntry): string {
// Description is intentionally omitted: it's untrusted author-supplied prose
// from the public catalog that the builder agent would read verbatim. Name +
// nodes + tags + source already disambiguate templates.
return [
'/**',
' * @template',
` * @name ${entry.name}`,
` * @nodes ${entry.nodes.join(', ')}`,
` * @tags ${entry.tags.join(', ')}`,
` * @source ${entry.source}`,
` * @author ${entry.author}`,
' */',
].join('\n');
}
function buildIndexLine(entry: ManifestEntry): string {
const truncatedNodes = truncateNodes(entry.nodes);
return [
`${entry.slug}.ts`,
entry.name,
truncatedNodes,
entry.tags.join(','),
`n8n:${entry.id}`,
].join(' | ');
}
function truncateNodes(nodes: string[]): string {
if (nodes.length <= NODES_INLINE_LIMIT) return nodes.join(INDEX_NODE_SEPARATOR);
const head = nodes.slice(0, NODES_INLINE_LIMIT).join(INDEX_NODE_SEPARATOR);
const remaining = nodes.length - NODES_INLINE_LIMIT;
return `${head} +${remaining} more`;
}

73
packages/@n8n/workflow-sdk/src/examples-zip.ts Normal file
View File

@ -0,0 +1,73 @@
/**
* Zip extraction utility for the curated workflow examples.
*
* The 106 workflow JSON files are committed as a single `examples/templates.zip`
* to keep the package small. At runtime the loader calls `ensureExtracted()`
* which extracts them on first use. The committed `manifest.json` is the
* source of truth and is NOT in the zip.
*/
import AdmZip from 'adm-zip';
import * as fs from 'fs';
import * as path from 'path';
const EXAMPLES_DIR = path.resolve(__dirname, '..', 'examples');
const ZIP_PATH = path.join(EXAMPLES_DIR, 'templates.zip');
const MANIFEST_PATH = path.join(EXAMPLES_DIR, 'manifest.json');
const WORKFLOWS_DIR = path.join(EXAMPLES_DIR, 'workflows');
interface ManifestEntry {
slug: string;
success: boolean;
skip?: boolean;
}
interface ManifestFile {
workflows: ManifestEntry[];
}
export function zipExists(): boolean {
return fs.existsSync(ZIP_PATH);
}
/**
* True if the zip exists and at least one workflow file expected by the
* manifest is missing on disk.
*/
export function needsExtraction(): boolean {
if (!fs.existsSync(ZIP_PATH)) return false;
if (!fs.existsSync(MANIFEST_PATH)) return false;
// eslint-disable-next-line n8n-local-rules/no-uncaught-json-parse -- Internal manifest file
const manifest = JSON.parse(fs.readFileSync(MANIFEST_PATH, 'utf-8')) as ManifestFile;
for (const entry of manifest.workflows ?? []) {
if (!entry.success || entry.skip) continue;
const filePath = path.join(WORKFLOWS_DIR, `${entry.slug}.json`);
if (!fs.existsSync(filePath)) return true;
}
return false;
}
/**
* Extract all workflow JSONs from the zip into `examples/workflows/`.
* The committed manifest.json is the source of truth and is not in the zip.
*/
export function extractFromZip(): void {
if (!fs.existsSync(ZIP_PATH)) {
throw new Error(`Examples zip not found: ${ZIP_PATH}`);
}
if (!fs.existsSync(WORKFLOWS_DIR)) {
fs.mkdirSync(WORKFLOWS_DIR, { recursive: true });
}
const zip = new AdmZip(ZIP_PATH);
for (const entry of zip.getEntries()) {
if (entry.isDirectory) continue;
zip.extractEntryTo(entry, WORKFLOWS_DIR, false, true);
}
}
export function ensureExtracted(): void {
if (needsExtraction()) {
extractFromZip();
}
}

1
packages/@n8n/workflow-sdk/src/index.ts
View File

@ -162,6 +162,7 @@ export {
// Code generation
export { generateWorkflowCode } from './codegen/index';
export { emitInstanceAi, type EmitInstanceAiOptions } from './codegen/index';
export { parseWorkflowCode, parseWorkflowCodeToBuilder } from './codegen/parse-workflow-code';
// Type generation utilities (for runtime type generation in CLI)

25
packages/cli/src/commands/export/__tests__/entities.test.ts
View File

@ -32,6 +32,7 @@ describe('ExportEntitiesCommand', () => {
'execution_metadata',
]),
undefined,
{ includeDataTableRows: undefined },
);
});
@ -53,6 +54,7 @@ describe('ExportEntitiesCommand', () => {
'./exports',
new Set<string>(),
undefined,
{ includeDataTableRows: undefined },
);
});
@ -80,6 +82,29 @@ describe('ExportEntitiesCommand', () => {
'execution_metadata',
]),
'key.txt',
{ includeDataTableRows: undefined },
);
});
it('should pass includeDataTableRows=false through to the export service', async () => {
const command = new ExportEntitiesCommand();
// @ts-expect-error Protected property
command.flags = {
outputDir: './exports',
includeDataTableRows: false,
};
// @ts-expect-error Protected property
command.logger = {
info: jest.fn(),
error: jest.fn(),
};
await command.run();
expect(mockExportService.exportEntities).toHaveBeenCalledWith(
'./exports',
expect.any(Set),
undefined,
{ includeDataTableRows: false },
);
});
});

11
packages/cli/src/commands/export/entities.ts
View File

@ -17,6 +17,12 @@ const flagsSchema = z.object({
'Include execution history data tables, these are excluded by default as they can be very large',
)
.default(false),
includeDataTableRows: z.coerce
.boolean()
.describe(
'Include user-data table row contents. When false, only the data-table schemas (registry rows) are exported.',
)
.default(true),
keyFile: z
.string()
.describe('Optional path to a file containing a custom encryption key')
@ -31,6 +37,7 @@ const flagsSchema = z.object({
'--outputDir=./exports',
'--outputDir=/path/to/backup',
'--includeExecutionHistoryDataTables=true',
'--includeDataTableRows=false',
'--keyFile=/path/to/key.txt',
'--outputDir=./exports --keyFile=/path/to/key.txt',
],
@ -50,7 +57,9 @@ export class ExportEntitiesCommand extends BaseCommand<z.infer<typeof flagsSchem
excludedDataTables.add('execution_metadata');
}
await Container.get(ExportService).exportEntities(outputDir, excludedDataTables, keyFilePath);
await Container.get(ExportService).exportEntities(outputDir, excludedDataTables, keyFilePath, {
includeDataTableRows: this.flags.includeDataTableRows,
});
}
catch(error: Error) {

4
packages/cli/src/controllers/oauth/__tests__/oauth1-credential.controller.test.ts
View File

@ -31,7 +31,7 @@ describe('OAuth1CredentialController', () => {
describe('getAuthUri', () => {
it('should return a valid auth URI', async () => {
const mockResolvedCredential = mock<CredentialsEntity>({ id: '1' });
oauthService.getCredential.mockResolvedValueOnce(mockResolvedCredential);
oauthService.getCredentialForUpdate.mockResolvedValueOnce(mockResolvedCredential);
oauthService.generateAOauth1AuthUri.mockResolvedValueOnce(
'https://example.domain/oauth/authorize?oauth_token=random-token',
);
@ -90,7 +90,7 @@ describe('OAuth1CredentialController', () => {
createdAt: timestamp,
data: 'encrypted-data',
};
oauthService.getCredential.mockResolvedValueOnce(mockResolvedCredential);
oauthService.getCredentialForUpdate.mockResolvedValueOnce(mockResolvedCredential);
// @ts-ignore
oauthService.getDecryptedData.mockResolvedValue({ csrfSecret: 'invalid' });
oauthService.getOAuthCredentials.mockResolvedValueOnce({

2
packages/cli/src/controllers/oauth/__tests__/oauth2-credential.controller.test.ts
View File

@ -49,7 +49,7 @@ describe('OAuth2CredentialController', () => {
);
const mockResolvedCredential = mock<CredentialsEntity>({ id: '1' });
oauthService.getCredential.mockResolvedValueOnce(mockResolvedCredential);
oauthService.getCredentialForUpdate.mockResolvedValueOnce(mockResolvedCredential);
oauthService.getOAuthCredentials.mockResolvedValueOnce({
clientId: 'client_id',
clientSecret: 'client_secret',

2
packages/cli/src/controllers/oauth/oauth1-credential.controller.ts
View File

@ -22,7 +22,7 @@ export class OAuth1CredentialController {
/** Get Authorization url */
@Get('/auth')
async getAuthUri(req: OAuthRequest.OAuth1Credential.Auth): Promise<string> {
const credential = await this.oauthService.getCredential(req);
const credential = await this.oauthService.getCredentialForUpdate(req);
const uri = await this.oauthService.generateAOauth1AuthUri(credential, {
cid: credential.id,

2
packages/cli/src/controllers/oauth/oauth2-credential.controller.ts
View File

@ -26,7 +26,7 @@ export class OAuth2CredentialController {
/** Get Authorization url */
@Get('/auth')
async getAuthUri(req: OAuthRequest.OAuth2Credential.Auth): Promise<string> {
const credential = await this.oauthService.getCredential(req);
const credential = await this.oauthService.getCredentialForUpdate(req);
const uri = await this.oauthService.generateAOauth2AuthUri(credential, {
cid: credential.id,

136
packages/cli/src/modules/agents/integrations/platforms/__tests__/telegram-integration.test.ts
View File

@ -1,6 +1,8 @@
/* eslint-disable @typescript-eslint/unbound-method -- mock-based tests intentionally reference unbound methods */
import type { Logger } from '@n8n/backend-common';
import { createHmac } from 'crypto';
import { mock } from 'jest-mock-extended';
import type { InstanceSettings } from 'n8n-core';
import { ConflictError } from '@/errors/response-errors/conflict.error';
import type { UrlService } from '@/services/url.service';
@ -8,8 +10,20 @@ import type { UrlService } from '@/services/url.service';
import type { Agent } from '../../../entities/agent.entity';
import type { AgentRepository } from '../../../repositories/agent.repository';
import type { AgentChatIntegrationContext } from '../../agent-chat-integration';
import { loadTelegramAdapter } from '../../esm-loader';
import { TelegramIntegration } from '../telegram-integration';
jest.mock('../../esm-loader', () => ({
loadTelegramAdapter: jest.fn(),
}));
const mockedLoadTelegramAdapter = loadTelegramAdapter as jest.MockedFunction<
typeof loadTelegramAdapter
>;
const expectedSecret = (encryptionKey: string, agentId: string, credentialId: string) =>
createHmac('sha256', encryptionKey).update(`telegram:${agentId}:${credentialId}`).digest('hex');
const makeAgent = (
id: string,
name: string,
@ -28,18 +42,42 @@ const makeContext = (
...overrides,
});
const makeIntegration = (
opts: {
urlService?: jest.Mocked<UrlService>;
agentRepository?: jest.Mocked<AgentRepository>;
encryptionKey?: string;
} = {},
) => {
const urlService =
opts.urlService ??
(() => {
const u = mock<UrlService>();
u.getWebhookBaseUrl.mockReturnValue('https://n8n.example.com/');
return u;
})();
const agentRepository = opts.agentRepository ?? mock<AgentRepository>();
const instanceSettings = mock<InstanceSettings>({
encryptionKey: opts.encryptionKey ?? 'test-encryption-key',
});
const integration = new TelegramIntegration(
mock<Logger>(),
urlService,
agentRepository,
instanceSettings,
);
return { integration, urlService, agentRepository, instanceSettings };
};
describe('TelegramIntegration.onBeforeConnect', () => {
let integration: TelegramIntegration;
let agentRepository: jest.Mocked<AgentRepository>;
let urlService: jest.Mocked<UrlService>;
let integration: TelegramIntegration;
let fetchSpy: jest.SpyInstance;
beforeEach(() => {
agentRepository = mock<AgentRepository>();
urlService = mock<UrlService>();
urlService.getWebhookBaseUrl.mockReturnValue('https://n8n.example.com/');
integration = new TelegramIntegration(mock<Logger>(), urlService, agentRepository);
const built = makeIntegration();
integration = built.integration;
agentRepository = built.agentRepository;
fetchSpy = jest.spyOn(globalThis, 'fetch');
});
@ -97,11 +135,7 @@ describe('TelegramIntegration.requiresLeader', () => {
const urlService = mock<UrlService>();
urlService.getWebhookBaseUrl.mockReturnValue('http://localhost:5678/');
const integration = new TelegramIntegration(
mock<Logger>(),
urlService,
mock<AgentRepository>(),
);
const { integration } = makeIntegration({ urlService });
expect(integration.requiresLeader()).toBe(true);
});
@ -110,12 +144,80 @@ describe('TelegramIntegration.requiresLeader', () => {
const urlService = mock<UrlService>();
urlService.getWebhookBaseUrl.mockReturnValue('https://n8n.example.com/');
const integration = new TelegramIntegration(
mock<Logger>(),
urlService,
mock<AgentRepository>(),
);
const { integration } = makeIntegration({ urlService });
expect(integration.requiresLeader()).toBe(false);
});
});
describe('TelegramIntegration secret token', () => {
const createTelegramAdapter = jest.fn();
beforeEach(() => {
createTelegramAdapter.mockReset();
createTelegramAdapter.mockReturnValue({ marker: 'adapter' });
mockedLoadTelegramAdapter.mockReset();
mockedLoadTelegramAdapter.mockResolvedValue({
createTelegramAdapter,
} as unknown as Awaited<ReturnType<typeof loadTelegramAdapter>>);
});
it('createAdapter passes a deterministic secretToken derived from encryptionKey + agentId + credentialId', async () => {
const { integration } = makeIntegration({ encryptionKey: 'super-secret-key' });
await integration.createAdapter(makeContext({ agentId: 'A', credentialId: 'C' }));
const expected = expectedSecret('super-secret-key', 'A', 'C');
expect(createTelegramAdapter).toHaveBeenCalledWith({
botToken: 'bot-token',
mode: 'webhook',
secretToken: expected,
});
});
it('two service instances configured with the same encryption key produce identical secrets — the multi-main invariant', async () => {
const a = makeIntegration({ encryptionKey: 'shared-cluster-key' });
const b = makeIntegration({ encryptionKey: 'shared-cluster-key' });
await a.integration.createAdapter(makeContext({ agentId: 'agent-X', credentialId: 'cred-Y' }));
await b.integration.createAdapter(makeContext({ agentId: 'agent-X', credentialId: 'cred-Y' }));
const firstSecret = createTelegramAdapter.mock.calls[0][0].secretToken;
const secondSecret = createTelegramAdapter.mock.calls[1][0].secretToken;
expect(firstSecret).toBe(secondSecret);
});
it('different (agentId, credentialId) pairs produce different secrets', async () => {
const { integration } = makeIntegration({ encryptionKey: 'k' });
await integration.createAdapter(makeContext({ agentId: 'A', credentialId: 'C1' }));
await integration.createAdapter(makeContext({ agentId: 'A', credentialId: 'C2' }));
expect(createTelegramAdapter.mock.calls[0][0].secretToken).not.toBe(
createTelegramAdapter.mock.calls[1][0].secretToken,
);
});
it('onAfterConnect sends secret_token to Telegram matching the secret used by the adapter', async () => {
const fetchSpy = jest
.spyOn(globalThis, 'fetch')
.mockResolvedValue({ ok: true, text: async () => 'ok' } as Response);
try {
const { integration } = makeIntegration({ encryptionKey: 'cluster-key' });
await integration.onAfterConnect(makeContext({ agentId: 'agent-Z', credentialId: 'cred-Z' }));
expect(fetchSpy).toHaveBeenCalledTimes(1);
const [url, init] = fetchSpy.mock.calls[0] as [string, RequestInit];
expect(url).toBe('https://api.telegram.org/botbot-token/setWebhook');
const body = JSON.parse(init.body as string) as { url: string; secret_token: string };
expect(body.url).toBe(
'https://n8n.example.com/rest/projects/proj-1/agents/v2/agent-1/webhooks/telegram',
);
expect(body.secret_token).toBe(expectedSecret('cluster-key', 'agent-Z', 'cred-Z'));
} finally {
fetchSpy.mockRestore();
}
});
});

22
packages/cli/src/modules/agents/integrations/platforms/telegram-integration.ts
View File

@ -1,6 +1,8 @@
import { Logger } from '@n8n/backend-common';
import { Service } from '@n8n/di';
import type { Thread } from 'chat';
import { createHmac } from 'crypto';
import { InstanceSettings } from 'n8n-core';
import { ConflictError } from '@/errors/response-errors/conflict.error';
import { UrlService } from '@/services/url.service';
@ -63,6 +65,7 @@ export class TelegramIntegration extends AgentChatIntegration {
private readonly logger: Logger,
private readonly urlService: UrlService,
private readonly agentRepository: AgentRepository,
private readonly instanceSettings: InstanceSettings,
) {
super();
}
@ -70,8 +73,9 @@ export class TelegramIntegration extends AgentChatIntegration {
async createAdapter(ctx: AgentChatIntegrationContext): Promise<unknown> {
const botToken = this.extractBotToken(ctx.credential);
const mode = this.getMode();
const secretToken = this.deriveSecretToken(ctx.agentId, ctx.credentialId);
const { createTelegramAdapter } = await loadTelegramAdapter();
return createTelegramAdapter({ botToken, mode });
return createTelegramAdapter({ botToken, mode, secretToken });
}
/**
@ -107,10 +111,11 @@ export class TelegramIntegration extends AgentChatIntegration {
if (this.getMode() !== 'webhook') return;
const botToken = this.extractBotToken(ctx.credential);
const webhookUrl = ctx.webhookUrlFor('telegram');
const secretToken = this.deriveSecretToken(ctx.agentId, ctx.credentialId);
const resp = await fetch(`https://api.telegram.org/bot${botToken}/setWebhook`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ url: webhookUrl }),
body: JSON.stringify({ url: webhookUrl, secret_token: secretToken }),
});
if (!resp.ok) {
throw new Error(`Failed to register Telegram webhook: ${await resp.text()}`);
@ -149,6 +154,19 @@ export class TelegramIntegration extends AgentChatIntegration {
return isPublic ? 'webhook' : 'polling';
}
/**
* Per-integration webhook secret derived deterministically from the cluster
* encryption key. In multi-main mode the webhook is registered exactly once by the
* leader, but every main must verify incoming POSTs. HMAC over an
* `(agentId, credentialId)` lets every main reach the identical secret with
* zero coordination.
*/
private deriveSecretToken(agentId: string, credentialId: string): string {
return createHmac('sha256', this.instanceSettings.encryptionKey)
.update(`telegram:${agentId}:${credentialId}`)
.digest('hex');
}
private extractBotToken(credential: Record<string, unknown>): string {
const token = credential.accessToken;
if (typeof token === 'string' && token) {

50
packages/cli/src/modules/data-table/utils/sql-utils.ts
View File

@ -1,5 +1,6 @@
import {
dataTableColumnNameSchema,
dataTableIdSchema,
DATA_TABLE_COLUMN_ERROR_MESSAGE,
type DataTableCreateColumnSchema,
} from '@n8n/api-types';
@ -79,6 +80,10 @@ export function isValidColumnName(name: string) {
return dataTableColumnNameSchema.safeParse(name).success;
}
export function isValidDataTableId(id: string) {
return dataTableIdSchema.safeParse(id).success;
}
export function addColumnQuery(
tableName: DataTableUserTableName,
column: DataTableCreateColumnSchema,
@ -187,8 +192,16 @@ export function extractInsertedIds(raw: unknown, dbType: DataSourceOptions['type
}
}
// Convert 0/1 or '0'/'1' to booleans, leaving other values unchanged
function normalizeBoolean<T>(value: T): T | boolean {
if (typeof value === 'boolean') return value;
if (value === 1 || value === '1') return true;
if (value === 0 || value === '0') return false;
return value;
}
// Convert date objects or strings to dates in UTC
function normalizeDate(value: DataTableColumnJsType): Date | null {
export function normalizeDate(value: unknown): Date | null {
if (value instanceof Date) return value;
if (typeof value === 'string') {
@ -229,14 +242,7 @@ export function normalizeRows(
const type = typeMap[key];
if (type === 'boolean') {
// Convert boolean values to true/false
if (typeof value === 'boolean') {
normalized[key] = value;
} else if (value === 1 || value === '1') {
normalized[key] = true;
} else if (value === 0 || value === '0') {
normalized[key] = false;
}
normalized[key] = normalizeBoolean(value);
}
if (type === 'date' && value !== null && value !== undefined) {
@ -296,6 +302,32 @@ export function normalizeValueForDatabase(
return value;
}
/**
* Normalize a raw value coming from a cross-database data-table export so it can
* be inserted into the destination database. Differs from `normalizeValueForDatabase`
* by also handling booleans (SQLite stores them as 0/1 INTEGER, Postgres as BOOLEAN)
* and SQLite-style date strings without timezone info (treated as UTC).
*/
export function normalizeUserRowValueForDatabase(
value: unknown,
columnType: string | undefined,
dbType?: DataSourceOptions['type'],
): unknown {
if (value === null || value === undefined) return null;
if (columnType === 'boolean') {
return normalizeBoolean(value);
}
if (columnType === 'date') {
const date = normalizeDate(value);
if (date === null) return value;
return normalizeValueForDatabase(date, columnType, dbType);
}
return value;
}
/**
* Convert a LIKE-style pattern (only % is wildcard) into a SQLite GLOB pattern.
*/

44
packages/cli/src/modules/instance-ai/__tests__/instance-ai-liveness.service.test.ts
View File

@ -10,7 +10,12 @@ import {
type InstanceAiLivenessTimeoutReason,
} from '@n8n/instance-ai';
import { INSTANCE_AI_RUN_TIMEOUT_REASON, InstanceAiLivenessService } from '../liveness';
import {
INSTANCE_AI_RUN_TIMEOUT_REASON,
InstanceAiLivenessService,
type InstanceAiLivenessSweepResult,
} from '../liveness';
import type { InstanceAiRunTimeoutDetails } from '../run-timeout-details';
type TestSuspendedRun = {
runId: string;
@ -22,11 +27,13 @@ function createLivenessService() {
const policyConfig = createInstanceAiLivenessPolicyConfig();
const policy = new InstanceAiLivenessPolicy(policyConfig);
const runState = {
sweepTimedOut: jest.fn((_policy: InstanceAiLivenessPolicy, _now?: number) => ({
activeThreadIds: [] as string[],
suspendedThreadIds: [] as string[],
confirmationRequestIds: [] as string[],
})),
sweepTimedOut: jest.fn(
(_policy: InstanceAiLivenessPolicy, _now?: number): InstanceAiLivenessSweepResult => ({
activeThreadIds: [] as string[],
suspendedThreadIds: [] as string[],
confirmationRequestIds: [] as string[],
}),
),
cancelActiveRun: jest.fn(
(_threadId: string): { runId: string; abortController: AbortController } | undefined =>
undefined,
@ -93,11 +100,27 @@ describe('InstanceAiLivenessService', () => {
threadId: 'thread-suspended',
abortController: suspendedAbortController,
};
const activeTimeout: InstanceAiRunTimeoutDetails = {
reason: 'idle_timeout',
surface: 'active-run',
timeoutMs: 600_000,
elapsedMs: 650_000,
idleMs: 606_000,
};
const suspendedTimeout: InstanceAiRunTimeoutDetails = {
reason: 'max_lifetime',
surface: 'suspended-run',
timeoutMs: 1_800_000,
elapsedMs: 1_801_000,
idleMs: 900_000,
};
runState.sweepTimedOut.mockReturnValue({
activeThreadIds: ['thread-active'],
suspendedThreadIds: ['thread-suspended'],
confirmationRequestIds: ['request-1'],
activeTimeouts: { 'thread-active': activeTimeout },
suspendedTimeouts: { 'thread-suspended': suspendedTimeout },
});
runState.cancelActiveRun.mockReturnValue({
runId: 'run-active',
@ -136,7 +159,14 @@ describe('InstanceAiLivenessService', () => {
'thread-confirmation',
expect.objectContaining({ responseId: 'run-timeout:run-confirmation' }),
);
expect(service.consumeRunTimedOut('run-active')).toBe(true);
expect(service.consumeRunTimeout('run-active')).toEqual({
timedOut: true,
details: activeTimeout,
});
expect(service.consumeRunTimeout('run-suspended')).toEqual({
timedOut: true,
details: suspendedTimeout,
});
expect(service.hasTimedOutActiveRunThread('thread-active')).toBe(true);
});

39
packages/cli/src/modules/instance-ai/__tests__/instance-ai.service.test.ts
View File

@ -533,6 +533,7 @@ type TerminalGuardOrderServiceInternals = {
getEventsForRuns: jest.Mock;
publish: jest.Mock;
};
liveness: { consumeRunTimeout: jest.Mock };
telemetry: { track: jest.Mock };
logger: { warn: jest.Mock; error: jest.Mock };
traceContextsByRunId: Map<string, { threadId: string; messageGroupId?: string }>;
@ -540,6 +541,7 @@ type TerminalGuardOrderServiceInternals = {
finalizeRunTracing: jest.Mock;
saveAgentTreeSnapshot: jest.Mock;
reapAiTemporaryFromRun: jest.Mock;
countCreditsIfFirst: jest.Mock;
maybeFinalizeRunTraceRoot: jest.Mock;
schedulePlannedTasks: jest.Mock;
drainPendingCheckpointReentries: jest.Mock;
@ -602,6 +604,7 @@ function createTerminalGuardOrderService(): TerminalGuardOrderServiceInternals {
events.push(event);
}),
};
service.liveness = { consumeRunTimeout: jest.fn(() => ({ timedOut: false })) };
service.telemetry = { track: jest.fn() };
service.logger = { warn: jest.fn(), error: jest.fn() };
service.traceContextsByRunId = new Map([
@ -611,6 +614,7 @@ function createTerminalGuardOrderService(): TerminalGuardOrderServiceInternals {
service.finalizeRunTracing = jest.fn(async () => {});
service.saveAgentTreeSnapshot = jest.fn(async () => {});
service.reapAiTemporaryFromRun = jest.fn(async () => []);
service.countCreditsIfFirst = jest.fn(async () => {});
service.maybeFinalizeRunTraceRoot = jest.fn(async () => {});
service.schedulePlannedTasks = jest.fn(async () => {});
service.drainPendingCheckpointReentries = jest.fn(async () => {});
@ -1218,6 +1222,10 @@ describe('InstanceAiService — agent tree snapshots', () => {
});
describe('InstanceAiService — terminal response guard wiring', () => {
beforeEach(() => {
jest.mocked(resumeAgentRun).mockReset();
});
it('publishes fallback output before run-finish on a silent completed run', () => {
const service = createTerminalGuardOrderService();
@ -1294,6 +1302,37 @@ describe('InstanceAiService — terminal response guard wiring', () => {
expect(service.eventBus.events.map((event) => event.type)).toEqual(['error', 'run-finish']);
expect(service.saveAgentTreeSnapshot).toHaveBeenCalledWith('thread-a', 'run-1', {});
});
it('counts credits when a resumed run completes', async () => {
const service = createTerminalGuardOrderService();
const abortController = new AbortController();
jest.mocked(resumeAgentRun).mockResolvedValueOnce({
status: 'completed',
mastraRunId: 'mastra-1',
text: Promise.resolve('done'),
workSummary: { toolCalls: [], totalToolCalls: 0, totalToolErrors: 0 },
});
await service.processResumedStream(
{},
{},
{
runId: 'run-1',
mastraRunId: 'mastra-1',
threadId: 'thread-a',
user: fakeUser,
toolCallId: 'tool-call-1',
signal: abortController.signal,
abortController,
snapshotStorage: {},
},
);
expect(service.countCreditsIfFirst).toHaveBeenCalledWith(fakeUser, 'thread-a', 'run-1');
expect(service.telemetry.track).toHaveBeenCalledWith('Builder satisfied user intent', {
thread_id: 'thread-a',
});
});
});
describe('InstanceAiService — AI temporary workflow cleanup', () => {

86
packages/cli/src/modules/instance-ai/__tests__/run-trace-metadata.test.ts Normal file
View File

@ -0,0 +1,86 @@
import type { InstanceAiEvent } from '@n8n/api-types';
import { buildInstanceAiRunTraceMetadata } from '../run-trace-metadata';
const baseEvent = {
runId: 'run-1',
agentId: 'agent-1',
};
describe('buildInstanceAiRunTraceMetadata', () => {
it('classifies a tool call followed by HITL as a contextless HITL first state', () => {
const events: InstanceAiEvent[] = [
{
type: 'run-start',
...baseEvent,
payload: { messageId: 'message-1' },
},
{
type: 'tool-call',
...baseEvent,
payload: { toolCallId: 'tool-1', toolName: 'credentials', args: { action: 'list' } },
},
{
type: 'confirmation-request',
...baseEvent,
payload: {
requestId: 'request-1',
toolCallId: 'tool-1',
toolName: 'credentials',
args: {},
severity: 'info',
message: 'Pick a credential',
},
},
];
expect(buildInstanceAiRunTraceMetadata(events, { status: 'cancelled' })).toMatchObject({
first_visible_state: 'contextless_hitl',
first_tool_name: 'credentials',
cancellation_type: 'explicit',
});
});
it('keeps the first tool name when assistant text appears first', () => {
const events: InstanceAiEvent[] = [
{
type: 'text-delta',
...baseEvent,
payload: { text: 'I will check the workflow first.' },
},
{
type: 'tool-call',
...baseEvent,
payload: { toolCallId: 'tool-1', toolName: 'workflows', args: { action: 'get' } },
},
];
expect(buildInstanceAiRunTraceMetadata(events, { status: 'completed' })).toEqual({
first_visible_state: 'assistant_text',
first_tool_name: 'workflows',
});
});
it('records timeout cancellation type and idle tail without extra fields', () => {
const metadata = buildInstanceAiRunTraceMetadata([], {
status: 'cancelled',
cancellationReason: 'timeout',
runTimeout: {
timedOut: true,
details: {
reason: 'idle_timeout',
surface: 'active-run',
timeoutMs: 600_000,
elapsedMs: 650_200,
idleMs: 606_400.4,
},
},
});
expect(metadata).toEqual({
first_visible_state: 'empty',
cancellation_type: 'idle_timeout',
idle_tail_ms: 606_400,
});
});
});

94
packages/cli/src/modules/instance-ai/instance-ai.service.ts
View File

@ -111,7 +111,12 @@ import { InstanceAiCompactionService } from './compaction.service';
import { ProxyTokenManager } from '@/services/proxy-token-manager';
import { InstanceAiThreadRepository } from './repositories/instance-ai-thread.repository';
import { TraceReplayState } from './trace-replay-state';
import { INSTANCE_AI_RUN_TIMEOUT_REASON, InstanceAiLivenessService } from './liveness';
import {
INSTANCE_AI_RUN_TIMEOUT_REASON,
InstanceAiLivenessService,
type InstanceAiConsumedRunTimeout,
} from './liveness';
import { buildInstanceAiRunTraceMetadata } from './run-trace-metadata';
function getErrorMessage(error: unknown): string {
return error instanceof Error ? error.message : String(error);
@ -902,6 +907,32 @@ export class InstanceAiService {
await this.finalizeMessageTraceRoot(runId, tracing, options);
}
private buildMessageTraceMetadata(
threadId: string,
runId: string,
options: {
status: MessageTraceFinalization['status'];
cancellationReason?: string;
runTimeout?: InstanceAiConsumedRunTimeout;
},
): Record<string, unknown> {
const traceOptions = {
status: options.status,
...(options.cancellationReason !== undefined
? { cancellationReason: options.cancellationReason }
: {}),
...(options.runTimeout !== undefined ? { runTimeout: options.runTimeout } : {}),
};
return {
completion_source: 'orchestrator',
...buildInstanceAiRunTraceMetadata(
this.eventBus.getEventsForRun(threadId, runId),
traceOptions,
),
};
}
private async finalizeRemainingMessageTraceRoots(
threadId: string,
options: MessageTraceFinalization,
@ -1854,7 +1885,9 @@ export class InstanceAiService {
return {
status: 'error',
reason: 'invalid_confirmation_payload',
metadata: { completion_source: 'orchestrator' },
metadata: this.buildMessageTraceMetadata(args.threadId, args.runId, {
status: 'error',
}),
};
}
@ -3123,7 +3156,7 @@ export class InstanceAiService {
status: finalStatus,
outputText,
modelId,
metadata: { completion_source: 'orchestrator' },
metadata: this.buildMessageTraceMetadata(threadId, runId, { status: finalStatus }),
};
const archivedWorkflowIds = await this.reapAiTemporaryFromRun(
threadId,
@ -3149,7 +3182,8 @@ export class InstanceAiService {
}
} catch (error) {
if (signal.aborted) {
const cancellationReason = this.liveness.consumeRunTimedOut(runId)
const runTimeout = this.liveness.consumeRunTimeout(runId);
const cancellationReason = runTimeout.timedOut
? INSTANCE_AI_RUN_TIMEOUT_REASON
: getAbortReason(signal);
if (cancellationReason === INSTANCE_AI_RUN_TIMEOUT_REASON) {
@ -3166,7 +3200,11 @@ export class InstanceAiService {
messageTraceFinalization = {
status: 'cancelled',
reason: cancellationReason,
metadata: { completion_source: 'orchestrator' },
metadata: this.buildMessageTraceMetadata(threadId, runId, {
status: 'cancelled',
cancellationReason,
runTimeout,
}),
};
const archivedWorkflowIds = await this.reapAiTemporaryFromRun(
threadId,
@ -3206,7 +3244,7 @@ export class InstanceAiService {
messageTraceFinalization = {
status: 'error',
reason: errorMessage,
metadata: { completion_source: 'orchestrator' },
metadata: this.buildMessageTraceMetadata(threadId, runId, { status: 'error' }),
};
const archivedWorkflowIds = await this.reapAiTemporaryFromRun(
@ -3237,6 +3275,9 @@ export class InstanceAiService {
this.domainAccessTrackersByThread.get(threadId)?.clearRun(runId);
if (messageTraceFinalization) {
await this.maybeFinalizeRunTraceRoot(runId, messageTraceFinalization);
if (messageTraceFinalization.status !== 'cancelled') {
this.liveness.consumeRunTimeout(runId);
}
}
// Clean up Mastra workflow snapshots unless the run is suspended (needed for resume).
// Mastra only persists snapshots on suspension and never deletes them on completion.
@ -3700,7 +3741,9 @@ export class InstanceAiService {
messageTraceFinalization = {
status: finalStatus,
outputText,
metadata: { completion_source: 'orchestrator' },
metadata: this.buildMessageTraceMetadata(opts.threadId, opts.runId, {
status: finalStatus,
}),
};
const archivedWorkflowIds = await this.reapAiTemporaryFromRun(
opts.threadId,
@ -3712,6 +3755,7 @@ export class InstanceAiService {
});
if (result.status === 'completed') {
await this.countCreditsIfFirst(opts.user, opts.threadId, opts.runId);
this.telemetry.track('Builder sent message', {
thread_id: opts.threadId,
message: outputText,
@ -3723,17 +3767,28 @@ export class InstanceAiService {
} catch (error) {
if (opts.signal.aborted) {
const messageGroupId = this.traceContextsByRunId.get(opts.runId)?.messageGroupId;
const runTimeout = this.liveness.consumeRunTimeout(opts.runId);
const cancellationReason = runTimeout.timedOut
? INSTANCE_AI_RUN_TIMEOUT_REASON
: getAbortReason(opts.signal);
if (cancellationReason === INSTANCE_AI_RUN_TIMEOUT_REASON) {
this.liveness.publishRunTimeoutNotice(opts.threadId, opts.runId);
}
this.evaluateTerminalResponse(opts.threadId, opts.runId, 'cancelled', {
messageGroupId,
});
await this.finalizeRunTracing(opts.runId, opts.tracing, {
status: 'cancelled',
reason: 'user_cancelled',
reason: cancellationReason,
});
messageTraceFinalization = {
status: 'cancelled',
reason: 'user_cancelled',
metadata: { completion_source: 'orchestrator' },
reason: cancellationReason,
metadata: this.buildMessageTraceMetadata(opts.threadId, opts.runId, {
status: 'cancelled',
cancellationReason,
runTimeout,
}),
};
const archivedWorkflowIds = await this.reapAiTemporaryFromRun(
opts.threadId,
@ -3744,7 +3799,7 @@ export class InstanceAiService {
opts.threadId,
opts.runId,
'cancelled',
'user_cancelled',
cancellationReason,
archivedWorkflowIds,
);
await this.saveAgentTreeSnapshot(opts.threadId, opts.runId, opts.snapshotStorage);
@ -3771,7 +3826,9 @@ export class InstanceAiService {
messageTraceFinalization = {
status: 'error',
reason: errorMessage,
metadata: { completion_source: 'orchestrator' },
metadata: this.buildMessageTraceMetadata(opts.threadId, opts.runId, {
status: 'error',
}),
};
const archivedWorkflowIds = await this.reapAiTemporaryFromRun(
@ -3796,6 +3853,9 @@ export class InstanceAiService {
// post-run planned-task dispatch.
if (messageTraceFinalization) {
await this.maybeFinalizeRunTraceRoot(opts.runId, messageTraceFinalization);
if (messageTraceFinalization.status !== 'cancelled') {
this.liveness.consumeRunTimeout(opts.runId);
}
}
// Post-run planned-task wiring — mirror the executeRun finally.
// Resumed ordinary-chat runs also need to drive the scheduler in case
@ -4137,6 +4197,10 @@ export class InstanceAiService {
suspended: SuspendedRunState<User>,
reason = 'user_cancelled',
): Promise<void> {
const runTimeout =
reason === INSTANCE_AI_RUN_TIMEOUT_REASON
? this.liveness.consumeRunTimeout(suspended.runId)
: undefined;
if (reason === INSTANCE_AI_RUN_TIMEOUT_REASON) {
this.liveness.publishRunTimeoutNotice(suspended.threadId, suspended.runId);
}
@ -4172,7 +4236,11 @@ export class InstanceAiService {
await this.maybeFinalizeRunTraceRoot(suspended.runId, {
status: 'cancelled',
reason,
metadata: { completion_source: 'orchestrator' },
metadata: this.buildMessageTraceMetadata(suspended.threadId, suspended.runId, {
status: 'cancelled',
cancellationReason: reason,
...(runTimeout ? { runTimeout } : {}),
}),
});
}

1
packages/cli/src/modules/instance-ai/liveness/index.ts
View File

@ -2,6 +2,7 @@ export {
INSTANCE_AI_RUN_TIMEOUT_REASON,
InstanceAiLivenessService,
type InstanceAiLivenessBackgroundTasks,
type InstanceAiConsumedRunTimeout,
type InstanceAiLivenessEventBus,
type InstanceAiLivenessLogger,
type InstanceAiLivenessPendingConfirmation,

52
packages/cli/src/modules/instance-ai/liveness/instance-ai-liveness.service.ts
View File

@ -2,6 +2,8 @@ import type { InstanceAiEvent } from '@n8n/api-types';
import { Time } from '@n8n/constants';
import type { InstanceAiLivenessPolicy, InstanceAiLivenessTimeoutReason } from '@n8n/instance-ai';
import type { InstanceAiRunTimeoutDetails } from '../run-timeout-details';
const ORCHESTRATOR_AGENT_ID = 'agent-001';
export const INSTANCE_AI_RUN_TIMEOUT_REASON = 'timeout';
@ -39,6 +41,14 @@ export type InstanceAiLivenessSweepResult = {
activeThreadIds: string[];
suspendedThreadIds: string[];
confirmationRequestIds: string[];
activeTimeouts?: Record<string, InstanceAiRunTimeoutDetails>;
suspendedTimeouts?: Record<string, InstanceAiRunTimeoutDetails>;
confirmationTimeouts?: Record<string, InstanceAiRunTimeoutDetails>;
};
export type InstanceAiConsumedRunTimeout = {
timedOut: boolean;
details?: InstanceAiRunTimeoutDetails;
};
export type InstanceAiLivenessRunState<
@ -86,7 +96,7 @@ export class InstanceAiLivenessService<
> {
private timeoutInterval?: NodeJS.Timeout;
private readonly timedOutRunIds = new Set<string>();
private readonly timedOutRunIds = new Map<string, InstanceAiRunTimeoutDetails | undefined>();
private readonly timedOutActiveRunThreads = new Set<string>();
@ -117,8 +127,8 @@ export class InstanceAiLivenessService<
this.timedOutActiveRunThreads.delete(threadId);
}
markRunTimedOut(runId: string): void {
this.timedOutRunIds.add(runId);
markRunTimedOut(runId: string, details?: InstanceAiRunTimeoutDetails): void {
this.timedOutRunIds.set(runId, details);
}
consumeRunTimedOut(runId: string): boolean {
@ -127,22 +137,37 @@ export class InstanceAiLivenessService<
return timedOut;
}
consumeRunTimeout(runId: string): InstanceAiConsumedRunTimeout {
const timedOut = this.timedOutRunIds.has(runId);
if (!timedOut) return { timedOut: false };
const details = this.timedOutRunIds.get(runId);
this.timedOutRunIds.delete(runId);
return details ? { timedOut: true, details } : { timedOut: true };
}
hasTimedOutActiveRunThread(threadId: string): boolean {
return this.timedOutActiveRunThreads.has(threadId);
}
async sweepTimedOutWork(now = Date.now()): Promise<void> {
const { activeThreadIds, suspendedThreadIds, confirmationRequestIds } =
this.options.runState.sweepTimedOut(this.options.policy, now);
const {
activeThreadIds,
suspendedThreadIds,
confirmationRequestIds,
activeTimeouts,
suspendedTimeouts,
confirmationTimeouts,
} = this.options.runState.sweepTimedOut(this.options.policy, now);
for (const threadId of activeThreadIds) {
this.options.logger.debug('Cancelling timed-out active run', { threadId });
this.cancelTimedOutActiveRun(threadId);
this.cancelTimedOutActiveRun(threadId, activeTimeouts?.[threadId]);
}
for (const threadId of suspendedThreadIds) {
this.options.logger.debug('Auto-rejecting timed-out suspended run', { threadId });
this.cancelTimedOutSuspendedRun(threadId);
this.cancelTimedOutSuspendedRun(threadId, suspendedTimeouts?.[threadId]);
}
for (const reqId of confirmationRequestIds) {
@ -152,7 +177,10 @@ export class InstanceAiLivenessService<
const pending = this.options.runState.getPendingConfirmation(reqId);
if (pending) {
const runId = this.options.runState.getActiveRunId(pending.threadId);
if (runId) this.publishRunTimeoutNotice(pending.threadId, runId);
if (runId) {
this.markRunTimedOut(runId, confirmationTimeouts?.[reqId]);
this.publishRunTimeoutNotice(pending.threadId, runId);
}
}
this.options.runState.rejectPendingConfirmation(reqId);
}
@ -177,21 +205,21 @@ export class InstanceAiLivenessService<
}
}
cancelTimedOutActiveRun(threadId: string): void {
cancelTimedOutActiveRun(threadId: string, details?: InstanceAiRunTimeoutDetails): void {
const active = this.options.runState.cancelActiveRun(threadId);
if (!active) return;
this.markRunTimedOut(active.runId);
this.markRunTimedOut(active.runId, details);
this.timedOutActiveRunThreads.add(threadId);
this.publishRunTimeoutNotice(threadId, active.runId);
active.abortController.abort();
}
cancelTimedOutSuspendedRun(threadId: string): void {
cancelTimedOutSuspendedRun(threadId: string, details?: InstanceAiRunTimeoutDetails): void {
const suspended = this.options.runState.cancelSuspendedRun(threadId);
if (!suspended) return;
this.markRunTimedOut(suspended.runId);
this.markRunTimedOut(suspended.runId, details);
suspended.abortController.abort();
this.options.finalizeCancelledSuspendedRun(suspended, INSTANCE_AI_RUN_TIMEOUT_REASON);
}

9
packages/cli/src/modules/instance-ai/run-timeout-details.ts Normal file
View File

@ -0,0 +1,9 @@
import type { InstanceAiLivenessSurface, InstanceAiLivenessTimeoutReason } from '@n8n/instance-ai';
export type InstanceAiRunTimeoutDetails = {
reason: InstanceAiLivenessTimeoutReason;
surface: InstanceAiLivenessSurface;
timeoutMs: number;
elapsedMs: number;
idleMs: number;
};

112
packages/cli/src/modules/instance-ai/run-trace-metadata.ts Normal file
View File

@ -0,0 +1,112 @@
import type { InstanceAiEvent } from '@n8n/api-types';
import type { InstanceAiLivenessTimeoutReason } from '@n8n/instance-ai';
import type { InstanceAiRunTimeoutDetails } from './run-timeout-details';
const RUN_TIMEOUT_REASON = 'timeout';
export type InstanceAiFirstVisibleState =
| 'assistant_text'
| 'contextless_hitl'
| 'tool_call'
| 'task_card'
| 'empty';
export type InstanceAiCancellationType = 'explicit' | InstanceAiLivenessTimeoutReason;
export type InstanceAiRunTimeoutTraceContext = {
timedOut: boolean;
details?: InstanceAiRunTimeoutDetails;
};
export type InstanceAiRunTraceMetadataOptions = {
status: 'completed' | 'cancelled' | 'error';
cancellationReason?: string;
runTimeout?: InstanceAiRunTimeoutTraceContext;
};
type FirstVisibleSummary = {
state: InstanceAiFirstVisibleState;
firstToolName?: string;
};
function withFirstToolName(
state: InstanceAiFirstVisibleState,
firstToolName: string | undefined,
): FirstVisibleSummary {
return firstToolName ? { state, firstToolName } : { state };
}
function getFirstToolName(events: InstanceAiEvent[]): string | undefined {
for (const event of events) {
if (event.type === 'tool-call') return event.payload.toolName;
}
return undefined;
}
function getFirstVisibleSummary(events: InstanceAiEvent[]): FirstVisibleSummary {
const firstToolName = getFirstToolName(events);
let sawToolCall = false;
for (const event of events) {
if (event.type === 'tool-call') {
sawToolCall = true;
continue;
}
if (event.type === 'confirmation-request') {
return {
state: 'contextless_hitl',
firstToolName: firstToolName ?? event.payload.toolName,
};
}
if (event.type === 'text-delta' && event.payload.text.trim().length > 0) {
return withFirstToolName('assistant_text', firstToolName);
}
if (event.type === 'agent-spawned' || event.type === 'tasks-update') {
return withFirstToolName('task_card', firstToolName);
}
}
return withFirstToolName(sawToolCall ? 'tool_call' : 'empty', firstToolName);
}
function getCancellationType(
options: InstanceAiRunTraceMetadataOptions,
): InstanceAiCancellationType | undefined {
if (options.status !== 'cancelled') return undefined;
if (options.runTimeout?.timedOut || options.cancellationReason === RUN_TIMEOUT_REASON) {
return options.runTimeout?.details?.reason ?? 'idle_timeout';
}
return 'explicit';
}
export function buildInstanceAiRunTraceMetadata(
events: InstanceAiEvent[],
options: InstanceAiRunTraceMetadataOptions,
): Record<string, unknown> {
const firstVisible = getFirstVisibleSummary(events);
const metadata: Record<string, unknown> = {
first_visible_state: firstVisible.state,
};
if (firstVisible.firstToolName) {
metadata.first_tool_name = firstVisible.firstToolName;
}
const cancellationType = getCancellationType(options);
if (cancellationType) {
metadata.cancellation_type = cancellationType;
}
if (options.runTimeout?.details?.idleMs !== undefined) {
metadata.idle_tail_ms = Math.round(options.runTimeout.details.idleMs);
}
return metadata;
}

2
packages/cli/src/modules/mcp/__tests__/mcp-oauth-token.service.test.ts
View File

@ -261,7 +261,7 @@ describe('McpOAuthTokenService', () => {
const result = await service.verifyOAuthAccessToken(accessToken);
expect(result).toEqual({ user });
expect(result).toEqual({ user, authType: 'oauth' });
expect(userRepository.findOne).toHaveBeenCalledWith({
where: { id: userId },
relations: ['role'],

24
packages/cli/src/modules/mcp/__tests__/mcp-server-middleware.service.test.ts
View File

@ -58,11 +58,11 @@ describe('McpServerMiddlewareService', () => {
meta: { isOAuth: true },
});
oauthTokenService.verifyOAuthAccessToken.mockResolvedValue({ user });
oauthTokenService.verifyOAuthAccessToken.mockResolvedValue({ user, authType: 'oauth' });
const result = await service.getUserForToken(oauthToken);
expect(result).toEqual({ user });
expect(result).toEqual({ user, authType: 'oauth' });
expect(oauthTokenService.verifyOAuthAccessToken).toHaveBeenCalledWith(oauthToken);
expect(mcpServerApiKeyService.verifyApiKey).not.toHaveBeenCalled();
});
@ -74,11 +74,11 @@ describe('McpServerMiddlewareService', () => {
aud: 'mcp-server-api',
});
mcpServerApiKeyService.verifyApiKey.mockResolvedValue({ user });
mcpServerApiKeyService.verifyApiKey.mockResolvedValue({ user, authType: 'api_key' });
const result = await service.getUserForToken(apiKeyToken);
expect(result).toEqual({ user });
expect(result).toEqual({ user, authType: 'api_key' });
expect(mcpServerApiKeyService.verifyApiKey).toHaveBeenCalledWith(apiKeyToken);
expect(oauthTokenService.verifyOAuthAccessToken).not.toHaveBeenCalled();
});
@ -91,11 +91,11 @@ describe('McpServerMiddlewareService', () => {
meta: { isOAuth: false },
});
mcpServerApiKeyService.verifyApiKey.mockResolvedValue({ user });
mcpServerApiKeyService.verifyApiKey.mockResolvedValue({ user, authType: 'api_key' });
const result = await service.getUserForToken(apiKeyToken);
expect(result).toEqual({ user });
expect(result).toEqual({ user, authType: 'api_key' });
expect(mcpServerApiKeyService.verifyApiKey).toHaveBeenCalledWith(apiKeyToken);
expect(oauthTokenService.verifyOAuthAccessToken).not.toHaveBeenCalled();
});
@ -234,13 +234,15 @@ describe('McpServerMiddlewareService', () => {
const res = mockDeep<Response>();
const next = jest.fn() as NextFunction;
oauthTokenService.verifyOAuthAccessToken.mockResolvedValue({ user });
oauthTokenService.verifyOAuthAccessToken.mockResolvedValue({ user, authType: 'oauth' });
const middleware = service.getAuthMiddleware();
await middleware(req, res, next);
const authenticatedReq = req as Request & { user?: User; mcpAuthType?: 'oauth' };
expect((req as any).user).toEqual(user);
expect(authenticatedReq.user).toEqual(user);
expect(authenticatedReq.mcpAuthType).toBe('oauth');
expect(next).toHaveBeenCalled();
expect(res.status).not.toHaveBeenCalled();
});
@ -256,13 +258,15 @@ describe('McpServerMiddlewareService', () => {
const res = mockDeep<Response>();
const next = jest.fn() as NextFunction;
mcpServerApiKeyService.verifyApiKey.mockResolvedValue({ user });
mcpServerApiKeyService.verifyApiKey.mockResolvedValue({ user, authType: 'api_key' });
const middleware = service.getAuthMiddleware();
await middleware(req, res, next);
const authenticatedReq = req as Request & { user?: User; mcpAuthType?: 'api_key' };
expect((req as any).user).toEqual(user);
expect(authenticatedReq.user).toEqual(user);
expect(authenticatedReq.mcpAuthType).toBe('api_key');
expect(next).toHaveBeenCalled();
expect(res.status).not.toHaveBeenCalled();
});

41
packages/cli/src/modules/mcp/__tests__/mcp.controller.test.ts
View File

@ -19,6 +19,8 @@ Container.set(McpServerMiddlewareService, mcpServerMiddlewareService);
import { McpController, type FlushableResponse } from '../mcp.controller';
import { McpService } from '../mcp.service';
import { McpSettingsService } from '../mcp.settings.service';
import { Telemetry } from '@/telemetry';
import type { UserConnectedToMCPEventPayload } from '../mcp.types';
const mockHandleRequest = jest.fn().mockResolvedValue(undefined);
jest.mock('@modelcontextprotocol/sdk/server/streamableHttp.js', () => {
@ -29,8 +31,12 @@ jest.mock('@modelcontextprotocol/sdk/server/streamableHttp.js', () => {
return { StreamableHTTPServerTransport };
});
const createReq = (overrides: Partial<AuthenticatedRequest> = {}): AuthenticatedRequest =>
({ user: { id: 'user-1' }, body: {}, ...overrides }) as unknown as AuthenticatedRequest;
type AuthenticatedMcpRequest = AuthenticatedRequest & {
mcpAuthType?: UserConnectedToMCPEventPayload['auth_type'];
};
const createReq = (overrides: Partial<AuthenticatedMcpRequest> = {}): AuthenticatedMcpRequest =>
({ user: { id: 'user-1' }, body: {}, ...overrides }) as unknown as AuthenticatedMcpRequest;
const createRes = (): FlushableResponse => {
const res = mock<FlushableResponse>();
@ -42,6 +48,7 @@ const createRes = (): FlushableResponse => {
describe('McpController', () => {
let controller: McpController;
const logger = mock<Logger>();
const telemetry = { track: jest.fn() } as unknown as Telemetry;
const mcpService = { getServer: jest.fn() } as unknown as McpService;
const mcpSettingsService = { getEnabled: jest.fn() } as unknown as McpSettingsService;
@ -49,6 +56,7 @@ describe('McpController', () => {
jest.clearAllMocks();
Container.set(Logger, logger);
Container.set(Telemetry, telemetry);
Container.set(McpService, mcpService);
Container.set(McpSettingsService, mcpSettingsService);
@ -75,6 +83,35 @@ describe('McpController', () => {
expect(mcpService.getServer as unknown as jest.Mock).toHaveBeenCalled();
});
test('tracks successful initialize connections with auth type', async () => {
(mcpSettingsService.getEnabled as jest.Mock).mockResolvedValue(true);
(mcpService.getServer as unknown as jest.Mock).mockReturnValue({
connect: jest.fn().mockResolvedValue(undefined),
close: jest.fn().mockResolvedValue(undefined),
});
const res = createRes();
await controller.build(
createReq({
mcpAuthType: 'oauth',
body: {
jsonrpc: '2.0',
method: 'initialize',
params: { clientInfo: { name: 'Claude', version: '1.0.0' } },
},
}),
res,
);
expect(telemetry.track).toHaveBeenCalledWith('User connected to MCP server', {
user_id: 'user-1',
client_name: 'Claude',
client_version: '1.0.0',
auth_type: 'oauth',
mcp_connection_status: 'success',
});
});
test('HEAD /http returns 401 with WWW-Authenticate header for auth scheme discovery', async () => {
const req = {} as Request;
const res = createRes();

1
packages/cli/src/modules/mcp/mcp-api-key.service.ts
View File

@ -78,6 +78,7 @@ export class McpServerApiKeyService {
return {
user: tokenGrant.actor ?? tokenGrant.subject,
actor: tokenGrant.actor,
authType: 'api_key',
};
}

2
packages/cli/src/modules/mcp/mcp-oauth-token.service.ts
View File

@ -188,7 +188,7 @@ export class McpOAuthTokenService {
return { user: null, context: { reason: 'user_not_found', auth_type: 'oauth' } };
}
return { user };
return { user, authType: 'oauth' };
} catch (error) {
const errorForSure = ensureError(error);
const reason =

2
packages/cli/src/modules/mcp/mcp-server-middleware.service.ts
View File

@ -92,6 +92,8 @@ export class McpServerMiddlewareService {
}
(req as AuthenticatedRequest).user = user;
(req as AuthenticatedRequest & { mcpAuthType?: UserWithContext['authType'] }).mcpAuthType =
result.authType;
next();
};

3
packages/cli/src/modules/mcp/mcp.controller.ts
View File

@ -123,6 +123,9 @@ export class McpController {
user_id: req.user.id,
client_name: clientInfo?.name,
client_version: clientInfo?.version,
auth_type: (
req as AuthenticatedRequest & { mcpAuthType?: UserConnectedToMCPEventPayload['auth_type'] }
).mcpAuthType,
};
// Deny if MCP access is disabled

2
packages/cli/src/modules/mcp/mcp.types.ts
View File

@ -71,6 +71,7 @@ export type UserConnectedToMCPEventPayload = {
user_id?: string;
client_name?: string;
client_version?: string;
auth_type?: Mcpauth_type;
mcp_connection_status: 'success' | 'error';
error?: string;
};
@ -128,4 +129,5 @@ export type UserWithContext = {
user: User | null;
actor?: User;
context?: TelemetryAuthContext;
authType?: Mcpauth_type;
};

106
packages/cli/src/modules/source-control.ee/__tests__/source-control-import.service.ee.test.ts
View File

@ -2746,7 +2746,7 @@ describe('SourceControlImportService', () => {
id: 'dt1',
name: 'Test Table 1',
projectId: 'project1',
columns: [{ id: 'col1', name: 'Column 1', type: 'string', index: 0 }],
columns: [{ id: 'col1', name: 'Column1', type: 'string', index: 0 }],
createdAt: '2024-01-01T00:00:00.000Z',
updatedAt: '2024-01-02T00:00:00.000Z',
};
@ -2754,7 +2754,7 @@ describe('SourceControlImportService', () => {
id: 'dt2',
name: 'Test Table 2',
projectId: 'project2',
columns: [{ id: 'col2', name: 'Column 2', type: 'number', index: 0 }],
columns: [{ id: 'col2', name: 'Column2', type: 'number', index: 0 }],
createdAt: '2024-01-03T00:00:00.000Z',
updatedAt: '2024-01-04T00:00:00.000Z',
};
@ -2824,7 +2824,7 @@ describe('SourceControlImportService', () => {
id: 'dt1',
name: 'Test Table',
projectId: 'project1',
columns: [{ id: 'col1', name: 'Column 1', type: 'string', index: 0 }],
columns: [{ id: 'col1', name: 'Column1', type: 'string', index: 0 }],
createdAt: new Date('2024-01-01'),
updatedAt: new Date('2024-01-02'),
project: {
@ -2851,7 +2851,7 @@ describe('SourceControlImportService', () => {
projectId: 'project1',
projectName: 'Team Project 1',
},
columns: [{ id: 'col1', name: 'Column 1', type: 'string', index: 0 }],
columns: [{ id: 'col1', name: 'Column1', type: 'string', index: 0 }],
filename: expect.stringContaining('dt1.json'),
createdAt: '2024-01-01T00:00:00.000Z',
updatedAt: '2024-01-02T00:00:00.000Z',
@ -2960,7 +2960,7 @@ describe('SourceControlImportService', () => {
teamId: 'project1',
teamName: 'Team Project 1',
},
columns: [{ id: 'col1', name: 'Column 1', type: 'string', index: 0 }],
columns: [{ id: 'col1', name: 'Column1', type: 'string', index: 0 }],
createdAt: '2024-01-01T00:00:00.000Z',
updatedAt: '2024-01-02T00:00:00.000Z',
};
@ -3003,7 +3003,7 @@ describe('SourceControlImportService', () => {
projectName: 'User Name',
personalEmail: 'user@example.com',
},
columns: [{ id: 'col1', name: 'Column 1', type: 'string', index: 0 }],
columns: [{ id: 'col1', name: 'Column1', type: 'string', index: 0 }],
createdAt: '2024-01-01T00:00:00.000Z',
updatedAt: '2024-01-02T00:00:00.000Z',
};
@ -3041,8 +3041,8 @@ describe('SourceControlImportService', () => {
teamName: 'Team Project 1',
},
columns: [
{ id: 'col1', name: 'Column 1', type: 'string', index: 0 },
{ id: 'col2', name: 'Column 2', type: 'number', index: 1 },
{ id: 'col1', name: 'Column1', type: 'string', index: 0 },
{ id: 'col2', name: 'Column2', type: 'number', index: 1 },
],
createdAt: '2024-01-01T00:00:00.000Z',
updatedAt: '2024-01-02T00:00:00.000Z',
@ -3052,12 +3052,12 @@ describe('SourceControlImportService', () => {
id: 'dt1',
name: 'Old Name',
projectId: 'project1',
columns: [{ id: 'col1', name: 'Column 1' }],
columns: [{ id: 'col1', name: 'Column1' }],
};
fsReadFile.mockResolvedValue(JSON.stringify(mockDataTable) as any);
dataTableRepository.findOne.mockResolvedValue(existingTable as any);
dataTableColumnRepository.find.mockResolvedValue([{ id: 'col1', name: 'Column 1' }] as any);
dataTableColumnRepository.find.mockResolvedValue([{ id: 'col1', name: 'Column1' }] as any);
dataTableColumnRepository.save.mockImplementation(async (col: any) => col);
projectRepository.findOne.mockResolvedValue({ id: 'project1', type: 'team' } as any);
@ -3126,7 +3126,7 @@ describe('SourceControlImportService', () => {
teamId: 'project1',
teamName: 'Team Project 1',
},
columns: [{ id: 'col1', name: 'Column 1', type: 'string', index: 0 }],
columns: [{ id: 'col1', name: 'Column1', type: 'string', index: 0 }],
createdAt: '2024-01-01T00:00:00.000Z',
updatedAt: '2024-01-02T00:00:00.000Z',
};
@ -3136,16 +3136,16 @@ describe('SourceControlImportService', () => {
name: 'Test Table',
projectId: 'project1',
columns: [
{ id: 'col1', name: 'Column 1' },
{ id: 'col2', name: 'Column 2' },
{ id: 'col1', name: 'Column1' },
{ id: 'col2', name: 'Column2' },
],
};
fsReadFile.mockResolvedValue(JSON.stringify(mockDataTable) as any);
dataTableRepository.findOne.mockResolvedValue(existingTable as any);
dataTableColumnRepository.find.mockResolvedValue([
{ id: 'col1', name: 'Column 1' },
{ id: 'col2', name: 'Column 2' },
{ id: 'col1', name: 'Column1' },
{ id: 'col2', name: 'Column2' },
] as any);
dataTableColumnRepository.save.mockResolvedValue({ id: 'col1' } as any);
projectRepository.findOne.mockResolvedValue({ id: 'project1', type: 'team' } as any);
@ -3156,7 +3156,7 @@ describe('SourceControlImportService', () => {
// Assert
expect(dataTableDDLService.dropColumnFromTable).toHaveBeenCalledWith(
'dt1',
'Column 2',
'Column2',
'sqlite',
expect.anything(),
);
@ -3183,7 +3183,7 @@ describe('SourceControlImportService', () => {
teamId: 'project1',
teamName: 'Team Project 1',
},
columns: [{ id: 'col1', name: 'Column 1', type: 'string', index: 0 }],
columns: [{ id: 'col1', name: 'Column1', type: 'string', index: 0 }],
createdAt: '2024-01-01T00:00:00.000Z',
updatedAt: '2024-01-02T00:00:00.000Z',
};
@ -3204,13 +3204,81 @@ describe('SourceControlImportService', () => {
expect(dataTableRepository.upsert).not.toHaveBeenCalled();
});
it('should skip columns with invalid names', async () => {
// Arrange
const mockDataTable = {
id: 'dt1',
name: 'Test Table',
ownedBy: {
type: 'team',
teamId: 'project1',
teamName: 'Team Project 1',
},
columns: [
{ id: 'col1', name: 'validName', type: 'string', index: 0 },
{
id: 'col2',
name: 'invalid" text); create INVALID table; --',
type: 'string',
index: 1,
},
],
createdAt: '2024-01-01T00:00:00.000Z',
updatedAt: '2024-01-02T00:00:00.000Z',
};
fsReadFile.mockResolvedValue(JSON.stringify(mockDataTable) as any);
dataTableRepository.findOne.mockResolvedValue(null);
dataTableColumnRepository.find.mockResolvedValue([]);
dataTableColumnRepository.save.mockImplementation(async (col: any) => col);
projectRepository.findOne.mockResolvedValue({ id: 'project1', type: 'team' } as any);
// Act
await service.importDataTablesFromWorkFolder([mockCandidate], mockUser.id);
// Assert
expect(dataTableDDLService.createTableWithColumns).toHaveBeenCalledWith(
'dt1',
expect.arrayContaining([expect.objectContaining({ id: 'col1', name: 'validName' })]),
expect.anything(),
);
const columns = (dataTableDDLService.createTableWithColumns as jest.Mock).mock.calls[0][1];
expect(columns).not.toEqual(
expect.arrayContaining([expect.objectContaining({ id: 'col2' })]),
);
});
it('should skip data tables with invalid IDs', async () => {
// Arrange
const mockDataTable = {
id: 'invalid"; create INVALID table;--',
name: 'Test Table',
ownedBy: {
type: 'team',
teamId: 'project1',
teamName: 'Team Project 1',
},
columns: [{ id: 'col1', name: 'validName', type: 'string', index: 0 }],
createdAt: '2024-01-01T00:00:00.000Z',
updatedAt: '2024-01-02T00:00:00.000Z',
};
fsReadFile.mockResolvedValue(JSON.stringify(mockDataTable) as any);
// Act
await service.importDataTablesFromWorkFolder([mockCandidate], mockUser.id);
// Assert
expect(dataTableRepository.upsert).not.toHaveBeenCalled();
});
it('should not partially import when a name collision exists among multiple tables', async () => {
// Arrange — two tables: dt1 is valid, dt2 has a name collision
const validTable = {
id: 'dt1',
name: 'Valid Table',
ownedBy: { type: 'team', teamId: 'project1', teamName: 'Team Project 1' },
columns: [{ id: 'col1', name: 'Column 1', type: 'string', index: 0 }],
columns: [{ id: 'col1', name: 'Column1', type: 'string', index: 0 }],
createdAt: '2024-01-01T00:00:00.000Z',
updatedAt: '2024-01-02T00:00:00.000Z',
};
@ -3218,7 +3286,7 @@ describe('SourceControlImportService', () => {
id: 'dt2',
name: 'Colliding Table',
ownedBy: { type: 'team', teamId: 'project1', teamName: 'Team Project 1' },
columns: [{ id: 'col2', name: 'Column 2', type: 'string', index: 0 }],
columns: [{ id: 'col2', name: 'Column2', type: 'string', index: 0 }],
createdAt: '2024-01-01T00:00:00.000Z',
updatedAt: '2024-01-02T00:00:00.000Z',
};

15
packages/cli/src/modules/source-control.ee/source-control-import.service.ee.ts
View File

@ -46,6 +46,7 @@ import { DataTableColumn } from '@/modules/data-table/data-table-column.entity';
import { DataTableColumnRepository } from '@/modules/data-table/data-table-column.repository';
import { DataTableDDLService } from '@/modules/data-table/data-table-ddl.service';
import { DataTableRepository } from '@/modules/data-table/data-table.repository';
import { isValidColumnName, isValidDataTableId } from '@/modules/data-table/utils/sql-utils';
import { RedactionEnforcementService } from '@/modules/redaction/redaction-enforcement.service';
import { isUniqueConstraintError } from '@/response-helper';
import { TagService } from '@/services/tag.service';
@ -1239,6 +1240,13 @@ export class SourceControlImportService {
continue;
}
if (!isValidDataTableId(dataTable.id)) {
this.logger.warn(
`Invalid data table ID "${dataTable.id}" in file ${candidate.file}. Skipping.`,
);
continue;
}
let targetProject: Project | null = null;
if (dataTable.ownedBy) {
@ -1354,6 +1362,13 @@ export class SourceControlImportService {
// Upsert columns
const columnEntities = [];
for (const column of dataTable.columns) {
if (!isValidColumnName(column.name)) {
this.logger.warn(
`Invalid column name "${column.name}" in data table ${dataTable.name}. Skipping column.`,
);
continue;
}
if (!isValidDataTableColumnType(column.type)) {
this.logger.warn(
`Invalid column type "${column.type}" in data table ${dataTable.name}, column ${column.name}. Skipping column.`,

51
packages/cli/src/oauth/__tests__/oauth.service.test.ts
View File

@ -136,7 +136,7 @@ describe('OauthService', () => {
enumerable: true,
});
const promise = service.getCredential(req);
const promise = service.getCredentialForUpdate(req);
await expect(promise).rejects.toThrow(BadRequestError);
await expect(promise).rejects.toThrow('Required credential ID is missing');
});
@ -149,10 +149,10 @@ describe('OauthService', () => {
credentialsFinderService.findCredentialForUser.mockResolvedValue(null);
await expect(service.getCredential(req)).rejects.toThrow(NotFoundError);
await expect(service.getCredentialForUpdate(req)).rejects.toThrow(NotFoundError);
expect(logger.error).toHaveBeenCalledWith(
'OAuth credential authorization failed because the current user does not have the correct permissions',
{ userId: '123' },
{ userId: '123', credentialId: 'credential-id' },
);
});
@ -165,13 +165,13 @@ describe('OauthService', () => {
credentialsFinderService.findCredentialForUser.mockResolvedValue(mockCredential);
const result = await service.getCredential(req);
const result = await service.getCredentialForUpdate(req);
expect(result).toBe(mockCredential);
expect(credentialsFinderService.findCredentialForUser).toHaveBeenCalledWith(
'credential-id',
req.user,
['credential:read'],
['credential:update'],
);
});
});
@ -404,20 +404,28 @@ describe('OauthService', () => {
};
const encodedState = Buffer.from(JSON.stringify(state)).toString('base64');
cipher.decryptV2.mockResolvedValue(JSON.stringify(csrfData));
const mockCredential = mock<CredentialsEntity>({ id: 'credential-id' });
credentialsFinderService.findCredentialForUser.mockResolvedValue(mockCredential);
const req = mock<AuthenticatedRequest>({
user: mock<User>({ id: 'user-id' }),
});
const result = await (service as any).decodeCsrfState(encodedState, req);
const [decodedState, credential] = await (service as any).decodeCsrfState(encodedState, req);
expect(cipher.decryptV2).toHaveBeenCalledWith('encrypted-data');
expect(result).toMatchObject({
expect(decodedState).toMatchObject({
token: 'token',
createdAt: timestamp,
cid: 'credential-id',
userId: 'user-id',
origin: 'static-credential',
});
expect(credential).toBe(mockCredential);
expect(credentialsFinderService.findCredentialForUser).toHaveBeenCalledWith(
'credential-id',
req.user,
['credential:update'],
);
});
it('should throw error when state format is invalid', async () => {
@ -529,20 +537,24 @@ describe('OauthService', () => {
};
const encodedState = Buffer.from(JSON.stringify(state)).toString('base64');
cipher.decryptV2.mockResolvedValue(JSON.stringify(csrfData));
const mockCredential = mock<CredentialsEntity>({ id: 'credential-id' });
credentialsRepository.findOneBy.mockResolvedValue(mockCredential as any);
const req = mock<AuthenticatedRequest>({
user: mock<User>({ id: 'user-id' }),
});
const result = await (service as any).decodeCsrfState(encodedState, req);
const [decodedState, credential] = await (service as any).decodeCsrfState(encodedState, req);
expect(result).toMatchObject({
expect(decodedState).toMatchObject({
token: 'token',
createdAt: timestamp,
cid: 'credential-id',
userId: 'different-user-id',
origin: 'dynamic-credential',
});
expect(credential).toBe(mockCredential);
expect(cipher.decryptV2).toHaveBeenCalledWith('encrypted-data');
expect(credentialsFinderService.findCredentialForUser).not.toHaveBeenCalled();
});
it('should bypass user validation for dynamic-credential origin even when req.user is undefined', async () => {
@ -558,19 +570,22 @@ describe('OauthService', () => {
};
const encodedState = Buffer.from(JSON.stringify(state)).toString('base64');
cipher.decryptV2.mockResolvedValue(JSON.stringify(csrfData));
const mockCredential = mock<CredentialsEntity>({ id: 'credential-id' });
credentialsRepository.findOneBy.mockResolvedValue(mockCredential as any);
const req = mock<AuthenticatedRequest>({
user: undefined,
});
const result = await (service as any).decodeCsrfState(encodedState, req);
const [decodedState, credential] = await (service as any).decodeCsrfState(encodedState, req);
expect(result).toMatchObject({
expect(decodedState).toMatchObject({
token: 'token',
createdAt: timestamp,
cid: 'credential-id',
userId: 'user-id',
origin: 'dynamic-credential',
});
expect(credential).toBe(mockCredential);
expect(cipher.decryptV2).toHaveBeenCalledWith('encrypted-data');
});
@ -760,7 +775,7 @@ describe('OauthService', () => {
});
cipher.decryptV2.mockResolvedValue(JSON.stringify(csrfData));
credentialsRepository.findOneBy.mockResolvedValue(mockCredential);
credentialsFinderService.findCredentialForUser.mockResolvedValue(mockCredential);
jest.mocked(WorkflowExecuteAdditionalData.getBase).mockResolvedValue(mockAdditionalData);
credentialsHelper.getDecrypted.mockResolvedValue(mockDecryptedData);
credentialsHelper.applyDefaultsAndOverwrites.mockResolvedValue(mockOAuthCredentials);
@ -781,7 +796,7 @@ describe('OauthService', () => {
});
});
it('should throw UnexpectedError when credential is not found', async () => {
it('should throw NotFoundError when credential is not found', async () => {
const csrfData = {
cid: 'credential-id',
userId: 'user-id',
@ -800,12 +815,10 @@ describe('OauthService', () => {
user: mock<User>({ id: 'user-id' }),
});
credentialsRepository.findOneBy.mockResolvedValue(null);
credentialsFinderService.findCredentialForUser.mockResolvedValue(null);
await expect(service.resolveCredential(req)).rejects.toThrow(UnexpectedError);
await expect(service.resolveCredential(req)).rejects.toThrow(
'OAuth callback failed because of insufficient permissions',
);
await expect(service.resolveCredential(req)).rejects.toThrow(NotFoundError);
await expect(service.resolveCredential(req)).rejects.toThrow('Credential not found');
});
it('should throw UnexpectedError when CSRF state is invalid', async () => {
@ -832,7 +845,7 @@ describe('OauthService', () => {
user: mock<User>({ id: 'user-id' }),
});
credentialsRepository.findOneBy.mockResolvedValue(mockCredential);
credentialsFinderService.findCredentialForUser.mockResolvedValue(mockCredential);
jest.mocked(WorkflowExecuteAdditionalData.getBase).mockResolvedValue(mockAdditionalData);
credentialsHelper.getDecrypted.mockResolvedValue(mockDecryptedData);
credentialsHelper.applyDefaultsAndOverwrites.mockResolvedValue(mockOAuthCredentials);

40
packages/cli/src/oauth/oauth.service.ts
View File

@ -90,7 +90,7 @@ export class OauthService {
return `${restUrl}/oauth${oauthVersion}-credential`;
}
async getCredential(
async getCredentialForUpdate(
req: OAuthRequest.OAuth1Credential.Auth | OAuthRequest.OAuth2Credential.Auth,
): Promise<CredentialsEntity> {
const { id: credentialId } = req.query;
@ -102,13 +102,13 @@ export class OauthService {
const credential = await this.credentialsFinderService.findCredentialForUser(
credentialId,
req.user,
['credential:read'],
['credential:update'],
);
if (!credential) {
this.logger.error(
'OAuth credential authorization failed because the current user does not have the correct permissions',
{ userId: req.user.id },
{ userId: req.user.id, credentialId },
);
throw new NotFoundError(RESPONSE_ERROR_MESSAGES.NO_CREDENTIAL);
}
@ -250,7 +250,7 @@ export class OauthService {
protected async decodeCsrfState(
encodedState: string,
req: AuthenticatedRequest,
): Promise<CsrfState & CreateCsrfStateData> {
): Promise<[CsrfState & CreateCsrfStateData, CredentialsEntity | null]> {
const errorMessage = 'Invalid state format';
const decodedState = Buffer.from(encodedState, 'base64').toString();
const decoded = jsonParse<CsrfState>(decodedState, {
@ -270,28 +270,31 @@ export class OauthService {
// Dynamic credentials: skip user validation (e.g. embed/iframe flows) as they do not contain an n8n user
if (decryptedState.origin === 'dynamic-credential') {
return {
...decoded,
...decryptedState,
};
return [
{ ...decoded, ...decryptedState },
await this.getCredentialWithoutUser(decryptedState.cid),
];
}
// Static credentials: skip user validation only when N8N_SKIP_AUTH_ON_OAUTH_CALLBACK is true (e.g. embed/iframe)
if (skipAuthOnOAuthCallback) {
return {
...decoded,
...decryptedState,
};
return [
{ ...decoded, ...decryptedState },
await this.getCredentialWithoutUser(decryptedState.cid),
];
}
if (req.user?.id === undefined || decryptedState.userId !== req.user.id) {
throw new AuthError('Unauthorized');
}
return {
...decoded,
...decryptedState,
};
const credential = await this.credentialsFinderService.findCredentialForUser(
decryptedState.cid,
req.user,
['credential:update'],
);
return [{ ...decoded, ...decryptedState }, credential];
}
protected verifyCsrfState(
@ -313,10 +316,9 @@ export class OauthService {
[CredentialsEntity, ICredentialDataDecryptedObject, T, CsrfState & CreateCsrfStateData]
> {
const { state: encodedState } = req.query;
const state = await this.decodeCsrfState(encodedState, req);
const credential = await this.getCredentialWithoutUser(state.cid);
const [state, credential] = await this.decodeCsrfState(encodedState, req);
if (!credential) {
throw new UnexpectedError('OAuth callback failed because of insufficient permissions');
throw new NotFoundError(RESPONSE_ERROR_MESSAGES.NO_CREDENTIAL);
}
const additionalData = await this.getAdditionalData();

139
packages/cli/src/services/__tests__/export.service.test.ts
View File

@ -3,6 +3,7 @@ import { ExportService } from '../export.service';
import { type DataSource } from '@n8n/typeorm';
import { mkdir, rm, readdir, appendFile, readFile } from 'fs/promises';
import { mock } from 'jest-mock-extended';
import type * as JestMockExtended from 'jest-mock-extended';
import type { Cipher } from 'n8n-core';
// Mock fs/promises with proper implementations
@ -25,9 +26,16 @@ jest.mock('@/utils/validate-database-type', () => ({
}));
// Mock @n8n/db
jest.mock('@n8n/db', () => ({
DataSource: mock<DataSource>(),
}));
jest.mock('@n8n/db', () => {
// `mock` from jest-mock-extended is not yet initialised when this hoisted
// factory runs, so resolve it lazily via require.
const jestMockExtended = require('jest-mock-extended') as typeof JestMockExtended;
return {
DataSource: jestMockExtended.mock<DataSource>(),
// `sql-utils` imports `DslColumn`; provide a no-op so the import resolves.
DslColumn: class {},
};
});
describe('ExportService', () => {
let exportService: ExportService;
@ -182,7 +190,7 @@ describe('ExportService', () => {
await exportService.exportEntities(outputDir);
expect(mockDataSource.query).toHaveBeenCalledTimes(5); // 1 migrations + 3 entity queries
expect(mockDataSource.query).toHaveBeenCalledTimes(6); // 2 migrations + 1 data_table + 3 entity queries
expect(appendFile).toHaveBeenCalled();
});
@ -210,10 +218,34 @@ describe('ExportService', () => {
await exportService.exportEntities(outputDir, new Set(['execution_data']));
expect(mockDataSource.query).toHaveBeenCalledTimes(4); // 1 migrations + 3 entity queries
expect(mockDataSource.query).toHaveBeenCalledTimes(5); // 2 migrations + 1 data_table + 2 entity queries
expect(appendFile).toHaveBeenCalled();
});
it('should skip data-table row export when includeDataTableRows is false', async () => {
const outputDir = '/test/output';
jest.mocked(mockDataSource.query).mockImplementation(async () => []);
jest.mocked(readdir).mockResolvedValue([]);
await exportService.exportEntities(outputDir, undefined, undefined, {
includeDataTableRows: false,
});
// Without the data_table SELECT id query, query count drops by 1 vs the
// includeDataTableRows=true path. Verify by ensuring no data_table query was
// issued and the schema-only log message was emitted.
const dataTableQueries = jest
.mocked(mockDataSource.query)
.mock.calls.filter(
([sql]) => typeof sql === 'string' && /SELECT id FROM "data_table"/.test(sql),
);
expect(dataTableQueries).toHaveLength(0);
expect(mockLogger.info).toHaveBeenCalledWith(
expect.stringContaining('Skipping data-table row export'),
);
});
it('should clear existing files before export', async () => {
const outputDir = '/test/output';
const existingFiles = ['user.jsonl', 'user.2.jsonl', 'other.txt'];
@ -338,6 +370,103 @@ describe('ExportService', () => {
});
});
describe('exportDataTableUserTables', () => {
it('should silently skip when the data_table registry is missing', async () => {
jest.mocked(mockDataSource.query).mockImplementation(async (query: string) => {
if (query.includes('data_table') && !query.includes('data_table_column')) {
throw new Error('relation does not exist');
}
return [];
});
// @ts-expect-error accessing private method for testing
const result = await exportService.exportDataTableUserTables('/test/output');
expect(result).toEqual({ totalTables: 0, totalRows: 0 });
});
it('should skip when there are no data tables', async () => {
jest.mocked(mockDataSource.query).mockResolvedValueOnce([]); // SELECT id FROM data_table -> empty
// @ts-expect-error accessing private method for testing
const result = await exportService.exportDataTableUserTables('/test/output');
expect(result).toEqual({ totalTables: 0, totalRows: 0 });
expect(appendFile).not.toHaveBeenCalled();
});
it('should keyset-paginate user rows by id and write JSONL files', async () => {
const dataTableId = 'abc';
const firstPage = Array.from({ length: 500 }, (_, i) => ({
id: i + 1,
createdAt: '2024-01-01 00:00:00',
updatedAt: '2024-01-01 00:00:00',
val: i,
}));
const secondPage = [
{
id: 501,
createdAt: '2024-01-01 00:00:00',
updatedAt: '2024-01-01 00:00:00',
val: 500,
},
];
let queryCount = 0;
jest.mocked(mockDataSource.query).mockImplementation(async (query: string) => {
queryCount++;
if (query.includes('FROM') && query.includes('data_table_column')) {
return [{ dataTableId, name: 'val', type: 'number', index: 0 }];
}
if (query.startsWith('SELECT id FROM') && query.includes('data_table')) {
return [{ id: dataTableId }];
}
if (query.includes('data_table_user_')) {
if (query.includes('WHERE "id" > 0')) return firstPage;
if (query.includes('WHERE "id" > 500')) return secondPage;
return [];
}
return [];
});
jest.mocked(readdir).mockResolvedValue([]);
// @ts-expect-error accessing private method for testing
const result = await exportService.exportDataTableUserTables('/test/output');
expect(result.totalTables).toBe(1);
expect(result.totalRows).toBe(501);
// First file appended once for first page; second page rolls into a new file
expect(appendFile).toHaveBeenCalledTimes(2);
// 1 list-tables + 2 page queries — second page has < pageSize
// rows, so the loop short-circuits without a terminator query.
expect(queryCount).toBe(3);
});
it('should handle a missing dynamic table gracefully without aborting', async () => {
jest.mocked(mockDataSource.query).mockImplementation(async (query: string) => {
if (query.startsWith('SELECT id FROM') && query.includes('data_table')) {
return [{ id: 'broken' }];
}
if (query.includes('data_table_column')) return [];
if (query.includes('data_table_user_')) {
throw new Error('no such table');
}
return [];
});
// @ts-expect-error accessing private method for testing
const result = await exportService.exportDataTableUserTables('/test/output');
expect(result).toEqual({ totalTables: 1, totalRows: 0 });
expect(mockLogger.warn).toHaveBeenCalledWith(
expect.stringContaining('Could not read rows'),
expect.objectContaining({ error: expect.any(Error) }),
);
});
});
describe('exportMigrationsTable', () => {
it('should export migrations table when it has data', async () => {
const outputDir = '/test/output';

145
packages/cli/src/services/__tests__/import.service.test.ts
View File

@ -281,6 +281,7 @@ describe('ImportService', () => {
workflowentity: ['/test/input/workflowentity.jsonl'],
},
tableNames: ['user', 'workflow_entity'],
dataTableFiles: {},
});
});
@ -301,6 +302,7 @@ describe('ImportService', () => {
user: ['/test/input/user.jsonl', '/test/input/user.2.jsonl', '/test/input/user.3.jsonl'],
},
tableNames: ['user'],
dataTableFiles: {},
});
});
@ -314,6 +316,7 @@ describe('ImportService', () => {
expect(result).toEqual({
entityFiles: {},
tableNames: [],
dataTableFiles: {},
});
});
@ -325,6 +328,7 @@ describe('ImportService', () => {
expect(result).toEqual({
entityFiles: {},
tableNames: [],
dataTableFiles: {},
});
});
@ -338,6 +342,7 @@ describe('ImportService', () => {
expect(result).toEqual({
entityFiles: {},
tableNames: [],
dataTableFiles: {},
});
});
@ -355,6 +360,37 @@ describe('ImportService', () => {
user: ['/test/input/user.jsonl'],
},
tableNames: ['user'],
dataTableFiles: {},
});
});
it('should route data-table user-row files into dataTableFiles', async () => {
const mockFiles = [
'user.jsonl',
'data_table_user_abc.jsonl',
'data_table_user_abc.2.jsonl',
'data_table_user_xyz.jsonl',
];
jest.mocked(readdir).mockResolvedValue(mockFiles as any);
jest
.mocked(safeJoinPath)
.mockReturnValueOnce('/test/input/user.jsonl')
.mockReturnValueOnce('/test/input/data_table_user_abc.jsonl')
.mockReturnValueOnce('/test/input/data_table_user_abc.2.jsonl')
.mockReturnValueOnce('/test/input/data_table_user_xyz.jsonl');
const result = await importService.getImportMetadata('/test/input');
expect(result).toEqual({
entityFiles: {
user: ['/test/input/user.jsonl'],
},
tableNames: ['user'],
dataTableFiles: {
abc: ['/test/input/data_table_user_abc.jsonl', '/test/input/data_table_user_abc.2.jsonl'],
xyz: ['/test/input/data_table_user_xyz.jsonl'],
},
});
});
});
@ -1105,6 +1141,115 @@ describe('ImportService', () => {
expect(mockDataTableDDLService.createTableWithColumns).not.toHaveBeenCalled();
expect(mockDataTableDDLService.dropTable).not.toHaveBeenCalled();
});
describe('row import', () => {
beforeEach(() => {
mockDataSource.driver.escapeQueryWithParameters = jest
.fn()
.mockImplementation((sql, params) => [sql, params]);
});
it('should insert rows into the recreated backing table when files are provided', async () => {
mockEntityManager.query = jest
.fn()
.mockResolvedValueOnce([{ id: 'abc' }])
.mockResolvedValueOnce([
{ id: 'col-1', dataTableId: 'abc', name: 'flag', type: 'boolean', index: 0 },
])
.mockResolvedValue(undefined); // subsequent INSERTs
const row = {
id: 1,
createdAt: '2024-01-01 12:00:00',
updatedAt: '2024-01-01 12:00:00',
flag: 0,
};
jest.mocked(readFile).mockResolvedValue(JSON.stringify(row));
await importService.recreateDataTableUserTablesFromRegistry(mockEntityManager, {
abc: ['/test/input/data_table_user_abc.jsonl'],
});
const insertCalls = (mockEntityManager.query as jest.Mock).mock.calls.filter(
([sql]) => typeof sql === 'string' && sql.startsWith('INSERT INTO'),
);
expect(insertCalls).toHaveLength(1);
const [, params] = insertCalls[0];
// Boolean must have been normalised from 0 -> false (matters for cross-DB inserts)
expect((params as Record<string, unknown>).flag).toBe(false);
// id is preserved
expect((params as Record<string, unknown>).id).toBe(1);
});
it('should warn when archive has row files but registry is empty', async () => {
mockEntityManager.query = jest.fn().mockResolvedValueOnce([]);
await importService.recreateDataTableUserTablesFromRegistry(mockEntityManager, {
orphan: ['/test/input/data_table_user_orphan.jsonl'],
});
expect(mockLogger.warn).toHaveBeenCalledWith(
expect.stringContaining('but no entries in the data_table registry'),
);
});
it('should reset the id sequence after inserts on Postgres', async () => {
// @ts-expect-error overriding for the test
mockDataSource.options = { type: 'postgres' };
// Mixed-case id matters: pg_get_serial_sequence folds unquoted
// identifiers to lowercase, so the param must be the quoted form.
const mixedCaseId = 'AbC123';
mockEntityManager.query = jest
.fn()
.mockResolvedValueOnce([{ id: mixedCaseId }])
.mockResolvedValueOnce([
{ id: 'col-1', dataTableId: mixedCaseId, name: 'foo', type: 'string', index: 0 },
])
.mockResolvedValueOnce(undefined) // INSERT row
.mockResolvedValueOnce([{ column_name: 'id' }]) // information_schema lookup
.mockResolvedValue(undefined);
jest.mocked(readFile).mockResolvedValue(
JSON.stringify({
id: 5,
createdAt: '2024-01-01T00:00:00.000Z',
updatedAt: '2024-01-01T00:00:00.000Z',
foo: 'x',
}),
);
await importService.recreateDataTableUserTablesFromRegistry(mockEntityManager, {
[mixedCaseId]: [`/test/input/data_table_user_${mixedCaseId}.jsonl`],
});
const setvalCalls = (mockEntityManager.query as jest.Mock).mock.calls.filter(
([sql]) => typeof sql === 'string' && sql.includes('setval('),
);
expect(setvalCalls).toHaveLength(1);
const [, params] = setvalCalls[0];
expect(params).toEqual([`"data_table_user_${mixedCaseId}"`, 'id']);
});
it('should NOT call setval on Postgres when no rows were inserted', async () => {
// @ts-expect-error overriding for the test
mockDataSource.options = { type: 'postgres' };
mockEntityManager.query = jest
.fn()
.mockResolvedValueOnce([{ id: 'abc' }])
.mockResolvedValueOnce([])
.mockResolvedValue(undefined);
await importService.recreateDataTableUserTablesFromRegistry(mockEntityManager, {});
const setvalCalls = (mockEntityManager.query as jest.Mock).mock.calls.filter(
([sql]) => typeof sql === 'string' && sql.includes('setval('),
);
expect(setvalCalls).toHaveLength(0);
});
});
});
describe('advanceIdentitySequences', () => {

156
packages/cli/src/services/export.service.ts
View File

@ -8,6 +8,9 @@ import { DataSource } from '@n8n/typeorm';
import { validateDbTypeForExportEntities } from '@/utils/validate-database-type';
import { Cipher } from 'n8n-core';
import { compressFolder } from '@/utils/compression.util';
import { quoteIdentifier, toTableName } from '@/modules/data-table/utils/sql-utils';
const DATA_TABLE_ROWS_FILE_PREFIX = 'data_table_user_';
@Service()
export class ExportService {
@ -90,11 +93,151 @@ export class ExportService {
return systemTablesExported;
}
private async loadDataTableIds(): Promise<string[]> {
const tablePrefix = this.dataSource.options.entityPrefix || '';
const dataTableTableName = `${tablePrefix}data_table`;
let dataTables: Array<{ id: string }>;
try {
dataTables = await this.dataSource.query(
`SELECT id FROM ${this.dataSource.driver.escape(dataTableTableName)}`,
);
} catch (error) {
this.logger.info(
` ⚠️ ${dataTableTableName} registry not found, skipping data-table row export...`,
{ error },
);
return [];
}
if (dataTables.length === 0) {
this.logger.info(' ℹ️ No data tables found, nothing to export.');
return [];
}
return dataTables.map((t) => t.id);
}
private async exportSingleDataTable(
dataTableId: string,
outputDir: string,
customEncryptionKey: string | undefined,
): Promise<number> {
const dbType = this.dataSource.options.type;
const userTableName = toTableName(dataTableId);
const fileBaseName = `${DATA_TABLE_ROWS_FILE_PREFIX}${dataTableId}`;
this.logger.info(`\n📊 Processing data table: ${userTableName}`);
await this.clearExistingEntityFiles(outputDir, fileBaseName);
const idCol = quoteIdentifier('id', dbType);
const escapedUserTable = quoteIdentifier(userTableName, dbType);
const pageSize = 500;
const entitiesPerFile = 500;
let lastId = 0;
let fileIndex = 1;
let currentFileEntityCount = 0;
let totalEntityCount = 0;
let hasNextPage = true;
do {
let pageRows: Array<Record<string, unknown>>;
try {
pageRows = await this.dataSource.query(
`SELECT * FROM ${escapedUserTable} WHERE ${idCol} > ${lastId} ORDER BY "id" LIMIT ${pageSize}`,
);
} catch (error) {
this.logger.warn(
` ⚠️ Could not read rows from ${userTableName}; skipping. The dynamic table may be missing on the source instance.`,
{ error },
);
break;
}
if (pageRows.length === 0) break;
const targetFileIndex = Math.floor(totalEntityCount / entitiesPerFile) + 1;
const fileName =
targetFileIndex === 1
? `${fileBaseName}.jsonl`
: `${fileBaseName}.${targetFileIndex}.jsonl`;
const filePath = safeJoinPath(outputDir, fileName);
if (targetFileIndex > fileIndex) {
this.logger.info(` ✅ Completed file ${fileIndex}: ${currentFileEntityCount} rows`);
fileIndex = targetFileIndex;
currentFileEntityCount = 0;
}
const rowsJsonl = pageRows.map((row) => JSON.stringify(row)).join('\n');
await appendFile(
filePath,
(await this.cipher.encryptV2(rowsJsonl, customEncryptionKey)) + '\n',
'utf8',
);
const lastRowId = Number((pageRows[pageRows.length - 1] as { id: unknown }).id);
if (!Number.isFinite(lastRowId)) {
throw new Error(
`Unexpected non-numeric id in ${userTableName}; cannot continue keyset pagination`,
);
}
lastId = lastRowId;
totalEntityCount += pageRows.length;
currentFileEntityCount += pageRows.length;
this.logger.info(
` Fetched ${pageRows.length} rows (last id: ${lastId}, total: ${totalEntityCount})`,
);
hasNextPage = pageRows.length >= pageSize;
} while (hasNextPage);
if (currentFileEntityCount > 0) {
this.logger.info(` ✅ Completed file ${fileIndex}: ${currentFileEntityCount} rows`);
}
this.logger.info(
` ✅ Completed export for ${userTableName}: ${totalEntityCount} rows in ${fileIndex} file(s)`,
);
return totalEntityCount;
}
private async exportDataTableUserTables(
outputDir: string,
customEncryptionKey?: string,
): Promise<{ totalTables: number; totalRows: number }> {
this.logger.info('\n📚 Exporting data table user rows:');
this.logger.info('==================================');
const dataTableIds = await this.loadDataTableIds();
if (dataTableIds.length === 0) {
return { totalTables: 0, totalRows: 0 };
}
let totalRows = 0;
for (const dataTableId of dataTableIds) {
totalRows += await this.exportSingleDataTable(dataTableId, outputDir, customEncryptionKey);
}
this.logger.info(
`\n📊 Data table row export summary: ${dataTableIds.length} table(s), ${totalRows} row(s)`,
);
return { totalTables: dataTableIds.length, totalRows };
}
async exportEntities(
outputDir: string,
excludedTables: Set<string> = new Set(),
keyFilePath?: string,
options: { includeDataTableRows?: boolean } = {},
) {
const { includeDataTableRows = true } = options;
this.logger.info('\n⚠️⚠️ This feature is currently under development. ⚠️⚠️');
validateDbTypeForExportEntities(this.dataSource.options.type);
@ -133,6 +276,19 @@ export class ExportService {
await this.exportMigrationsTable(outputDir, customEncryptionKey);
if (includeDataTableRows) {
const dataTableRowsExported = await this.exportDataTableUserTables(
outputDir,
customEncryptionKey,
);
totalEntitiesExported += dataTableRowsExported.totalRows;
totalTablesProcessed += dataTableRowsExported.totalTables;
} else {
this.logger.info(
'\nℹ️ Skipping data-table row export (only schemas will be in the archive).',
);
}
for (const metadata of entityMetadatas) {
// Get table name and entity name
const tableName = metadata.tableName;

170
packages/cli/src/services/import.service.ts
View File

@ -33,7 +33,14 @@ import { ActiveWorkflowManager } from '@/active-workflow-manager';
import type { IWorkflowWithVersionMetadata } from '@/interfaces';
import { WorkflowIndexService } from '@/modules/workflow-index/workflow-index.service';
import { DataTableDDLService } from '@/modules/data-table/data-table-ddl.service';
import { DataTableColumn } from '@/modules/data-table/data-table-column.entity';
import type { DataTableColumn } from '@/modules/data-table/data-table-column.entity';
import {
normalizeUserRowValueForDatabase,
quoteIdentifier,
toTableName,
} from '@/modules/data-table/utils/sql-utils';
const DATA_TABLE_ROWS_FILE_PREFIX = 'data_table_user_';
@Service()
export class ImportService {
@ -328,49 +335,64 @@ export class ImportService {
/**
* Get import metadata including table names and entity files
* @param inputDir - Directory containing exported entity files
* @returns Object containing table names and entity files grouped by entity name
* @returns Object containing table names, entity files grouped by entity name,
* and data-table row files grouped by data table id.
*/
async getImportMetadata(inputDir: string): Promise<{
tableNames: string[];
entityFiles: Record<string, string[]>;
dataTableFiles: Record<string, string[]>;
}> {
const files = await readdir(inputDir);
const entityTableNamesMap: Record<string, string> = {};
const entityFiles: Record<string, string[]> = {};
const dataTableFiles: Record<string, string[]> = {};
for (const file of files) {
if (file.endsWith('.jsonl')) {
const entityName = file.replace(/\.\d+\.jsonl$/, '.jsonl').replace('.jsonl', '');
// Exclude migrations from regular entity import
if (entityName === 'migrations') {
if (!file.endsWith('.jsonl')) continue;
const baseName = file.replace(/\.\d+\.jsonl$/, '.jsonl').replace('.jsonl', '');
// Exclude migrations from regular entity import
if (baseName === 'migrations') {
continue;
}
// Route data-table user-row files into their own bucket keyed by data table id.
if (baseName.startsWith(DATA_TABLE_ROWS_FILE_PREFIX)) {
const dataTableId = baseName.slice(DATA_TABLE_ROWS_FILE_PREFIX.length);
if (!dataTableFiles[dataTableId]) dataTableFiles[dataTableId] = [];
dataTableFiles[dataTableId].push(safeJoinPath(inputDir, file));
continue;
}
const entityName = baseName;
// Build table names map (only need to do this once per entity)
if (!entityTableNamesMap[entityName]) {
const entityMetadata = this.dataSource.entityMetadatas.find(
(meta) => meta.name.toLowerCase() === entityName,
);
if (!entityMetadata) {
this.logger.warn(`⚠️ No entity metadata found for ${entityName}, skipping...`);
continue;
}
// Build table names map (only need to do this once per entity)
if (!entityTableNamesMap[entityName]) {
const entityMetadata = this.dataSource.entityMetadatas.find(
(meta) => meta.name.toLowerCase() === entityName,
);
if (!entityMetadata) {
this.logger.warn(`⚠️ No entity metadata found for ${entityName}, skipping...`);
continue;
}
entityTableNamesMap[entityName] = entityMetadata.tableName;
}
// Build entity files map (only for entities with valid metadata)
if (!entityFiles[entityName]) {
entityFiles[entityName] = [];
}
entityFiles[entityName].push(safeJoinPath(inputDir, file));
entityTableNamesMap[entityName] = entityMetadata.tableName;
}
// Build entity files map (only for entities with valid metadata)
if (!entityFiles[entityName]) {
entityFiles[entityName] = [];
}
entityFiles[entityName].push(safeJoinPath(inputDir, file));
}
return {
tableNames: Object.values(entityTableNamesMap),
entityFiles,
dataTableFiles,
};
}
@ -463,7 +485,7 @@ export class ImportService {
// Get import metadata after migration validation
const importMetadata = await this.getImportMetadata(inputDir);
const { tableNames, entityFiles } = importMetadata;
const { tableNames, entityFiles, dataTableFiles } = importMetadata;
const entityNames = Object.keys(entityFiles);
if (truncateTables) {
@ -506,8 +528,12 @@ export class ImportService {
await this.advanceIdentitySequences(transactionManager, tableNames);
// After the data_table / data_table_column registry rows are imported,
// recreate the dynamic backing tables (empty) so the imported tables work.
await this.recreateDataTableUserTablesFromRegistry(transactionManager);
// recreate the dynamic backing tables and load any user rows from the archive.
await this.recreateDataTableUserTablesFromRegistry(
transactionManager,
dataTableFiles,
customEncryptionKey,
);
if (!skipTogglingForeignKeyConstraints) {
await this.enableForeignKeyConstraints(transactionManager);
@ -710,15 +736,20 @@ export class ImportService {
/**
* Recreate dynamic data-table backing tables for every entry in the imported
* `data_table` registry. The export bug (ADO-5143) means archives produced
* before this fix only contain registry rows; without recreating the backing
* tables the imported data tables would reference tables that don't exist on
* the destination.
* `data_table` registry, and insert user rows from any matching JSONL files.
*
* Tables are dropped before recreation so the operation is idempotent and
* handles stale tables left over from a partially-failed prior import.
* handles stale tables left over from a partially-failed prior import. When
* no row file exists for a table (e.g. an archive produced with
* `--includeDataTableRows=false` or before this feature) the backing table
* is recreated empty so the system stays functional.
*/
async recreateDataTableUserTablesFromRegistry(transactionManager: EntityManager): Promise<void> {
async recreateDataTableUserTablesFromRegistry(
transactionManager: EntityManager,
dataTableFiles: Record<string, string[]> = {},
customEncryptionKey?: string,
): Promise<void> {
const dbType = this.dataSource.options.type;
const tablePrefix = this.dataSource.options.entityPrefix || '';
const dataTableTableName = `${tablePrefix}data_table`;
const dataTableColumnTableName = `${tablePrefix}data_table_column`;
@ -736,41 +767,92 @@ export class ImportService {
return;
}
if (dataTables.length === 0) return;
if (dataTables.length === 0) {
if (Object.keys(dataTableFiles).length > 0) {
this.logger.warn(
` ⚠️ Found ${Object.keys(dataTableFiles).length} data-table row file(s) in the archive but no entries in the data_table registry; skipping.`,
);
}
return;
}
const escapedColumnTable = this.dataSource.driver.escape(dataTableColumnTableName);
const escapedDataTableId = this.dataSource.driver.escape('dataTableId');
const escapedIndex = this.dataSource.driver.escape('index');
const columnRows: Array<{
id: string;
dataTableId: string;
name: string;
type: 'string' | 'number' | 'boolean' | 'date';
index: number;
}> = await transactionManager.query(
const columnRows: DataTableColumn[] = await transactionManager.query(
`SELECT id, ${escapedDataTableId}, name, type, ${escapedIndex} FROM ${escapedColumnTable}`,
);
const columnsByDataTableId = new Map<string, DataTableColumn[]>();
for (const row of columnRows) {
const list = columnsByDataTableId.get(row.dataTableId) ?? [];
list.push(transactionManager.create(DataTableColumn, row));
list.push(row);
columnsByDataTableId.set(row.dataTableId, list);
}
this.logger.info(`\n📚 Recreating ${dataTables.length} data-table backing table(s)...`);
let totalRowsImported = 0;
for (const { id: dataTableId } of dataTables) {
const userTable = toTableName(dataTableId);
const cols = (columnsByDataTableId.get(dataTableId) ?? [])
.slice()
.sort((a, b) => a.index - b.index);
await this.dataTableDDLService.dropTable(dataTableId, transactionManager);
await this.dataTableDDLService.createTableWithColumns(dataTableId, cols, transactionManager);
const files = dataTableFiles[dataTableId];
if (!files || files.length === 0) continue;
const columnTypeMap = new Map<string, string>(cols.map((c) => [c.name, c.type]));
columnTypeMap.set('createdAt', 'date');
columnTypeMap.set('updatedAt', 'date');
const escapedTable = quoteIdentifier(userTable, dbType);
let rowsForTable = 0;
for (const filePath of files) {
const rows = await this.readEntityFile(filePath, customEncryptionKey);
for (const row of rows) {
const normalizedRow: Record<string, unknown> = {};
for (const [key, value] of Object.entries(row)) {
normalizedRow[key] = normalizeUserRowValueForDatabase(
value,
columnTypeMap.get(key),
dbType,
);
}
const columnNames = Object.keys(normalizedRow);
const escapedCols = columnNames.map((c) => quoteIdentifier(c, dbType)).join(', ');
const placeholders = columnNames.map((c) => `:${c}`).join(', ');
const [query, params] = this.dataSource.driver.escapeQueryWithParameters(
`INSERT INTO ${escapedTable} (${escapedCols}) VALUES (${placeholders})`,
normalizedRow,
{},
);
await transactionManager.query(query, params);
rowsForTable++;
}
}
// Postgres IDENTITY columns don't auto-advance when explicit ids are
// inserted, so the next implicit insert would collide. Reset to MAX(id).
if (rowsForTable > 0 && dbType === 'postgres') {
await this.advanceIdentitySequences(transactionManager, [userTable]);
}
if (rowsForTable > 0) {
this.logger.info(` ✅ ${userTable}: ${rowsForTable} row(s) imported`);
}
totalRowsImported += rowsForTable;
}
this.logger.info(`✅ Recreated ${dataTables.length} data-table backing table(s)`);
this.logger.info(
`✅ Recreated ${dataTables.length} data-table backing table(s); imported ${totalRowsImported} row(s)`,
);
}
/**

Some files were not shown because too many files have changed in this diff Show More