mirror of
https://github.com/n8n-io/n8n.git
synced 2026-10-19 16:55:02 +02:00
docs: Add n8n-test PoC v2 spec and plan
This commit is contained in:
James Gee
parent
97225bc0cf
commit
267de8a650
147
.agents/specs/n8n-test-poc-v2.md
Normal file
147
.agents/specs/n8n-test-poc-v2.md
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
# Spec: `n8n-test` PoC — version 2
|
||||
|
||||
> Extends [`n8n-test-poc.md`](n8n-test-poc.md) (v1, feasibility proven on branch
|
||||
> `ligo-workflow-testing-poc`). Same ground rules: throwaway spike on branch
|
||||
> `ligo-workflow-testing-v2`, never merged, no tests for the harness itself.
|
||||
> Source of intent: the "Hack - Workflow testing demo" Notion page, §Version 2.
|
||||
|
||||
## Objective
|
||||
|
||||
Two phases:
|
||||
|
||||
- **Phase A — `mockNode`.** Swap one named node's execution for a canned result
|
||||
while everything else runs real, with input capture for assertions. Star
|
||||
demo: mocking an Execute Sub-workflow node — which v1 proved cannot run for
|
||||
real under the test harness, so mocking it is both the honest and the useful
|
||||
move.
|
||||
- **Phase B — the CI story v1 kept as talk-track.** A git repo holding an
|
||||
exploded project package export; a PR changing a workflow runs the workflow
|
||||
tests in a GitHub Action, fast; the reviewer watches the check go red → green.
|
||||
|
||||
## Public API additions (Phase A)
|
||||
|
||||
```ts
|
||||
import { mockNode, runWorkflow } from 'n8n-test';
|
||||
|
||||
// Registers an override for the named node in this workflow. Does not mutate the JSON.
|
||||
const sub = mockNode(workflowJson, 'Execute sub-workflow', { subworkflowOutput: 'sfoutput' });
|
||||
|
||||
const result = await runWorkflow(workflowJson);
|
||||
|
||||
expect(result.subworkflowOutput).toBe('sfoutput');
|
||||
// What the mocked node received (first item's json) — the capture half of the API.
|
||||
expect(sub.input()).toMatchObject({ data: 'Hello world' });
|
||||
```
|
||||
|
||||
Multiple nodes in one workflow can be mocked independently — each call returns
|
||||
its own handle, and with the HTTP node mocked too the test needs no nock at all:
|
||||
|
||||
```ts
|
||||
const http = mockNode(workflowJson, 'Http request', { data: 'canned response' });
|
||||
const sub = mockNode(workflowJson, 'Execute sub-workflow', { subworkflowOutput: 'sfoutput' });
|
||||
|
||||
const result = await runWorkflow(workflowJson);
|
||||
|
||||
expect(result.subworkflowOutput).toBe('sfoutput');
|
||||
expect(sub.input()).toMatchObject({ 'test-output': 'canned response' });
|
||||
```
|
||||
|
||||
Semantics:
|
||||
|
||||
- Matched by **node name** within the given workflow object; the override lives
|
||||
in a module-level registry keyed by the workflow object (WeakMap), applied at
|
||||
`runWorkflow` time — the JSON is never mutated.
|
||||
- **Any number of nodes per workflow** may be mocked; mocking the same node name
|
||||
again replaces the previous override (last wins) and returns a fresh handle.
|
||||
- The mocked node emits the canned object as a single item; the handle's
|
||||
`input()` returns the first item's `json` the node received (undefined until run).
|
||||
- `clearNodeMocks()` is exported and called from the demo `afterEach` so mocks
|
||||
never leak between tests.
|
||||
- Mechanism: at run time the target node's `type` is swapped for an in-process
|
||||
mock node implementation (registered alongside the loader's node types, the
|
||||
way `core/nodes-testing`'s `testData` node already is) whose `execute` records
|
||||
`getInputData()` and returns the canned item.
|
||||
|
||||
## Demo additions (Phase A)
|
||||
|
||||
- New `demo/workflow-with-subworkflow.json`: trigger → HTTP Request →
|
||||
Edit Fields → **Execute Sub-workflow** (last node). Kept separate from the v1
|
||||
`workflow.json` so the three v1 tests stay untouched — and because without a
|
||||
mock this workflow *cannot* run, which is exactly the story.
|
||||
- New demo test `workflow with subworkflow mocked`, per the API sketch above.
|
||||
- New demo test `workflow with every dependency mocked`: HTTP node **and**
|
||||
sub-workflow node both mocked, no nock in the test — node-level mocking
|
||||
standing on its own.
|
||||
|
||||
## Phase B — repo, artifact, Action
|
||||
|
||||
- **Demo repo** on James's personal GitHub (disposable): an **exploded project
|
||||
package export** (the Ligo `.n8np` layout untarred: `manifest.json`,
|
||||
`projects/<project>/workflows/*.json`), a `tests/` folder whose test files
|
||||
import workflow JSONs from that layout, a vanilla vitest setup, and one
|
||||
GitHub Action running on `pull_request`.
|
||||
- **Prebaked artifact, npm runtime.** CI must be fast (~2-4 min), so the Action
|
||||
does NOT build the monorepo. Instead:
|
||||
- `n8n-test` is bundled locally (esbuild/tsdown) into a self-contained
|
||||
tarball: our `src/` **plus vendored `core/nodes-testing` components** (they
|
||||
are not in the published `n8n-core` npm tarball; their relative `../dist/*`
|
||||
imports get rewritten to `n8n-core/dist/*` deep requires).
|
||||
- The tarball is attached to a GitHub release on the demo repo; the demo
|
||||
repo's `package.json` installs it by URL, alongside **released npm**
|
||||
`n8n-core` / `n8n-workflow` / `n8n-nodes-base` / `nock` / `vitest`.
|
||||
- In a plain node_modules world the DI single-container problem disappears
|
||||
(everything loads as CJS dist), so the demo repo's vitest config is vanilla
|
||||
plus `reflect-metadata` setup.
|
||||
- **Demo script:** PR changes the workflow's field mapping → check fails with a
|
||||
readable assertion diff → fix commit → green.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
# Phase A (monorepo, as v1)
|
||||
pnpm --filter n8n-test demo
|
||||
|
||||
# Phase B (monorepo side)
|
||||
pnpm --filter n8n-test bundle # produce the self-contained tarball for the demo repo
|
||||
```
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. Phase A: `pnpm --filter n8n-test demo` runs 5+ tests green, including the
|
||||
sub-workflow mock with an `input()` capture assertion and the multi-node
|
||||
test (HTTP + sub-workflow both mocked, nock-free); v1's three tests
|
||||
unchanged.
|
||||
2. Phase A honesty check: running the sub-workflow demo workflow *without*
|
||||
`mockNode` rejects (it cannot execute for real) — asserted in a demo test.
|
||||
3. Phase B: a real PR on the demo repo shows the workflow-test check failing on
|
||||
a broken workflow change and passing once fixed, in under ~4 minutes.
|
||||
4. The README gains a v2 section: `mockNode` usage + how the demo repo is wired.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- **Always:** keep monorepo changes inside `packages/n8n-test` (+ specs/plan);
|
||||
spike branch only.
|
||||
- **Ask first:** anything that must run against the spike branch's n8n-core
|
||||
rather than released npm (that flips the CI strategy); creating anything
|
||||
under the n8n-io org; publishing anything to npm.
|
||||
- **Never:** merge; expose secrets in the demo repo; real network calls in tests.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
`replaceNode` / `fetchWorkflow`, non-throw error modes, credentials injection,
|
||||
multi-item canned outputs, mocking by node type (name-only), npm publishing.
|
||||
The SSRF-helper seam remains the documented production direction for HTTP
|
||||
mocking; `mockNode` is orthogonal (node-level, not transport-level).
|
||||
|
||||
## Risks
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|------|--------|------------|
|
||||
| Released npm `n8n-core` engine API drifts from master (v1 was built against master source) | High — bundle may not run | Validate the bundle against npm versions locally before creating the repo; if it fights back, fall back to a GitHub Actions cache of a built monorepo checkout (slower first run, fast after) |
|
||||
| Vendored `nodes-testing` imports (`../dist/*` → `n8n-core/dist/*`) break if core ships an exports map | Med | Check `n8n-core` package.json `exports` up front; deep-require works today |
|
||||
| Execute Sub-workflow node's parameters demand a real workflow reference before our mock swaps it | Low | The swap happens before the engine touches the node; parameters go unread by the mock implementation |
|
||||
|
||||
## Open Questions
|
||||
|
||||
- None blocking. Demo repo name and the npm version pin are picked during
|
||||
implementation (latest stable n8n release).
|
||||
105
tasks/plan.md
105
tasks/plan.md
|
|
@ -1,70 +1,79 @@
|
|||
# Implementation Plan: `n8n-test` workflow-testing PoC
|
||||
# Implementation Plan: `n8n-test` PoC — version 2
|
||||
|
||||
Spec: `.agents/specs/n8n-test-poc.md`. Spike branch off master (suggested:
|
||||
`ligo-workflow-testing-poc`); never merged.
|
||||
Spec: `.agents/specs/n8n-test-poc-v2.md`. Branch `ligo-workflow-testing-v2`
|
||||
(off the v1 spike); never merged. v1 plan is in this file's git history.
|
||||
|
||||
## Overview
|
||||
|
||||
Build `packages/n8n-test`, a throwaway monorepo package exposing
|
||||
`await runWorkflow(workflowJson, input?)`, which executes a workflow through the
|
||||
real n8n engine with real `nodes-base` node types, HTTP intercepted by nock, and
|
||||
demo it as a two-test vitest suite run locally.
|
||||
Phase A adds `mockNode` (multi-node, canned output, input capture) to
|
||||
`packages/n8n-test`, starring an Execute Sub-workflow mock. Phase B builds the
|
||||
real CI story: a disposable GitHub repo (James's account) holding an exploded
|
||||
project package export, whose PRs run the workflow tests in under ~4 minutes
|
||||
against a prebaked `n8n-test` bundle + released npm n8n packages.
|
||||
|
||||
## Architecture Decisions
|
||||
|
||||
- **Reuse `packages/core/nodes-testing/` wholesale.** `node-test-harness.ts`
|
||||
already proves the exact mechanics: `LoadNodesAndCredentials` + `NodeTypes` +
|
||||
`CredentialsHelper` (stub), `mock<IWorkflowExecuteAdditionalData>()`, seed the
|
||||
start node on `nodeExecutionStack` with `main: [[{ json: input }]]`, run
|
||||
`new WorkflowExecute(...).processRunExecutionData(workflow)`, await the
|
||||
`workflowExecuteAfter` hook. `runWorkflow` is a ~100-line rearrangement of
|
||||
`executeWorkflow` (node-test-harness.ts:199-264), not new machinery.
|
||||
- **Import the harness components via a tsconfig path alias**, exactly as
|
||||
nodes-base does (`"@nodes-testing/*": ["../core/nodes-testing/*"]` in
|
||||
`packages/nodes-base/tsconfig.json:8`). No changes to `packages/core`.
|
||||
- **Result contract:** read `result.data.resultData`; on `error` → throw it
|
||||
(throw-mode); otherwise return
|
||||
`runData[lastNodeExecuted].at(-1).data.main[0][0].json`.
|
||||
- **nock, not the SSRF request-helper seam.** Zero engine changes for the PoC;
|
||||
the helper seam is the documented production direction (spec, Out of Scope).
|
||||
- **Demo assets fixed for consistency:** the Notion sample workflow POSTs to
|
||||
requestcatcher while the sample test nocks GET `https://test-endpoint.com/test`;
|
||||
the packaged `demo/workflow.json` is retargeted so they line up.
|
||||
- **Mock mechanism: per-run node-type swap.** `mockNode` writes an entry into a
|
||||
module-level `WeakMap<workflowJson, Map<nodeName, entry>>`. `runWorkflow`
|
||||
builds, per run, a mock `INodeType` closing over those entries and an
|
||||
`INodeTypes` wrapper that serves it for the sentinel type; mocked nodes get
|
||||
their `type` rewritten to the sentinel when the `Workflow` is constructed.
|
||||
The mock's `execute` records `getInputData()` into the entry (feeding
|
||||
`handle.input()`) and returns the canned object as one item. JSON never
|
||||
mutates; last `mockNode` per name wins; `clearNodeMocks()` empties the map.
|
||||
- **Sub-workflow demo is a separate JSON** (`demo/workflow-with-subworkflow.json`,
|
||||
trigger → HTTP → Edit Fields → Execute Sub-workflow @1.3) so v1's tests stay
|
||||
untouched — and the unmockable-for-real node doubles as the honesty test
|
||||
(without `mockNode` the run must reject).
|
||||
- **Prebake: esbuild bundle, npm runtime.** `n8n-core` has no `exports` map
|
||||
(verified: `main` only), so the vendored `core/nodes-testing` sources'
|
||||
`../dist/*` imports can be rewritten to `n8n-core/dist/*` deep requires by a
|
||||
small esbuild `onResolve` plugin. Bundle externals: `n8n-core`,
|
||||
`n8n-workflow`, `n8n-nodes-base`, `nock`, `vitest`, `reflect-metadata`.
|
||||
Runtime pins: npm `n8n-core@2.16.x`, `n8n-workflow@2.16.x`,
|
||||
`n8n-nodes-base@2.15.x` (current releases).
|
||||
- **Validate the prebake before building the repo** (risk-first): npm releases
|
||||
lag master's engine; if the bundle can't run on them (e.g. a missing
|
||||
`createRunExecutionData`, changed `establishExecutionContext` shape), first
|
||||
try shimming the gap inside the bundle; if the engine itself diverges, stop
|
||||
and ask — that flips the CI strategy (spec's "Ask first" boundary).
|
||||
- **Demo repo anatomy:** exploded `.n8np` layout (`manifest.json`,
|
||||
`projects/<project>/workflows/*.json`) authentic to the Ligo export format,
|
||||
`tests/*.test.ts` importing those JSONs, vanilla vitest + `reflect-metadata`
|
||||
setup (no DI-container gymnastics outside the monorepo), one `pull_request`
|
||||
Action with npm caching.
|
||||
|
||||
## Task List
|
||||
|
||||
### Phase 1: Foundation
|
||||
- [x] Task 1: Scaffold `packages/n8n-test` (builds; empty vitest suite runs)
|
||||
### Phase A: mockNode
|
||||
- [ ] Task 1: `mockNode` + sub-workflow demo (mock, capture, honesty test)
|
||||
- [ ] Task 2: Multi-node demo test (HTTP + sub-workflow mocked, nock-free)
|
||||
|
||||
### Checkpoint: Foundation
|
||||
- [x] `pnpm --filter n8n-test build` and `pnpm --filter n8n-test demo` both exit 0
|
||||
### Checkpoint: Phase A
|
||||
- [ ] `pnpm --filter n8n-test demo` — 6 tests green (3 × v1 untouched + 3 new)
|
||||
- [ ] Typecheck + lint clean
|
||||
|
||||
### Phase 2: Feasibility core (highest risk first)
|
||||
- [x] Task 2: `runWorkflow` happy path through the real engine; demo happy-day test green
|
||||
- [x] Task 3: Throw-mode errors + network lockdown; demo unhappy-day test green
|
||||
### Phase B: CI story (risk first)
|
||||
- [ ] Task 3: `bundle` command + validation against released npm packages (go/no-go)
|
||||
- [ ] Task 4: Demo repo — exploded export, tests, Action, release asset, first green run
|
||||
- [ ] Task 5: The demo PR — red on a broken workflow change, green on the fix
|
||||
|
||||
### Checkpoint: Feasibility proven (the go/no-go moment)
|
||||
- [x] Both demo tests green via real `WorkflowExecute` + real node types
|
||||
- [x] Un-mocked HTTP fails loudly (`nock.disableNetConnect`)
|
||||
- [x] Review with James before polish
|
||||
### Checkpoint: Phase B
|
||||
- [ ] PR check red → green in under ~4 minutes end to end
|
||||
|
||||
### Phase 3: Demo polish
|
||||
- [x] Task 4: `input` parameter, demo-day tidy-up, gotchas write-up
|
||||
|
||||
### Checkpoint: Complete
|
||||
- [x] Demo dry-run from a clean checkout of the spike branch matches the spec's success criteria
|
||||
### Phase C: Wrap
|
||||
- [ ] Task 6: README v2 section, gotchas, verification fan-out, push
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|------|--------|------------|
|
||||
| `core/nodes-testing` imports from `packages/core/dist` → needs a built monorepo | Med | Build `n8n-core`/`nodes-base` once up front; document in the package README |
|
||||
| Full nodes-base load is slow (~30s `beforeAll` in the existing harness) | Med (demo latency) | Accept for PoC; load once per suite; note as a gotcha for the real feature |
|
||||
| Alias/vitest resolution for `@nodes-testing/*` from a new package | Low | Copy nodes-base's tsconfig-paths setup; fall back to relative imports |
|
||||
| HTTP Request node's client bypasses nock | Low | The existing harness already nocks this node's traffic; nock 14 covers http/fetch |
|
||||
| Set node expression needs full expression context | Low | Real engine provides it; proven by harness usage across nodes-base tests |
|
||||
| npm-released engine lags master APIs the bundle relies on | High | Task 3 validates first; shim small gaps in-bundle; stop-and-ask if the engine diverges (CI strategy flip) |
|
||||
| Mock node type must satisfy engine plumbing (inputs/outputs, versioning) | Med | Mirror `core/nodes-testing/test-data-node.ts`, the in-repo precedent for an in-process node |
|
||||
| Handle capture across vitest worker/module boundaries | Low | Registry and handles live in one module instance per test file — same-process by construction |
|
||||
| Demo repo Action flakes on npm install time | Low | `actions/setup-node` npm cache; pin exact versions |
|
||||
|
||||
## Open Questions
|
||||
|
||||
- None blocking (spec §Open Questions). Branch name and demo timing are
|
||||
James's call.
|
||||
- None blocking. Demo repo name decided at Task 4 (suggestion:
|
||||
`n8n-workflow-tests-demo`).
|
||||
|
|
|
|||
208
tasks/todo.md
208
tasks/todo.md
|
|
@ -1,121 +1,159 @@
|
|||
# Tasks: `n8n-test` workflow-testing PoC
|
||||
# Tasks: `n8n-test` PoC — version 2
|
||||
|
||||
Spec: `.agents/specs/n8n-test-poc.md` · Plan: `tasks/plan.md`
|
||||
Spec: `.agents/specs/n8n-test-poc-v2.md` · Plan: `tasks/plan.md` · v1 tasks in git history
|
||||
|
||||
## Task 1: Scaffold `packages/n8n-test`
|
||||
## Task 1: `mockNode` + sub-workflow demo
|
||||
|
||||
**Description:** Create the workspace package so it builds and runs an empty
|
||||
vitest suite. Wires the `@nodes-testing/*` tsconfig path alias (copied from
|
||||
`packages/nodes-base/tsconfig.json:8`) and workspace deps on `n8n-core`,
|
||||
`n8n-workflow`, `n8n-nodes-base`, `nock`, `vitest`.
|
||||
**Description:** Add `mockNode(workflowJson, nodeName, cannedOutput)` returning
|
||||
a handle with `input()`, plus `clearNodeMocks()`. Implementation per plan: a
|
||||
`WeakMap` registry, a per-run mock `INodeType` (modelled on
|
||||
`core/nodes-testing/test-data-node.ts`) served by an `INodeTypes` wrapper, and
|
||||
node-type rewriting inside `runWorkflow`. Add
|
||||
`demo/workflow-with-subworkflow.json` (trigger → HTTP → Edit Fields →
|
||||
Execute Sub-workflow @1.3) and two demo tests: the mocked sub-workflow test
|
||||
(canned output + `input()` capture) and the honesty test (same workflow
|
||||
without the mock rejects — sub-workflows cannot run for real in the harness).
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [x] `pnpm --filter n8n-test build` exits 0
|
||||
- [x] `pnpm --filter n8n-test demo` runs a placeholder test green
|
||||
- [x] No other package is modified
|
||||
- [ ] `workflow with subworkflow mocked` green: canned output surfaces in the
|
||||
result and `sub.input()` captures what Edit Fields produced
|
||||
- [ ] Honesty test green: un-mocked run of the same workflow rejects
|
||||
- [ ] v1's three tests untouched and green
|
||||
|
||||
**Verification:**
|
||||
- [x] Build succeeds: `pnpm --filter n8n-test build`
|
||||
- [x] Tests pass: `pnpm --filter n8n-test demo`
|
||||
- [ ] Tests pass: `pnpm --filter n8n-test demo`
|
||||
- [ ] Build succeeds: `pnpm --filter n8n-test build`
|
||||
|
||||
**Dependencies:** None
|
||||
|
||||
**Files likely touched:**
|
||||
- `packages/n8n-test/package.json`
|
||||
- `packages/n8n-test/tsconfig.json`
|
||||
- `packages/n8n-test/vitest.config.ts`
|
||||
- `packages/n8n-test/src/mock-node.ts`
|
||||
- `packages/n8n-test/src/run-workflow.ts`
|
||||
- `packages/n8n-test/src/index.ts`
|
||||
- `packages/n8n-test/demo/workflow-with-subworkflow.json`
|
||||
- `packages/n8n-test/demo/workflow.test.ts` (or a new demo test file)
|
||||
|
||||
**Estimated scope:** S
|
||||
**Estimated scope:** M
|
||||
|
||||
## Task 2: `runWorkflow` happy path through the real engine
|
||||
## Task 2: Multi-node demo test
|
||||
|
||||
**Description:** Implement `runWorkflow(workflowJson, input?)` by rearranging
|
||||
`executeWorkflow` from `packages/core/nodes-testing/node-test-harness.ts:199-264`:
|
||||
load real node types once (`LoadNodesAndCredentials`/`NodeTypes`), build a
|
||||
`Workflow`, seed the start node's `nodeExecutionStack` with
|
||||
`main: [[{ json: input ?? {} }]]`, run `WorkflowExecute.processRunExecutionData`,
|
||||
await the `workflowExecuteAfter` hook, and return
|
||||
`runData[lastNodeExecuted].at(-1).data.main[0][0].json`. Add
|
||||
`demo/workflow.json` (Notion sample retargeted to GET
|
||||
`https://test-endpoint.com/test`) and the happy-day demo test.
|
||||
**Description:** Demo test `workflow with every dependency mocked`: HTTP node
|
||||
and sub-workflow node both mocked on the sub-workflow demo JSON, no nock in
|
||||
the test; assert the real Edit Fields node's output flowed between the mocks
|
||||
via `sub.input()`, and re-mocking a name replaces the previous override.
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [x] Happy-day demo test green: nock 200 `{ data: 'Hello world' }` →
|
||||
`output['test-output'] === 'Hello world'` via the real Set-node expression
|
||||
- [x] No mocked execution: real `WorkflowExecute`, real `nodes-base` node types
|
||||
- [ ] Test green with zero nock usage
|
||||
- [ ] `sub.input()` proves the real middle node ran between two mocks
|
||||
|
||||
**Verification:**
|
||||
- [x] Tests pass: `pnpm --filter n8n-test demo`
|
||||
- [x] Manual check: temporarily change the nock reply body and watch the
|
||||
assertion fail (proves the engine, not a canned value, produced output)
|
||||
- [ ] Tests pass: `pnpm --filter n8n-test demo` (6 green)
|
||||
- [ ] Manual check: lint + typecheck clean
|
||||
|
||||
**Dependencies:** Task 1
|
||||
|
||||
**Files likely touched:**
|
||||
- `packages/n8n-test/src/run-workflow.ts`
|
||||
- `packages/n8n-test/src/index.ts`
|
||||
- `packages/n8n-test/demo/workflow.json`
|
||||
- `packages/n8n-test/demo/workflow.test.ts`
|
||||
- `packages/n8n-test/demo/workflow.test.ts` (or the new demo test file)
|
||||
|
||||
**Estimated scope:** S
|
||||
|
||||
## Checkpoint: Phase A — 6 tests green, typecheck + lint clean
|
||||
|
||||
## Task 3: `bundle` command + npm-runtime validation (go/no-go)
|
||||
|
||||
**Description:** Add `pnpm --filter n8n-test bundle`: an esbuild script
|
||||
(`scripts/bundle.mjs`) producing a self-contained CJS tarball of `n8n-test` —
|
||||
vendoring the `@nodes-testing/*` sources with their relative `../dist/*`
|
||||
imports rewritten to `n8n-core/dist/*` via an `onResolve` plugin; externals:
|
||||
`n8n-core`, `n8n-workflow`, `n8n-nodes-base`, `nock`, `vitest`,
|
||||
`reflect-metadata`. Validate in a scratch dir outside the monorepo: npm-install
|
||||
the tarball + released `n8n-core@2.16.x` / `n8n-workflow@2.16.x` /
|
||||
`n8n-nodes-base@2.15.x` / `nock` / `vitest`, copy the demo tests, run them.
|
||||
Shim small API gaps inside the bundle; stop and ask if the released engine
|
||||
diverges structurally.
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [ ] Tarball builds reproducibly from the package dir
|
||||
- [ ] All 6 demo tests pass in the scratch dir against released npm packages
|
||||
|
||||
**Verification:**
|
||||
- [ ] Manual check: scratch-dir vitest run output attached to the task notes
|
||||
|
||||
**Dependencies:** Task 2 (bundles the final API)
|
||||
|
||||
**Files likely touched:**
|
||||
- `packages/n8n-test/scripts/bundle.mjs`
|
||||
- `packages/n8n-test/package.json`
|
||||
|
||||
**Estimated scope:** M
|
||||
|
||||
## Task 3: Throw-mode errors and network lockdown
|
||||
## Task 4: Demo repo with exploded export, Action, release
|
||||
|
||||
**Description:** When the run finishes with `result.data.resultData.error`,
|
||||
`runWorkflow` rejects with that error (n8n default `onError=stopWorkflow`).
|
||||
Add a vitest setup file calling `nock.disableNetConnect()` (mirroring
|
||||
`node-test-harness.ts:58`) so un-mocked HTTP fails loudly. Add the unhappy-day
|
||||
demo test (500 → `await expect(...).rejects.toThrowError()`).
|
||||
**Description:** Create a disposable repo under James's GitHub (suggestion:
|
||||
`n8n-workflow-tests-demo`): exploded `.n8np` layout (`manifest.json`,
|
||||
`projects/<project>/workflows/*.json` — the two demo workflows, authentic to
|
||||
the Ligo export format), `tests/*.test.ts` importing those JSONs,
|
||||
vanilla vitest config + `reflect-metadata`/nock setup, README, and a
|
||||
`pull_request` GitHub Action (node 22, npm cache) running vitest. Attach the
|
||||
Task 3 tarball as a GitHub release asset; `package.json` installs it by URL.
|
||||
First push must show a green run on the default branch.
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [x] Unhappy-day demo test green: nock 500 → `runWorkflow` rejects with the
|
||||
HTTP Request node's error
|
||||
- [x] Removing a nock intercept makes the test fail with a blocked-connection
|
||||
error, not a real network call
|
||||
- [ ] Fresh clone + `npm ci` + `npx vitest run` passes locally
|
||||
- [ ] Action green on the default branch, wall-clock under ~4 min
|
||||
|
||||
**Verification:**
|
||||
- [x] Tests pass: `pnpm --filter n8n-test demo` (both tests)
|
||||
- [x] Manual check: comment out a nock intercept → loud failure
|
||||
|
||||
**Dependencies:** Task 2
|
||||
|
||||
**Files likely touched:**
|
||||
- `packages/n8n-test/src/run-workflow.ts`
|
||||
- `packages/n8n-test/demo/workflow.test.ts`
|
||||
- `packages/n8n-test/vitest.config.ts` (setup file)
|
||||
|
||||
**Estimated scope:** S
|
||||
|
||||
## Checkpoint: Feasibility proven — review with James before Task 4
|
||||
|
||||
- [x] Both demo tests green through the real engine
|
||||
- [x] Un-mocked network blocked
|
||||
- [x] Go/no-go on polish
|
||||
|
||||
## Task 4: `input` parameter, demo tidy-up, gotchas write-up
|
||||
|
||||
**Description:** Prove the `input` seeding end-to-end (a demo assertion that a
|
||||
field from `input` flows through), make the demo test file read like the Notion
|
||||
"Sample test" (it is the artifact on screen), add a short package README with
|
||||
the demo commands, and write the gotchas list (node-load time, dist-build
|
||||
requirement, error-mode semantics, SSRF-helper seam as production direction)
|
||||
into the spike branch/PR description.
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [x] `runWorkflow(workflowJson, { some: 'field' })` demonstrably influences output
|
||||
- [x] README documents: build prerequisite, `pnpm --filter n8n-test demo`
|
||||
- [x] Gotchas list exists (branch/PR description)
|
||||
|
||||
**Verification:**
|
||||
- [x] Tests pass: `pnpm --filter n8n-test demo`
|
||||
- [x] Manual check: demo dry-run from clean checkout following only the README
|
||||
- [ ] Manual check: Actions run URL green
|
||||
|
||||
**Dependencies:** Task 3
|
||||
|
||||
**Files likely touched:**
|
||||
- `packages/n8n-test/demo/workflow.test.ts`
|
||||
- `packages/n8n-test/demo/workflow.json`
|
||||
- `packages/n8n-test/README.md`
|
||||
**Files likely touched:** (new repo, not the monorepo)
|
||||
|
||||
**Estimated scope:** M
|
||||
|
||||
## Task 5: The demo PR — red → green
|
||||
|
||||
**Description:** In the demo repo, raise a PR that breaks a workflow (change
|
||||
the Edit Fields mapping in the exploded workflow JSON) — the check must fail
|
||||
with a readable assertion diff. Push a fix commit — the check must go green.
|
||||
Leave the PR open as the demo artifact.
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [ ] Failing check on the breaking commit, readable diff in the log
|
||||
- [ ] Green check on the fix commit, each run under ~4 min
|
||||
|
||||
**Verification:**
|
||||
- [ ] Manual check: PR URL with both check runs
|
||||
|
||||
**Dependencies:** Task 4
|
||||
|
||||
**Files likely touched:** (demo repo)
|
||||
|
||||
**Estimated scope:** S
|
||||
|
||||
## Checkpoint: Phase B — PR shows red → green in under ~4 minutes
|
||||
|
||||
## Task 6: README v2, gotchas, verification fan-out, push
|
||||
|
||||
**Description:** README v2 section (`mockNode` usage, bundle command, demo-repo
|
||||
wiring + links), extend the gotchas with what v2 surfaced (npm-runtime deltas,
|
||||
mock-node plumbing), run the adversarial verification fan-out over the v2
|
||||
diff, apply must-fixes, push, and update the draft-PR description with a v2
|
||||
summary.
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [ ] README covers phase A API and phase B wiring with real links
|
||||
- [ ] Verification fan-out findings triaged; must-fixes applied
|
||||
- [ ] Branch pushed; draft PR body updated
|
||||
|
||||
**Verification:**
|
||||
- [ ] Tests pass: `pnpm --filter n8n-test demo`
|
||||
- [ ] Manual check: draft PR renders the v2 summary
|
||||
|
||||
**Dependencies:** Task 5
|
||||
|
||||
**Files likely touched:**
|
||||
- `packages/n8n-test/README.md`
|
||||
- PR #36120 body
|
||||
|
||||
**Estimated scope:** S
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user