github-mirror/n8n
1
0
Fork 0
mirror of https://github.com/n8n-io/n8n.git synced 2026-10-18 05:59:42 +02:00
Code Issues Actions 13 Packages Projects Releases Wiki Activity Graph
workshop/raw-sql
n8n/.github/WORKFLOWS.md

1281 lines
76 KiB
Markdown
Raw Permalink Blame History

# GitHub Actions & CI/CD Documentation
Complete reference for n8n's `.github/` folder.
---
## Folder Structure
```
.github/
├── WORKFLOWS.md # This document
├── CI-TELEMETRY.md # Telemetry & metrics guide
├── pull_request_template.md # PR description template
├── pull_request_title_conventions.md # Title format rules (Angular)
├── actionlint.yml # Workflow linter config
├── docker-compose.yml # DB services for local testing
├── test-metrics/
│ └── playwright.json # E2E performance baselines
├── ISSUE_TEMPLATE/
│ ├── config.yml # Routes to community/security
│ └── 01-bug.yml # Structured bug report form
├── scripts/ # Automation scripts
│ ├── owners/ # Owners scripts (the OWNERS file lives at the repo root)
│ ├── bump-versions.mjs # Calculate next version
│ ├── update-changelog.mjs # Generate CHANGELOG
│ ├── trim-fe-packageJson.js # Strip frontend devDeps
│ ├── ensure-provenance-fields.mjs # Add license/author fields
│ ├── validate-docs-links.js # Check documentation URLs
│ ├── send-build-stats.mjs # Turbo build telemetry → webhook
│ └── docker/
│ ├── docker-tags.mjs # Generate image tags
│ └── docker-config.mjs # Build context config
├── actions/ # Custom composite actions
│ ├── setup-nodejs/ # pnpm + Node + Turbo cache
│ └── docker-registry-login/ # GHCR + DockerHub auth
└── workflows/ # GitHub Actions workflows
```
---
## Architecture Overview
```
┌────────────────────────────────────────────────────────────────────────────┐
│ n8n CI/CD ARCHITECTURE │
├────────────────────────────────────────────────────────────────────────────┤
│ │
│ TRIGGERS PIPELINES OUTPUTS │
│ ──────── ───────── ─────── │
│ │
│ ┌──────────┐ ┌──────────────────────────────────┐ ┌────────────┐ │
│ │ PR │───▶│ ci-pull-requests.yml │───▶│ Checks │ │
│ └──────────┘ │ ├─ build + paths-filter │ │ Gate │ │
│ │ ├─ unit-test (reusable) │ └────────────┘ │
│ ┌──────────┐ │ ├─ typecheck │ │
│ │ Push │───▶│ ├─ lint (reusable) │ ┌────────────┐ │
│ │ master │ │ ├─ e2e-tests (reusable) │───▶│ Coverage │ │
│ └──────────┘ │ └─ security (if .github/**) │ └────────────┘ │
│ └──────────────────────────────────┘ │
│ │
│ ┌──────────┐ ┌──────────────────────────────────┐ ┌────────────┐ │
│ │ Merge │───▶│ release-publish.yml │───▶│ NPM │ │
│ │release/* │ │ ├─ publish-to-npm │ ├────────────┤ │
│ └──────────┘ │ ├─ publish-to-docker-hub │───▶│ Docker │ │
│ │ ├─ create-github-release │ ├────────────┤ │
│ │ ├─ create-sentry-release │───▶│ Sentry │ │
│ │ └─ generate-sbom │ ├────────────┤ │
│ └──────────────────────────────────┘───▶│ SBOM │ │
│ └────────────┘ │
│ ┌──────────┐ ┌──────────────────────────────────┐ │
│ │ Schedule │───▶│ Nightly/Weekly Jobs │ ┌────────────┐ │
│ │ (cron) │ │ ├─ docker-build-push (nightly) │───▶│ Images │ │
│ └──────────┘ │ ├─ test-benchmark-nightly │───▶│ Metrics │ │
│ │ └─ test-e2e-coverage-weekly │ │
│ └──────────────────────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────────────┘
```
---
## Quick Reference
| Prefix | Purpose |
|------------|-----------------------------------------|
| `test-` | Testing (E2E, unit, visual, benchmarks) |
| `ci-` | Continuous integration |
| `util-` | Utilities (notifications, sync, Claude) |
| `build-` | Build processes |
| `release-` | Release automation |
| `sec-` | Security scanning |
| Other | Docker, SBOM, patch releases |
---
## PR Title Conventions
Commits drive changelog generation. Follow Angular convention:
```
Format: <type>(<scope>): <summary>
Types: feat | fix | perf | test | docs | refactor | build | ci | chore
Scopes: API | benchmark | core | editor | * Node (optional)
Examples:
feat(editor): Add dark mode toggle
fix(Slack Node): Handle rate limiting correctly
perf(core): Optimize workflow execution by 20%
refactor: Migrate to TypeScript strict mode (no-changelog)
Breaking Changes: Add "BREAKING CHANGE:" footer with migration guide
Deprecations: Add "DEPRECATED:" footer with update path
Skip Changelog: Add "(no-changelog)" to PR title
```
See `pull_request_title_conventions.md` for full spec.
---
## What Runs When You Open a PR
### Flow Diagram
```
┌──────────────────────────────────────────────────────────────────────────────┐
│ PR OPENED / UPDATED │
└─────────────────────────────────────┬────────────────────────────────────────┘
│
┌───────────────────────────┴───────────────────────┐
▼ ▼
┌───────────────────────────┐ ┌───────────────────────────┐
│ ci-pull-requests.yml │ │ ci-check-pr-title.yml │
│ (main orchestrator) │ │ (validates title format) │
└─────────────┬─────────────┘ └───────────────────────────┘
│
▼
┌───────────────────────────┐
│ install-and-build │
│ └─ paths-filter │──────────────────────────────────────────┐
└─────────────┬─────────────┘ │
│ │
│ [if non-Python files changed] │ [if .github/** changed]
│ │
┌─────────┼─────────┬─────────────┬─────────────┐ │
│ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼
┌───────┐ ┌───────┐ ┌───────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐
│ unit │ │ type │ │ lint │ │ e2e-tests │ │ security │ │ security │
│ test │ │ check │ │ │ │ │ │ checks │ │ checks │
└───┬───┘ └───┬───┘ └───┬───┘ └─────┬──────┘ └─────┬──────┘ └─────┬──────┘
│ │ │ │ │ │
│ │ │ ┌─────┴─────┐ │ │
│ │ │ ▼ ▼ │ │
│ │ │ Internal Fork PR │ │
│ │ │ 14 shards 6 shards │ │
│ │ │ Docker SQLite │ │
│ │ │ │ │
└─────────┴─────────┴──────────┬───────────────┴────────────────┘
│
▼
┌──────────────────────────────┐
│ required-checks │
│ (merge gate) │
└──────────────────────────────┘
```
### Path-Filtered Workflows
These only run if specific files changed:
| Files Changed | Workflow | Branch |
|------------------------------------------------------------------------|-----------------------------|------------|
| `packages/@n8n/task-runner-python/**` | `ci-python.yml` | any |
| `packages/cli/src/databases/**`, `*.entity.ts`, `*.repository.ts` | `test-db.yml` | any |
| `packages/frontend/@n8n/storybook/**`, design-system, chat | `release-storybook.yml` | master |
| `docker/images/n8n-base/Dockerfile` | `build-base-image.yml` | any |
| `**/package.json`, `**/turbo.json` | `build-windows.yml` | master |
| `packages/@n8n/ai-workflow-builder.ee/evaluations/programmatic/python/**` | `test-evals-python.yml` | any |
| `packages/quality/efficiency/scale/benchmark/**` | `build-benchmark-image.yml` | master |
| `packages/cli/src/public-api/**/*.yml`, `packages/cli/src/public-api/**/*.yaml`, `packages/cli/src/public-api/**/*.css`, `packages/cli/src/public-api/v1/openapi-gen/**/*.ts`, `packages/cli/scripts/build.mjs`, `packages/cli/package.json` | `util-publish-api-schema.yml` | master |
| `packages/@n8n/instance-ai/src/**`, `packages/@n8n/instance-ai/skills/**`, `packages/@n8n/instance-ai/knowledge-base/**`, `packages/@n8n/instance-ai/evaluations/**`, `packages/cli/src/modules/instance-ai/**`, `packages/core/src/execution-engine/eval-mock-helpers.ts`, `packages/@n8n/agents/src/**` | `ci-instance-ai-evals.yml` | on PR `opened` / `reopened` / `ready_for_review` |
| `docker/get-n8n.sh`, `docker/get-n8n-compose.yml`, `docker/test-get-n8n.sh` | `test-get-n8n.yml` | any |
### On PR Review
| Event | Workflow | Condition |
|----------------------------|-----------------------------|------------------------------------------------------|
| Review approved | `release-chromatic.yml` | + design files changed |
| Any review | `util-notify-pr-status.yml` | not community-labeled |
**Why Instance AI evals fire once per PR state-change, not per push:** the
workflow eval is the most expensive job in PR CI (LLM-bound builds). Running it
on every push made cost untenable; firing on every review approval cascaded
through the dismiss-stale-on-push → re-approve loop, which also blew up.
The current trigger fires once per `opened` / `reopened` / `ready_for_review`
on a non-fork PR touching the eval surface, and runs the `pr` test-case dataset
(a small set of high-reliability, capability-diverse cases) instead of the full
suite. Test cases are pulled at run time from the LangTracer suite
`baseline` — the source of truth; CI has no disk fallback (local runs
keep `--source disk` for authoring). To
re-run after pushing a fix, dispatch `ci-instance-ai-evals.yml` with the PR
number (optionally `tier: full` for broader coverage) — results post back to
the PR. The lighter `test-evals-discovery.yml` still runs on every push as part
of `ci-pull-requests.yml`.
**`ci-instance-ai-evals.yml` is the PR gate; `test-evals-instance-ai.yml` is
the lab bench.** The gate deliberately exposes only PR re-runs. Anything that
isn't PR gating — baselines, model experiments, arbitrary branch runs — goes
through `test-evals-instance-ai.yml`'s own dispatch form ("Instance AI
Evals: Experiments"): full knob set (branch, filter, tier, suite,
iterations, experiment-name, model, model-url, model-key, reasoning-effort, supports-structured-outputs), no per-PR cancellation (dispatches run in parallel, e.g. concurrent
model-comparison arms), and SHA-keyed docker cache hits on master. Evals never
run on fork PRs: the event trigger gates on `head.repo.fork`, and the `pr`
re-run path refuses fork PRs in `resolve` (dispatched runs carry secrets).
**MCP workflow evals (`ci-mcp-evals.yml`) are manual only (`workflow_dispatch`),
never per-PR or scheduled.** They reuse the Instance AI verifier but build each
workflow through the instance MCP server by driving the `claude` CLI, which adds
Anthropic build cost on top of the verifier — too expensive to run
automatically. The job boots `lanes` n8n containers on one runner and runs a
single `eval:instance-ai --build-via-mcp` process: each case is built by driving
its lane's own MCP server with `claude`, then verified on that same lane
(work-stealing across lanes, capped per-lane). One process → one experiment in
the isolated `mcp-workflow-evals` LangSmith dataset, so there is no shard/merge
step. Dispatch from the Actions tab (set `experiment-name=mcp-baseline` to
refresh the baseline, `filter=<slug>` to run a single case, or `lanes` to widen
parallelism). See the `--build-via-mcp` section in
`packages/@n8n/instance-ai/evaluations/README.md`.
### On PR Close/Merge
| Event | Workflow |
|------------------------------------|--------------------------------|
| PR closed (any) | `util-notify-pr-status.yml` |
| PR merged to `release/*` | `release-publish.yml` |
| PR closed with `codespace-preview` | `util-codespace-preview.yml` |
### Manual Triggers (PR Comments)
| Command | Workflow | Permissions |
|--------------------|------------------------------|---------------------|
| `/test-workflows` | `test-workflows-callable.yml`| admin/write/maintain|
**Why:** Re-run tests without pushing commits. Useful for flaky test investigation.
### Label Triggers
| Label | Workflow | Effect |
|-----------------------|------------------------------|----------------------------------------------------|
| `codespace-preview` | `util-codespace-preview.yml` | Runs the PR in a Codespace, comments the URL |
| `preview:enterprise` | `util-codespace-preview.yml` | Re-serves the instance with an enterprise licence |
| `preview:debug` | `util-codespace-preview.yml` | Re-serves the instance with `N8N_LOG_LEVEL=debug` |
**Why:** A reviewer gets a running instance of the PR without a Docker build or a
cloud deploy. The workflow calls `scripts/codespace-preview/preview.mjs`, which keeps one codespace
for each PR (display name `preview/pr-<number>`) and shares port 5678 with the
organization. A later push serves the new head in the same box. Removing the
label, or closing the PR, deletes the box.
Only a PR from a branch in this repository is eligible: a codespace token is
scoped to `n8n-io/n8n` and cannot check out a fork head.
#### Live progress on the PR
`up` and `refresh` take minutes, and an absent comment looks the same as a broken
preview. So the comment goes up before the box work starts, as a checklist of the
phases in `scripts/codespace-preview/preview-phases.mjs`, and is edited for each phase and once a
minute after that. The final URL replaces it in place.
A comment **edit sends no notification** — only a create does. That is what makes a
heartbeat on a sticky comment acceptable at all.
A `refresh` keeps the URL of the previous run in the checklist. The URL does not
change between runs, and taking a working link off the PR for several minutes is
worse than saying it is briefly down.
A cancelled job — including the 45-minute `timeout-minutes` — kills the script while
its checklist is up, so a last `if: cancelled()` step writes the outcome instead. It
writes **only over a checklist**: a run cancelled while it was queued never started
work, and must leave the previous run's URL alone.
#### Running it by hand
A box sleeps after 2 hours of no use, and GitHub makes every forwarded port
private again at each start. So a preview that slept reaches nobody until
something shares port 5678 again, and its backend is gone with the container.
**Actions → Util: Codespace Preview → Run workflow** does both without a commit
and without a label toggle:
| Input | Meaning |
|---|---|
| `pr_number` | The pull request to act on. |
| `operation` | `up` (default) creates the box if it is gone, starts it if it sleeps, serves the head and shares the port again. `refresh` re-serves the head in a box that already exists. `down` deletes the box. |
Prefer `up` unless you mean to delete: it covers create, wake and re-share.
The button needs write access, and it appears only once the trigger is on
`master` — GitHub lists dispatchable workflows from the default branch. A manual
run takes the branch picked in the dropdown, which is the branch GitHub read the
workflow from, so a branch can test a change to the preview scripts. A fork head
is still refused, by `preview.mjs` rather than by the job's `if`. There is no
`ls` operation: it needs no PR and posts no comment — run `pnpm preview ls`
locally.
#### Preview toggles
A `preview:*` label configures an instance that already exists, so adding or
removing one re-serves the box instead of creating or deleting it. It does
nothing on a PR without `codespace-preview`.
The vocabulary lives in `scripts/codespace-preview/preview-labels.mjs`, which both ends import:
`preview.mjs` turns the PR's labels into slugs, and `preview-serve.mjs` turns
those slugs into environment inside the box. Add a toggle there, in one place.
Two things cross the gap, and both are shape-checked rather than trusted: a
`preview:*` slug, and a phase key from `scripts/codespace-preview/preview-phases.mjs`. The `gh
codespace ssh` command is a shell string that appears in the box's process list, so
a value is never passed through it — `preview:enterprise` resolves to a licence key
inside the box, not on the runner.
`preview:enterprise` needs a **Codespaces** secret named
`N8N_LICENSE_ACTIVATION_KEY`, scoped to `n8n-io/n8n`. That is a Codespaces
secret, not an Actions secret, and it is unrelated to `CODESPACE_PREVIEW_TOKEN`.
Use the sandbox key: the preview sets tenant `1001` to match, and the default
tenant (`1`) rejects it. Without the secret the preview still serves, unlicensed,
and says so in the log.
Note what the label does and does not control. Codespaces injects the secret into
**every** preview box, so the label decides whether the licence reaches n8n, not
whether the key reaches the box. Anyone who can run PR-head code can read
`/workspaces/.codespaces/shared/.env-secrets`. Previews are limited to branches
in this repository, so that is the set of people who already have write access.
#### Preview environment from a webhook
A preview can also take environment from an n8n webhook we control, so a value
can change without a commit and a merge. `scripts/codespace-preview/preview-remote-env.mjs` fetches
it, and `preview-serve.mjs` hands the result to the backend.
It needs three **Codespaces** secrets on `n8n-io/n8n`, again not Actions secrets:
| Secret | Purpose |
| ------------------------ | ------------------------------------------------- |
| `CODESPACE_ENV_URL` | The webhook URL |
| `CODESPACE_ENV_USER` | Basic auth user. Optional, defaults to `preview`. |
| `CODESPACE_ENV_PASSWORD` | Basic auth password |
The webhook answers with a flat JSON object. Its keys become environment
variables and its values are used as-is, so a number or a boolean is stringified
and a nested object is dropped. The request carries the PR number and the head
SHA as query parameters, so one endpoint can answer per PR.
The fetch runs in the box, not on the runner. Codespaces secrets are unreadable
from Actions, so neither the password nor a returned value can reach a CI log.
The log prints key names only.
**Every key is passed through.** A response containing `NODE_OPTIONS`,
`EXTERNAL_HOOK_FILES` or `PATH` runs code inside the preview box, so whoever can
edit that workflow can run code there. The one exception is the preview's own
wiring — the sign-in hook and the owner credentials — which is applied last and
wins. The `.env-secrets` note above applies to these secrets too.
Nothing here is required. Without the secrets, an unreachable webhook or a
rejected password, the preview serves as usual and says so in the log. A wrong
password is not retried: it cannot fix itself.
#### The `CODESPACE_PREVIEW_TOKEN` secret
The job needs `CODESPACE_PREVIEW_TOKEN`, a **fine-grained** personal access token,
held in the `codespaces` environment. Set the resource owner to `n8n-io` and limit
repository access to `n8n-io/n8n`. Grant these repository permissions:
| Permission | Level | What it unlocks |
|---|---|---|
| Metadata | Read | Mandatory, selected for you |
| Codespaces | Read and write | Create, list and delete a box |
| Codespaces metadata | Read | `GET .../codespaces/machines`, which resolves the machine type |
| Codespaces lifecycle admin | Read and write | Start a stopped box, which is what `gh codespace ssh` does |
| Contents | Read | Read the repository |
| Pull requests | Read | `gh pr view`, to resolve the head ref and SHA |
`Codespaces metadata` is a **different permission** from `Codespaces`. Without it
the run fails with `HTTP 403: Resource not accessible by personal access token` on
the `machines` endpoint, after the org billing check has already printed a tick —
so the failure looks unrelated to permissions.
An organization owner may have to approve the token. It stays pending until then.
No other credential can do this:
- `GITHUB_TOKEN` has no Codespaces access.
- A GitHub App **installation** token cannot create a codespace at all. The
Codespaces API belongs to a user, not to an installation.
- A **classic** token is refused by org policy:
`` `n8n-io` forbids access via a personal access token (classic) ``. So no
combination of classic scopes works, whatever the API reference says about the
`codespace` scope.
The environment keeps the token away from every other workflow: only a job that
names `codespaces` can read it. **The environment must allow every branch.** A
`pull_request` run has the ref `refs/pull/<n>/merge`, which no deployment branch
policy matches, so a branch rule would block every preview. Add required
reviewers only if a click for each preview is acceptable.
The codespace belongs to whoever owns the token, and shows up in that account's
codespace list. Billing still goes to the organization, because the repository is
organization-owned and has a Codespaces budget. A service account is therefore
better than a person's account for quota attribution, though the token is scoped
to one repository either way.
The job checks out the base branch, never the PR head, so a PR cannot supply the
script that reads that token. A manual run checks out the branch chosen in the
Run-workflow dropdown, which only a user with write access can pick.
### Other Manual Workflows
| Workflow | Purpose |
|-----------------------------|---------------------------------------------------------|
| `util-codespace-preview.yml`| Wake, re-serve or delete a PR preview instance by hand |
| `util-data-tooling.yml` | SQLite/PostgreSQL export/import validation (manual) |
| `util-probe-registry.yml` | Diagnose slow npm metadata fetches (temporary) |
| `sec-sync-bundle-branches.yml` | Sync both bundle branches on demand, including before a bundle cut |
---
## Workflow Call Graph
Shows which workflows call which reusable workflows:
```
CALLER REUSABLE WORKFLOW
───────────────────────────────────────────────────────────────────────────────
ci-pull-requests.yml
├──────────────────────────▶ test-unit-reusable.yml
├──────────────────────────▶ test-linting-reusable.yml
├──────────────────────────▶ test-e2e-reusable.yml
└──────────────────────────▶ sec-ci-reusable.yml
└──────────▶ sec-poutine-reusable.yml
ci-master.yml
├──────────────────────────▶ test-unit-reusable.yml
├──────────────────────────▶ test-linting-reusable.yml
└──────────────────────────▶ test-single-instance-npm.yml
release-publish.yml
├──────────────────────────▶ docker-build-push.yml
│ └──────────▶ security-trivy-scan-callable.yml
└──────────────────────────▶ sbom-generation-callable.yml
test-sbom-nightly.yml
└──────────────────────────▶ sbom-validation-callable.yml
test-workflows-nightly.yml (manual dispatch only — nightly schedule disabled, DEVP-544)
└──────────────────────────▶ test-workflows-callable.yml
PR Comment Dispatchers (triggered by /command in PR comments):
test-workflows-pr-comment.yml
└──────────────────────────▶ test-workflows-callable.yml
```
---
## Release Lifecycle
```
┌────────────────────────────────────────────────────────────────────────────┐
│ RELEASE LIFECYCLE │
├────────────────────────────────────────────────────────────────────────────┤
│ │
│ STAGE 1: Create Release PR │
│ ─────────────────────────── │
│ Trigger: Manual workflow_dispatch │
│ │
│ release-create-pr.yml │
│ ├─ bump-versions.mjs ────────▶ Calculate X.Y.Z │
│ ├─ update-changelog.mjs ─────▶ Generate CHANGELOG │
│ └─ Create PR: release-pr/X.Y.Z → release/X.Y.Z │
│ │
│ Inputs: │
│ ├─ release-type: patch │ minor │ major │ experimental │ premajor │
│ └─ base-branch: default master │
│ │ │
│ ▼ │
│ STAGE 2: CI Validation │
│ ─────────────────────── │
│ ci-pull-requests.yml runs full suite │
│ ├─ NO ci-check-pr-title.yml (skipped for release branches) │
│ └─ NO release-chromatic.yml (skipped) │
│ │ │
│ ▼ [Merge PR] │
│ STAGE 3: Publish │
│ ─────────────── │
│ release-publish.yml (triggered on merge to release/*) │
│ ├─ publish-to-npm │
│ │ ├─ trim-fe-packageJson.js ───▶ Strip devDeps │
│ │ ├─ ensure-provenance-fields.mjs ───▶ Add license fields │
│ │ └─ npm publish (tag: rc or latest) │
│ ├─ publish-to-docker-hub ────────▶ docker-build-push.yml │
│ │ ├─ Build the application once on amd64 │
│ │ ├─ Build/push images on native amd64 + arm64 runners │
│ │ └─ Merge manifests, then add release provenance │
│ │ └─ Release VEX/SBOM attestations follow provenance │
│ ├─ create-github-release │
│ ├─ create-sentry-release (sourcemaps) │
│ ├─ generate-sbom ────────────────▶ sbom-generation-callable.yml │
│ │ └─ CycloneDX + Cosign signing │
│ └─ trigger-release-note (stable only) │
│ │ │
│ ▼ │
│ STAGE 4: Channel Promotion (optional) │
│ ────────────────────────────────────── │
│ Trigger: Manual release-push-to-channel.yml │
│ ├─ beta ─────▶ npm tags: next, beta │
│ └─ stable ───▶ npm tags: latest, stable │
│ │
└────────────────────────────────────────────────────────────────────────────┘
```
### Recovering a failed release
If the pipeline publishes `n8n@X.Y.Z` to npm and then fails, that version is
burned — npm versions are immutable. Recovery depends on how far it got:
| Failure point | Recovery |
|---|---|
| Before `n8n` reached npm | **Re-run failed jobs** on the original run. Sub-packages publish before `n8n` and `pnpm publish -r` skips versions already on npm, so a retry is safe. |
| After `n8n` reached npm | Dispatch **`release-recreate-failed-release.yml`** with `failed-version: X.Y.Z`. |
`release-recreate-failed-release.yml` checks out `release/X.Y.Z`, bumps only the
root and `packages/cli` versions to `X.Y.(Z+1)`, pushes `release/X.Y.(Z+1)` from
the same commit, and opens an auto-merging `release-pr/X.Y.(Z+1)`. Merging it
fires `release-publish.yml` as normal — `release-publish.yml` has no
`workflow_dispatch`, so a release PR is the only way to re-drive it.
Only those two `package.json` files move: the root version drives every publish
output (git tag, Docker tags, GitHub Release, SBOM) and `packages/cli` drives
the runtime `N8N_VERSION`. Every other package keeps its version, so the publish
step skips the ones already on npm and publishes whichever ones the failed run
never reached.
It refuses to run unless `n8n@X.Y.Z` is on npm and `n8n@X.Y.(Z+1)` is not. That
check fails closed: if the registry can't be reached, the run stops rather than
guessing. Dispatch with `force: true` to skip it.
The burned version stays on npm. Deprecate it by hand once the re-release is
out: `npm deprecate n8n@X.Y.Z "Failed release, use X.Y.(Z+1)"`.
### Other Release Workflows
| Workflow | Trigger | Purpose |
|-------------------------------------------|-----------------|------------------------------------------------|
| `release-standalone-package.yml` | Manual dispatch | Release individual packages (@n8n/codemirror-lang, @n8n/create-node, etc.) |
| `release-create-patch-pr.yml` | Manual dispatch | Open a patch release PR for one track |
| `release-recreate-failed-release.yml` | Manual dispatch | Re-release a version whose publish failed after it reached npm |
---
## Fork vs Internal PR
| Aspect | Internal PR | Fork PR |
|--------------------|----------------------------------|-------------------------|
| E2E Runner | `blacksmith-2vcpu-ubuntu-2204` | `ubuntu-latest` |
| E2E Mode | `docker-build` (multi-main) | `local` (SQLite) |
| E2E Shards | 14 + 2 | 6 + 2 |
| Test Command | `test:container:multi-main:*` | `test:local:*` |
| Secrets | Full access | None |
| Currents Recording | Yes | No |
| Failure Artifacts | No | Yes |
**Why:** Fork PRs cannot access repository secrets. Local mode with SQLite provides feedback without paid services.
---
## ci-master.yml
Runs on push to `master`:
```
Push to master
├─ build-and-format (Blacksmith: build, then format check; populate master cache)
├─ unit-test (matrix: Node 24.18.1, 26.5.1)
│ └─ Coverage only on 24.18.1
├─ lint
├─ (performance: CodSpeed benchmarks, paused until the plugin supports Vitest 5)
├─ verify-single-instance-npm (advisory; packages changed by this push)
└─ notify-on-failure (Slack #alerts-build)
```
---
## Scheduled Jobs
| Schedule (UTC) | Workflow | Purpose |
|---------------------------|-----------------------------------|--------------------------|
| Hourly :00 | `sec-sync-public-to-private.yml` | Mirror public → private |
| Daily 03:00 | `sec-sync-bundle-branches.yml` | Merge the base into `bundle/*` |
| Daily 00:00 | `docker-build-push.yml` | Nightly Docker images |
| Daily 00:00 | `test-db.yml` | Database compatibility |
| Daily 00:00 | `test-e2e-performance-reusable.yml`| Performance E2E |
| Daily 00:00 | `release-storybook.yml` | Storybook deploy |
| Daily 00:00 | `release-chromatic.yml` | Visual regression |
| Daily 00:00 | `util-check-docs-urls.yml` | Doc link validation |
| Daily 01:30, 02:30, 03:30 | `test-benchmark-nightly.yml` | Performance benchmarks |
| Daily 02:00 | `test-get-n8n.yml` | get.n8n.io installer health |
| Daily 02:00 | `test-e2e-pc-nightly.yml` | E2E on the `-pc` image |
| Daily 04:00 | `test-sbom-nightly.yml` | Release and image SBOM license validation |
| Daily 05:00 | `test-benchmark-destroy-nightly.yml`| Cleanup benchmark env |
| Daily 06:00 | `util-sync-master-to-3x.yml` | Replay 3.x onto master (v3) |
| Daily 08:00 | `build-v3-nightly.yml` | Nightly v3 Docker images |
| Monday 00:00 | `util-update-node-popularity.yml` | Node usage stats |
| Monday 02:00 | `test-e2e-coverage-weekly.yml` | Weekly E2E coverage |
| Saturday 22:00 | `test-evals-ai.yml` | AI workflow evals |
| 1st of month 04:00 | `util-refresh-cubic-schema.yml` | Refresh vendored cubic schema |
---
## v3 development (master + 3.x)
The sync runs automation code from the triggering `master` SHA while its working checkout
stays on `3.x`.
During the v3 release window, `master` carries normal feature work (behind opt-in
flags) and the long-lived `3.x` branch carries breaking changes. `util-sync-master-to-3x.yml`
syncs daily by **replaying the `3.x`-only commits onto `master` and force-pushing `3.x`**, so a
clean sync adds no commit and nothing is squashed. What it pushes is always verified to be
exactly the tree a merge of `3.x` and `master` produces, and marker-free. Conflicts confined
to non-lockfile mechanical files (bot-maintained data files — see `MECHANICAL_PATHS` in
`sync-master-to-3x.mjs`) are auto-resolved during the replay. On a code or `pnpm-lock.yaml`
conflict, `3.x` is left untouched and a draft PR carrying the conflict markers (labeled
`automation:v3-sync`, with other mechanical files pre-resolved) is opened on
`sync/master-to-3x`. The lockfile is always left for the resolver. The PR names both ends of
the conflict — the breaking-commit authors and the `master` commits that touched the same
files — via `sync-conflict-owners.mjs`, posts to `#alerts-v3-sync`, and pauses further syncs
until it is resolved and merged normally. Delete/modify conflicts have no markers to carry,
so they are resolved toward `3.x` and listed as an explicit decision in the PR body.
`build-v3-nightly.yml` publishes `n8nio/n8n:v3-nightly[-<date>]` images from `3.x`
by calling `docker-build-push.yml` with `ref: 3.x` + `date_tag`. On Mondays it also
retags that run's n8n + runners manifests as a release candidate (by digest on GHCR, so
the RC is exactly what was built), giving a self-consistent set to trial. Any manual run
can promote too via the `force_rc` dispatch input, several times a day: each publish
claims the next free `v3-rc-<date>.N` as its immutable tag and moves the floating `v3-rc`
and `v3-rc-<date>` onto it. The counter is derived by probing the registry, and the job
is serialized on a `v3-rc-tagging` concurrency group so two runs can't claim one number.
See **[`DEVELOPING_V3.md`](./DEVELOPING_V3.md)** for the full model.
---
## Custom Actions
Composite actions in `.github/actions/`:
| Action | Purpose | Used By |
|--------------------------|----------------------------------------------|--------------------|
| `setup-nodejs` | pnpm + Node.js + Turbo cache + Docker (opt) | Most CI workflows |
| `run-workflow-script` | Run a `.github/scripts` module with no setup or install | Owners and PR quality checks |
| `docker-registry-login` | GHCR + DockerHub + DHI authentication | Docker workflows |
### setup-nodejs
```yaml
inputs:
node-version: # default: '26.5.1'
enable-docker-cache: # default: 'false' (Blacksmith Buildx)
docker-cache-key: # required when enable-docker-cache is true
build-command: # default: 'pnpm build'
```
The pnpm version comes from the `packageManager` field in the root
`package.json`, through `resolve-pnpm-version.mjs`. There is no version input to
keep in sync: a bump in `package.json` moves the setup step, the cache key and
the version check together.
The action caches the pnpm executable in `~/setup-pnpm` by OS, architecture and
version. A cache miss runs `pnpm/setup` and retries once if the registry request
fails. The action verifies the version and saves the cache before the rest of
the job can fail.
Concurrent jobs can all miss a new key before the first save completes. Jobs
that start after the save use the cached executable. Windows keeps the standard
`pnpm/setup` path because its runner cannot activate the cached POSIX home path.
The existing `actions/setup-node` cache continues to store the pnpm package
store.
The Blacksmith layer cache lives on a sticky disk identified by
`docker-cache-key`, and commits are last-writer-wins. Splitting the key per
image would avoid that, but Blacksmith currently never populates a
newly created sticky disk - it stays at 0 bytes however many runs commit to it,
while the build reports a successful commit. Every job therefore shares the
`n8n-io/n8n` key, which is the only disk that actually retains layers. Revisit
once new-disk retention works.
### run-workflow-script
```yaml
inputs:
script: # path of the module, relative to the repository root
github-token: # token for the Octokit client, also exported as GITHUB_TOKEN
```
Runs one `.github/scripts` module through `actions/github-script`. That action
brings its own Node.js and an Octokit client, so the job needs no
`setup-nodejs` step and no dependency install. The action loads
`github-helpers.mjs`, hands the client to `setOctokit`, then imports the module
and awaits its exported `main()`.
Use it for a module that imports only node builtins and other `.github/scripts`
modules. `github-helpers.mjs` loads `@actions/github` and `semver` only when
they are present, so it works in both modes. A module that needs an npm
package (`semver`, `yaml`, `minimatch`, ...) keeps the `setup-nodejs` path with
the `.github/scripts` install command.
Pair it with a sparse checkout of `.github` when the module reads nothing else
from the tree. Cone mode always includes the root files, so `OWNERS` and
`package.json` stay available.
### docker-registry-login
```yaml
inputs:
login-ghcr: # default: 'true'
login-dockerhub: # default: 'false'
login-dhi: # default: 'false'
```
### External actions
Actions consumed from other n8n-io repositories, SHA-pinned like any third-party
action:
| Action | Purpose | Used By |
|-----------------------------------|-------------------------------------------------------------------------------|--------------------|
| `n8n-io/github-actions/cla-check` | CLA signature check: `CLA Check` commit status, in-place PR comment, `cla-signed` label | `ci-cla-check.yml` |
Behaviour changes belong in that repo; bumping the pin here is what picks them up.
A `/cla-check` comment on a PR re-runs the check without a push.
---
## Reusable Workflows
Workflows with `workflow_call` trigger:
| Workflow | Inputs | Purpose |
|------------------------------------|-----------------------------------------------|-----------------------|
| `test-unit-reusable.yml` | `ref`, `nodeVersion`, `collectCoverage` | Unit tests |
| `test-linting-reusable.yml` | `ref`, `nodeVersion` | ESLint |
| `test-e2e-reusable.yml` | `branch`, `test-mode`, `shards`, `runner` | Core E2E executor |
| `test-workflows-callable.yml` | `git_ref`, `compare_schemas` | Workflow tests |
| `docker-build-push.yml` | `n8n_version`, `release_type`, `push_enabled`, `ref`, `date_tag`, `create_attestations` | Docker build |
| `sec-ci-reusable.yml` | `ref` | Security orchestrator |
| `sec-poutine-reusable.yml` | `ref` | Poutine scanner |
| `sec-sync-retarget-prs.yml` | none | Move bundle PRs back onto `bundle/*` |
| `security-trivy-scan-callable.yml` | `image_ref` | Trivy scan |
| `sbom-generation-callable.yml` | `n8n_version`, `release_tag_ref` | SBOM generation |
| `sbom-validation-callable.yml` | `sha` | Read-only SBOM validation |
| `test-single-instance-npm.yml` | `scope`, `base-ref`, `base-branch`, `blocking`, `timeout-minutes` | Dependency duplication |
---
## Scripts
Scripts in `.github/scripts/`:
### Release Scripts
| Script | Purpose | Called By |
|-------------------------------|----------------------------|-------------------------|
| `bump-versions.mjs` | Calculate next version | `release-create-pr.yml` |
| `update-changelog.mjs` | Generate CHANGELOG | `release-create-pr.yml` |
| `prepare-rerelease.mjs` | Bump root + cli for a re-release | `release-recreate-failed-release.yml` |
| `trim-fe-packageJson.js` | Strip frontend devDeps | `release-publish.yml` |
| `ensure-provenance-fields.mjs`| Add license/author fields | `release-publish.yml` |
### Docker Scripts
| Script | Purpose | Called By |
|-------------------------|-------------------|------------------------|
| `docker/docker-config.mjs`| Build context | `docker-build-push.yml`|
| `docker/docker-tags.mjs` | Image tags | `docker-build-push.yml`|
| `docker/kafka-native-smoke-check.mjs`| Verify librdkafka binary loads in built image | `docker-build-smoke.yml`|
| `docker/assert-manifest-format.mjs`| Assert a merged manifest is an OCI image index with the expected platforms | `docker-build-push.yml`|
| `docker/should-smoke-build.mjs`| Narrow the `pnpm-workspace.yaml` smoke trigger to native dependency pins | `docker-build-smoke.yml`|
| `attest-image-sbom.mjs` | Generate, validate, and optionally attest image SBOMs | `docker-build-push.yml`, `test-sbom-nightly.yml` |
### Validation Scripts
| Script | Purpose | Called By |
|-------------------------|-------------------|---------------------------|
| `validate-docs-links.js`| Check doc URLs | `util-check-docs-urls.yml`|
| `send-build-stats.mjs` | Build telemetry | `setup-nodejs` action |
| `resolve-pnpm-version.mjs` | Publish the pinned pnpm version and its executable cache key | `setup-nodejs` action |
| `nightly-sbom-context.mjs` | Resolve the source SHA and image tag for nightly SBOM validation | `test-sbom-nightly.yml` |
| `db-test-matrix.mjs` | DB test matrix from `postgres-versions.json` | `ci-pull-requests.yml` |
| `quality/check-cubic-config.mjs` | Validate `cubic.yaml` against the vendored cubic schema; enforce its silent agent/character limits. `--refresh` re-pulls the schema | `test-workflow-scripts-reusable.yml`, `util-refresh-cubic-schema.yml` |
| `glob.mjs` | Builtin-only glob matcher for changed-file paths (`**`, `*`, dot segments) | `quality/check-pr-size.mjs` |
| `probe-registry.mjs` | Registry path throughput probe (temporary) | `util-probe-registry.yml` |
### Preview Scripts
| Script | Purpose | Called By |
|---------------------------------|-------------------------------------------------------------------------|--------------------------------|
| `codespace-preview.mjs` | Map a `pull_request` event or a manual operation onto a preview operation, comment the result | `util-codespace-preview.yml` |
| `../../scripts/codespace-preview/preview.mjs` | One codespace for each PR: `up`, `refresh`, `down`, `ls`. `--json` for CI | `codespace-preview.mjs`, developers |
| `../../scripts/codespace-preview/preview-remote-env.mjs` | Fetch extra environment for a preview from the webhook, inside the box | `../../scripts/codespace-preview/preview-serve.mjs` |
| `../../scripts/codespace-preview/preview-phases.mjs` | The phase vocabulary and its one-line marker, so the runner, the box and the comment cannot drift | `codespace-preview.mjs`, `../../scripts/codespace-preview/preview.mjs`, `../../scripts/codespace-preview/preview-serve.mjs` |
`scripts/codespace-preview/preview.mjs` is also the developer entry point (`pnpm preview up <pr>`).
In `--json` mode stdout carries one line for each phase and then the report object,
and all human progress goes to stderr. So a workflow can follow a run that also
streams an in-box build log. The reader tells the two apart by the `url` field: the
report has one, a phase line never does.
### Branch Replay Scripts
Both keep a long-lived branch that is "base + its own commits" in sync by rebasing those
commits onto the base and force-pushing, sharing the merge-tree content guard that makes the
rewrite safe.
| Script | Purpose | Called By |
|----------------------------|----------------------------------------------------------------------|------------------------------------|
| `branch-replay.mjs` | Shared primitives: merge-tree, tree guard, marker scan | the two scripts below |
| `sync-master-to-3x.mjs` | master → `3.x`, rebased; auto-resolves mechanical files, opens a conflict PR | `util-sync-master-to-3x.yml` |
| `sync-bundle-branch.mjs` | base → `bundle/*` in n8n-private, merged; skips while the base is unpublished; fail-loud, never resolves conflicts | `sec-sync-bundle-branches.yml` |
### Slack Scripts
See [Slack Notifications](#slack-notifications) for the calling pattern.
| Script | Purpose |
|---------------------------------|-------------------------------------------------------------------------------|
| `slack/notify.mjs` | CLI + `sendSlackMessage` export. POSTs `chat.postMessage`, fails on `ok:false`. |
| `slack/build-trivy-blocks.mjs` | `--blocks trivy` — vulnerability digest |
---
## Telemetry
CI metrics are collected via webhooks to n8n, then stored in BigQuery for analysis.
See **[CI-TELEMETRY.md](CI-TELEMETRY.md)** for:
- Common data points (git, CI context, runner info)
- Existing implementations (build stats, container stack)
- How to add new telemetry
- BigQuery schema patterns and queries
---
## OWNERS
Team ownership lives in the top-level `OWNERS` file. The scripts that consume
it live in `.github/scripts/owners/`. Line format:
```
<pattern> <@org/team> [required]
```
Patterns are a catch-all (`*`), a directory prefix (`packages/x/`), or an
exact file path. Matching is last-match-wins, so specific rules must come
after general rules.
The format is strict, enforced by `node .github/scripts/owners/owners.mjs --check`:
- Tokens on a line come in a fixed order: pattern, one team, then options
such as `required`.
- Directory patterns end with `/` and must be existing directories; all other
patterns must be existing files. Duplicate patterns are rejected.
Team existence is not checked by `--check` (it needs an org read token, which
fork PRs do not have); a separate workflow covers it (DEVP-891).
The file drives four workflows:
| Workflow | Purpose |
|-----------------------------------|--------------------------------------------------------------------------|
| `ci-owners-validation.yml` | Validates OWNERS (syntax, dead paths) via `owners.mjs --check` |
| `ci-owners-review-recommendations.yml` | Advisory PR comment: reviewer teams, line stats, required reviews |
| `ci-owners-assign-reviewers.yml` | Opt-in reviewer auto-assignment (label-triggered) |
| `ci-owners-required-reviews.yml` | Enforces `required` entries via the "Required Reviews" commit status |
### Required reviews
An entry with the `required` option makes team approval mandatory: when a PR
changes a non-test file whose winning entry carries `required`, a member of each
listed team must approve the PR. Test files match the shared patterns in
`test-files.mjs` and do not trigger required reviews.
`ci-owners-required-reviews.yml` evaluates this on
PR changes and review events, and reports a commit status
named **Required Reviews** on the head SHA. A missing approval reports
`pending` ("Waiting for approval from: …"), not `failure`, so an unreviewed PR
does not show red CI; any non-success state blocks the merge equally. The
ruleset for a branch must list that status as a required check for the block
to take effect. Merge-queue runs report success on the queue head without
re-evaluating: a PR cannot enter the queue unless the status is green on its
head, and the queue does not change approvals.
The status is evaluated for a PR into any base branch, from a same-repo head
or a fork. Both matter because the ruleset that gates a PR is not always the
one on its base branch: GitHub applies the ruleset of a stack's target branch
to every PR in the stack, so a stacked PR into a feature branch is gated by
the `master` ruleset. A required status that no run ever writes leaves the PR
blocked on "Expected". Routes that skip the evaluation are listed in
`REQUIRED_REVIEW_EXEMPTIONS` in
`required-reviews.mjs`. An entry is `<head> -> <base>` or just `<base>`
(any head); `*` matches any run of characters. An exempt PR reports
`success` with the route in the description. Only heads in this repository
can match, so a fork branch with a matching name is still evaluated. Add a
route only when every commit it carries was already reviewed elsewhere, as
with `sync/master-to-3x -> 3.x`: its commits landed on `master` first.
The workflow reads OWNERS, its scripts and the exemption routes from `master`
only, never from the base branch or the PR: any writable branch can be a base,
so only `master` is trusted input. A PR cannot lift its own review
requirement. A retarget re-evaluates the PR, so a verdict computed against the
old base does not carry over.
Members of the `large-scale-changes` GitHub team can make the **Required
Reviews** status succeed without required OWNERS team approvals. A team member
must apply the `large-scale-change` label. The gate checks the actor from the
label event and verifies their current team membership. The label alone is not
sufficient. Label changes re-evaluate the status. Removing the label removes
the exemption.
Every path that writes the status runs in the base repository context, because
a fork-context run has no secrets and a read-only token. PR changes arrive
through `pull_request_target`, which is safe here because no step checks out
or runs PR code. Review events on a same-repo PR arrive through
`pull_request_review`. Review events on a fork PR arrive through the
`workflow_run` of `ci-pull-request-review.yml`, which runs on every submitted
or dismissed review; the owners workflow looks the PR up from that run's head
and skips same-repo heads, which the direct event already covers. A first
contribution whose runs still wait for approval gets no review-event
re-evaluation until a maintainer approves the runs; `workflow_dispatch` with
the PR number is the manual fallback.
---
## Runner Selection
| Runner | vCPU | Use Case |
|-------------------------------------|------|-----------------------------|
| `ubuntu-slim` | 1 | Gate jobs (required-checks) |
| `ubuntu-latest` | 2 | Simple jobs, fork PR E2E |
| `blacksmith-2vcpu-ubuntu-2204` | 2 | Standard builds, E2E shards |
| `blacksmith-4vcpu-ubuntu-2204` | 4 | Unit tests, typecheck, lint |
| `blacksmith-8vcpu-ubuntu-2204` | 8 | Heavy parallel workloads |
| `blacksmith-8vcpu-ubuntu-2204-arm` | 8 | ARM64 Docker builds |
### Selection Guidelines
**`ubuntu-slim`** - Status check aggregation, gate/required-check jobs, notifications
**`ubuntu-latest`** - Simple build verification, scheduled maintenance, PR comment handlers, release tagging, Docker manifest creation, any job where speed is not critical
**`blacksmith-2vcpu-ubuntu-2204`** - Initial build/install (benefits from Blacksmith caching), database integration tests (I/O bound), Chromatic/Storybook builds
**`blacksmith-4vcpu-ubuntu-2204`** - Unit tests (parallelized), linting (parallel file processing), typechecking (CPU-intensive), E2E test shards
**`blacksmith-8vcpu-ubuntu-2204`** - Heavy parallel workloads. The PR `install-and-build` job uses it with the default runner provider. Most PR jobs wait for that job, and a cold build keeps all 4 vCPUs of a smaller runner busy.
### Runner Provider Toggle
The `RUNNER_PROVIDER` repository variable controls runner selection across workflows:
| Value | Behavior |
|-------|----------|
| (unset) | Use Blacksmith runners (default) |
| `github` | Use GitHub-hosted `ubuntu-latest` |
**Note:** When set to `github`, all jobs use `ubuntu-latest` regardless of any runner inputs or defaults specified in reusable workflows. GitHub runners have fewer vCPUs (2 vs 4), so jobs may run slower.
---
## Security
### Why We Do This
Supply chain security ensures artifacts haven't been tampered with. We provide three types of signed attestations:
```
ATTESTATION (signed statement)
│
┌─────────────────┼─────────────────┐
│ │ │
▼ ▼ ▼
PROVENANCE SBOM VEX
"Trust the "Know the "Understand
build" contents" the risk"
```
| Attestation | Question It Answers |
|-------------|--------------------------------|
| **Provenance** | "Can we trust this artifact came from n8n's CI and wasn't tampered with?" |
| **SBOM** | "What dependencies are inside?" (license compliance, vulnerability scanning) |
| **VEX** | "The scanner found CVE-X - does it actually affect us or is it a false positive?" |
**How they relate:**
- **SBOM** is the ingredients list - input for both license checks AND security scanning
- **VEX** is the security triage output - "we investigated CVE-X, here's our assessment"
- **Provenance** proves the SBOM and VEX came from our CI, not an attacker
---
### Poutine (Supply Chain)
- **Runs on:** PR changes to `.github/**`
- **Detects:** Exposed secrets, insecure workflow configs
- **Output:** SARIF to GitHub Security tab
### Trivy (Container)
- **Runs on:** stable/nightly/rc Docker builds
- **Scans:** n8n image, runners image
- **Output:** GitHub Actions step summary (`$GITHUB_STEP_SUMMARY`) and run logs
### SBOM
There are two, with different subjects and different consumers. They are not duplicates.
| | Release SBOM | Image SBOM |
|---|---|---|
| **Job** | `generate-and-attach-sbom` (`sbom-generation-callable.yml`) | `sbom-attestation` (`docker-build-push.yml`) |
| **Scans** | the deployed npm closure in `compiled/` (`cdxgen -t pnpm`) | each pushed image, by digest (`syft`) |
| **Covers** | npm only | OS packages **and** npm, as laid down in the image |
| **Signing** | GitHub Attestation API, subject `./package.json` | `cosign attest`, subject = image digest |
| **Output** | `sbom-source.cdx.json`, `THIRD_PARTY_LICENSES.md`, `vex.openvex.json` on the GitHub Release | attestation in the registry beside the image |
| **Consumer** | humans — legal/license compliance; backs `/third-party-licenses` | machines — `cosign verify-attestation`, admission control |
Format is CycloneDX JSON 1.6 for both. Each pipeline pins the schema version, so a
scanner upgrade cannot change the shape of a signed artifact without a visible diff.
The two use different scanners on purpose. The release SBOM runs `cdxgen -t pnpm` over the
resolved pnpm closure with `FETCH_LICENSE=true`, because a lockfile scan has no package files
to read licenses from. The image SBOM runs `syft` over the pushed image, which resolves
licenses from the LICENSE files on disk and so needs no network at all.
The image job used to run `cdxgen -t docker --profile license-compliance`. That profile sets
`FETCH_LICENSE=true` and nothing else, so it made one sequential npm registry call per
component — roughly 3,700 per release, about half the job's runtime. syft resolves the same
licenses locally in a fraction of the time, and catalogues more of the image besides.
A/B any scanner change against the current output before shipping it. The gate only enforces
`pkg:npm/`, so a change can silently degrade PyPI or OS license coverage while CI stays green.
Compare the licenses resolved per component, not just the component counts.
`enrich-sbom.mjs --drop-phantom-npm` removes scan artefacts that would otherwise assert
components the image does not contain: nested test/fixture `package.json` and `exports`
subpaths. It reads the component's source path from either scanner's property name (`SrcFile`
for cdxgen, `syft:location:0:path` for syft) and treats syft's `version: "UNKNOWN"` the same as
a missing version.
Packages whose license cannot be resolved from disk go in
`scripts/licenses/license-overrides.json` with a verified `source` citation — the upstream
LICENSE file, not registry metadata.
`test-sbom-nightly.yml` runs at 04:00 UTC. It waits up to two hours for the current scheduled
Docker build to complete. It builds the production deployment closure at that run's SHA and
validates the release SBOM. It also resolves the four immutable SHA image tags from that
build and validates each image SBOM. The validation uses the same enrichment and SPDX gates
as a release. It does not publish, attest, or upload an artifact. A failure reports to the
Developer Platform Slack channel.
### SLSA L3 Provenance
SLSA (Supply-chain Levels for Software Artifacts) Level 3 provides cryptographic proof of build integrity.
| Artifact | Generator | Level |
|----------|-----------|-------|
| Docker images | `slsa-framework/slsa-github-generator` | L3 |
| npm packages | `NPM_CONFIG_PROVENANCE=true` | L3 |
**Docker provenance** uses the SLSA GitHub Generator as a reusable workflow (not an action). This is required for L3 because provenance must be generated in an isolated environment the build can't tamper with.
```yaml
# IMPORTANT: Must use semantic version tags (@vX.Y.Z), NOT commit SHAs.
# The slsa-verifier requires tagged versions to verify authenticity.
uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v2.1.0
```
**Verify provenance:**
```bash
# Docker
slsa-verifier verify-image ghcr.io/n8n-io/n8n:VERSION \
--source-uri github.com/n8n-io/n8n
# npm
npm audit signatures n8n@VERSION
```
### VEX (Vulnerability Exploitability eXchange)
VEX documents which CVEs actually affect n8n vs false positives from scanners.
- **File:** `security/vex.openvex.json`
- **Format:** OpenVEX (broad scanner compatibility - Trivy, Docker Scout, etc.)
- **Attached to:** GitHub Release, Docker image attestations
- **Used by:** Trivy scans (via `security/trivy.yaml`)
**VEX Status Types:**
| Status | Meaning |
|--------|---------|
| `not_affected` | CVE doesn't impact n8n (code not reachable, etc.) |
| `affected` | CVE impacts n8n, tracking fix |
| `fixed` | CVE was present, now fixed |
| `under_investigation` | Assessing impact |
**Verify VEX attestation:**
```bash
cosign verify-attestation --type openvex \
--certificate-identity-regexp '.*github.com/n8n-io/n8n.*' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
ghcr.io/n8n-io/n8n:VERSION
```
**Adding a CVE statement to security/vex.openvex.json:**
```json
{
"statements": [
{
"vulnerability": { "name": "CVE-2024-XXXXX" },
"products": [{ "@id": "pkg:github/n8n-io/n8n" }],
"status": "not_affected",
"justification": "vulnerable_code_not_in_execute_path",
"statement": "n8n does not use the affected code path in this dependency"
}
]
}
```
### Public ↔ private sync (bundle branches)
Embargoed security work happens in `n8n-io/n8n-private`. `sec-sync-public-to-private.yml`
runs hourly there (and on `workflow_dispatch` with `force` for conflict recovery),
mirroring public `master` and `1.x` into private with `reset --hard` +
`--force-with-lease` — skipping a branch when private is ahead, ignoring `chore: Bundle`
commits when judging "ahead". A skipped branch, or a hard failure, is reported to
`#alerts-build`: a non-`chore: Bundle` commit on private `master`/`1.x` leaves the mirror
stuck every hour until it is removed or a `force` dispatch overwrites it. Fixes are never
committed to private `master`/`1.x` directly: `ci-restrict-private-merges.yml` requires
PRs into them to come from the long-lived integration
branches `bundle/2.x` and `bundle/1.x` (a `bundle/2.x` merge is
backported to `bundle/1.x` by `util-backport-bundle.yml`). Once a bundle branch is merged
into private `master`/`1.x` as a `chore: Bundle/*` PR, `sec-publish-fix.yml` /
`sec-publish-fix-1x.yml` cherry-pick that commit onto a fresh branch in the public repo and
open the PR there. That PR **must stay a single-parent squash** — the publish step is a bare
`git cherry-pick` of `HEAD`, which aborts on a merge commit. A `chore: Bundle/*` PR whose
*Required Checks* go red holds back every fix batched into it, so `ci-pull-requests.yml`
posts to `#alerts-build` when that gate fails on a PR opened *from* `bundle/2.x` or
`bundle/1.x` (link only, no PR title, since the branch is embargoed).
`sec-sync-bundle-branches.yml` keeps those branches current nightly at 03:00 UTC and on
`workflow_dispatch`, including before a bundle cut. It **merges the public base into** the
bundle branch via
[`scripts/sync-bundle-branch.mjs`](scripts/sync-bundle-branch.mjs) and pushes without forcing.
Every push is verified to carry exactly the tree a merge of the two sides would produce (`git
merge-tree`); a mismatch, or a conflict marker, fails the run instead of pushing.
**A bundle branch is only ever built on published history.** The sync fetches `master` or `1.x`
directly from `https://github.com/n8n-io/n8n.git`. It creates a missing bundle branch at that
exact public SHA and merges that public SHA into an existing bundle branch. It never reads the
private base branch. This matters because the `chore: Bundle/*` squash on private `master` is
private-only until publication. The mirror can replace that commit with the public copy after
the bundle branch is deleted. The next bundle branch must not inherit the replaced commit.
Deleting a bundle branch takes its open PRs with it: GitHub moves each one onto the deleted
branch's own base, and re-creating the branch does not move them back. The next nightly or manual
sync re-creates the branch and then calls
[`sec-sync-retarget-prs.yml`](workflows/sec-sync-retarget-prs.yml), which moves every open PR
on `master` onto `bundle/2.x` and every one on `1.x` onto `bundle/1.x`. It skips a bundle
branch that does not exist, and skips PRs whose *head* is `bundle/*` — those are the
`chore: Bundle/*` cut PRs, which target the base on purpose. It is also dispatchable on its
own. The retarget runs inside the bundle sync after a missing branch is recreated.
**`bundle/*` is append-only — never rebase it, never force-push it.** These branches receive
PRs, and rewriting a branch that receives PRs orphans the copies of its commits that the open
PR branches already contain: every such PR's merge base regresses to an old base commit, so
GitHub shows it carrying everyone else's fixes, in the commit list *and* in the diff (which
can then trip required checks like *PR Size Limit*). It compounds — each refresh between
rewrites picks up another duplicate generation of the same fixes and starts conflicting with
itself. To refresh a fix branch, use GitHub's **Update branch** button or
`git merge origin/bundle/2.x`; squash-merging a fix *into* the bundle branch leaves every
sibling PR's merge base untouched, which is why only a rewrite breaks this.
The costs of merging are deliberate and paid for: a merge commit per run, and fixes that have
already been published staying in the branch's log. Neither reaches anything downstream,
because a bundle publishes as one squashed commit taken from the tree rather than the history —
the `chore: Bundle/*` PR's **diff** stays
exactly the pending fixes even when its commit list does not. For a list of what a bundle
actually carries, read the fix PRs merged into the branch since the last cut, not `base..bundle`.
There is **one job per bundle branch**. A conflict is detected from the merge tree before the
working tree is touched, so the branch is left exactly as it was, that job **fails** (no
green runs hiding a stalled branch) and `#alerts-security` gets a run link — while the other
branch still syncs. Recovery is deliberate: merge the public base into the branch locally,
resolve, push, then re-run the workflow — and that resolution then lives in the merge commit
instead of being re-litigated on every later run. The sync never resolves a conflict itself,
unlike `util-sync-master-to-3x.yml`.
See **[`../AGENTS.md`](../AGENTS.md)** ("Security Fix Hygiene") for the naming rules that
keep the vulnerability out of public branch names, commits, and test descriptions.
---
## Slack Notifications
All workflows post via `.github/scripts/slack/notify.mjs` — a direct `fetch` to `chat.postMessage` that exits non-zero on any Slack error. No third-party action; no silent swallowing.
```yaml
notify-on-failure:
runs-on: ubuntu-latest
needs: [build]
if: ${{ always() && contains(needs.*.result, 'failure') }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
sparse-checkout: .github/scripts/slack
sparse-checkout-cone-mode: false
- name: Notify Slack
env:
SLACK_TOKEN: ${{ secrets.QBOT_SLACK_TOKEN }}
run: |
node .github/scripts/slack/notify.mjs \
--channel '#alerts-build' \
--text 'Build failed - ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}'
```
If notify is a step inside an existing checked-out job, skip the `checkout` and gate with step-level `if: failure()` instead.
> `if: failure()` at the **step** level of a dedicated notify job is a no-op when a `needs:` dependency fails (the job is skipped before steps evaluate). Always gate the **job** with `if: ${{ always() && contains(needs.*.result, 'failure') }}`.
**Rich payloads (Block Kit):** add `build-<name>-blocks.mjs` whose default export returns a blocks array, then pass `--blocks <name>` plus any workflow-specific args. Builders read repo / run context from `GITHUB_*` runner env vars. Kebab-case flags become camelCase keys for the builder (`--image-ref` → `imageRef`).
| Token | Bot | Channels |
|------------------------------|----------------|-------------------------------------------------------------|
| `QBOT_SLACK_TOKEN` | QBot | Default — engineering / build / security |
| `RELEASE_HELPER_SLACK_TOKEN` | Release Helper | `#releases` (C036AELNMV0) |
Adding a new channel requires inviting the bot first; the first run otherwise fails loudly with `not_in_channel`. Private-repo workflows (`sec-publish-fix*.yml`, `sec-sync-public-to-private.yml`, and the bundle-PR alert in `ci-pull-requests.yml`) need `QBOT_SLACK_TOKEN` set in `n8n-io/n8n-private`; the scripts themselves are mirrored by `sec-sync-public-to-private.yml`.
---
## Secrets
### By Category
| Category | Secrets |
|---------------------|-------------------------------------------------------------|
| Package Publishing | `NPM_TOKEN`, `DOCKER_USERNAME`, `DOCKER_PASSWORD` |
| Notifications | `QBOT_SLACK_TOKEN`, `RELEASE_HELPER_SLACK_TOKEN` |
| Code Quality | `CODECOV_TOKEN`, `CHROMATIC_PROJECT_TOKEN`, `CURRENTS_RECORD_KEY` |
| Error Tracking | `SENTRY_AUTH_TOKEN`, `SENTRY_ORG`, `SENTRY_*_PROJECT` |
| Cloud/CDN | `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID` |
| GitHub Automation | `N8N_ASSISTANT_APP_ID`, `N8N_ASSISTANT_PRIVATE_KEY` |
| Benchmarking | `BENCHMARK_ARM_*`, `N8N_BENCHMARK_LICENSE_CERT` |
| AI/Evals | `EVALS_ANTHROPIC_KEY`, `EVALS_OPENAI_KEY`, `EVALS_OPENROUTER_KEY`, `EVALS_XAI_KEY`, `EVALS_BASETEN_KEY`, `EVALS_FIREWORKS_KEY`, `EVALS_TOGETHER_KEY`, `EVALS_DATABRICKS_KEY`, `EVALS_MODAL_KEY`, `EVALS_LYCEUM_KEY`, `EVALS_AZURE_FOUNDRY_KEY`, `EVALS_VERTEX_KEY`, `EVALS_VERTEX_PROJECT_ID`, `EVALS_VERTEX_LOCATION`, `EVALS_LANGSMITH_*` |
### Scoping
- **`secrets: inherit`** - passes all secrets to reusable workflows
- **Explicit passing** - for minimal exposure
- **Environment: `benchmarking`** - Azure OIDC credentials
---
## Future Vision
### Redundancy Review
Comment trigger (`/test-workflows`) is a workaround.
Long-term: Main CI should be reliable enough to not need these.
### Workflow Testability
- Tools like `act` for local testing
- Unit tests for `.github/scripts/*.mjs`
- Validation with `actionlint`
Reference in New Issue View Git Blame Copy Permalink