n8n/scripts/mutation-health

scripts/mutation-health/

Phase 1 substrate for the Mutation Health Observability initiative.

What is mutation testing?

Line coverage tells you which lines your tests execute. Mutation testing tells you which behavioural changes your tests catch. A file can have 100% line coverage and a 0% mutation score: every line runs during the test suite, but no test would fail if the code were silently broken.

How it works

A mutation testing tool (n8n uses Stryker) does this for each source file:

1. Parse the source into an AST.

2. Generate small variants ("mutants") by changing nodes in the AST. Examples:

   Mutator	Original	Mutated
   Conditional	if (item.mode === 'everyX')	if (true), if (false)
   Equality	a === b	a !== b
   Boundary	value > 0	value >= 0
   Arithmetic	return a + b	return a - b
   String literal	'hello'	'', "Stryker was here!"
   Block statement	{ x(); return; }	{}
   Conditional (ternary)	cond ? a : b	a, b, cond ? a : a, cond ? b : b

   There are ~40 mutator categories. One source line typically produces several mutants.

3. For each mutant, run the test suite against the mutated code. One of these outcomes:

   Outcome	Meaning
   Killed	At least one test failed → tests caught the change. ✓
   Survived	All tests passed → tests didn't catch the change. ✗
   NoCoverage	No test even ran the mutated line.
   Timeout	Tests hung (counted as detected).

4. Mutation score = (killed + timeout) / (killed + timeout + survived + no_coverage). Higher = more load-bearing assertions.

Line coverage vs mutation score — a real example

packages/workflow/src/workflow-checksum.ts:

• Line coverage: 87.09%
• Mutation score: 38.64%

Mutating let hexString = '' to let hexString = "Stryker was here!" survived the test suite. The tests assert that two similar workflows produce different checksums — but never pin the actual output format. Line coverage calls this fine; mutation testing flags it as assertion-light test theatre.

That divergence is exactly why this project exists.

---

What's in this directory

File	Purpose
pick-next.mjs	Walk <pkg>/src/ (per-package mode) or every vitest-eligible package (global mode, --global), merge with the live ledger, return the next source file(s) to mutate
mutate.mjs	Run Stryker on one source file of any vitest package, write summary.json
stryker.default.mjs	Default Stryker config for onboarded packages (points at the package's own vitest.config.*)
emit-payload.mjs	Turn a Stryker summary.json into a BQ-ready writer payload
ledger.mjs	Read-all ledger access: readLedger({ path, pkg? }) returns every row across every package in one pass, optionally narrowed to one package without re-reading
signals.mjs	Per-file git-derived risk signals (churn + fix-density) used by the global picker's value formula
build-matrix.mjs	Convert the global picker's top-N output (or an on-demand SOURCE_FILE) into the GHA mutate matrix; runs the die-loud ELIGIBLE_PACKAGES ↔ ledger guard

mutate.mjs is package-agnostic — run pnpm mutate <repo-relative-file> from the repo root and the package is inferred from the path (or pass --package-dir <pkg> for a package-relative target, as the nightly does). It uses the package's own stryker.config.mjs if one exists (e.g. packages/workflow carves out the isolated-vm engine), otherwise stryker.default.mjs.

The reader and writer webhooks are plain HTTP — the GHA hits them with curl. There is no fetch/post wrapper script; if you want to call them locally, see Local usage.

The BQ table schema lives with the writer workflow (in n8n's internal Quality project), not in this repo — the writer owns the MERGE statement and is the single source of truth.

End-to-end pipeline

[GHA nightly cron, .github/workflows/mutation-health-nightly.yml]
       │
       ├─► setup job  (fetch-depth: 0, one process per night)
       │     │
       │     ├─► curl GET reader webhook    → live-ledger.json (read-all BQ state)
       │     │       │
       │     │       └─► [n8n: QA Mutation Health Reader] ──► SELECT from BQ ledger
       │     │
       │     ├─► signals.mjs                → signals.json (churn + fix-density)
       │     │
       │     ├─► build-matrix.mjs           → matrix={ include: [top-N picks] }
       │     │     │
       │     │     ├─► die-loud guard: empty ledger throws (strict mode); each
       │     │     │   ELIGIBLE_PACKAGES.name must have ≥1 non-`new` row.
       │     │     │   Escape hatch: `bootstrap_packages` workflow_dispatch input.
       │     │     │
       │     │     └─► pick-next.mjs --global --top-n N
       │     │           walks every eligible package's src/, merges with ledger,
       │     │           ranks: w_churn·churn + w_fix·fixDensity + w_cov·(1−cov)
       │     │           priority: new → red → stale → skip green
       │     │
       │     └─► outputs.matrix = { include: [{name, dir, slug, mode, source_file, file_slug}, ...] }
       │
       └─► mutate job (one per matrix include)
             │
             ├─► mutate.mjs --package-dir <pkg> <source_file>   → summary.json
             │
             ├─► emit-payload.mjs                               → bq-payload.json
             │
             └─► curl POST writer webhook  → INSERTs event + MERGEs ledger row
                                              ↓
                              [n8n writer workflow: QA: Mutation Health Writer]
                                              ↓
                              ┌───────────────────────────────────┐
                              │ qa_mutation_health_ledger (MERGE) │
                              │ qa_performance_metrics  (INSERT)  │
                              └───────────────────────────────────┘

The writer workflow lives in n8n's internal Quality project. It's created and maintained outside this repo. This README documents the contract it implements.

Passes, packages & onboarding

The nightly runs a dynamic matrix of top-N picks (built once in the setup job of mutation-health-nightly.yml). The setup job fetches the read-all live ledger, gathers git-derived signals, and calls pick-next.mjs --global --top-n N once across every eligible package. Each picked row becomes one mutate job; jobs run independently, the ledger is keyed by package + file, so they don't collide. Two passes, selectable via the mode dispatch input (both on schedule):

• baseline — files with no result yet (the new bucket). Builds out coverage. Maps from effective_status: new on a picked row.
• coverage — revisits the weakest scored files (red/stale, lowest first). Strengthens existing tests. Maps from effective_status: red | stale on a picked row.

To onboard a vitest package: add one { name, dir } entry to ELIGIBLE_PACKAGES in pick-next.mjs. The nightly setup job derives its matrix from that single source of truth, so the picker and the workflow can't drift. No per-package config needed — stryker.default.mjs auto-resolves the package's own vitest.config.* (verified on plain and DI-decorator packages). Add a local stryker.config.mjs only if the package needs special handling.

Then, once, trigger the nightly with workflow_dispatch and set bootstrap_packages to the new package name (e.g. @n8n/decorators). The divergence guard otherwise refuses to schedule for a package that has zero non-new ledger rows — that's how it catches an ELIGIBLE_PACKAGES.name ↔ ledger.package rename mismatch, but it can't tell that from a legitimate new onboarding. The one-off bootstrap_packages value skips the guard for that package on that run only; subsequent scheduled nightlies revert to strict mode once the new package's rows have been populated.

For a genuine cold-start (first run after the ledger is provisioned, the read-all returns []), trigger with bootstrap_packages: '*' instead — it acknowledges the empty ledger and skips the per-package divergence check for every entry.

Not yet covered: jest packages (need Stryker's jest-runner — different setup) and @n8n/expression-runtime (it is the isolated-vm engine; blocked on the patch in DEVP-257).

State transitions

Trigger	Stored status
Source file in src/ but no row yet	synthesised as new at pick time; not stored
Last run passed the gate (score ≥ threshold_at_run AND no unjustified survivors)	green
Last run failed the gate (score < threshold_at_run OR ≥1 Survived/NoCoverage mutant)	red

Stored statuses are just two: red and green. new is computed in-memory by the picker for any file in the source tree that has no ledger row yet — the row is only persisted after that file's first scored run. The picker also computes a transient stale state — any green row whose last_checked_at is older than 4 weeks is treated as stale for that pick. No last_checked_sha is needed; no git history is consulted.

Picker priority: new → red → stale → skip fresh green.

• Within new: alphabetical (rows exit the bucket as they're scored)
• Within red: lowest score first (weakest tests revisited first)
• Within stale: oldest last_checked_at first (natural cycling of long-stable files)

If every row is green and fresh, the picker exits 0 with {"picked": null, "reason": "all-green"} — a healthy "nothing to do" state, not a failure.

Webhook contracts

Two n8n workflows back the pipeline. Both live in the internal Quality project (L8csxtEbFpFOWlf8) and are created/maintained outside this repo. Both run unauthenticated (URL-as-secret pattern, matching existing qa_* workers).

Endpoint	Method	Workflow	Purpose
https://internal.users.n8n.cloud/webhook/mutation-health-writer	POST	QA: Mutation Health Writer (iYEBmBat8OscRTVq)	INSERT events + MERGE ledger
https://internal.users.n8n.cloud/webhook/mutation-health-ledger?package=<name>	GET	QA: Mutation Health Reader (ZmRsNUwvgfCSq0JI)	Read current ledger state

Writer webhook

POST https://internal.users.n8n.cloud/webhook/mutation-health-writer with Content-Type: application/json:

{
  "ledger": [
    {
      "source_file_path": "packages/workflow/src/cron.ts",
      "package": "n8n-workflow",
      "last_score": 95.12,
      "coverage": 0.93,
      "churn": 7,
      "fix_density": 2.41,
      "threshold_at_run": 80,
      "last_checked_at": "2026-05-22T10:03:55.660Z",
      "status": "green",
      "mutants_killed": 39,
      "mutants_survived": 2,
      "mutants_no_coverage": 0,
      "mutants_timeout": 0
    }
  ],
  "events": [
    {
      "benchmark_name": "mutation_health",
      "value": 95.12,
      "timestamp": "2026-05-22T10:03:55.660Z",
      "dimensions": {
        "package": "n8n-workflow",
        "source_file": "packages/workflow/src/cron.ts",
        "sha": "095239e175",
        "status_after": "green",
        "threshold": 80,
        "coverage": 0.93,
        "churn": 7,
        "fix_density": 2.41,
        "mutants_killed": 39,
        "mutants_survived": 2,
        "mutants_no_coverage": 0,
        "mutants_timeout": 0
      }
    }
  ]
}

Either array may be empty (manual smoke tests sometimes send only events).

Each ledger row also carries coverage — the scored file's line-coverage proxy in [0,1], the share of mutants a test actually exercised — written back by mutate.mjs after each run. The global picker reads it next cycle as the (1 − coverage) term of its value formula, so a file no test touches (coverage 0) gets the strongest urge to be scored.

Rows also carry churn — the count of commits touching the source file within a recent window (default 90 days, override with --churn-window), derived from git in emit-payload.mjs. It feeds the picker's churn term so hot files outrank cold ones. On a shallow clone history is truncated and the count would undercount, so churn is emitted as null rather than a misleading low number.

The third value-formula term, fix_density, is emitted the same way — the file's time-decayed, delta-weighted fix-density (the same git-derived signal signals.mjs computes: lines changed by fix:-shaped commits, decayed by a half-life on commit age). emit-payload.mjs reads one per-package git log --numstat pass (bounded by --fix-density-window, default 1 year; half-life via --fix-density-half-life, default 90 days) and scores each file. A file with no fix commits scores 0 (known: no fixes); a shallow clone or git failure emits null (unknown), the same guard as churn. A richer variant joining against the bug taxonomy is a possible future enhancement on the writer side, but is not required to feed the picker.

The writer:

1. For each events[] row → INSERT into qa_performance_metrics.
2. For each ledger[] row → MERGE into qa_mutation_health_ledger on source_file_path. Status is always red or green — the picker synthesises new in-memory and never posts it.

The webhook URL is delivered to GHA via the MUTATION_HEALTH_WEBHOOK repo secret. The secret URL itself is the only auth (matches existing qa_* writer pattern); rotate the secret if leaked.

Reader webhook

GET https://internal.users.n8n.cloud/webhook/mutation-health-ledger?package=<pkg>:

{
  "ledger": [
    {
      "source_file_path": "packages/workflow/src/cron.ts",
      "package": "n8n-workflow",
      "last_score": 95.12,
      "coverage": 0.93,
      "churn": 7,
      "fix_density": 2.41,
      "threshold_at_run": 80,
      "last_checked_at": "2026-05-22T10:03:55.660Z",
      "status": "green",
      "mutants_killed": 39,
      "mutants_survived": 2,
      "mutants_no_coverage": 0,
      "mutants_timeout": 0
    }
  ]
}

The package query param is validated server-side against the same pnpm-workspace allowlist regex used elsewhere in the pipeline; invalid input returns 500. No SQL is constructed or accepted on the client side — the SELECT is hardcoded in the workflow.

Unauthenticated — the URL is not a secret. The data isn't sensitive (file paths + integer scores), but treat the URL as low-trust: anyone with it can read all current ledger state for the queried package.

Gate semantics

A run passes only when both:

1. Mutation score meets STRYKER_THRESHOLD (default 80), and
2. Zero Survived / NoCoverage mutants remain — every unkilled mutant must be explicitly justified as Ignored via a // Stryker disable next-line <Mutator>: <reason> comment in the source.

Stryker excludes Ignored mutants from both numerator and denominator of the score (see scoreFromCounts in mutate.mjs), so marking a genuine equivalent as ignored is not padding — it's the documented mechanism for "this mutant is equivalent / not behaviour-bearing, here's why". The score becomes a coarse floor; the real gate is "no unjustified survivors". This stops agents from padding the suite with trivial tests to clear 80% while leaving real behaviour gaps unasserted. See DEVP-442 for the motivation.

summary.json surfaces every Ignored mutant alongside its disable-comment reason so reviewers can spot-check the justifications — those become the high-signal review artifact rather than N padding tests.

Threshold (provisional)

Runs use STRYKER_THRESHOLD=80 as a placeholder. The threshold moves to evidence-based after ~4 weeks of accumulated data. Until then, treat red/green verdicts as preliminary.

Local usage

# Run Stryker on one file (the inner loop — also invokable via /mutant-score skill).
# Package is inferred from the repo-relative path; works for any vitest package.
pnpm mutate packages/workflow/src/cron.ts
pnpm mutate packages/@n8n/crdt/src/utils.ts

# Pull current ledger from BQ
curl --fail -sS \
  'https://internal.users.n8n.cloud/webhook/mutation-health-ledger?package=n8n-workflow' \
  -o /tmp/ledger.json

# Pick the next file to score (per-package mode)
node scripts/mutation-health/pick-next.mjs \
  --package-dir packages/workflow \
  --ledger-file /tmp/ledger.json

# Pick top-N files across every vitest-eligible package, ranked by the
# global value formula w_churn·churn + w_fix·fixDensity + w_cov·(1 − coverage).
# Signals and coverage files are optional — missing terms contribute 0
# except `(1 − coverage)`, which degrades to 1 (worst-case) so untracked
# files float to the top.
node scripts/mutation-health/pick-next.mjs \
  --global \
  --ledger-file /tmp/ledger.json \
  --signals-file /tmp/signals.json \
  --coverage-file /tmp/coverage.json \
  --top-n 5 \
  --block '@n8n/expression-runtime'

# Build a BQ payload from a Stryker run
node scripts/mutation-health/emit-payload.mjs \
  --summary packages/workflow/reports/mutation/summary.json \
  --package n8n-workflow

# POST the result (requires MUTATION_HEALTH_WEBHOOK to be set)
curl --fail -sS -X POST \
  -H 'Content-Type: application/json' \
  --data @packages/workflow/reports/mutation/bq-payload.json \
  "$MUTATION_HEALTH_WEBHOOK"