n8n/docs/hypervisor.md

Hypervisor: queue mode on one host without Redis

tl;dr: n8n hypervisor boots one supervisor process that forks 2 mains and 2 workers via node:cluster and hosts every coordination primitive (leader election, pubsub, instance registry, cache, execution queue) over the fork IPC channel. A workflow execution can only kill a worker child, which the supervisor respawns; HTTP and the sibling processes keep running. No Redis anywhere. Built during the Aug 2026 hackmation on branch hackmation-unfied-mode; PoC quality, gated behind the explicit n8n hypervisor command and N8N_TRANSPORT_* env flags, zero behavior change for every existing deployment.

Context: Unify modes of operation RFC. The RFC proposed a single unified operation mode with a transport abstraction (messaging + storage facets) selected by infrastructure. This branch implements that architecture end to end for a single host.

Process topology

The primary never executes workflows. It forks the children, routes their IPC messages to per-feature hosts, respawns crashed children, and answers /cluster-info with live per-process stats. Mains share port 5678 through the cluster module's socket distribution.

flowchart TB
    subgraph Primary["hypervisor primary (n8n hypervisor)"]
        Router["HypervisorMessageRouter<br/>dispatch by message-type prefix"]
        LE["LeaderElectionHost<br/>leader:"]
        REG["InstanceRegistryHost<br/>registry:"]
        PS["PubSubHost<br/>pubsub:"]
        CA["CacheHost<br/>cache:"]
        JQ["JobQueueHost<br/>queue:"]
        SUP["ChildSupervisor<br/>respawn on exit, graceful SIGTERM drain"]
        INFO["SupervisorInfoHost<br/>live memory per child"]
        Router --> LE & REG & PS & CA & JQ & INFO
    end

    M1["main #1<br/>(Start command)"]
    M2["main #2<br/>(Start command)"]
    W1["worker #1<br/>(Worker command)"]
    W2["worker #2<br/>(Worker command)"]
    DB[("Postgres / SQLite<br/>shared application DB")]

    Primary <-->|"fork IPC (process.send)"| M1 & M2 & W1 & W2
    M1 & M2 & W1 & W2 --> DB

Each child runs the ordinary Start or Worker command with EXECUTIONS_MODE=queue; the only difference is the env the primary forks it with:

N8N_HYPERVISOR_MODE=1
N8N_HYPERVISOR_ROLE=main|worker
N8N_TRANSPORT_LEADER_ELECTION=ipc
N8N_TRANSPORT_INSTANCE_REGISTRY=ipc
N8N_TRANSPORT_PUBSUB=ipc
N8N_TRANSPORT_CACHE=ipc
N8N_TRANSPORT_QUEUE=ipc

Transport selection

TransportModeService (packages/cli/src/scaling/transport-mode.service.ts) is the single place that dictates which transport each subsystem uses. It only reads the explicit per-subsystem config; there is no detection and no fallback branch. validateAtBoot() rejects any ipc selection outside a hypervisor child, since only the fork channel makes ipc possible.

Subsystem	Env flag	Default	ipc implementation	Redis implementation
Leader election	N8N_TRANSPORT_LEADER_ELECTION	redis	HypervisorLeaderElection + LeaderElectionHost (primary assigns leadership, heartbeat failover)	MultiMainSetup lease
Pub/sub	N8N_TRANSPORT_PUBSUB	redis	HypervisorMessageTransport + PubSubHost	RedisMessageTransport
Instance registry	N8N_TRANSPORT_INSTANCE_REGISTRY	memory	IpcInstanceStorage + InstanceRegistryHost	RedisInstanceStorage
Cache	N8N_TRANSPORT_CACHE	memory	IpcCacheStore + CacheHost (KV in the primary, writes visible to all children)	Redis backend
Execution queue	N8N_TRANSPORT_QUEUE	redis	IpcJobQueue + JobQueueHost	BullJobQueue (Bull, unchanged behavior)

Every host follows the same pattern: it implements HypervisorMessageHandler, owns one message-type prefix, and gets onMessage, onExit, and onTick from the router. The primary is authoritative for liveness because it observes cluster.on('exit') directly; there are no TTLs or stale-member sweeps.

The execution queue

IJobQueue (packages/cli/src/scaling/queue/job-queue.interface.ts) is the storage facet of the RFC's transport abstraction: enqueue, process with concurrency, counts, statuses, plus a job message channel (QueueJob.sendMessage / IJobQueue.onMessage) that carries the in-flight worker-to-main messages (respond-to-webhook, job-finished, send-chunk, abort-job, mcp-response). ScalingService selects the implementation via resolve('queue'); everything above the interface (JobProcessor, WorkflowRunner, webhook response relay) is implementation-agnostic.

Three implementations, one behavioral contract (__tests__/job-queue.contract.ts runs against all of them): single consumer per job, priority ordering, bounded per-worker concurrency, message broadcast to every attached process including the sender, and no automatic retries.

• BullJobQueue: today's production path, byte-for-byte (patched bull@4.16.4, maxStalledCount: 0, retention from queueRetention).
• InMemoryJobQueue: single-process reference implementation that carries the contract suite.
• IpcJobQueue + JobQueueHost: the hypervisor path. Queue state lives in the primary; children talk fire-and-forget for writes and requestId round-trips for reads, same as the instance registry.

sequenceDiagram
    participant Main as main (enqueuer)
    participant HV as JobQueueHost (primary)
    participant Worker as worker (processor)

    Main->>HV: queue:enqueue {data, priority, requestId}
    HV-->>Main: queue:reply {job}
    HV->>Worker: queue:dispatch {job}
    Note over Worker: runs the execution
    Worker->>HV: queue:job-message {respond-to-webhook, ...}
    HV-->>Main: broadcast to all attached children
    alt execution finishes
        Worker->>HV: queue:job-done {jobId}
        HV-->>Main: queue:job-outcome {completed}
        Note over Main: job.finished() resolves
    else worker process dies
        Note over HV: cluster exit observed
        HV-->>Main: queue:job-outcome {stalled}
        Note over Main: job.finished() rejects with MaxStalledCountError
    end

Recovery semantics were decided deliberately: they mirror today's Bull configuration, not the RFC's aspiration. A job whose worker is lost fails immediately (as with maxStalledCount: 0); there is no re-delivery, and the leader main's existing queue-recovery DB-diff handles dangling executions. Re-delivery would re-run half-executed workflows, which n8n executions are not generally idempotent against.

Crash recovery and memory limits

The ChildSupervisor in commands/hypervisor.ts tracks each child's role and env, respawns it identically on exit (infinite respawn, no backoff; PoC), classifies the crash for the log line (SIGKILL from the OS OOM-killer vs V8 heap OOM vs plain crash), and drives graceful shutdown: SIGTERM to each child triggers n8n's own drain, with a 10s SIGKILL backstop.

For the OOM demo, N8N_HYPERVISOR_OOM_DEMO_HEAP_MB_WORKER / ..._MAIN heap-limit children per role via --max-old-space-size, and the limit survives respawn. A memory-hogging workflow kills only its worker: the execution fails fast via the stalled outcome, the other worker keeps processing, HTTP never blips, and the worker is back within seconds. That is the process-isolation claim of the RFC, demonstrated.

SupervisorInfoHost aggregates live per-child memory plus respawn counts, exposed for the demo cluster view.

What still needs Redis (known gaps)

The queue was the last execution-critical subsystem; what remains is the KV/locking facet. Under the hypervisor these fall back to per-process behavior, explicitly guarded with !TransportModeService.isUnderHypervisor():

• Cross-process locks: base-command normally swaps in RedisLockService for queue mode; hypervisor children keep the in-process provider. Lock scope is per-child until a lock host lands on the router.
• MCP session store: needs a shared KV (Publisher's Redis-only key-value utils); skipped under ipc, so MCP triggers do not relay sessions across mains yet. The CacheHost KV is the natural future home.
• Chat-hub execution/stream stores: fall back to their existing in-memory path (per-process).

Multi-host stays Redis by design: the same code paths run with N8N_TRANSPORT_*=redis and nothing on this branch changes them.

Open decisions

• JobMessage placement (tbd): both queue implementations currently carry the job message channel themselves (option B). Moving it onto MessageTransport (option A) would delete a small duplicated broadcast and open the path to dropping the pnpm patch on bull/lib/job.js, at the cost of the transport carrying webhook-sized payloads. Recommendation on record: keep B through the hackmation, adopt A afterwards. See docs/generated/plans/job-message-placement-options.md (local) for the full trade-off.
• Lock host and shared KV over the router (tbd): closes the three gaps above.
• Crash-loop backoff, execution recovery on respawn, and forcing EXECUTIONS_MODE=queue from the hypervisor (tbd): currently the operator sets queue mode in the shell env.

Running it

pnpm build > build.log 2>&1
pnpm --filter n8n-containers services --services postgres   # no redis

EXECUTIONS_MODE=queue DB_TYPE=postgresdb \
  DB_POSTGRESDB_HOST=localhost DB_POSTGRESDB_PORT=<port> \
  DB_POSTGRESDB_DATABASE=n8n_db DB_POSTGRESDB_USER=n8n_user DB_POSTGRESDB_PASSWORD=<pw> \
  ./packages/cli/bin/n8n hypervisor

Checks that prove the interesting properties: run a workflow and watch a [worker pid=...] line pick it up; curl a webhook workflow with Respond to Webhook (response relayed worker to main over IPC); stop a running execution from the UI (abort over IPC); kill -9 a busy worker and watch the execution fail fast while the instance survives and the worker respawns; redis-cli keys 'bull:*' against any Redis you have lying around stays empty because nothing connects to it.

Commit map (hackmation-unfied-mode)

Commit	What
5af87ca	Hypervisor command, TransportModeService, IPC leader election, instance registry extension
f4e944c / 9d51648	Pub/sub behind swappable MessageTransport, reconciled with transport selection
afec703	Execution queue behind IJobQueue (Bull + in-memory + contract suite)
edae34a / f8009f4	Router + hosts refactor; pub/sub and registry over the cluster IPC channel
b6a8166	Cache over IPC (IpcCacheStore + CacheHost)
b3548da	IpcJobQueue + JobQueueHost; queue over IPC
2c3eb1a	Child supervisor: memory limits, crash classification, respawn, graceful shutdown
bf0d27d	No-Redis guards for locking, Publisher KV, chat-hub stores
37537ce	Cluster live view for the demo