mirror of
https://github.com/n8n-io/n8n.git
synced 2026-10-19 16:55:02 +02:00
docs: Add hypervisor architecture documentation
This commit is contained in:
Phyllis Noester
parent
37537cef51
commit
aa8d4c1efa
146
docs/hypervisor.md
Normal file
146
docs/hypervisor.md
Normal file
|
|
@ -0,0 +1,146 @@
|
|||
# Hypervisor: queue mode on one host without Redis
|
||||
|
||||
tl;dr: `n8n hypervisor` boots one supervisor process that forks 2 mains and 2 workers via `node:cluster` and hosts every coordination primitive (leader election, pubsub, instance registry, cache, execution queue) over the fork IPC channel. A workflow execution can only kill a worker child, which the supervisor respawns; HTTP and the sibling processes keep running. No Redis anywhere. Built during the Aug 2026 hackmation on branch `hackmation-unfied-mode`; PoC quality, gated behind the explicit `n8n hypervisor` command and `N8N_TRANSPORT_*` env flags, zero behavior change for every existing deployment.
|
||||
|
||||
Context: [Unify modes of operation RFC](https://app.notion.com/p/n8n/Unify-modes-of-operation-3225b6e0c94f809b8a9ff037adcde248). The RFC proposed a single unified operation mode with a transport abstraction (messaging + storage facets) selected by infrastructure. This branch implements that architecture end to end for a single host.
|
||||
|
||||
## Process topology
|
||||
|
||||
The primary never executes workflows. It forks the children, routes their IPC messages to per-feature hosts, respawns crashed children, and answers `/cluster-info` with live per-process stats. Mains share port 5678 through the cluster module's socket distribution.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Primary["hypervisor primary (n8n hypervisor)"]
|
||||
Router["HypervisorMessageRouter<br/>dispatch by message-type prefix"]
|
||||
LE["LeaderElectionHost<br/>leader:"]
|
||||
REG["InstanceRegistryHost<br/>registry:"]
|
||||
PS["PubSubHost<br/>pubsub:"]
|
||||
CA["CacheHost<br/>cache:"]
|
||||
JQ["JobQueueHost<br/>queue:"]
|
||||
SUP["ChildSupervisor<br/>respawn on exit, graceful SIGTERM drain"]
|
||||
INFO["SupervisorInfoHost<br/>live memory per child"]
|
||||
Router --> LE & REG & PS & CA & JQ & INFO
|
||||
end
|
||||
|
||||
M1["main #1<br/>(Start command)"]
|
||||
M2["main #2<br/>(Start command)"]
|
||||
W1["worker #1<br/>(Worker command)"]
|
||||
W2["worker #2<br/>(Worker command)"]
|
||||
DB[("Postgres / SQLite<br/>shared application DB")]
|
||||
|
||||
Primary <-->|"fork IPC (process.send)"| M1 & M2 & W1 & W2
|
||||
M1 & M2 & W1 & W2 --> DB
|
||||
```
|
||||
|
||||
Each child runs the ordinary `Start` or `Worker` command with `EXECUTIONS_MODE=queue`; the only difference is the env the primary forks it with:
|
||||
|
||||
```
|
||||
N8N_HYPERVISOR_MODE=1
|
||||
N8N_HYPERVISOR_ROLE=main|worker
|
||||
N8N_TRANSPORT_LEADER_ELECTION=ipc
|
||||
N8N_TRANSPORT_INSTANCE_REGISTRY=ipc
|
||||
N8N_TRANSPORT_PUBSUB=ipc
|
||||
N8N_TRANSPORT_CACHE=ipc
|
||||
N8N_TRANSPORT_QUEUE=ipc
|
||||
```
|
||||
|
||||
## Transport selection
|
||||
|
||||
`TransportModeService` (`packages/cli/src/scaling/transport-mode.service.ts`) is the single place that dictates which transport each subsystem uses. It only reads the explicit per-subsystem config; there is no detection and no fallback branch. `validateAtBoot()` rejects any `ipc` selection outside a hypervisor child, since only the fork channel makes ipc possible.
|
||||
|
||||
| Subsystem | Env flag | Default | ipc implementation | Redis implementation |
|
||||
|---|---|---|---|---|
|
||||
| Leader election | `N8N_TRANSPORT_LEADER_ELECTION` | `redis` | `HypervisorLeaderElection` + `LeaderElectionHost` (primary assigns leadership, heartbeat failover) | `MultiMainSetup` lease |
|
||||
| Pub/sub | `N8N_TRANSPORT_PUBSUB` | `redis` | `HypervisorMessageTransport` + `PubSubHost` | `RedisMessageTransport` |
|
||||
| Instance registry | `N8N_TRANSPORT_INSTANCE_REGISTRY` | `memory` | `IpcInstanceStorage` + `InstanceRegistryHost` | `RedisInstanceStorage` |
|
||||
| Cache | `N8N_TRANSPORT_CACHE` | `memory` | `IpcCacheStore` + `CacheHost` (KV in the primary, writes visible to all children) | Redis backend |
|
||||
| Execution queue | `N8N_TRANSPORT_QUEUE` | `redis` | `IpcJobQueue` + `JobQueueHost` | `BullJobQueue` (Bull, unchanged behavior) |
|
||||
|
||||
Every host follows the same pattern: it implements `HypervisorMessageHandler`, owns one message-type prefix, and gets `onMessage`, `onExit`, and `onTick` from the router. The primary is authoritative for liveness because it observes `cluster.on('exit')` directly; there are no TTLs or stale-member sweeps.
|
||||
|
||||
## The execution queue
|
||||
|
||||
`IJobQueue` (`packages/cli/src/scaling/queue/job-queue.interface.ts`) is the storage facet of the RFC's transport abstraction: enqueue, process with concurrency, counts, statuses, plus a job message channel (`QueueJob.sendMessage` / `IJobQueue.onMessage`) that carries the in-flight worker-to-main messages (`respond-to-webhook`, `job-finished`, `send-chunk`, `abort-job`, `mcp-response`). `ScalingService` selects the implementation via `resolve('queue')`; everything above the interface (`JobProcessor`, `WorkflowRunner`, webhook response relay) is implementation-agnostic.
|
||||
|
||||
Three implementations, one behavioral contract (`__tests__/job-queue.contract.ts` runs against all of them): single consumer per job, priority ordering, bounded per-worker concurrency, message broadcast to every attached process including the sender, and no automatic retries.
|
||||
|
||||
- `BullJobQueue`: today's production path, byte-for-byte (patched `bull@4.16.4`, `maxStalledCount: 0`, retention from `queueRetention`).
|
||||
- `InMemoryJobQueue`: single-process reference implementation that carries the contract suite.
|
||||
- `IpcJobQueue` + `JobQueueHost`: the hypervisor path. Queue state lives in the primary; children talk fire-and-forget for writes and requestId round-trips for reads, same as the instance registry.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Main as main (enqueuer)
|
||||
participant HV as JobQueueHost (primary)
|
||||
participant Worker as worker (processor)
|
||||
|
||||
Main->>HV: queue:enqueue {data, priority, requestId}
|
||||
HV-->>Main: queue:reply {job}
|
||||
HV->>Worker: queue:dispatch {job}
|
||||
Note over Worker: runs the execution
|
||||
Worker->>HV: queue:job-message {respond-to-webhook, ...}
|
||||
HV-->>Main: broadcast to all attached children
|
||||
alt execution finishes
|
||||
Worker->>HV: queue:job-done {jobId}
|
||||
HV-->>Main: queue:job-outcome {completed}
|
||||
Note over Main: job.finished() resolves
|
||||
else worker process dies
|
||||
Note over HV: cluster exit observed
|
||||
HV-->>Main: queue:job-outcome {stalled}
|
||||
Note over Main: job.finished() rejects with MaxStalledCountError
|
||||
end
|
||||
```
|
||||
|
||||
Recovery semantics were decided deliberately: they mirror today's Bull configuration, not the RFC's aspiration. A job whose worker is lost fails immediately (as with `maxStalledCount: 0`); there is no re-delivery, and the leader main's existing queue-recovery DB-diff handles dangling executions. Re-delivery would re-run half-executed workflows, which n8n executions are not generally idempotent against.
|
||||
|
||||
## Crash recovery and memory limits
|
||||
|
||||
The `ChildSupervisor` in `commands/hypervisor.ts` tracks each child's role and env, respawns it identically on exit (infinite respawn, no backoff; PoC), classifies the crash for the log line (SIGKILL from the OS OOM-killer vs V8 heap OOM vs plain crash), and drives graceful shutdown: SIGTERM to each child triggers n8n's own drain, with a 10s SIGKILL backstop.
|
||||
|
||||
For the OOM demo, `N8N_HYPERVISOR_OOM_DEMO_HEAP_MB_WORKER` / `..._MAIN` heap-limit children per role via `--max-old-space-size`, and the limit survives respawn. A memory-hogging workflow kills only its worker: the execution fails fast via the stalled outcome, the other worker keeps processing, HTTP never blips, and the worker is back within seconds. That is the process-isolation claim of the RFC, demonstrated.
|
||||
|
||||
`SupervisorInfoHost` aggregates live per-child memory plus respawn counts, exposed for the demo cluster view.
|
||||
|
||||
## What still needs Redis (known gaps)
|
||||
|
||||
The queue was the last execution-critical subsystem; what remains is the KV/locking facet. Under the hypervisor these fall back to per-process behavior, explicitly guarded with `!TransportModeService.isUnderHypervisor()`:
|
||||
|
||||
- Cross-process locks: `base-command` normally swaps in `RedisLockService` for queue mode; hypervisor children keep the in-process provider. Lock scope is per-child until a lock host lands on the router.
|
||||
- MCP session store: needs a shared KV (`Publisher`'s Redis-only key-value utils); skipped under ipc, so MCP triggers do not relay sessions across mains yet. The `CacheHost` KV is the natural future home.
|
||||
- Chat-hub execution/stream stores: fall back to their existing in-memory path (per-process).
|
||||
|
||||
Multi-host stays Redis by design: the same code paths run with `N8N_TRANSPORT_*=redis` and nothing on this branch changes them.
|
||||
|
||||
## Open decisions
|
||||
|
||||
- JobMessage placement (tbd): both queue implementations currently carry the job message channel themselves (option B). Moving it onto `MessageTransport` (option A) would delete a small duplicated broadcast and open the path to dropping the pnpm patch on `bull/lib/job.js`, at the cost of the transport carrying webhook-sized payloads. Recommendation on record: keep B through the hackmation, adopt A afterwards. See `docs/generated/plans/job-message-placement-options.md` (local) for the full trade-off.
|
||||
- Lock host and shared KV over the router (tbd): closes the three gaps above.
|
||||
- Crash-loop backoff, execution recovery on respawn, and forcing `EXECUTIONS_MODE=queue` from the hypervisor (tbd): currently the operator sets queue mode in the shell env.
|
||||
|
||||
## Running it
|
||||
|
||||
```bash
|
||||
pnpm build > build.log 2>&1
|
||||
pnpm --filter n8n-containers services --services postgres # no redis
|
||||
|
||||
EXECUTIONS_MODE=queue DB_TYPE=postgresdb \
|
||||
DB_POSTGRESDB_HOST=localhost DB_POSTGRESDB_PORT=<port> \
|
||||
DB_POSTGRESDB_DATABASE=n8n_db DB_POSTGRESDB_USER=n8n_user DB_POSTGRESDB_PASSWORD=<pw> \
|
||||
./packages/cli/bin/n8n hypervisor
|
||||
```
|
||||
|
||||
Checks that prove the interesting properties: run a workflow and watch a `[worker pid=...]` line pick it up; curl a webhook workflow with Respond to Webhook (response relayed worker to main over IPC); stop a running execution from the UI (abort over IPC); `kill -9` a busy worker and watch the execution fail fast while the instance survives and the worker respawns; `redis-cli keys 'bull:*'` against any Redis you have lying around stays empty because nothing connects to it.
|
||||
|
||||
## Commit map (hackmation-unfied-mode)
|
||||
|
||||
| Commit | What |
|
||||
|---|---|
|
||||
| `5af87ca` | Hypervisor command, `TransportModeService`, IPC leader election, instance registry extension |
|
||||
| `f4e944c` / `9d51648` | Pub/sub behind swappable `MessageTransport`, reconciled with transport selection |
|
||||
| `afec703` | Execution queue behind `IJobQueue` (Bull + in-memory + contract suite) |
|
||||
| `edae34a` / `f8009f4` | Router + hosts refactor; pub/sub and registry over the cluster IPC channel |
|
||||
| `b6a8166` | Cache over IPC (`IpcCacheStore` + `CacheHost`) |
|
||||
| `b3548da` | `IpcJobQueue` + `JobQueueHost`; queue over IPC |
|
||||
| `2c3eb1a` | Child supervisor: memory limits, crash classification, respawn, graceful shutdown |
|
||||
| `bf0d27d` | No-Redis guards for locking, Publisher KV, chat-hub stores |
|
||||
| `37537ce` | Cluster live view for the demo |
|
||||
Loading…
Reference in New Issue
Block a user