github-mirror/n8n
1
0
Fork 0
mirror of https://github.com/n8n-io/n8n.git synced 2026-10-18 14:56:50 +02:00
Code Issues Actions 13 Packages Projects Releases Wiki Activity Graph

fix(core): Fence agent conversation writes with database ownership

Browse Source
This commit is contained in:
Robin Braumann 2026-09-17 08:59:30 +02:00
parent f649a98692
commit eaafb307da
No known key found for this signature in database
54 changed files with 2465 additions and 726 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

154
.agents/specs/agent-conversation-coordination.md Normal file
View File

@ -0,0 +1,154 @@
# Agent conversation coordination and fencing
Status: Implemented from the approved specification and plan.
Scope: The execution lease comment on [PR #38778](https://github.com/n8n-io/n8n/pull/38778#discussion_r4028193318).
## Purpose
Make the database the authority for agent conversation execution ownership. Define which database writes a former owner must no longer be able to make.
The previous implementation used `LockService` to coordinate execution. Queue and execution state were stored in the database. An expired lease could leave a former owner running. The database now rejects protected writes from that attempt.
Moving lease storage into SQL does not by itself provide fencing. A fence must reject an outdated owner when the protected write occurs.
## Terms
- A main is a backend server process.
- A conversation is the existing logical chat thread.
- An owner is one execution attempt that holds the current right to process that conversation. A main process can own different conversations at the same time.
- A former owner is an attempt whose lease has expired, been released, or been replaced. Its main process can still be alive. If that process resumes after a pause, it must not write protected state with the former ownership identity.
- A lease is the database record of that right and its expiry.
- A fence is an ownership condition that the database enforces with a protected mutation. An earlier ownership check does not provide the same guarantee.
## Required behavior
### One authority for each conversation
1. The database grants at most one current execution owner for a conversation.
2. Competing mains must receive a consistent result when they try to claim the same conversation.
3. Different conversations can execute concurrently.
4. Ownership must work before the first execution or history record exists.
5. Queue processing, HITL resumption, wake execution, automatic workflow continuations, and recovery must use the same ownership rules. Existing conversation busy checks must reflect these rules.
6. The database determines lease expiry. Differences between main-process clocks must not decide ownership.
7. Notifications can prompt work. Receiving or missing a notification must not grant ownership.
### Preserve queue and HITL behavior
1. Ordinary messages keep their current durable FIFO order. Eligible HITL responses keep priority over waiting ordinary messages.
2. An incoming message or HITL response can enter the queue while another turn owns execution.
3. Editing, removal, and Stop keep their existing contracts in the preview follow-up PR.
4. A suspended run releases active execution ownership. Its persisted HITL state continues to block ordinary messages and wakes.
5. A valid response acquires ownership before resuming the matching suspension. Another suspension keeps ordinary messages blocked.
6. Wake requests remain in the existing wake system. This change does not assign them a FIFO position among queued messages.
### Ownership loss and recovery
1. A current owner can renew its lease. An expired or replaced owner cannot revive its old claim.
2. A process that detects ownership loss must stop starting further work for that attempt and request cancellation of work in progress.
3. Protected database writes from that attempt must fail even if the process has not handled its cancellation signal.
4. A late heartbeat, completion, or release from a former owner must not change a successor's ownership or queue entry.
5. Recovery must recheck ownership and liveness when it changes state. It must not interrupt a healthy owner based on an earlier stale read.
6. Preserve the current interrupted-run policy. Recovery marks abandoned active work interrupted and does not replay that input.
7. Preserve waiting-message recovery for integrations. Recovery of waiting previews after the accepting main dies remains outside this change.
8. An explicit Stop retains the execution slot until cancellation finishes. A stale Stop cannot cancel a successor.
9. An unavailable database must not cause execution to continue through a fallback ownership mechanism. A rejected renewal requests cancellation.
## Fence coverage
The fence covers the state below. Each protected write must enforce the ownership condition.
| State | Required guarantee | Decision |
| --- | --- | --- |
| Conversation ownership | Claim, renewal, takeover, and release reject an outdated ownership identity. | Core scope |
| Queue processing state | Execution claims, execution links, owner heartbeats, completion, and recovery cleanup obey current ownership. User edits and removal retain their queued-status conditions. | Core scope |
| Execution records | A former owner cannot update the active timeline, heartbeat, or terminal result after ownership loss. | Included; agreed |
| HITL checkpoints | A former owner cannot restore, replace, resume, or expire checkpoint state after ownership loss. | Included; agreed |
| Saved conversation messages | A former owner cannot add, replace, or delete messages in the conversation after ownership loss. This includes messages used as context by later turns. | Included; agreed |
The ownership condition and each protected mutation must form one atomic database decision. A lease check followed by an unguarded write is insufficient.
## Boundaries
- This specification addresses the conversation execution lease. The other two human review comments on #38778 are separate.
- Preview work stays on its accepting main. The transport changes in #38836 and the frontend in #38820 keep their agreed scope.
- Admission, session rotation, callback handling, and other `LockService` consumers are separate unless they directly use the conversation execution lease.
- No new broker, queue framework, message limits, reordering, execution retries, or admission deduplication is added.
- External actions already started can still finish after ownership loss. This change does not guarantee that a tool or platform API action occurs exactly once.
- Database transactions must not remain open while an agent waits for a model, tool, or human response.
- SQLite and PostgreSQL must support the same ownership and recovery contract. Multi-main deployments must remain supported.
- Any required schema change belongs to the unmerged base PR. Amend its existing migration, as agreed. The existing queue migration also creates the conversation lease table.
## Acceptance scenarios
| Scenario | Required result |
| --- | --- |
| Two mains claim the same conversation | Only one receives current execution ownership. |
| Two different conversations have waiting work | Both can progress independently. |
| A queue turn races with a wake or automatic continuation | Both paths observe the same owner. They cannot each receive valid ownership. |
| A first message has no execution record yet | It still receives the same ownership protection. |
| A lease expires and another main takes over | The former owner cannot renew, release, or mutate protected state with its old identity. |
| A stale process resumes after recovery | Its protected writes are rejected. It cannot overwrite the successor's state. |
| A former owner sends an execution heartbeat, timeline update, or terminal result | The database rejects the write, even if the process has not handled its cancellation signal. |
| A former owner saves, resumes, or expires a HITL checkpoint | The database rejects the write. The current suspension or resolved state remains intact. |
| A former owner adds, replaces, or deletes saved conversation messages | The database rejects the mutation. Stale output does not enter the history that later turns read. |
| Recovery races with a valid renewal or completion | Recovery does not act on a stale ownership decision. |
| A run suspends for HITL, resumes, and suspends again | Ordinary messages and wakes remain blocked until the suspension is resolved. |
| Stop races with completion and the next queued turn | Cancellation finishes before the slot is released. The request cannot target the next turn. |
| The accepting main dies during execution | The active input is interrupted without replay. Existing waiting-input recovery rules apply. |
| A notification is duplicated or missed | Ownership remains correct. Existing recovery can prompt pending work. |
| Mains have different local clocks | Lease validity follows database time. |
Run the ownership and stale-writer scenarios against SQLite and PostgreSQL. Run the queue integration suite with Redis to verify notifications between mains.
## Open decisions
None remain.
## Implementation TODO
- [x] Agree on fence coverage: ownership, queue state, execution records, HITL checkpoints, and saved conversation messages.
- [x] Review and agree on the complete specification.
- [x] Create and agree on a separate implementation plan.
- [x] Implement and validate the approved plan.
The approved plan uses a database lease of two minutes and renews it every 30 seconds. Backend execution context carries each attempt's ownership identity. The unmerged queue migration creates the lease table. Preview admission, session rotation, and callback locks remain separate.
## Execution context
`AgentConversationLeaseService` carries the owner with `AsyncLocalStorage`. It uses a two-minute lease and renews it every 30 seconds. Each acquisition receives a new UUID. The service locks the ownership row before it checks expiry and writes protected state in the same transaction.
The PostgreSQL expiry check uses `clock_timestamp()`. This reads the current database time after a row-lock wait. Queue and execution recovery also use database time.
Chat runs, task runs, saved agents invoked by workflows, and persisted child agents use this context. A foreground child keeps its parent cancellation signal. A detached child starts outside the parent context and receives its own lease. Cached runtimes store no ownership token. Each stream chunk and deferred persistence callback keeps the attempt that created it.
Builder operations, explicit history deletion, and inline workflow agents use a separate persistence path. An inline workflow agent has no stored agent row for the lease foreign key. Title, observation, and episodic-memory writes keep their existing behavior.
Existing telemetry keeps its event names and payloads. `Agent execution count` retains its message, tool, and token counters. `Agent session metrics` records a turn only after its terminal database update succeeds. A rejected former-owner completion does not add a completed turn.
## Deployment and draft databases
Stop or drain all older mains before starting this implementation. The old `LockService` implementation and the database implementation do not share ownership. Do not run both implementations against the same conversations during deployment.
The migration is still part of an unmerged PR. It replaces the earlier draft of `1789499956927-CreateAgentMessageQueue.ts`. It does not add a second migration.
Refresh a disposable development database that already applied the earlier draft:
1. Stop every n8n process connected to that database.
2. For SQLite, delete the disposable database file and its `-wal` and `-shm` files.
3. For PostgreSQL, drop and recreate the disposable database.
4. Start n8n from the updated branch. Let it apply the migrations to the empty database.
Keep a database that contains required data separate from this reset procedure. Removing only the migration-history row does not refresh the schema.
## Pull request stack
1. #38778 contains the queue and database coordination. Its migration creates both tables.
2. #38836 adds preview admission, push delivery, editing, removal, and cancellation. It retains its separate admission lock.
3. #38820 adds the preview queue interface.
## Validation
The focused queue and ownership suites pass on SQLite and PostgreSQL. PostgreSQL cases use separate connections to test a protected write against takeover and expiry during a row-lock wait. Redis covers queue notifications between mains.
The affected unit suites, migration up/down checks, shared build, affected package typechecks, and changed-file lint pass. The migration checks use fresh databases.

8
docs/generated/postgres-schema/README.md generated
View File

@ -14,6 +14,7 @@ Auto-generated from the PostgreSQL migrations in @n8n/db. Do not edit by hand.
| [public.agent_chat_attachments](public.agent_chat_attachments.md) | 12 | | BASE TABLE |
| [public.agent_chat_subscriptions](public.agent_chat_subscriptions.md) | 6 | | BASE TABLE |
| [public.agent_checkpoints](public.agent_checkpoints.md) | 6 | | BASE TABLE |
| [public.agent_conversation_lease](public.agent_conversation_lease.md) | 4 | | BASE TABLE |
| [public.agent_credential_dependency](public.agent_credential_dependency.md) | 3 | | BASE TABLE |
| [public.agent_eval_dataset](public.agent_eval_dataset.md) | 10 | | BASE TABLE |
| [public.agent_eval_rating](public.agent_eval_rating.md) | 8 | | BASE TABLE |
@ -180,6 +181,7 @@ erDiagram
"public.agent_chat_attachments" }o--o| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
"public.agent_chat_subscriptions" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
"public.agent_checkpoints" }o--o| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
"public.agent_conversation_lease" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
"public.agent_credential_dependency" }o--|| "public.credentials_entity" : "FOREIGN KEY (#quot;credentialId#quot;) REFERENCES credentials_entity(id) ON DELETE CASCADE"
"public.agent_credential_dependency" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
"public.agent_eval_dataset" }o--o| "public.user" : "FOREIGN KEY (#quot;createdById#quot;) REFERENCES #quot;user#quot;(id) ON DELETE SET NULL"
@ -454,6 +456,12 @@ erDiagram
text state
timestamp_3__with_time_zone updatedAt
}
"public.agent_conversation_lease" {
varchar_36_ agentId FK
timestamp_3__with_time_zone expiresAt
uuid ownerToken
varchar_128_ threadId
}
"public.agent_credential_dependency" {
varchar_36_ agentId FK
timestamp_3__with_time_zone createdAt

63
docs/generated/postgres-schema/public.agent_conversation_lease.md generated Normal file
View File

@ -0,0 +1,63 @@
# public.agent_conversation_lease
## Columns
| Name | Type | Default | Nullable | Children | Parents | Comment |
| ---- | ---- | ------- | -------- | -------- | ------- | ------- |
| agentId | varchar(36) | | false | | [public.agents](public.agents.md) | |
| expiresAt | timestamp(3) with time zone | | false | | | Lease expiry on the database clock. |
| ownerToken | uuid | | false | | | Fresh identity for one execution owner. |
| threadId | varchar(128) | | false | | | Conversation routing ID. |
## Constraints
| Name | Type | Definition |
| ---- | ---- | ---------- |
| FK_828d2a046ac7a8565ac2d273d1c | FOREIGN KEY | FOREIGN KEY ("agentId") REFERENCES agents(id) ON DELETE CASCADE |
| PK_1f4fe0ff37a0ebe5023c2f294f7 | PRIMARY KEY | PRIMARY KEY ("threadId") |
| agent_conversation_lease_agentId_not_null | n | NOT NULL "agentId" |
| agent_conversation_lease_expiresAt_not_null | n | NOT NULL "expiresAt" |
| agent_conversation_lease_ownerToken_not_null | n | NOT NULL "ownerToken" |
| agent_conversation_lease_threadId_not_null | n | NOT NULL "threadId" |
## Indexes
| Name | Definition |
| ---- | ---------- |
| IDX_828d2a046ac7a8565ac2d273d1 | CREATE INDEX "IDX_828d2a046ac7a8565ac2d273d1" ON public.agent_conversation_lease USING btree ("agentId") |
| PK_1f4fe0ff37a0ebe5023c2f294f7 | CREATE UNIQUE INDEX "PK_1f4fe0ff37a0ebe5023c2f294f7" ON public.agent_conversation_lease USING btree ("threadId") |
## Relations
```mermaid
erDiagram
"public.agent_conversation_lease" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
"public.agent_conversation_lease" {
varchar_36_ agentId FK
timestamp_3__with_time_zone expiresAt
uuid ownerToken
varchar_128_ threadId
}
"public.agents" {
varchar_36_ activeVersionId FK
boolean availableInMCP
timestamp_3__with_time_zone createdAt
varchar_36_ id
json integrations
varchar_128_ name
varchar_255_ projectId FK
integer revision
json schema
timestamp_3__with_time_zone setupCompletedAt
json skills
json tools
timestamp_3__with_time_zone updatedAt
varchar_36_ versionId
}
```
---
> Generated by [tbls](https://github.com/k1LoW/tbls)

9
docs/generated/postgres-schema/public.agents.md generated
View File

@ -7,7 +7,7 @@
| activeVersionId | varchar(36) | | true | | [public.agent_history](public.agent_history.md) | |
| availableInMCP | boolean | false | false | | | Whether MCP clients granted agent scopes may operate on this agent |
| createdAt | timestamp(3) with time zone | CURRENT_TIMESTAMP(3) | false | | | |
| id | varchar(36) | | false | [public.agent_background_job](public.agent_background_job.md) [public.agent_channel_status](public.agent_channel_status.md) [public.agent_chat_attachments](public.agent_chat_attachments.md) [public.agent_chat_subscriptions](public.agent_chat_subscriptions.md) [public.agent_checkpoints](public.agent_checkpoints.md) [public.agent_credential_dependency](public.agent_credential_dependency.md) [public.agent_eval_dataset](public.agent_eval_dataset.md) [public.agent_execution_threads](public.agent_execution_threads.md) [public.agent_files](public.agent_files.md) [public.agent_history](public.agent_history.md) [public.agent_message_queue](public.agent_message_queue.md) [public.agent_task_definition](public.agent_task_definition.md) [public.agent_task_run_lock](public.agent_task_run_lock.md) [public.agent_workflow_dependency](public.agent_workflow_dependency.md) [public.agents_memory_entries](public.agents_memory_entries.md) [public.agents_memory_entry_candidates](public.agents_memory_entry_candidates.md) [public.agents_memory_entry_locks](public.agents_memory_entry_locks.md) [public.agents_memory_entry_sources](public.agents_memory_entry_sources.md) [public.agents_observation_cursors](public.agents_observation_cursors.md) [public.agents_observation_locks](public.agents_observation_locks.md) [public.agents_observations](public.agents_observations.md) | | |
| id | varchar(36) | | false | [public.agent_background_job](public.agent_background_job.md) [public.agent_channel_status](public.agent_channel_status.md) [public.agent_chat_attachments](public.agent_chat_attachments.md) [public.agent_chat_subscriptions](public.agent_chat_subscriptions.md) [public.agent_checkpoints](public.agent_checkpoints.md) [public.agent_conversation_lease](public.agent_conversation_lease.md) [public.agent_credential_dependency](public.agent_credential_dependency.md) [public.agent_eval_dataset](public.agent_eval_dataset.md) [public.agent_execution_threads](public.agent_execution_threads.md) [public.agent_files](public.agent_files.md) [public.agent_history](public.agent_history.md) [public.agent_message_queue](public.agent_message_queue.md) [public.agent_task_definition](public.agent_task_definition.md) [public.agent_task_run_lock](public.agent_task_run_lock.md) [public.agent_workflow_dependency](public.agent_workflow_dependency.md) [public.agents_memory_entries](public.agents_memory_entries.md) [public.agents_memory_entry_candidates](public.agents_memory_entry_candidates.md) [public.agents_memory_entry_locks](public.agents_memory_entry_locks.md) [public.agents_memory_entry_sources](public.agents_memory_entry_sources.md) [public.agents_observation_cursors](public.agents_observation_cursors.md) [public.agents_observation_locks](public.agents_observation_locks.md) [public.agents_observations](public.agents_observations.md) | | |
| integrations | json | '[]'::json | false | | | |
| name | varchar(128) | | false | | | |
| projectId | varchar(255) | | false | | [public.project](public.project.md) | |
@ -56,6 +56,7 @@ erDiagram
"public.agent_chat_attachments" }o--o| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
"public.agent_chat_subscriptions" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
"public.agent_checkpoints" }o--o| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
"public.agent_conversation_lease" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
"public.agent_credential_dependency" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
"public.agent_eval_dataset" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
"public.agent_execution_threads" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
@ -165,6 +166,12 @@ erDiagram
text state
timestamp_3__with_time_zone updatedAt
}
"public.agent_conversation_lease" {
varchar_36_ agentId FK
timestamp_3__with_time_zone expiresAt
uuid ownerToken
varchar_128_ threadId
}
"public.agent_credential_dependency" {
varchar_36_ agentId FK
timestamp_3__with_time_zone createdAt

8
docs/generated/sqlite-schema/README.md generated
View File

@ -14,6 +14,7 @@ Auto-generated from the SQLite migrations in @n8n/db. Do not edit by hand.
| [agent_chat_attachments](agent_chat_attachments.md) | 12 | | table |
| [agent_chat_subscriptions](agent_chat_subscriptions.md) | 6 | | table |
| [agent_checkpoints](agent_checkpoints.md) | 6 | | table |
| [agent_conversation_lease](agent_conversation_lease.md) | 4 | | table |
| [agent_credential_dependency](agent_credential_dependency.md) | 3 | | table |
| [agent_eval_dataset](agent_eval_dataset.md) | 10 | | table |
| [agent_eval_rating](agent_eval_rating.md) | 8 | | table |
@ -163,6 +164,7 @@ erDiagram
"agent_chat_attachments" }o--o| "agents" : "FOREIGN KEY (agentId) REFERENCES agents (id) ON UPDATE NO ACTION ON DELETE CASCADE MATCH NONE"
"agent_chat_subscriptions" |o--|| "agents" : "FOREIGN KEY (agentId) REFERENCES agents (id) ON UPDATE NO ACTION ON DELETE CASCADE MATCH NONE"
"agent_checkpoints" }o--o| "agents" : "FOREIGN KEY (agentId) REFERENCES agents (id) ON UPDATE NO ACTION ON DELETE CASCADE MATCH NONE"
"agent_conversation_lease" }o--|| "agents" : "FOREIGN KEY (agentId) REFERENCES agents (id) ON UPDATE NO ACTION ON DELETE CASCADE MATCH NONE"
"agent_credential_dependency" |o--|| "credentials_entity" : "FOREIGN KEY (credentialId) REFERENCES credentials_entity (id) ON UPDATE NO ACTION ON DELETE CASCADE MATCH NONE"
"agent_credential_dependency" |o--|| "agents" : "FOREIGN KEY (agentId) REFERENCES agents (id) ON UPDATE NO ACTION ON DELETE CASCADE MATCH NONE"
"agent_eval_dataset" }o--o| "user" : "FOREIGN KEY (createdById) REFERENCES user (id) ON UPDATE NO ACTION ON DELETE SET NULL MATCH NONE"
@ -441,6 +443,12 @@ erDiagram
TEXT state
datetime_3_ updatedAt
}
"agent_conversation_lease" {
varchar_36_ agentId FK
datetime_3_ expiresAt
varchar ownerToken
varchar_128_ threadId PK
}
"agent_credential_dependency" {
varchar_36_ agentId PK
datetime_3_ createdAt

71
docs/generated/sqlite-schema/agent_conversation_lease.md generated Normal file
View File

@ -0,0 +1,71 @@
# agent_conversation_lease
## Description
<details>
<summary><strong>Table Definition</strong></summary>
```sql
CREATE TABLE "agent_conversation_lease" ("threadId" varchar(128) PRIMARY KEY NOT NULL, "agentId" varchar(36) NOT NULL, "ownerToken" varchar NOT NULL, "expiresAt" datetime(3) NOT NULL, CONSTRAINT "FK_828d2a046ac7a8565ac2d273d1c" FOREIGN KEY ("agentId") REFERENCES "agents" ("id") ON DELETE CASCADE)
```
</details>
## Columns
| Name | Type | Default | Nullable | Children | Parents | Comment |
| ---- | ---- | ------- | -------- | -------- | ------- | ------- |
| agentId | varchar(36) | | false | | [agents](agents.md) | |
| expiresAt | datetime(3) | | false | | | |
| ownerToken | varchar | | false | | | |
| threadId | varchar(128) | | false | | | |
## Constraints
| Name | Type | Definition |
| ---- | ---- | ---------- |
| - (Foreign key ID: 0) | FOREIGN KEY | FOREIGN KEY (agentId) REFERENCES agents (id) ON UPDATE NO ACTION ON DELETE CASCADE MATCH NONE |
| sqlite_autoindex_agent_conversation_lease_1 | PRIMARY KEY | PRIMARY KEY (threadId) |
| threadId | PRIMARY KEY | PRIMARY KEY (threadId) |
## Indexes
| Name | Definition |
| ---- | ---------- |
| IDX_828d2a046ac7a8565ac2d273d1 | CREATE INDEX "IDX_828d2a046ac7a8565ac2d273d1" ON "agent_conversation_lease" ("agentId") |
| sqlite_autoindex_agent_conversation_lease_1 | PRIMARY KEY (threadId) |
## Relations
```mermaid
erDiagram
"agent_conversation_lease" }o--|| "agents" : "FOREIGN KEY (agentId) REFERENCES agents (id) ON UPDATE NO ACTION ON DELETE CASCADE MATCH NONE"
"agent_conversation_lease" {
varchar_36_ agentId FK
datetime_3_ expiresAt
varchar ownerToken
varchar_128_ threadId PK
}
"agents" {
varchar_36_ activeVersionId FK
boolean availableInMCP
datetime_3_ createdAt
varchar_36_ id PK
TEXT integrations
varchar_128_ name
varchar_255_ projectId FK
INTEGER revision
TEXT schema
datetime_3_ setupCompletedAt
TEXT skills
TEXT tools
datetime_3_ updatedAt
varchar_36_ versionId
}
```
---
> Generated by [tbls](https://github.com/k1LoW/tbls)

9
docs/generated/sqlite-schema/agents.md generated
View File

@ -18,7 +18,7 @@ CREATE TABLE "agents" ("id" varchar(36) PRIMARY KEY NOT NULL, "name" varchar(128
| activeVersionId | varchar(36) | | true | | [agent_history](agent_history.md) | |
| availableInMCP | boolean | false | false | | | |
| createdAt | datetime(3) | STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW') | false | | | |
| id | varchar(36) | | false | [agent_background_job](agent_background_job.md) [agent_channel_status](agent_channel_status.md) [agent_chat_attachments](agent_chat_attachments.md) [agent_chat_subscriptions](agent_chat_subscriptions.md) [agent_checkpoints](agent_checkpoints.md) [agent_credential_dependency](agent_credential_dependency.md) [agent_eval_dataset](agent_eval_dataset.md) [agent_execution_threads](agent_execution_threads.md) [agent_files](agent_files.md) [agent_history](agent_history.md) [agent_message_queue](agent_message_queue.md) [agent_task_definition](agent_task_definition.md) [agent_task_run_lock](agent_task_run_lock.md) [agent_workflow_dependency](agent_workflow_dependency.md) [agents_memory_entries](agents_memory_entries.md) [agents_memory_entry_candidates](agents_memory_entry_candidates.md) [agents_memory_entry_locks](agents_memory_entry_locks.md) [agents_memory_entry_sources](agents_memory_entry_sources.md) [agents_observation_cursors](agents_observation_cursors.md) [agents_observation_locks](agents_observation_locks.md) [agents_observations](agents_observations.md) | | |
| id | varchar(36) | | false | [agent_background_job](agent_background_job.md) [agent_channel_status](agent_channel_status.md) [agent_chat_attachments](agent_chat_attachments.md) [agent_chat_subscriptions](agent_chat_subscriptions.md) [agent_checkpoints](agent_checkpoints.md) [agent_conversation_lease](agent_conversation_lease.md) [agent_credential_dependency](agent_credential_dependency.md) [agent_eval_dataset](agent_eval_dataset.md) [agent_execution_threads](agent_execution_threads.md) [agent_files](agent_files.md) [agent_history](agent_history.md) [agent_message_queue](agent_message_queue.md) [agent_task_definition](agent_task_definition.md) [agent_task_run_lock](agent_task_run_lock.md) [agent_workflow_dependency](agent_workflow_dependency.md) [agents_memory_entries](agents_memory_entries.md) [agents_memory_entry_candidates](agents_memory_entry_candidates.md) [agents_memory_entry_locks](agents_memory_entry_locks.md) [agents_memory_entry_sources](agents_memory_entry_sources.md) [agents_observation_cursors](agents_observation_cursors.md) [agents_observation_locks](agents_observation_locks.md) [agents_observations](agents_observations.md) | | |
| integrations | TEXT | '[]' | false | | | |
| name | varchar(128) | | false | | | |
| projectId | varchar(255) | | false | | [project](project.md) | |
@ -58,6 +58,7 @@ erDiagram
"agent_chat_attachments" }o--o| "agents" : "FOREIGN KEY (agentId) REFERENCES agents (id) ON UPDATE NO ACTION ON DELETE CASCADE MATCH NONE"
"agent_chat_subscriptions" |o--|| "agents" : "FOREIGN KEY (agentId) REFERENCES agents (id) ON UPDATE NO ACTION ON DELETE CASCADE MATCH NONE"
"agent_checkpoints" }o--o| "agents" : "FOREIGN KEY (agentId) REFERENCES agents (id) ON UPDATE NO ACTION ON DELETE CASCADE MATCH NONE"
"agent_conversation_lease" }o--|| "agents" : "FOREIGN KEY (agentId) REFERENCES agents (id) ON UPDATE NO ACTION ON DELETE CASCADE MATCH NONE"
"agent_credential_dependency" |o--|| "agents" : "FOREIGN KEY (agentId) REFERENCES agents (id) ON UPDATE NO ACTION ON DELETE CASCADE MATCH NONE"
"agent_eval_dataset" }o--|| "agents" : "FOREIGN KEY (agentId) REFERENCES agents (id) ON UPDATE NO ACTION ON DELETE CASCADE MATCH NONE"
"agent_execution_threads" }o--|| "agents" : "FOREIGN KEY (agentId) REFERENCES agents (id) ON UPDATE NO ACTION ON DELETE CASCADE MATCH NONE"
@ -167,6 +168,12 @@ erDiagram
TEXT state
datetime_3_ updatedAt
}
"agent_conversation_lease" {
varchar_36_ agentId FK
datetime_3_ expiresAt
varchar ownerToken
varchar_128_ threadId PK
}
"agent_credential_dependency" {
varchar_36_ agentId PK
datetime_3_ createdAt

15
packages/@n8n/db/src/migrations/common/1789499956927-CreateAgentMessageQueue.ts
View File

@ -41,9 +41,24 @@ export class CreateAgentMessageQueue1789499956927 implements ReversibleMigration
columnName: 'id',
onDelete: 'SET NULL',
});
await createTable('agent_conversation_lease')
.withColumns(
// Ownership must work before the first execution thread exists.
column('threadId')
.varchar(128)
.primary.comment('Conversation routing ID.'),
column('agentId').varchar(36).notNull,
column('ownerToken').uuid.notNull.comment('Fresh identity for one execution owner.'),
column('expiresAt')
.timestampTimezone()
.notNull.comment('Lease expiry on the database clock.'),
)
.withIndexOn(['agentId'])
.withForeignKey('agentId', { tableName: 'agents', columnName: 'id', onDelete: 'CASCADE' });
}
async down({ schemaBuilder: { dropTable } }: MigrationContext) {
await dropTable('agent_conversation_lease');
await dropTable('agent_message_queue');
}
}

21
packages/cli/src/modules/agents/__tests__/agent-checkpoint.repository.test.ts
View File

@ -1,10 +1,11 @@
import { mock } from 'vitest-mock-extended';
/* eslint-disable @typescript-eslint/unbound-method -- mock-based tests intentionally reference unbound methods */
import { mockEntityManager } from '@test/mocking';
import { AgentCheckpoint } from '../entities/agent-checkpoint.entity';
import { AgentCheckpointRepository } from '../repositories/agent-checkpoint.repository';
const entityManager = mockEntityManager(AgentCheckpoint);
const entityManager = { ...mockEntityManager(AgentCheckpoint), update: vi.fn() };
const mockDataSource = { manager: entityManager };
describe('AgentCheckpointRepository', () => {
@ -12,12 +13,12 @@ describe('AgentCheckpointRepository', () => {
beforeEach(() => {
vi.clearAllMocks();
repository = new AgentCheckpointRepository(mockDataSource as never);
repository = new AgentCheckpointRepository(mockDataSource as never, mock());
});
describe('claimForResume', () => {
it('claims only the checkpoint row that still has the original suspended state', async () => {
vi.spyOn(repository, 'update').mockResolvedValue({ affected: 1 } as never);
vi.spyOn(entityManager, 'update').mockResolvedValue({ affected: 1 } as never);
await expect(
repository.claimForResume(
@ -25,10 +26,12 @@ describe('AgentCheckpointRepository', () => {
'agent-1',
'{"status":"suspended"}',
'{"status":"running"}',
{},
),
).resolves.toBe(true);
expect(repository.update).toHaveBeenCalledWith(
expect(entityManager.update).toHaveBeenCalledWith(
AgentCheckpoint,
{
runId: 'run-1',
agentId: 'agent-1',
@ -40,7 +43,7 @@ describe('AgentCheckpointRepository', () => {
});
it('returns false when another process already changed the checkpoint state', async () => {
vi.spyOn(repository, 'update').mockResolvedValue({ affected: 0 } as never);
vi.spyOn(entityManager, 'update').mockResolvedValue({ affected: 0 } as never);
await expect(
repository.claimForResume(
@ -48,6 +51,7 @@ describe('AgentCheckpointRepository', () => {
'agent-1',
'{"status":"suspended"}',
'{"status":"running"}',
{},
),
).resolves.toBe(false);
});
@ -55,13 +59,14 @@ describe('AgentCheckpointRepository', () => {
describe('cancelSuspended', () => {
it('matches checkpoints scoped to the current agent', async () => {
vi.spyOn(repository, 'update').mockResolvedValue({ affected: 1 } as never);
vi.spyOn(entityManager, 'update').mockResolvedValue({ affected: 1 } as never);
await expect(
repository.cancelSuspended('run-1', 'agent-1', '{"status":"suspended"}'),
repository.cancelSuspended('run-1', 'agent-1', '{"status":"suspended"}', {}),
).resolves.toBe(true);
expect(repository.update).toHaveBeenCalledWith(
expect(entityManager.update).toHaveBeenCalledWith(
AgentCheckpoint,
{
runId: 'run-1',
agentId: 'agent-1',

38
packages/cli/src/modules/agents/__tests__/agent-execution-orchestrator.service.test.ts
View File

@ -1,3 +1,4 @@
import { mockConversationLeases } from './mock-conversation-lease';
import type {
Agent as RuntimeAgent,
JSONValue,
@ -180,6 +181,8 @@ function makeService(sandboxEnabled = false) {
agentSandboxRuntimeService,
agentRepository,
aiConfigMock,
mockConversationLeases(),
);
return {
@ -461,7 +464,7 @@ describe('AgentExecutionOrchestratorService', () => {
],
genieAttribution,
);
checkpointStorage.getStatus.mockResolvedValueOnce({
checkpointStorage.getStatus.mockResolvedValue({
status: 'active',
checkpoint: { persistence: { threadId: 'thread-1', resourceId: 'resource-1' } },
} as never);
@ -1432,7 +1435,7 @@ describe('AgentExecutionOrchestratorService', () => {
} = makeService();
const runtime = makeRuntime([{ type: 'finish', finishReason: 'stop' }]);
checkpointStorage.getStatus.mockResolvedValueOnce({ status: 'expired' });
checkpointStorage.getStatus.mockResolvedValue({ status: 'expired' });
await expect(
collect(
service.resumeForChat({
@ -1446,7 +1449,7 @@ describe('AgentExecutionOrchestratorService', () => {
).rejects.toThrow(UserError);
expect(checkpointStorage.getStatus).toHaveBeenLastCalledWith('expired-run', agentId);
checkpointStorage.getStatus.mockResolvedValueOnce({
checkpointStorage.getStatus.mockResolvedValue({
status: 'active',
checkpoint: { persistence: { threadId: 'thread-1', resourceId: 'platform-user-1' } },
} as never);
@ -1474,7 +1477,7 @@ describe('AgentExecutionOrchestratorService', () => {
expect.objectContaining({
runId: 'run-1',
toolCallId: 'tc-1',
abortSignal: abortController.signal,
abortSignal: expect.any(AbortSignal),
onResumeClaimed,
}),
);
@ -1777,12 +1780,12 @@ describe('AgentExecutionOrchestratorService', () => {
}),
).resolves.toBe(true);
expect(checkpointStorage.delete).toHaveBeenCalledTimes(3);
expect(checkpointStorage.delete).toHaveBeenCalledWith(
expect(checkpointStorage.deleteSuspended).toHaveBeenCalledTimes(2);
expect(checkpointStorage.deleteSuspended).toHaveBeenCalledWith(
'configured-child-run',
'configured-agent',
);
expect(checkpointStorage.delete).toHaveBeenCalledWith('inline-child-run', agentId);
expect(checkpointStorage.deleteSuspended).toHaveBeenCalledWith('inline-child-run', agentId);
expect(checkpointStorage.delete).toHaveBeenCalledWith('run-1', agentId);
});
@ -1799,9 +1802,10 @@ describe('AgentExecutionOrchestratorService', () => {
});
checkpointStorage.getStatus
.mockResolvedValueOnce({ status: 'active', checkpoint })
.mockResolvedValueOnce({ status: 'expired', checkpoint });
.mockResolvedValueOnce({ status: 'active', checkpoint })
.mockResolvedValue({ status: 'expired', checkpoint });
checkpointStorage.cancelSuspended.mockResolvedValue(true);
checkpointStorage.delete
checkpointStorage.deleteSuspended
.mockRejectedValueOnce(new Error('child checkpoint unavailable'))
.mockResolvedValue(undefined);
@ -1814,9 +1818,9 @@ describe('AgentExecutionOrchestratorService', () => {
await expect(service.cancelChatRun(request)).resolves.toBe(true);
expect(checkpointStorage.cancelSuspended).toHaveBeenCalledOnce();
expect(checkpointStorage.delete).toHaveBeenNthCalledWith(1, 'child-run-1', agentId);
expect(checkpointStorage.delete).toHaveBeenNthCalledWith(2, 'child-run-1', agentId);
expect(checkpointStorage.delete).toHaveBeenNthCalledWith(3, 'run-1', agentId);
expect(checkpointStorage.deleteSuspended).toHaveBeenNthCalledWith(1, 'child-run-1', agentId);
expect(checkpointStorage.deleteSuspended).toHaveBeenNthCalledWith(2, 'child-run-1', agentId);
expect(checkpointStorage.delete).toHaveBeenCalledWith('run-1', agentId);
});
it('passes tracing telemetry returned by AgentRunTracingService into stream() and resume()', async () => {
@ -1849,7 +1853,7 @@ describe('AgentExecutionOrchestratorService', () => {
expect.objectContaining({ telemetry: fakeTelemetry }),
);
checkpointStorage.getStatus.mockResolvedValueOnce({
checkpointStorage.getStatus.mockResolvedValue({
status: 'active',
checkpoint: { persistence: { threadId: 'thread-1', resourceId: 'platform-user-1' } },
} as never);
@ -1882,7 +1886,7 @@ describe('AgentExecutionOrchestratorService', () => {
} = makeService();
const runtime = makeRuntime([{ type: 'finish', finishReason: 'stop' }]);
checkpointStorage.getStatus.mockResolvedValueOnce({
checkpointStorage.getStatus.mockResolvedValue({
status: 'active',
checkpoint: { persistence: { threadId: 'thread-1', resourceId: 'platform-user-1' } },
} as never);
@ -1924,7 +1928,7 @@ describe('AgentExecutionOrchestratorService', () => {
} = makeService();
const runtime = makeRuntime([{ type: 'finish', finishReason: 'stop' }]);
checkpointStorage.getStatus.mockResolvedValueOnce({
checkpointStorage.getStatus.mockResolvedValue({
status: 'active',
checkpoint: { persistence: { threadId: 'thread-1', resourceId: 'platform-user-1' } },
} as never);
@ -1958,7 +1962,7 @@ describe('AgentExecutionOrchestratorService', () => {
const runtime = makeRuntime([{ type: 'finish', finishReason: 'stop' }]);
Object.defineProperty(agentRunTracingService, 'enabled', { value: false });
checkpointStorage.getStatus.mockResolvedValueOnce({
checkpointStorage.getStatus.mockResolvedValue({
status: 'active',
checkpoint: { persistence: { threadId: 'thread-1', resourceId: 'platform-user-1' } },
} as never);
@ -1992,7 +1996,7 @@ describe('AgentExecutionOrchestratorService', () => {
},
]);
checkpointStorage.getStatus.mockResolvedValueOnce({
checkpointStorage.getStatus.mockResolvedValue({
status: 'active',
checkpoint: { persistence: { threadId: 'thread-1', resourceId: 'platform-user-1' } },
} as never);

3
packages/cli/src/modules/agents/__tests__/agent-execution.repository.test.ts
View File

@ -1,3 +1,4 @@
import { mock } from 'vitest-mock-extended';
/* eslint-disable @typescript-eslint/unbound-method -- mock-based tests intentionally reference unbound methods */
import { mockEntityManager } from '@test/mocking';
@ -12,7 +13,7 @@ describe('AgentExecutionRepository', () => {
beforeEach(() => {
vi.clearAllMocks();
repository = new AgentExecutionRepository(mockDataSource as never);
repository = new AgentExecutionRepository(mockDataSource as never, mock());
});
describe('findLatestSuspendedByThreadId', () => {

115
packages/cli/src/modules/agents/__tests__/agent-execution.service.test.ts
View File

@ -1,3 +1,4 @@
import { mockConversationLeases } from './mock-conversation-lease';
import type { Mocked } from 'vitest';
import { mockLogger } from '@n8n/backend-test-utils';
import { mock } from 'vitest-mock-extended';
@ -73,10 +74,17 @@ describe('AgentExecutionService', () => {
vi.clearAllMocks();
agentExecutionRepository = mock<AgentExecutionRepository>();
agentExecutionRepository.createExecution.mockResolvedValue(
mock<AgentExecution>({ id: 'execution-1' }),
);
agentExecutionRepository.updateIfRunning.mockResolvedValue(true);
agentExecutionRepository.updateIfAbandoned.mockResolvedValue(true);
agentExecutionRepository.updateTimelineIfRunning.mockResolvedValue(true);
agentExecutionThreadRepository = mock<AgentExecutionThreadRepository>();
agentExecutionThreadRepository.findOrCreate.mockResolvedValue({
thread: makeThread(),
created: false,
});
n8nMemory = mock<N8nMemory>();
memoryBackend = mock<N8nMemoryImplementation>();
n8nMemory.getImplementation.mockReturnValue(memoryBackend);
@ -100,9 +108,29 @@ describe('AgentExecutionService', () => {
errorReporter,
executionUpdateBroadcaster,
agentMessageQueueRepository,
mockConversationLeases(),
);
});
afterEach(() => {
for (const timer of service['heartbeatTimers'].values()) clearInterval(timer);
});
async function startRecording() {
await service.startExecutionRecording(
{
threadId: 'thread-1',
agentId: 'agent-1',
agentName: 'Agent',
projectId: 'project-1',
userMessage: 'Run',
},
new Date(),
);
executionUpdateBroadcaster.notify.mockClear();
}
async function recordExecution(params: RecordMessageParams): Promise<string> {
const { record, ...startParams } = params;
const executionId = await service.startExecutionRecording(
@ -119,8 +147,7 @@ describe('AgentExecutionService', () => {
created: false,
});
const execution = mock<AgentExecution>({ id: 'execution-1' });
agentExecutionRepository.create.mockReturnValue(execution);
agentExecutionRepository.save.mockResolvedValue(execution);
agentExecutionRepository.createExecution.mockResolvedValue(execution);
const initialTimeline: TimelineEvent[] = [
{
type: 'background-task-signal',
@ -139,13 +166,14 @@ describe('AgentExecutionService', () => {
initialTimeline,
};
executionUpdateBroadcaster.notify.mockImplementation(() => {
expect(agentExecutionRepository.save).toHaveBeenCalled();
expect(agentExecutionRepository.create).toHaveBeenCalledWith(
expect(agentExecutionRepository.createExecution).toHaveBeenCalled();
expect(agentExecutionRepository.createExecution).toHaveBeenCalledWith(
expect.objectContaining({
timeline: initialTimeline,
userMessage: null,
status: 'running',
}),
{},
);
});
const id = await service.startExecutionRecording(params, new Date(100));
@ -157,6 +185,7 @@ describe('AgentExecutionService', () => {
expect(agentExecutionRepository.updateIfRunning).toHaveBeenCalledWith(
id,
expect.objectContaining({ timeline: initialTimeline }),
{},
);
});
@ -167,8 +196,7 @@ describe('AgentExecutionService', () => {
thread: makeThread(),
created: true,
});
agentExecutionRepository.create.mockImplementation((data) => data as AgentExecution);
agentExecutionRepository.save.mockResolvedValue({
agentExecutionRepository.createExecution.mockResolvedValue({
id: 'execution-1',
} as AgentExecution);
agentExecutionRepository.touchRunning.mockResolvedValue();
@ -192,10 +220,11 @@ describe('AgentExecutionService', () => {
threadId: 'thread-1',
executionId,
});
expect(agentExecutionRepository.create).toHaveBeenCalledWith(
expect(agentExecutionRepository.createExecution).toHaveBeenCalledWith(
expect.objectContaining({ status: 'running' }),
{},
);
expect(agentExecutionRepository.touchRunning).toHaveBeenCalledWith(executionId);
expect(agentExecutionRepository.touchRunning).toHaveBeenCalledWith(executionId, {});
await service.finalizeExecution(executionId, {
threadId: 'thread-1',
@ -215,6 +244,7 @@ describe('AgentExecutionService', () => {
});
it('serializes timeline snapshot updates', async () => {
await startRecording();
let releaseFirstWrite!: () => void;
agentExecutionRepository.updateTimelineIfRunning
.mockImplementationOnce(
@ -252,10 +282,12 @@ describe('AgentExecutionService', () => {
expect(agentExecutionRepository.updateTimelineIfRunning).toHaveBeenLastCalledWith(
'execution-1',
second,
{},
);
});
it('notifies only after a timeline snapshot retry persists', async () => {
await startRecording();
vi.useFakeTimers();
try {
agentExecutionRepository.updateTimelineIfRunning
@ -288,6 +320,7 @@ describe('AgentExecutionService', () => {
});
it('does not notify when a late timeline snapshot is rejected', async () => {
await startRecording();
agentExecutionRepository.updateTimelineIfRunning.mockResolvedValue(false);
service.recordTimelineSnapshot({
@ -305,6 +338,7 @@ describe('AgentExecutionService', () => {
});
it('does not notify when execution finalization loses the running-state race', async () => {
await startRecording();
agentExecutionRepository.updateIfRunning.mockResolvedValue(false);
await service.finalizeExecution('execution-1', {
@ -334,6 +368,8 @@ describe('AgentExecutionService', () => {
errorReporter,
executionUpdateBroadcaster,
agentMessageQueueRepository,
mockConversationLeases(),
);
const record = makeMessageRecord({
@ -355,8 +391,9 @@ describe('AgentExecutionService', () => {
thread: makeThread(),
created: true,
});
agentExecutionRepository.create.mockImplementation((data) => data as AgentExecution);
agentExecutionRepository.save.mockResolvedValue({ id: 'execution-1' } as AgentExecution);
agentExecutionRepository.createExecution.mockResolvedValue({
id: 'execution-1',
} as AgentExecution);
await recordExecution({
threadId: 'thread-1',
@ -382,6 +419,7 @@ describe('AgentExecutionService', () => {
},
},
}),
{},
);
expect(agentExecutionLogStore.write).toHaveBeenCalledWith(
{ agentId: 'agent-1', threadId: 'thread-1', executionId: 'execution-1' },
@ -411,6 +449,8 @@ describe('AgentExecutionService', () => {
errorReporter,
executionUpdateBroadcaster,
agentMessageQueueRepository,
mockConversationLeases(),
);
const record = makeMessageRecord({
@ -432,8 +472,9 @@ describe('AgentExecutionService', () => {
thread: makeThread(),
created: true,
});
agentExecutionRepository.create.mockImplementation((data) => data as AgentExecution);
agentExecutionRepository.save.mockResolvedValue({ id: 'execution-1' } as AgentExecution);
agentExecutionRepository.createExecution.mockResolvedValue({
id: 'execution-1',
} as AgentExecution);
agentExecutionLogStore.write.mockRejectedValue(new Error('disk full'));
await recordExecution({
@ -453,6 +494,7 @@ describe('AgentExecutionService', () => {
storedAt: 'db',
failureSummary: null,
}),
{},
);
});
@ -470,8 +512,9 @@ describe('AgentExecutionService', () => {
error: null,
};
agentExecutionThreadRepository.findOrCreate.mockResolvedValue({ thread, created: true });
agentExecutionRepository.create.mockImplementation((entity) => entity as AgentExecution);
agentExecutionRepository.save.mockResolvedValue({ id: 'execution-1' } as AgentExecution);
agentExecutionRepository.createExecution.mockResolvedValue({
id: 'execution-1',
} as AgentExecution);
await recordExecution({
threadId: 'thread-1',
@ -506,8 +549,9 @@ describe('AgentExecutionService', () => {
thread: makeThread({ title: 'Task run' }),
created: false,
});
agentExecutionRepository.create.mockImplementation((data) => data as AgentExecution);
agentExecutionRepository.save.mockResolvedValue({ id: 'execution-1' } as AgentExecution);
agentExecutionRepository.createExecution.mockResolvedValue({
id: 'execution-1',
} as AgentExecution);
await recordExecution({
threadId: 'thread-1',
@ -537,8 +581,9 @@ describe('AgentExecutionService', () => {
thread: makeThread({ title: null }),
created: false,
});
agentExecutionRepository.create.mockImplementation((data) => data as AgentExecution);
agentExecutionRepository.save.mockResolvedValue({ id: 'execution-1' } as AgentExecution);
agentExecutionRepository.createExecution.mockResolvedValue({
id: 'execution-1',
} as AgentExecution);
memoryBackend.getThread.mockResolvedValue({
id: 'thread-1',
resourceId: 'user-1',
@ -566,8 +611,9 @@ describe('AgentExecutionService', () => {
thread: makeThread({ title: 'Existing title' }),
created: false,
});
agentExecutionRepository.create.mockImplementation((data) => data as AgentExecution);
agentExecutionRepository.save.mockResolvedValue({ id: 'execution-1' } as AgentExecution);
agentExecutionRepository.createExecution.mockResolvedValue({
id: 'execution-1',
} as AgentExecution);
await recordExecution({
threadId: 'thread-1',
@ -587,8 +633,9 @@ describe('AgentExecutionService', () => {
thread: makeThread(),
created: false,
});
agentExecutionRepository.create.mockImplementation((data) => data as AgentExecution);
agentExecutionRepository.save.mockResolvedValue({ id: 'execution-1' } as AgentExecution);
agentExecutionRepository.createExecution.mockResolvedValue({
id: 'execution-1',
} as AgentExecution);
await recordExecution({
threadId: 'thread-1',
@ -652,8 +699,9 @@ describe('AgentExecutionService', () => {
thread: makeThread(),
created: false,
});
agentExecutionRepository.create.mockImplementation((data) => data as AgentExecution);
agentExecutionRepository.save.mockResolvedValue({ id: 'execution-1' } as AgentExecution);
agentExecutionRepository.createExecution.mockResolvedValue({
id: 'execution-1',
} as AgentExecution);
telemetry.trackAgentTurnFinished.mockImplementation(() => {
throw new Error('telemetry failed');
});
@ -698,8 +746,9 @@ describe('AgentExecutionService', () => {
thread: makeThread(),
created: false,
});
agentExecutionRepository.create.mockImplementation((data) => data as AgentExecution);
agentExecutionRepository.save.mockResolvedValue({ id: 'execution-1' } as AgentExecution);
agentExecutionRepository.createExecution.mockResolvedValue({
id: 'execution-1',
} as AgentExecution);
await recordExecution({
threadId: 'thread-1',
@ -739,8 +788,9 @@ describe('AgentExecutionService', () => {
thread: makeThread(),
created: false,
});
agentExecutionRepository.create.mockImplementation((data) => data as AgentExecution);
agentExecutionRepository.save.mockResolvedValue({ id: 'execution-1' } as AgentExecution);
agentExecutionRepository.createExecution.mockResolvedValue({
id: 'execution-1',
} as AgentExecution);
await recordExecution({
threadId: 'thread-1',
@ -773,6 +823,7 @@ describe('AgentExecutionService', () => {
describe('finalizeExecution', () => {
it('makes the terminal timeline authoritative', async () => {
await startRecording();
const record = makeMessageRecord({
finishReason: 'cancelled',
timeline: [{ type: 'text', content: 'Done', timestamp: 1, endTime: 2 }],
@ -807,6 +858,7 @@ describe('AgentExecutionService', () => {
storedAt: 'db',
failureSummary: null,
}),
{},
);
expect(telemetry.trackAgentTurnFinished).toHaveBeenCalledWith(
expect.objectContaining({ turn_status: 'failed' }),
@ -827,12 +879,14 @@ describe('AgentExecutionService', () => {
errorReporter,
executionUpdateBroadcaster,
agentMessageQueueRepository,
mockConversationLeases(),
);
const partial = [{ type: 'text', content: 'Partial', timestamp: 1, endTime: 2 }] as const;
agentExecutionRepository.updateIfAbandoned.mockResolvedValue(true);
agentExecutionThreadRepository.findOneBy.mockResolvedValue(makeThread());
const staleBefore = new Date();
const staleBefore = 120_000;
await service.finalizeInterruptedExecution(
{
id: 'execution-1',
@ -870,6 +924,7 @@ describe('AgentExecutionService', () => {
},
},
}),
{},
);
expect(agentExecutionLogStore.write).not.toHaveBeenCalled();
});
@ -1078,7 +1133,7 @@ describe('AgentExecutionService', () => {
projectId: 'project-1',
agentId: 'agent-1',
});
expect(n8nMemory.getImplementation).toHaveBeenCalledWith('agent-1');
expect(n8nMemory.getImplementation).toHaveBeenCalledWith('agent-1', false);
expect(memoryBackend.deleteThread).toHaveBeenCalledWith('thread-1');
expect(agentChatAttachmentService.deleteByThread).toHaveBeenCalledWith('thread-1', {
projectId: 'project-1',

42
packages/cli/src/modules/agents/__tests__/agent-interrupted-execution-sweeper.test.ts
View File

@ -1,5 +1,6 @@
import type { AgentExecutionThreadRepository } from '../repositories/agent-execution-thread.repository';
import { mockConversationLeases } from './mock-conversation-lease';
import { mockLogger } from '@n8n/backend-test-utils';
import type { LockService } from '@n8n/backend-common';
import type { AgentMessageQueueService } from '../agent-message-queue.service';
import type { AgentsConfig } from '@n8n/config';
import { mock } from 'vitest-mock-extended';
@ -19,12 +20,12 @@ function setup(options: { backgroundTasksEnabled?: boolean } = {}) {
const agentsConfig = mock<AgentsConfig>({
backgroundTasksEnabled: options.backgroundTasksEnabled ?? false,
});
const lockService = mock<LockService>();
lockService.withLease.mockImplementation(
const leases = mockConversationLeases();
leases.withLease.mockImplementation(
async (_ns, _key, execute) => await execute(new AbortController().signal),
);
repository.findRunningById.mockImplementation(
async (id) => (await repository.findRunning()).find((row) => row.id === id) ?? null,
async (id) => (await repository.findStaleRunning(120_000)).find((row) => row.id === id) ?? null,
);
const sweeper = new AgentInterruptedExecutionSweeper(
mockLogger(),
@ -34,7 +35,10 @@ function setup(options: { backgroundTasksEnabled?: boolean } = {}) {
agentWakeService,
agentsConfig,
mock<AgentMessageQueueService>(),
lockService,
leases,
mock<AgentExecutionThreadRepository>({
findOneBy: vi.fn().mockResolvedValue({ agentId: 'agent-1' }),
}),
);
return { sweeper, repository, executionService, backgroundJobService, agentWakeService };
}
@ -49,43 +53,35 @@ describe('AgentInterruptedExecutionSweeper', () => {
startedAt: new Date(0),
updatedAt: new Date(0),
} as AgentExecution;
repository.findRunning.mockResolvedValue([execution]);
repository.findStaleRunning.mockResolvedValue([execution]);
executionService.finalizeInterruptedExecution.mockResolvedValue(true);
await sweeper.sweep();
expect(executionService.finalizeInterruptedExecution).toHaveBeenCalledWith(
execution,
expect.any(Date),
AgentInterruptedExecutionSweeper.LIVENESS_GRACE_MS,
);
});
it('leaves a recently active execution running in another process', async () => {
it('leaves an execution running when the database liveness recheck fails', async () => {
const { sweeper, repository, executionService } = setup();
repository.findRunning.mockResolvedValue([
{
id: 'execution-1',
threadId: 'thread-1',
status: 'running',
startedAt: new Date(Date.now() - AgentInterruptedExecutionSweeper.LIVENESS_GRACE_MS * 2),
updatedAt: new Date(),
} as AgentExecution,
]);
const execution = mock<AgentExecution>({ id: 'execution-1', threadId: 'thread-1' });
repository.findStaleRunning.mockResolvedValue([execution]);
executionService.finalizeInterruptedExecution.mockResolvedValue(false);
await sweeper.sweep();
expect(executionService.finalizeInterruptedExecution).not.toHaveBeenCalled();
expect(executionService.finalizeInterruptedExecution).toHaveBeenCalledOnce();
});
it('runs full reconciliation when the feature is on, and still reconciles workflow jobs when it is off', async () => {
const disabled = setup();
disabled.repository.findRunning.mockResolvedValue([]);
disabled.repository.findStaleRunning.mockResolvedValue([]);
await disabled.sweeper.sweep();
expect(disabled.backgroundJobService.reconcile).not.toHaveBeenCalled();
expect(disabled.backgroundJobService.reconcileWorkflowJobs).toHaveBeenCalled();
const enabled = setup({ backgroundTasksEnabled: true });
enabled.repository.findRunning.mockResolvedValue([]);
enabled.repository.findStaleRunning.mockResolvedValue([]);
await enabled.sweeper.sweep();
expect(enabled.backgroundJobService.reconcile).toHaveBeenCalled();
expect(enabled.backgroundJobService.reconcileWorkflowJobs).not.toHaveBeenCalled();
@ -93,7 +89,7 @@ describe('AgentInterruptedExecutionSweeper', () => {
it('checks for pending job results after reconciliation', async () => {
const { sweeper, repository, agentWakeService } = setup();
repository.findRunning.mockResolvedValue([]);
repository.findStaleRunning.mockResolvedValue([]);
await sweeper.sweep();

3
packages/cli/src/modules/agents/__tests__/agent-runtime-reconstruction-integration-tools.test.ts
View File

@ -1,3 +1,4 @@
import { mockConversationLeases } from './mock-conversation-lease';
import type { Mocked } from 'vitest';
import { type AgentJsonConfig } from '@n8n/api-types';
import type { Logger } from '@n8n/backend-common';
@ -283,6 +284,8 @@ describe('AgentRuntimeReconstructionService integration tools', () => {
agentSandboxRuntimeService,
agentRepository,
mock<AiConfig>(),
mockConversationLeases(),
);
agentIntegrationPersistenceService = new AgentIntegrationPersistenceService(
agentRepository,

2
packages/cli/src/modules/agents/__tests__/agent-workflow-execution.service.test.ts
View File

@ -1,3 +1,4 @@
import { mockConversationLeases } from './mock-conversation-lease';
import type { Agent as RuntimeAgent, StreamChunk } from '@n8n/agents';
import type { AgentJsonConfig } from '@n8n/api-types';
import { mockLogger } from '@n8n/backend-test-utils';
@ -143,6 +144,7 @@ function makeService() {
executionLevelTracer,
nodeToolAiGatewayService,
aiConfigMock,
mockConversationLeases(),
);
return {

9
packages/cli/src/modules/agents/__tests__/agent-workflow-tool-resume.service.test.ts
View File

@ -1,5 +1,6 @@
import { mockConversationLeases } from './mock-conversation-lease';
import { N8N_CHAT_INTEGRATION_TYPE } from '@n8n/api-types';
import type { LockService, Logger } from '@n8n/backend-common';
import type { Logger } from '@n8n/backend-common';
import type { User, UserRepository } from '@n8n/db';
import type { WorkflowExecuteAfterContext } from '@n8n/decorators';
import type { InstanceSettings } from 'n8n-core';
@ -54,8 +55,8 @@ function setup() {
checkpoint: { status: 'suspended' },
} as never);
const backgroundJobService = mock<AgentBackgroundJobService>();
const lockService = mock<LockService>();
lockService.withLease.mockImplementation(
const leases = mockConversationLeases();
leases.withLease.mockImplementation(
async (_ns, _key, execute) => await execute(new AbortController().signal),
);
const messageQueue = mock<AgentMessageQueueService>();
@ -70,7 +71,7 @@ function setup() {
instanceSettings,
publisher,
backgroundJobService,
lockService,
leases,
messageQueue,
);
return {

29
packages/cli/src/modules/agents/__tests__/mock-conversation-lease.ts Normal file
View File

@ -0,0 +1,29 @@
import { mock } from 'vitest-mock-extended';
import type { AgentConversationLeaseService } from '../agent-conversation-lease.service';
// Persistence and async context are exercised against real leases in the integration suite.
export function mockConversationLeases() {
const leases = mock<AgentConversationLeaseService>();
const controller = new AbortController();
leases.isHeld.mockResolvedValue(false);
leases.requireOwner.mockImplementation((threadId) => ({
lease: { threadId, agentId: 'agent-1', ownerToken: 'owner-1' },
signal: controller.signal,
lose: (error) => controller.abort(error),
}));
leases.withLease.mockImplementation(
async (_agentId, _threadId, run, options) =>
await run(
options?.signal ? AbortSignal.any([controller.signal, options.signal]) : controller.signal,
),
);
leases.stream.mockImplementation((_agentId, _threadId, create, options) =>
create(
options?.signal ? AbortSignal.any([controller.signal, options.signal]) : controller.signal,
),
);
leases.write.mockImplementation(async (_owner, write) => await write({}));
leases.outsideConversation.mockImplementation((run) => run());
return leases;
}

185
packages/cli/src/modules/agents/agent-conversation-lease.service.ts Normal file
View File

@ -0,0 +1,185 @@
import { Logger } from '@n8n/backend-common';
import { TransactionRunner, type OperationContext } from '@n8n/db';
import { Service } from '@n8n/di';
import { AsyncLocalStorage } from 'node:async_hooks';
import { setTimeout as delay } from 'node:timers/promises';
import { UnexpectedError } from 'n8n-workflow';
import {
AgentConversationLeaseLostError,
AgentConversationLeaseTimeoutError,
CONVERSATION_LEASE_RENEW_MS,
type AgentConversationLeaseHandle,
} from './agent-conversation-lease.types';
import { AgentConversationLeaseRepository } from './repositories/agent-conversation-lease.repository';
export type AgentConversationOwner = Readonly<{
lease: AgentConversationLeaseHandle;
signal: AbortSignal;
lose: (error: unknown) => void;
}>;
const executionOwner = new AsyncLocalStorage<AgentConversationOwner>();
@Service()
export class AgentConversationLeaseService {
constructor(
private readonly repository: AgentConversationLeaseRepository,
private readonly transactionRunner: TransactionRunner,
private readonly logger: Logger,
) {}
currentOwner(): AgentConversationOwner | undefined {
return executionOwner.getStore();
}
requireOwner(threadId: string, agentId?: string): AgentConversationOwner {
const owner = this.currentOwner();
if (
!owner ||
owner.lease.threadId !== threadId ||
(agentId !== undefined && owner.lease.agentId !== agentId)
) {
throw new UnexpectedError('Agent conversation execution has no matching owner');
}
return owner;
}
async write<T>(
owner: AgentConversationOwner,
write: (ctx: OperationContext) => Promise<T>,
): Promise<T> {
if (owner.signal.aborted) throw new AgentConversationLeaseLostError(owner.lease.threadId);
try {
return await this.transactionRunner.run({}, async (ctx) => {
await this.repository.assertOwner(owner.lease, ctx);
return await write(ctx);
});
} catch (error) {
if (error instanceof AgentConversationLeaseLostError) owner.lose(error);
throw error;
}
}
async isHeld(threadId: string): Promise<boolean> {
return await this.repository.isHeld(threadId);
}
async withLease<T>(
agentId: string,
threadId: string,
run: (signal: AbortSignal) => Promise<T>,
options: { signal?: AbortSignal; waitTimeoutMs?: number } = {},
): Promise<T> {
const active = await this.acquire(agentId, threadId, options);
try {
return await executionOwner.run(active.owner, async () => await run(active.signal));
} finally {
await active.release();
}
}
stream<T>(
agentId: string,
threadId: string,
create: (signal: AbortSignal) => AsyncGenerator<T>,
options: { signal?: AbortSignal } = {},
): AsyncGenerator<T> {
const current = this.currentOwner();
return this.consumeStream(agentId, threadId, create, options, current);
}
private async *consumeStream<T>(
agentId: string,
threadId: string,
create: (signal: AbortSignal) => AsyncGenerator<T>,
options: { signal?: AbortSignal },
current: AgentConversationOwner | undefined,
): AsyncGenerator<T> {
const borrowed = current?.lease.threadId === threadId && current.lease.agentId === agentId;
const acquired = borrowed ? undefined : await this.acquire(agentId, threadId, options);
const owner = borrowed ? current : acquired!.owner;
const signal = options.signal ? AbortSignal.any([owner.signal, options.signal]) : owner.signal;
let iterator: AsyncGenerator<T> | undefined;
try {
await this.write(owner, async () => undefined);
signal.throwIfAborted();
iterator = executionOwner.run(owner, () => create(signal));
while (true) {
// The caller can request the next chunk from a different async context.
const next = await executionOwner.run(owner, async () => await iterator!.next());
if (next.done) return;
yield next.value;
}
} finally {
try {
await executionOwner.run(owner, async () => await iterator?.return(undefined));
} finally {
await acquired?.release();
}
}
}
outsideConversation<T>(run: () => T): T {
return executionOwner.exit(run);
}
private async acquire(
agentId: string,
threadId: string,
options: { signal?: AbortSignal; waitTimeoutMs?: number },
) {
const started = performance.now();
let handle: AgentConversationLeaseHandle | null;
do {
options.signal?.throwIfAborted();
handle = await this.repository.acquire(agentId, threadId);
if (handle) break;
const remaining = (options.waitTimeoutMs ?? Infinity) - (performance.now() - started);
if (remaining <= 0) throw new AgentConversationLeaseTimeoutError(threadId);
await delay(Math.min(250, remaining), undefined, { signal: options.signal });
} while (!handle);
const controller = new AbortController();
const owner: AgentConversationOwner = {
lease: handle,
signal: controller.signal,
lose: (error) => controller.abort(error),
};
let renewing: Promise<void> | undefined;
const timer = setInterval(() => {
if (renewing || controller.signal.aborted) return;
renewing = this.repository
.renew(owner.lease)
.then((renewed) => {
if (!renewed) owner.lose(new AgentConversationLeaseLostError(threadId));
})
.catch((error: unknown) => {
owner.lose(error);
this.logger.warn('Agent conversation lease renewal failed', { threadId, error });
})
.finally(() => {
renewing = undefined;
});
}, CONVERSATION_LEASE_RENEW_MS);
timer.unref();
controller.signal.addEventListener('abort', () => clearInterval(timer), { once: true });
return {
owner,
signal: options.signal
? AbortSignal.any([controller.signal, options.signal])
: controller.signal,
release: async () => {
clearInterval(timer);
await renewing;
if (controller.signal.aborted) return;
owner.lose(new AgentConversationLeaseLostError(threadId));
try {
await this.repository.release(owner.lease);
} catch (error) {
this.logger.warn('Failed to release agent conversation ownership', { threadId, error });
}
},
};
}
}

22
packages/cli/src/modules/agents/agent-conversation-lease.types.ts Normal file
View File

@ -0,0 +1,22 @@
import { OperationalError } from 'n8n-workflow';
export const CONVERSATION_LEASE_TTL_MS = 120_000;
export const CONVERSATION_LEASE_RENEW_MS = 30_000;
export type AgentConversationLeaseHandle = Readonly<{
agentId: string;
threadId: string;
ownerToken: string;
}>;
export class AgentConversationLeaseLostError extends OperationalError {
constructor(threadId: string) {
super('Agent conversation ownership has ended', { extra: { threadId } });
}
}
export class AgentConversationLeaseTimeoutError extends OperationalError {
constructor(threadId: string) {
super('Agent conversation is busy', { extra: { threadId } });
}
}

110
packages/cli/src/modules/agents/agent-execution-orchestrator.service.ts
View File

@ -44,6 +44,11 @@ import { buildAgentConfigurationTelemetry } from './agent-telemetry';
import { buildToolCallDetails, ExecutionRecorder, type MessageRecord } from './execution-recorder';
import { IntegrationMessageContextService } from './integrations/integration-message-context.service';
import { N8NCheckpointStorage } from './integrations/n8n-checkpoint-storage';
import { AgentConversationLeaseService } from './agent-conversation-lease.service';
import {
AgentConversationLeaseLostError,
AgentConversationLeaseTimeoutError,
} from './agent-conversation-lease.types';
import { modelStreamStallOptions } from './model-stream-stall-options';
import { AgentRepository } from './repositories/agent.repository';
import type { ToolRegistry } from './tool-registry';
@ -152,6 +157,7 @@ export interface ResumeForChatConfig {
}
export interface ExecuteForTaskPublishedConfig {
abortSignal?: AbortSignal;
agentId: string;
projectId: string;
message: string;
@ -164,6 +170,7 @@ export interface ExecuteForTaskPublishedConfig {
}
export interface ExecuteForTaskNowConfig {
abortSignal?: AbortSignal;
agentId: string;
projectId: string;
/**
@ -336,6 +343,7 @@ export class AgentExecutionOrchestratorService {
private readonly agentSandboxRuntimeService: AgentSandboxRuntimeService,
private readonly agentRepository: AgentRepository,
private readonly aiConfig: AiConfig,
private readonly leases: AgentConversationLeaseService,
) {}
/**
@ -361,6 +369,36 @@ export class AgentExecutionOrchestratorService {
runId: string;
resourceId: string;
onCancelled?: (threadId: string) => void;
}): Promise<boolean> {
const initial = await this.n8nCheckpointStorage.getStatus(params.runId, params.agentId);
const memory = initial.status !== 'not-found' ? initial.checkpoint?.persistence : undefined;
if (!memory || memory.delegated || memory.resourceId !== params.resourceId) return false;
const current = this.leases.currentOwner();
try {
if (current?.lease.threadId === memory.threadId && current.lease.agentId === params.agentId) {
return await this.cancelChatRunInsideLease(params);
}
return await this.leases.withLease(
params.agentId,
memory.threadId,
async () => await this.cancelChatRunInsideLease(params),
{ waitTimeoutMs: 250 },
);
} catch (error) {
if (
error instanceof AgentConversationLeaseTimeoutError ||
error instanceof AgentConversationLeaseLostError
)
return false;
throw error;
}
}
private async cancelChatRunInsideLease(params: {
agentId: string;
runId: string;
resourceId: string;
onCancelled?: (threadId: string) => void;
}): Promise<boolean> {
const checkpointStatus = await this.n8nCheckpointStorage.getStatus(
params.runId,
@ -391,7 +429,8 @@ export class AgentExecutionOrchestratorService {
await Promise.all(
childCheckpoints.map(
async ({ runId, agentId }) => await this.n8nCheckpointStorage.delete(runId, agentId),
async ({ runId, agentId }) =>
await this.n8nCheckpointStorage.deleteSuspended(runId, agentId),
),
);
await this.n8nCheckpointStorage.delete(params.runId, params.agentId);
@ -405,6 +444,25 @@ export class AgentExecutionOrchestratorService {
* a human-in-the-loop action (button click, modal submission).
*/
async *resumeForChat(config: ResumeForChatConfig): AsyncGenerator<StreamChunk> {
const initial = await this.n8nCheckpointStorage.getStatus(config.runId, config.agentId);
if (initial.status === 'expired') {
throw new UserError(`Checkpoint ${config.runId} is expired and cannot be resumed`);
}
if (initial.status === 'not-found') {
throw new UserError(`Checkpoint ${config.runId} not found and cannot be resumed`);
}
const threadId = initial.checkpoint.persistence?.threadId;
if (!threadId)
throw new UserError(`Checkpoint ${config.runId} has no memory data and cannot be resumed`);
yield* this.leases.stream(
config.agentId,
threadId,
(signal) => this.resumeForChatOwned({ ...config, abortSignal: signal }),
{ signal: config.abortSignal },
);
}
private async *resumeForChatOwned(config: ResumeForChatConfig): AsyncGenerator<StreamChunk> {
const {
agentId,
projectId,
@ -605,7 +663,16 @@ export class AgentExecutionOrchestratorService {
/**
* Execute an agent for the in-app test chat and yield stream chunks.
*/
async *executeForChat(config: ExecuteForChatConfig): AsyncGenerator<StreamChunk> {
executeForChat(config: ExecuteForChatConfig): AsyncGenerator<StreamChunk> {
return this.leases.stream(
config.agentId,
config.memory.threadId,
(signal) => this.executeForChatOwned({ ...config, abortSignal: signal }),
{ signal: config.abortSignal },
);
}
private async *executeForChatOwned(config: ExecuteForChatConfig): AsyncGenerator<StreamChunk> {
const {
agentId,
projectId,
@ -674,7 +741,16 @@ export class AgentExecutionOrchestratorService {
*
* Loads the published snapshot — never the draft.
*/
async *executeForChatPublished(
executeForChatPublished(config: ExecuteForChatPublishedConfig): AsyncGenerator<StreamChunk> {
return this.leases.stream(
config.agentId,
config.memory.threadId,
(signal) => this.executeForChatPublishedOwned({ ...config, abortSignal: signal }),
{ signal: config.abortSignal },
);
}
private async *executeForChatPublishedOwned(
config: ExecuteForChatPublishedConfig,
): AsyncGenerator<StreamChunk> {
const {
@ -739,7 +815,16 @@ export class AgentExecutionOrchestratorService {
* Execute a published agent for a scheduled task, stamping `source='task'`
* and the originating `taskId` on the recorded session for traceability.
*/
async *executeForTaskPublished(
executeForTaskPublished(config: ExecuteForTaskPublishedConfig): AsyncGenerator<StreamChunk> {
return this.leases.stream(
config.agentId,
config.memory.threadId,
(signal) => this.executeForTaskPublishedOwned({ ...config, abortSignal: signal }),
{ signal: config.abortSignal },
);
}
private async *executeForTaskPublishedOwned(
config: ExecuteForTaskPublishedConfig,
): AsyncGenerator<StreamChunk> {
const { agentId, projectId, message, memory, taskId, taskVersionId } = config;
@ -761,6 +846,7 @@ export class AgentExecutionOrchestratorService {
try {
yield* this.streamChatResponse({
abortSignal: config.abortSignal,
agentInstance: runtime.agent,
toolRegistry: runtime.toolRegistry,
mcpServerAttributions: runtime.mcpServerAttributions,
@ -786,7 +872,18 @@ export class AgentExecutionOrchestratorService {
* Execute a task on demand against the current (draft) config as the
* requesting user.
*/
async *executeForTaskNow(config: ExecuteForTaskNowConfig): AsyncGenerator<StreamChunk> {
executeForTaskNow(config: ExecuteForTaskNowConfig): AsyncGenerator<StreamChunk> {
return this.leases.stream(
config.agentId,
config.memory.threadId,
(signal) => this.executeForTaskNowOwned({ ...config, abortSignal: signal }),
{ signal: config.abortSignal },
);
}
private async *executeForTaskNowOwned(
config: ExecuteForTaskNowConfig,
): AsyncGenerator<StreamChunk> {
const { agentId, projectId, user, message, memory, taskId } = config;
// `user` is always set (see ExecuteForTaskNowConfig) — manual "Run now"
@ -806,6 +903,7 @@ export class AgentExecutionOrchestratorService {
try {
yield* this.streamChatResponse({
abortSignal: config.abortSignal,
agentInstance: runtime.agent,
toolRegistry: runtime.toolRegistry,
mcpServerAttributions: runtime.mcpServerAttributions,
@ -1168,6 +1266,7 @@ export class AgentExecutionOrchestratorService {
try {
return await this.agentExecutionService.startExecutionRecording(params, startedAt);
} catch (error) {
if (error instanceof AgentConversationLeaseLostError) throw error;
this.logger.warn(failureMessage, {
agentId: params.agentId,
threadId: params.threadId,
@ -1189,6 +1288,7 @@ export class AgentExecutionOrchestratorService {
const recordedId = await this.agentExecutionService.finalizeExecution(executionId, params);
onExecutionRecorded?.(recordedId);
} catch (error) {
if (error instanceof AgentConversationLeaseLostError) throw error;
this.logger.warn(failureMessage, {
agentId: params.agentId,
threadId: params.threadId,

211
packages/cli/src/modules/agents/agent-execution.service.ts
View File

@ -18,6 +18,11 @@ import {
type StoredAttachmentRef,
} from './agent-chat-attachment.service';
import { AgentExecutionUpdateBroadcaster } from './agent-execution-update-broadcaster';
import {
AgentConversationLeaseService,
type AgentConversationOwner,
} from './agent-conversation-lease.service';
import { AgentConversationLeaseLostError } from './agent-conversation-lease.types';
import { AgentExecutionThread } from './entities/agent-execution-thread.entity';
import { AgentExecution, type AgentExecutionStatus } from './entities/agent-execution.entity';
import type { MessageRecord, TimelineEvent } from './execution-recorder';
@ -99,10 +104,11 @@ export class AgentExecutionService {
private static readonly heartbeatIntervalMs = 30_000;
private readonly heartbeatTimers = new Map<string, NodeJS.Timeout>();
private readonly recordingOwners = new Map<string, AgentConversationOwner>();
private readonly pendingTimelineSnapshots = new Map<
string,
Omit<TimelineSnapshotParams, 'executionId'>
Omit<TimelineSnapshotParams, 'executionId'> & { owner: AgentConversationOwner }
>();
private readonly timelineSnapshotWrites = new Map<string, Promise<void>>();
@ -121,36 +127,44 @@ export class AgentExecutionService {
private readonly errorReporter: ErrorReporter,
private readonly executionUpdateBroadcaster: AgentExecutionUpdateBroadcaster,
private readonly agentMessageQueueRepository: AgentMessageQueueRepository,
private readonly leases: AgentConversationLeaseService,
) {}
async startExecutionRecording(params: StartExecutionParams, startedAt: Date): Promise<string> {
const owner = this.leases.requireOwner(params.threadId, params.agentId);
const { userMessage, created } = await this.prepareThread(params);
const inserted = await this.agentExecutionRepository.save(
this.agentExecutionRepository.create({
threadId: params.threadId,
status: 'running',
startedAt,
stoppedAt: null,
duration: 0,
userMessage,
author: params.author ?? null,
model: null,
promptTokens: null,
completionTokens: null,
totalTokens: null,
cost: null,
// Save the background job signal before notifying clients that the execution started.
timeline: params.initialTimeline?.length ? params.initialTimeline : null,
storedAt: 'db',
error: null,
failureSummary: null,
hitlStatus: null,
source: params.source ?? null,
attachments: params.attachments?.length ? params.attachments : null,
}),
const inserted = await this.leases.write(
owner,
async (ctx) =>
await this.agentExecutionRepository.createExecution(
{
threadId: params.threadId,
status: 'running',
startedAt,
stoppedAt: null,
duration: 0,
userMessage,
author: params.author ?? null,
model: null,
promptTokens: null,
completionTokens: null,
totalTokens: null,
cost: null,
// Save the background job signal before notifying clients that the execution started.
timeline: params.initialTimeline?.length ? params.initialTimeline : null,
storedAt: 'db',
error: null,
failureSummary: null,
hitlStatus: null,
source: params.source ?? null,
attachments: params.attachments?.length ? params.attachments : null,
},
ctx,
),
);
this.recordingOwners.set(inserted.id, owner);
if (created) this.executionsNeedingTitleSync.add(inserted.id);
this.startHeartbeat(inserted.id);
this.startHeartbeat(inserted.id, owner);
this.executionUpdateBroadcaster.notify({
projectId: params.projectId,
agentId: params.agentId,
@ -161,11 +175,15 @@ export class AgentExecutionService {
}
recordTimelineSnapshot({ executionId, ...snapshot }: TimelineSnapshotParams): void {
this.pendingTimelineSnapshots.set(executionId, snapshot);
const owner = this.recordingOwners.get(executionId);
if (owner === undefined) return;
this.pendingTimelineSnapshots.set(executionId, { ...snapshot, owner });
this.ensureTimelineSnapshotWrite(executionId);
}
async finalizeExecution(executionId: string, params: RecordMessageParams): Promise<string> {
const owner = this.recordingOwners.get(executionId);
if (owner === undefined) throw new UnexpectedError('Agent execution has no recording owner');
const { record, hitlStatus } = params;
const status = executionStatus(record);
const stoppedAt = new Date(record.startTime + record.duration);
@ -192,21 +210,29 @@ export class AgentExecutionService {
}
}
const finalized = await this.agentExecutionRepository.updateIfRunning(executionId, {
status,
stoppedAt,
duration: record.duration,
model: record.model,
promptTokens: record.usage?.promptTokens ?? null,
completionTokens: record.usage?.completionTokens ?? null,
totalTokens: record.usage?.totalTokens ?? null,
cost: record.totalCost,
timeline: storedAt === 'db' && record.timeline.length > 0 ? record.timeline : null,
storedAt,
error: record.error,
failureSummary,
hitlStatus: hitlStatus ?? null,
});
const finalized = await this.leases.write(
owner,
async (ctx) =>
await this.agentExecutionRepository.updateIfRunning(
executionId,
{
status,
stoppedAt,
duration: record.duration,
model: record.model,
promptTokens: record.usage?.promptTokens ?? null,
completionTokens: record.usage?.completionTokens ?? null,
totalTokens: record.usage?.totalTokens ?? null,
cost: record.totalCost,
timeline: storedAt === 'db' && record.timeline.length > 0 ? record.timeline : null,
storedAt,
error: record.error,
failureSummary,
hitlStatus: hitlStatus ?? null,
},
ctx,
),
);
if (!finalized) return executionId;
this.executionUpdateBroadcaster.notify({
@ -215,7 +241,7 @@ export class AgentExecutionService {
threadId: params.threadId,
executionId,
});
await this.completeRecordedExecution(params, executionId, status);
await this.completeRecordedExecution(params, executionId, status, owner);
return executionId;
} catch (error) {
this.errorReporter.error(error);
@ -223,12 +249,13 @@ export class AgentExecutionService {
} finally {
this.stopHeartbeat(executionId);
this.executionsNeedingTitleSync.delete(executionId);
this.recordingOwners.delete(executionId);
}
}
async finalizeInterruptedExecution(
execution: RunningAgentExecution,
staleBefore: Date,
graceMs: number,
): Promise<boolean> {
const timeline = execution.timeline ?? [];
const stoppedAt = new Date();
@ -236,23 +263,29 @@ export class AgentExecutionService {
const duration = execution.startedAt
? Math.max(0, stoppedAt.getTime() - execution.startedAt.getTime())
: 0;
const finalized = await this.agentExecutionRepository.updateIfAbandoned(
execution.id,
staleBefore,
{
status: 'interrupted',
stoppedAt,
duration,
timeline: timeline.length > 0 ? timeline : null,
storedAt: 'db',
error,
failureSummary: computeExecutionFailureSummary({
timeline,
status: 'interrupted',
error,
stoppedAt: stoppedAt.getTime(),
}),
},
const owner = this.leases.requireOwner(execution.threadId);
const finalized = await this.leases.write(
owner,
async (ctx) =>
await this.agentExecutionRepository.updateIfAbandoned(
execution.id,
graceMs,
{
status: 'interrupted',
stoppedAt,
duration,
timeline: timeline.length > 0 ? timeline : null,
storedAt: 'db',
error,
failureSummary: computeExecutionFailureSummary({
timeline,
status: 'interrupted',
error,
stoppedAt: stoppedAt.getTime(),
}),
},
ctx,
),
);
if (finalized) void this.notifyInterruptedExecution(execution);
return finalized;
@ -279,14 +312,23 @@ export class AgentExecutionService {
}
}
private startHeartbeat(executionId: string): void {
private startHeartbeat(executionId: string, owner: AgentConversationOwner): void {
const timer = setInterval(() => {
void this.agentExecutionRepository.touchRunning(executionId).catch((error: unknown) => {
this.logger.warn('Failed to heartbeat a running agent execution', {
executionId,
error: error instanceof Error ? error.message : String(error),
if (owner.signal.aborted) {
this.stopHeartbeat(executionId);
return;
}
void this.leases
.write(
owner,
async (ctx) => await this.agentExecutionRepository.touchRunning(executionId, ctx),
)
.catch((error: unknown) => {
this.logger.warn('Failed to heartbeat a running agent execution', {
executionId,
error: error instanceof Error ? error.message : String(error),
});
});
});
}, AgentExecutionService.heartbeatIntervalMs);
timer.unref();
this.heartbeatTimers.set(executionId, timer);
@ -319,9 +361,14 @@ export class AgentExecutionService {
this.pendingTimelineSnapshots.delete(executionId);
try {
if (
!(await this.agentExecutionRepository.updateTimelineIfRunning(
executionId,
snapshot.timeline,
!(await this.leases.write(
snapshot.owner,
async (ctx) =>
await this.agentExecutionRepository.updateTimelineIfRunning(
executionId,
snapshot.timeline,
ctx,
),
))
) {
this.pendingTimelineSnapshots.delete(executionId);
@ -334,6 +381,10 @@ export class AgentExecutionService {
executionId,
});
} catch (error) {
if (error instanceof AgentConversationLeaseLostError) {
this.pendingTimelineSnapshots.delete(executionId);
return;
}
if (!this.pendingTimelineSnapshots.has(executionId)) {
this.pendingTimelineSnapshots.set(executionId, snapshot);
}
@ -372,10 +423,11 @@ export class AgentExecutionService {
params: RecordMessageParams,
executionId: string,
status: AgentExecution['status'],
owner: AgentConversationOwner,
): Promise<void> {
const { threadId, agentId, record, hitlStatus } = params;
if (hitlStatus === 'resumed' && record.model) {
await this.backfillSuspendedExecutions(threadId, record.model);
await this.backfillSuspendedExecutions(threadId, record.model, owner);
}
if (record.usage) {
await this.agentExecutionThreadRepository.incrementUsage(
@ -447,12 +499,21 @@ export class AgentExecutionService {
* Called when the resumed run finishes — the model applies to the whole
* suspend/resume cycle but only arrives once the resume completes.
*/
private async backfillSuspendedExecutions(threadId: string, model: string): Promise<void> {
private async backfillSuspendedExecutions(
threadId: string,
model: string,
owner: AgentConversationOwner,
): Promise<void> {
const candidates = await this.agentExecutionRepository.findSuspendedWithoutModel(threadId);
if (candidates.length === 0) return;
await this.agentExecutionRepository.backfillModel(
candidates.map((c) => c.id),
model,
await this.leases.write(
owner,
async (ctx) =>
await this.agentExecutionRepository.backfillModel(
candidates.map((c) => c.id),
model,
ctx,
),
);
}
@ -497,7 +558,7 @@ export class AgentExecutionService {
await this.agentExecutionRepository.findBlobRefsByThreadId(threadId),
);
await this.n8nMemory.getImplementation(agentId).deleteThread(threadId);
await this.n8nMemory.getImplementation(agentId, false).deleteThread(threadId);
await this.agentChatAttachmentService.deleteByThread(threadId, { projectId });
await Promise.all([
this.agentExecutionThreadRepository.delete({ id: threadId }),

35
packages/cli/src/modules/agents/agent-interrupted-execution-sweeper.ts
View File

@ -1,15 +1,12 @@
import {
LockAcquisitionTimeoutError,
LockNamespace,
LockService,
Logger,
} from '@n8n/backend-common';
import { Logger } from '@n8n/backend-common';
import { AgentsConfig } from '@n8n/config';
import { Service } from '@n8n/di';
import { AgentExecutionService } from './agent-execution.service';
import { AgentMessageQueueService } from './agent-message-queue.service';
import { agentConversationLockKey } from './agent-message-queue.types';
import { AgentConversationLeaseService } from './agent-conversation-lease.service';
import { AgentConversationLeaseTimeoutError } from './agent-conversation-lease.types';
import { AgentExecutionThreadRepository } from './repositories/agent-execution-thread.repository';
import { AgentBackgroundJobService } from './background/agent-background-job.service';
import { AgentWakeService } from './background/agent-wake.service';
import { AgentExecutionRepository } from './repositories/agent-execution.repository';
@ -26,16 +23,17 @@ export class AgentInterruptedExecutionSweeper {
private readonly agentWakeService: AgentWakeService,
private readonly agentsConfig: AgentsConfig,
private readonly messageQueue: AgentMessageQueueService,
private readonly lockService: LockService,
private readonly leases: AgentConversationLeaseService,
private readonly threadRepository: AgentExecutionThreadRepository,
) {
this.logger = this.logger.scoped('agents');
}
async sweep(): Promise<void> {
const staleBefore = new Date(Date.now() - AgentInterruptedExecutionSweeper.LIVENESS_GRACE_MS);
const graceMs = AgentInterruptedExecutionSweeper.LIVENESS_GRACE_MS;
let running;
try {
running = await this.executionRepository.findRunning();
running = await this.executionRepository.findStaleRunning(graceMs);
} catch (error) {
this.logger.error('Failed to query running agent executions', { error });
return;
@ -43,16 +41,15 @@ export class AgentInterruptedExecutionSweeper {
for (const execution of running) {
try {
if (execution.updatedAt > staleBefore) {
continue;
}
await this.lockService.withLease(
LockNamespace.KNOWN_LOCKS,
agentConversationLockKey(execution.threadId),
const thread = await this.threadRepository.findOneBy({ id: execution.threadId });
if (!thread) continue;
await this.leases.withLease(
thread.agentId,
execution.threadId,
async () => {
const current = await this.executionRepository.findRunningById(execution.id);
if (!current || current.updatedAt > staleBefore) return;
if (await this.executionService.finalizeInterruptedExecution(current, staleBefore)) {
if (!current) return;
if (await this.executionService.finalizeInterruptedExecution(current, graceMs)) {
this.logger.info('Marked abandoned agent execution as interrupted', {
executionId: execution.id,
threadId: execution.threadId,
@ -62,7 +59,7 @@ export class AgentInterruptedExecutionSweeper {
{ waitTimeoutMs: 250 },
);
} catch (error) {
if (error instanceof LockAcquisitionTimeoutError) continue;
if (error instanceof AgentConversationLeaseTimeoutError) continue;
this.logger.error('Failed to finalize interrupted agent execution', {
executionId: execution.id,
threadId: execution.threadId,

118
packages/cli/src/modules/agents/agent-message-queue.service.ts
View File

@ -1,9 +1,4 @@
import {
LockAcquisitionTimeoutError,
LockNamespace,
LockService,
Logger,
} from '@n8n/backend-common';
import { Logger } from '@n8n/backend-common';
import { OnPubSubEvent, OnShutdown } from '@n8n/decorators';
import { Service } from '@n8n/di';
import { createDeferredPromise, type IDeferredPromise } from '@n8n/utils/promise/deferred-promise';
@ -13,12 +8,13 @@ import { UserError } from 'n8n-workflow';
import { Publisher } from '@/scaling/pubsub/publisher.service';
import { AgentChatAttachmentService } from './agent-chat-attachment.service';
import { AgentExecutionService } from './agent-execution.service';
import {
agentConversationLockKey,
type AgentQueueInput,
type QueueExecutionContext,
} from './agent-message-queue.types';
AgentConversationLeaseService,
type AgentConversationOwner,
} from './agent-conversation-lease.service';
import { AgentConversationLeaseTimeoutError } from './agent-conversation-lease.types';
import { AgentExecutionService } from './agent-execution.service';
import { type AgentQueueInput, type QueueExecutionContext } from './agent-message-queue.types';
import { ChatIntegrationService } from './integrations/chat-integration.service';
import { N8NCheckpointStorage } from './integrations/n8n-checkpoint-storage';
import { AgentExecutionRepository } from './repositories/agent-execution.repository';
@ -36,7 +32,7 @@ type PreviewWaiter = {
export class AgentMessageQueueService {
static readonly LIVENESS_GRACE_MS = 2 * 60_000;
private readonly previews = new Map<string, PreviewWaiter>();
private readonly processing = new Set<string>();
private readonly processing = new Map<string, AgentConversationOwner>();
private readonly drains = new Map<string, Promise<void>>();
private readonly requested = new Set<string>();
private readonly stopping = new AbortController();
@ -45,7 +41,7 @@ export class AgentMessageQueueService {
constructor(
private readonly logger: Logger,
private readonly repository: AgentMessageQueueRepository,
private readonly lockService: LockService,
private readonly leases: AgentConversationLeaseService,
private readonly checkpoints: N8NCheckpointStorage,
private readonly integrations: ChatIntegrationService,
private readonly agents: AgentRepository,
@ -163,11 +159,15 @@ export class AgentMessageQueueService {
this.requested.add(threadId);
const existing = this.drains.get(threadId);
if (existing) return await existing;
const drain = this.drainThread(threadId).finally(() => {
this.drains.delete(threadId);
// An enqueue can arrive after the loop's last check but before this cleanup.
if (this.requested.has(threadId)) this.handleDrainRequest({ threadId });
});
const drain = this.drainThread(threadId)
.catch((error: unknown) => {
if (!this.stopping.signal.aborted) throw error;
})
.finally(() => {
this.drains.delete(threadId);
// An enqueue can arrive after the loop's last check but before this cleanup.
if (this.requested.has(threadId)) this.handleDrainRequest({ threadId });
});
this.drains.set(threadId, drain);
await drain;
}
@ -176,10 +176,13 @@ export class AgentMessageQueueService {
do {
this.requested.delete(threadId);
await this.settleRemovedPreviews(threadId);
const processed = await this.lockService.withLease(
LockNamespace.KNOWN_LOCKS,
agentConversationLockKey(threadId),
const next = await this.repository.findNext(threadId);
if (!next) break;
const processed = await this.leases.withLease(
next.agentId,
threadId,
async (signal) => await this.processNext(threadId, signal),
{ signal: this.stopping.signal },
);
if (processed) this.requested.add(threadId);
} while (this.requested.has(threadId) && !this.stopping.signal.aborted);
@ -189,8 +192,10 @@ export class AgentMessageQueueService {
if (this.stopping.signal.aborted || leaseSignal.aborted) return false;
if (await this.repository.hasProcessing(threadId)) return false;
if (await this.executionRepository.existsRunningByThread(threadId)) return false;
const entry = await this.repository.findNext(threadId);
let entry = await this.repository.findNext(threadId);
if (!entry) return false;
const owner = this.leases.requireOwner(threadId);
if (owner.lease.agentId !== entry.agentId) return true;
const payload = entry.payload;
const preview = this.previews.get(entry.id);
@ -214,7 +219,8 @@ export class AgentMessageQueueService {
id: entry.id,
threadId,
});
await this.repository.removeEntry(entry.id);
const id = entry.id;
await this.leases.write(owner, async (ctx) => await this.repository.removeEntry(id, ctx));
return true;
}
const bridge = this.integrations.getBridge(
@ -223,15 +229,25 @@ export class AgentMessageQueueService {
payload.credentialId,
);
if (!bridge) return false;
execute = async (context) => await bridge.processQueuedInput(payload, threadId, context);
execute = async (context) => {
if (entry?.payload.source === 'integration') {
await bridge.processQueuedInput(entry.payload, threadId, context);
}
};
}
if (
entry.kind === 'message' &&
(await this.checkpoints.findSuspendedForThread(entry.agentId, threadId))
)
return false;
if (!(await this.repository.markProcessing(entry.id))) return true;
this.processing.add(entry.id);
const selectedId = entry.id;
entry = await this.leases.write(owner, async (ctx) => {
if (!(await this.repository.markProcessing(selectedId, ctx))) return null;
return await this.repository.findById(selectedId, ctx);
});
if (!entry) return true;
const id = entry.id;
this.processing.set(id, owner);
const abortSignal = AbortSignal.any([
leaseSignal,
this.stopping.signal,
@ -241,15 +257,19 @@ export class AgentMessageQueueService {
abortSignal.throwIfAborted();
await execute({
abortSignal,
onExecutionStarted: async (id) => await this.repository.linkExecution(entry.id, id),
onExecutionStarted: async (executionId) =>
await this.leases.write(
owner,
async (ctx) => await this.repository.linkExecution(id, executionId, ctx),
),
});
preview?.done.resolve();
} catch (error) {
preview?.done.reject(ensureError(error));
this.logger.warn('Queued agent input ended with an error', { id: entry.id, threadId, error });
} finally {
this.processing.delete(entry.id);
await this.repository.removeEntry(entry.id);
this.processing.delete(id);
await this.leases.write(owner, async (ctx) => await this.repository.removeEntry(id, ctx));
this.notifyPeers(threadId);
}
return true;
@ -267,7 +287,11 @@ export class AgentMessageQueueService {
}
private async heartbeat(): Promise<void> {
const ids = [...new Set([...this.processing, ...this.previews.keys()])];
for (const [id, owner] of this.processing) {
if (owner.signal.aborted) continue;
await this.leases.write(owner, async (ctx) => await this.repository.touchProcessing(id, ctx));
}
const ids = [...new Set([...this.processing.keys(), ...this.previews.keys()])];
const present = new Set(await this.repository.touchLiveEntries(ids));
for (const id of ids) {
if (!present.has(id))
@ -276,51 +300,55 @@ export class AgentMessageQueueService {
}
async recover(): Promise<void> {
const staleBefore = new Date(Date.now() - AgentMessageQueueService.LIVENESS_GRACE_MS);
for (const threadId of await this.repository.findStaleThreads(staleBefore)) {
const graceMs = AgentMessageQueueService.LIVENESS_GRACE_MS;
for (const threadId of await this.repository.findStaleThreads(graceMs)) {
try {
await this.lockService.withLease(
LockNamespace.KNOWN_LOCKS,
agentConversationLockKey(threadId),
const [candidate] = await this.repository.findStale(threadId, graceMs);
if (!candidate) continue;
await this.leases.withLease(
candidate.agentId,
threadId,
async (leaseSignal) => {
for (const entry of await this.repository.findStale(threadId, staleBefore)) {
const owner = this.leases.requireOwner(threadId);
for (const entry of await this.repository.findStale(threadId, graceMs)) {
if (leaseSignal.aborted) return;
if (entry.executionId) {
const execution = await this.executionRepository.findRunningById(entry.executionId);
if (execution && execution.updatedAt > staleBefore) continue;
if (execution) {
if (leaseSignal.aborted) return;
if (
!(await this.executionService.finalizeInterruptedExecution(
execution,
staleBefore,
))
!(await this.executionService.finalizeInterruptedExecution(execution, graceMs))
)
continue;
}
}
if (
!(await this.leases.write(
owner,
async (ctx) => await this.repository.removeStale(entry.id, graceMs, ctx),
))
)
continue;
if (
entry.status === 'queued' &&
entry.payload.source === 'preview' &&
entry.payload.kind === 'message'
) {
if (leaseSignal.aborted) return;
await this.attachments.deleteByIds(
entry.payload.attachments?.map(({ id }) => id) ?? [],
);
}
if (leaseSignal.aborted) return;
await this.repository.removeEntry(entry.id);
this.logger.info('Removed interrupted agent queue entry without replay', {
id: entry.id,
threadId,
});
}
},
{ waitTimeoutMs: 250 },
{ waitTimeoutMs: 250, signal: this.stopping.signal },
);
} catch (error) {
if (!(error instanceof LockAcquisitionTimeoutError))
if (!(error instanceof AgentConversationLeaseTimeoutError))
this.logger.warn('Agent queue recovery failed', { threadId, error });
}
}

2
packages/cli/src/modules/agents/agent-message-queue.types.ts
View File

@ -2,8 +2,6 @@ import type { ActionEvent, SerializedMessage, SerializedThread } from 'chat';
import type { StoredAttachmentRef } from './agent-chat-attachment.service';
export const agentConversationLockKey = (threadId: string) => `agent-conversation:${threadId}`;
export interface QueueExecutionContext {
abortSignal: AbortSignal;
onExecutionStarted: (executionId: string) => Promise<void>;

19
packages/cli/src/modules/agents/agent-runtime-reconstruction.service.ts
View File

@ -102,6 +102,7 @@ export interface SubAgentDelegationConfig {
}
export interface ReconstructAgentRuntimeParams {
requireConversationOwnership?: boolean;
config: AgentJsonConfig;
memoryOwnerAgentId: string;
projectId: string;
@ -239,7 +240,9 @@ export class AgentRuntimeReconstructionService {
supportsHitl,
previewChat,
allowBackgroundTasks = true,
requireConversationOwnership = true,
}: {
requireConversationOwnership?: boolean;
/** Pass false when the caller cannot resume a suspended run (workflow executions). */
supportsHitl?: boolean;
/** Set by the in-app preview chat only — see `BuildFromJsonOptions.previewChat`. */
@ -293,6 +296,7 @@ export class AgentRuntimeReconstructionService {
toolCodeByName: toolsByName,
skills: agentEntity.skills ?? {},
runtimeProfile: 'top-level',
requireConversationOwnership,
supportsHitl,
runType,
workflowToolExecutionMode,
@ -484,6 +488,7 @@ export class AgentRuntimeReconstructionService {
}
private async reconstructRuntime(options: {
requireConversationOwnership?: boolean;
config: AgentJsonConfig;
memoryOwnerAgentId: string;
projectId: string;
@ -539,6 +544,7 @@ export class AgentRuntimeReconstructionService {
parentWorkspace,
allowBackgroundTasks = true,
previewChat,
requireConversationOwnership = true,
} = options;
const unavailable = [...(options.unavailableTools ?? [])];
const backgroundTasksEnabled =
@ -626,7 +632,7 @@ export class AgentRuntimeReconstructionService {
return resolved;
},
skills,
memoryFactory: this.getMemoryFactory(memoryOwnerAgentId),
memoryFactory: this.getMemoryFactory(memoryOwnerAgentId, requireConversationOwnership),
buildMcpClient,
resolveManagedEmbeddingProviderOptions: async () =>
await this.resolveManagedEmbeddingProviderOptions(projectId),
@ -647,6 +653,7 @@ export class AgentRuntimeReconstructionService {
}
await this.injectRuntimeDependencies({
requireConversationOwnership,
agent: reconstructed,
agentId: memoryOwnerAgentId,
projectId,
@ -705,8 +712,9 @@ export class AgentRuntimeReconstructionService {
return { sourcesById, availableSubAgents };
}
private getMemoryFactory(agentId: string): MemoryFactory {
return (_params: AgentJsonMemoryConfig) => this.n8nMemory.getImplementation(agentId);
private getMemoryFactory(agentId: string, requireOwnership: boolean): MemoryFactory {
return (_params: AgentJsonMemoryConfig) =>
this.n8nMemory.getImplementation(agentId, requireOwnership);
}
/**
@ -824,6 +832,7 @@ export class AgentRuntimeReconstructionService {
}
private async injectRuntimeDependencies(params: {
requireConversationOwnership: boolean;
agent: RuntimeAgent;
agentId: string;
projectId: string;
@ -1019,7 +1028,9 @@ export class AgentRuntimeReconstructionService {
// Inline agents get no checkpoint storage: `agent_checkpoints.agentId`
// is an FK to `agents`, and a synthetic inline id has no entity row.
if (runtimeProfile !== 'inline' && !agent.hasCheckpointStorage()) {
agent.checkpoint(this.n8nCheckpointStorage.getStorage(agentId));
agent.checkpoint(
this.n8nCheckpointStorage.getStorage(agentId, params.requireConversationOwnership),
);
}
// Attachment lookups are agent-scoped, so a synthetic inline id would

4
packages/cli/src/modules/agents/agent-test-chat.service.ts
View File

@ -48,14 +48,14 @@ export class AgentTestChatService {
*/
async clearTestChatMessages(agentId: string, userId: string) {
const threadId = chatThreadId(agentId, userId);
await this.n8nMemory.getImplementation(agentId).deleteThread(threadId);
await this.n8nMemory.getImplementation(agentId, false).deleteThread(threadId);
await this.agentChatAttachmentService.deleteByThread(threadId, { agentId });
}
/** Delete all test-chat messages + the thread row — used when the agent itself is deleted. */
async clearAllTestChatMessages(agentId: string) {
const threadId = chatThreadId(agentId);
const memory = this.n8nMemory.getImplementation(agentId);
const memory = this.n8nMemory.getImplementation(agentId, false);
await memory.deleteThreadsByPrefix(threadId);
await memory.deleteMessagesByThread(threadId);
await memory.deleteThread(threadId);

167
packages/cli/src/modules/agents/agent-workflow-execution.service.ts
View File

@ -1,3 +1,5 @@
import { AgentConversationLeaseService } from './agent-conversation-lease.service';
import { AgentConversationLeaseLostError } from './agent-conversation-lease.types';
import type { Agent as RuntimeAgent, BuiltAgent, BuiltTool, CredentialProvider } from '@n8n/agents';
import type { AgentJsonConfig, AgentSkill } from '@n8n/api-types';
import {
@ -101,6 +103,7 @@ export class AgentWorkflowExecutionService {
private readonly executionLevelTracer: ExecutionLevelTracer,
private readonly nodeToolAiGatewayService: NodeToolAiGatewayService,
private readonly aiConfig: AiConfig,
private readonly leases: AgentConversationLeaseService,
) {}
private normalizeWorkflowStreamError(error: unknown, outputSchema?: JSONSchema7): Error {
@ -232,6 +235,7 @@ export class AgentWorkflowExecutionService {
toolCodeByName: {},
skills,
runtimeProfile: 'inline',
requireConversationOwnership: false,
runType,
});
return this.applyPerCallAgentExtras(reconstructed, outputSchema, extraTools);
@ -258,6 +262,7 @@ export class AgentWorkflowExecutionService {
/** Stream one workflow-invoked agent run and collect its outcome. */
private async streamWorkflowAgent(params: {
abortSignal?: AbortSignal;
agentInstance: BuiltAgent;
message: string;
threadId: string;
@ -278,6 +283,7 @@ export class AgentWorkflowExecutionService {
}): Promise<WorkflowAgentRunOutcome> {
const {
agentInstance,
abortSignal,
message,
threadId,
telemetryAgentId,
@ -337,6 +343,7 @@ export class AgentWorkflowExecutionService {
});
const resultStream = await agentInstance.stream(message, {
abortSignal,
// The memory store scopes message reads by `resourceId` (the
// "per-user scope"; chat integrations pass the chat user id there).
// Workflow runs have no user, so key the scope by the thread
@ -365,6 +372,8 @@ export class AgentWorkflowExecutionService {
startedAt,
);
} catch (error) {
if (error instanceof AgentConversationLeaseLostError || abortSignal?.aborted)
throw error;
this.logger.warn('Failed to start agent execution recording from workflow', {
agentId: recordingParams.agentId,
threadId,
@ -392,6 +401,7 @@ export class AgentWorkflowExecutionService {
}
}
} catch (error) {
if (error instanceof AgentConversationLeaseLostError || abortSignal?.aborted) throw error;
const normalizedError = this.normalizeWorkflowStreamError(error, outputSchema);
recorder.record({ type: 'error', error: normalizedError });
recorder.record({ type: 'finish', finishReason: 'error' });
@ -406,6 +416,7 @@ export class AgentWorkflowExecutionService {
await run();
}
abortSignal?.throwIfAborted();
if (streamError && recordingParams && !agentExecutionId) {
try {
agentExecutionId = await this.agentExecutionService.startExecutionRecording(
@ -540,100 +551,104 @@ export class AgentWorkflowExecutionService {
throw new OperationalError('Agent not found or not accessible.');
}
const credentialProvider = createAgentCredentialProvider(this.credentialsService, projectId);
return await this.leases.withLease(agentId, threadId, async (abortSignal) => {
const credentialProvider = createAgentCredentialProvider(this.credentialsService, projectId);
let agentData: Agent = agentEntity;
let agentData: Agent = agentEntity;
if (!useDraftVersion) {
agentData = getPublishedAgentSnapshot(agentEntity);
}
const telemetryConfiguration = buildAgentConfigurationTelemetry(agentData);
const runType: AgentRunTelemetryType = useDraftVersion ? 'test' : 'production';
if (!useDraftVersion) {
agentData = getPublishedAgentSnapshot(agentEntity);
}
const telemetryConfiguration = buildAgentConfigurationTelemetry(agentData);
const runType: AgentRunTelemetryType = useDraftVersion ? 'test' : 'production';
const extraTools = this.buildWorkflowExtraTools(workflowContext);
const compiled = await this.compileIsolated(
agentData,
credentialProvider,
runType,
outputSchema,
extraTools?.length ? extraTools : undefined,
sandboxScope?.principalHash,
);
if (!compiled.ok || !compiled.agent) {
throw new OperationalError(`Failed to compile agent: ${compiled.error ?? 'unknown error'}`);
}
const extraTools = this.buildWorkflowExtraTools(workflowContext);
const compiled = await this.compileIsolated(
agentData,
credentialProvider,
runType,
outputSchema,
extraTools?.length ? extraTools : undefined,
sandboxScope?.principalHash,
);
if (!compiled.ok || !compiled.agent) {
throw new OperationalError(`Failed to compile agent: ${compiled.error ?? 'unknown error'}`);
}
const agentInstance = compiled.agent;
const run = await this.streamWorkflowAgent({
agentInstance,
message,
threadId,
telemetryAgentId: agentId,
telemetryUserId,
runType,
outputSchema,
tracing: {
projectId,
executionId,
workflowId: workflowContext?.workflowId,
nodeId: workflowContext?.callingNodeId,
nodeName: workflowContext?.callingNodeName,
},
recordingParams: {
const agentInstance = compiled.agent;
const run = await this.streamWorkflowAgent({
agentInstance,
abortSignal,
message,
threadId,
agentId,
agentName: agentInstance.name,
projectId,
userMessage: message,
source: AGENT_WORKFLOW_TRIGGER_TYPE,
telemetry: {
userId: telemetryUserId,
runType,
configuration: telemetryConfiguration,
telemetryAgentId: agentId,
telemetryUserId,
runType,
outputSchema,
tracing: {
projectId,
executionId,
workflowId: workflowContext?.workflowId,
nodeId: workflowContext?.callingNodeId,
nodeName: workflowContext?.callingNodeName,
},
},
streamObserver,
...(sandboxScope
? {
sandboxScope: {
projectId,
principalHash: sandboxScope.principalHash,
},
}
: {}),
});
if (run.agentExecutionId) {
try {
await this.agentExecutionService.finalizeExecution(run.agentExecutionId, {
recordingParams: {
threadId,
agentId,
agentName: agentInstance.name,
projectId,
userMessage: message,
record: run.messageRecord,
source: AGENT_WORKFLOW_TRIGGER_TYPE,
telemetry: {
userId: telemetryUserId,
runType,
configuration: telemetryConfiguration,
},
});
} catch (error) {
this.logger.warn('Failed to record agent execution from workflow', {
agentId,
threadId,
error: error instanceof Error ? error.message : String(error),
});
}
}
},
streamObserver,
...(sandboxScope
? {
sandboxScope: {
projectId,
principalHash: sandboxScope.principalHash,
},
}
: {}),
});
return this.buildWorkflowResult({
run,
// sessionId here is still the scoped thread key — executeAgent remaps it
// to the caller-facing id; this method never sees the unscoped one.
session: { agentId, projectId, sessionId: threadId, threadId },
outputSchema,
if (run.agentExecutionId) {
try {
await this.agentExecutionService.finalizeExecution(run.agentExecutionId, {
threadId,
agentId,
agentName: agentInstance.name,
projectId,
userMessage: message,
record: run.messageRecord,
source: AGENT_WORKFLOW_TRIGGER_TYPE,
telemetry: {
userId: telemetryUserId,
runType,
configuration: telemetryConfiguration,
},
});
} catch (error) {
if (error instanceof AgentConversationLeaseLostError || abortSignal.aborted) throw error;
this.logger.warn('Failed to record agent execution from workflow', {
agentId,
threadId,
error: error instanceof Error ? error.message : String(error),
});
}
}
return this.buildWorkflowResult({
run,
// sessionId here is still the scoped thread key — executeAgent remaps it
// to the caller-facing id; this method never sees the unscoped one.
session: { agentId, projectId, sessionId: threadId, threadId },
outputSchema,
});
});
}

26
packages/cli/src/modules/agents/agent-workflow-tool-resume.service.ts
View File

@ -1,5 +1,5 @@
import { N8N_CHAT_INTEGRATION_TYPE } from '@n8n/api-types';
import { LockNamespace, LockService, Logger } from '@n8n/backend-common';
import { Logger } from '@n8n/backend-common';
import { UserRepository } from '@n8n/db';
import { OnLifecycleEvent, OnPubSubEvent, type WorkflowExecuteAfterContext } from '@n8n/decorators';
import { Service } from '@n8n/di';
@ -12,7 +12,7 @@ import { Publisher } from '@/scaling/pubsub/publisher.service';
import { AgentExecutionUpdateBroadcaster } from './agent-execution-update-broadcaster';
import { AgentTestRunService } from './agent-test-run.service';
import { AgentMessageQueueService } from './agent-message-queue.service';
import { agentConversationLockKey } from './agent-message-queue.types';
import { AgentConversationLeaseService } from './agent-conversation-lease.service';
import {
AgentBackgroundJobService,
collectResultData,
@ -42,7 +42,7 @@ export class AgentWorkflowToolResumeService {
private readonly instanceSettings: InstanceSettings,
private readonly publisher: Publisher,
private readonly backgroundJobService: AgentBackgroundJobService,
private readonly lockService: LockService,
private readonly leases: AgentConversationLeaseService,
private readonly messageQueue: AgentMessageQueueService,
) {
this.logger = this.logger.scoped('agents');
@ -142,20 +142,12 @@ export class AgentWorkflowToolResumeService {
/** The tool handler re-reads the execution, so this payload only says why it woke. */
async resume(agentRun: RelatedAgentRun, status: string): Promise<void> {
try {
await this.lockService.withLease(
LockNamespace.KNOWN_LOCKS,
agentConversationLockKey(agentRun.threadId),
async (signal) => {
const checkpoint = await this.checkpointStorage.getStatus(
agentRun.runId,
agentRun.agentId,
);
if (checkpoint.status !== 'active' || checkpoint.checkpoint.status !== 'suspended')
return;
signal.throwIfAborted();
await this.resumeInsideLease(agentRun, status, signal);
},
);
await this.leases.withLease(agentRun.agentId, agentRun.threadId, async (signal) => {
const checkpoint = await this.checkpointStorage.getStatus(agentRun.runId, agentRun.agentId);
if (checkpoint.status !== 'active' || checkpoint.checkpoint.status !== 'suspended') return;
signal.throwIfAborted();
await this.resumeInsideLease(agentRun, status, signal);
});
} finally {
this.messageQueue.notify(agentRun.threadId);
}

4
packages/cli/src/modules/agents/agents.module.ts
View File

@ -233,6 +233,9 @@ export class AgentsModule implements ModuleInterface {
const { AgentExecution } = await import('./entities/agent-execution.entity.js');
const { AgentBackgroundJob } = await import('./entities/agent-background-job.entity.js');
const { AgentMessageQueue } = await import('./entities/agent-message-queue.entity.js');
const { AgentConversationLease } = await import(
'./entities/agent-conversation-lease.entity.js'
);
const { AgentHistory } = await import('./entities/agent-history.entity.js');
const { AgentCredentialDependency } = await import(
'./entities/agent-credential-dependency.entity.js'
@ -275,6 +278,7 @@ export class AgentsModule implements ModuleInterface {
AgentExecution,
AgentBackgroundJob,
AgentMessageQueue,
AgentConversationLease,
AgentHistory,
AgentCredentialDependency,
AgentWorkflowDependency,

48
packages/cli/src/modules/agents/background/__tests__/agent-wake.service.test.ts
View File

@ -1,4 +1,6 @@
import { LockAcquisitionTimeoutError, type LockService, type Logger } from '@n8n/backend-common';
import { mockConversationLeases } from '../../__tests__/mock-conversation-lease';
import type { Logger } from '@n8n/backend-common';
import { AgentConversationLeaseTimeoutError } from '../../agent-conversation-lease.types';
import type { AgentsConfig } from '@n8n/config';
import type { UserRepository } from '@n8n/db';
import { createDeferredPromise } from '@n8n/utils/promise/deferred-promise';
@ -71,7 +73,7 @@ function setup(options: { worker?: boolean; enabled?: boolean } = {}) {
const checkpointStorage = mock<N8NCheckpointStorage>();
const integrationRegistry = mock<ChatIntegrationRegistry>();
const orchestrator = mock<AgentExecutionOrchestratorService>();
const lockService = mock<LockService>();
const leases = mockConversationLeases();
const publisher = mock<Publisher>();
const instanceSettings = mock<InstanceSettings>({ isWorker: options.worker ?? false });
const agentsConfig = mock<AgentsConfig>({ backgroundTasksEnabled: options.enabled ?? true });
@ -86,7 +88,7 @@ function setup(options: { worker?: boolean; enabled?: boolean } = {}) {
agentRepository.findById.mockResolvedValue({ id: 'agent-1', projectId: 'project-1' } as never);
userRepository.findByIdWithRole.mockResolvedValue(user as never);
integrationRegistry.get.mockReturnValue({} as never);
lockService.withLease.mockImplementation(async (_namespace, _key, callback) => {
leases.withLease.mockImplementation(async (_namespace, _key, callback) => {
return await callback(new AbortController().signal);
});
@ -98,7 +100,7 @@ function setup(options: { worker?: boolean; enabled?: boolean } = {}) {
checkpointStorage,
integrationRegistry,
orchestrator,
lockService,
leases,
publisher,
instanceSettings,
agentsConfig,
@ -116,7 +118,7 @@ function setup(options: { worker?: boolean; enabled?: boolean } = {}) {
userRepository,
checkpointStorage,
orchestrator,
lockService,
leases,
publisher,
integrationRegistry,
logger,
@ -158,22 +160,22 @@ describe('AgentWakeService', () => {
it('debounces wakes for one thread', async () => {
vi.useFakeTimers();
try {
const { service, lockService } = setup();
const { service, leases } = setup();
await service.requestWake('thread-1');
await service.requestWake('thread-1');
expect(lockService.withLease).not.toHaveBeenCalled();
expect(leases.withLease).not.toHaveBeenCalled();
await vi.advanceTimersByTimeAsync(WAKE_DEBOUNCE_MS);
expect(lockService.withLease).toHaveBeenCalledTimes(1);
expect(leases.withLease).toHaveBeenCalledTimes(1);
} finally {
vi.useRealTimers();
}
});
it('routes worker wakes through pubsub', async () => {
const { service, publisher, lockService } = setup({ worker: true });
const { service, publisher, leases } = setup({ worker: true });
await service.requestWake('thread-1');
@ -181,20 +183,20 @@ describe('AgentWakeService', () => {
command: 'wake-agent-background-job',
payload: { threadId: 'thread-1' },
});
expect(lockService.withLease).not.toHaveBeenCalled();
expect(leases.withLease).not.toHaveBeenCalled();
});
it('schedules a wake when a main instance receives the pubsub request', async () => {
vi.useFakeTimers();
try {
const { service, lockService } = setup();
const { service, leases } = setup();
service.handleWakeRelay({ threadId: 'thread-1' });
await vi.advanceTimersByTimeAsync(WAKE_DEBOUNCE_MS);
expect(lockService.withLease).toHaveBeenCalledWith(
expect(leases.withLease).toHaveBeenCalledWith(
expect.anything(),
'agent-conversation:thread-1',
'thread-1',
expect.any(Function),
expect.anything(),
);
@ -206,27 +208,27 @@ describe('AgentWakeService', () => {
it('schedules a wake for each thread with pending results', async () => {
vi.useFakeTimers();
try {
const { service, jobRepository, lockService } = setup();
const { service, jobRepository, leases } = setup();
jobRepository.findThreadsWithUnconsumedMail.mockResolvedValue(['thread-1', 'thread-2']);
await service.drainUnconsumed();
await vi.advanceTimersByTimeAsync(WAKE_DEBOUNCE_MS);
expect(lockService.withLease).toHaveBeenCalledTimes(2);
expect(leases.withLease).toHaveBeenCalledTimes(2);
} finally {
vi.useRealTimers();
}
});
it('does nothing while background tasks are disabled', async () => {
const { service, publisher, lockService, jobRepository } = setup({ enabled: false });
const { service, publisher, leases, jobRepository } = setup({ enabled: false });
await service.requestWake('thread-1');
await service.drainUnconsumed();
await service.attemptWake('thread-1');
expect(publisher.publishCommand).not.toHaveBeenCalled();
expect(lockService.withLease).not.toHaveBeenCalled();
expect(leases.withLease).not.toHaveBeenCalled();
expect(jobRepository.findThreadsWithUnconsumedMail).not.toHaveBeenCalled();
});
@ -363,7 +365,7 @@ describe('AgentWakeService', () => {
it('delivers results for one author and schedules another wake for the remaining authors', async () => {
vi.useFakeTimers();
try {
const { service, orchestrator, jobRepository, lockService } = setup();
const { service, orchestrator, jobRepository, leases } = setup();
const otherJob = makeJob({
id: 'job-2',
parentResourceId: `draft-chat:${otherUser.id}`,
@ -382,7 +384,7 @@ describe('AgentWakeService', () => {
expect(jobRepository.markMailConsumed).toHaveBeenCalledWith('thread-1', ['job-1']);
await vi.advanceTimersByTimeAsync(WAKE_DEBOUNCE_MS);
expect(lockService.withLease).toHaveBeenCalledTimes(2);
expect(leases.withLease).toHaveBeenCalledTimes(2);
} finally {
vi.useRealTimers();
}
@ -413,9 +415,9 @@ describe('AgentWakeService', () => {
});
it('does not mark results as delivered after the wake loses its lease', async () => {
const { service, lockService, orchestrator, jobRepository } = setup();
const { service, leases, orchestrator, jobRepository } = setup();
const controller = new AbortController();
lockService.withLease.mockImplementation(async (_namespace, _key, callback) => {
leases.withLease.mockImplementation(async (_namespace, _key, callback) => {
return await callback(controller.signal);
});
orchestrator.executeForWake.mockImplementation(async () => {
@ -428,8 +430,8 @@ describe('AgentWakeService', () => {
});
it('leaves results pending when the wake cannot acquire the lease', async () => {
const { service, lockService, orchestrator, jobRepository, messageQueue, logger } = setup();
lockService.withLease.mockRejectedValue(new LockAcquisitionTimeoutError('lock unavailable'));
const { service, leases, orchestrator, jobRepository, messageQueue, logger } = setup();
leases.withLease.mockRejectedValue(new AgentConversationLeaseTimeoutError('lock unavailable'));
await service.attemptWake('thread-1');

8
packages/cli/src/modules/agents/background/__tests__/sub-agent-background-runner.test.ts
View File

@ -1,3 +1,4 @@
import { mockConversationLeases } from '../../__tests__/mock-conversation-lease';
import type { CredentialProvider } from '@n8n/agents';
import type { Logger } from '@n8n/backend-common';
import type { Mock } from 'vitest';
@ -49,7 +50,12 @@ function setup() {
jobService.settle.mockResolvedValue(true);
runner.run.mockResolvedValue(completedRunResult());
const backgroundRunner = new SubAgentBackgroundRunner(runner, jobService, logger);
const backgroundRunner = new SubAgentBackgroundRunner(
runner,
jobService,
logger,
mockConversationLeases(),
);
const context = {
projectId: 'project-1',
parentAgentId: 'agent-1',

25
packages/cli/src/modules/agents/background/agent-wake.service.ts
View File

@ -1,9 +1,4 @@
import {
LockAcquisitionTimeoutError,
LockNamespace,
LockService,
Logger,
} from '@n8n/backend-common';
import { Logger } from '@n8n/backend-common';
import { AgentsConfig } from '@n8n/config';
import { UserRepository } from '@n8n/db';
import { OnPubSubEvent } from '@n8n/decorators';
@ -40,10 +35,10 @@ export const WAKE_DEBOUNCE_MS = 5_000;
export const MAX_CONSECUTIVE_FAILED_WAKES = 3;
import { AgentMessageQueueService } from '../agent-message-queue.service';
import { agentConversationLockKey } from '../agent-message-queue.types';
import { AgentConversationLeaseService } from '../agent-conversation-lease.service';
import { AgentConversationLeaseTimeoutError } from '../agent-conversation-lease.types';
const WAKE_LOCK_WAIT_MS = 250;
const WAKE_LOCK_TTL_MS = 30_000;
const HINT_TITLE_MAX_CHARS = 80;
type FailureState = { generation: string; count: number };
@ -65,7 +60,7 @@ export class AgentWakeService {
private readonly checkpointStorage: N8NCheckpointStorage,
private readonly integrationRegistry: ChatIntegrationRegistry,
private readonly orchestrator: AgentExecutionOrchestratorService,
private readonly lockService: LockService,
private readonly leases: AgentConversationLeaseService,
private readonly publisher: Publisher,
private readonly instanceSettings: InstanceSettings,
private readonly agentsConfig: AgentsConfig,
@ -148,9 +143,11 @@ export class AgentWakeService {
if (!this.agentsConfig.backgroundTasksEnabled) return;
try {
await this.lockService.withLease(
LockNamespace.KNOWN_LOCKS,
agentConversationLockKey(threadId),
const [first] = await this.jobRepository.findWakeableUnconsumedSettled(threadId);
if (!first) return;
await this.leases.withLease(
first.parentAgentId,
threadId,
async (signal) => {
try {
await this.deliverInsideLease(threadId, signal);
@ -158,10 +155,10 @@ export class AgentWakeService {
this.messageQueue.notify(threadId);
}
},
{ waitTimeoutMs: WAKE_LOCK_WAIT_MS, leaseTtlMs: WAKE_LOCK_TTL_MS },
{ waitTimeoutMs: WAKE_LOCK_WAIT_MS },
);
} catch (error) {
if (error instanceof LockAcquisitionTimeoutError) {
if (error instanceof AgentConversationLeaseTimeoutError) {
this.logger.debug('Skipped background job wake because the conversation lease is busy', {
threadId,
});

148
packages/cli/src/modules/agents/background/sub-agent-background-runner.ts
View File

@ -10,6 +10,7 @@ import {
type BackgroundJobReceipt,
} from './agent-background-job.service';
import type { AgentBackgroundJobSettlement } from '../repositories/agent-background-job.repository';
import { AgentConversationLeaseService } from '../agent-conversation-lease.service';
import { formatSubAgentToolOutput } from '../sub-agents/format-sub-agent-tool-output';
import {
SubAgentRunner,
@ -46,6 +47,7 @@ export class SubAgentBackgroundRunner {
private readonly runner: SubAgentRunner,
private readonly jobService: AgentBackgroundJobService,
private readonly logger: Logger,
private readonly leases: AgentConversationLeaseService,
) {
this.logger = this.logger.scoped('agents');
}
@ -84,81 +86,83 @@ export class SubAgentBackgroundRunner {
});
if (receipt.status !== 'started') return receipt;
// The job runs on its own abort scope: the parent's signal dies with the
// chat connection, and the parent's live telemetry does not outlive its
// turn — neither is forwarded.
const abortController = new AbortController();
this.jobService.registerAbortController(jobId, abortController);
const timeout = setTimeout(() => {
// Settle first so the timeout is recorded as the reason; the aborted
// run's own settle then loses to this one.
void this.jobService
.settle(jobId, {
status: 'failed',
error: `Timed out after ${Math.round(SUB_AGENT_BACKGROUND_TIMEOUT_MS / 60_000)} minutes`,
this.leases.outsideConversation(() => {
// The job runs on its own abort scope: the parent's signal dies with the
// chat connection, and the parent's live telemetry does not outlive its
// turn — neither is forwarded.
const abortController = new AbortController();
this.jobService.registerAbortController(jobId, abortController);
const timeout = setTimeout(() => {
// Settle first so the timeout is recorded as the reason; the aborted
// run's own settle then loses to this one.
void this.jobService
.settle(jobId, {
status: 'failed',
error: `Timed out after ${Math.round(SUB_AGENT_BACKGROUND_TIMEOUT_MS / 60_000)} minutes`,
})
.catch((error: unknown) => {
// A rejection escaping this detached chain would surface as an
// unhandled rejection; the sweeper reconciles the row later.
this.logger.error('Failed to settle timed-out background job', { jobId, error });
})
.finally(() => abortController.abort());
}, SUB_AGENT_BACKGROUND_TIMEOUT_MS);
timeout.unref();
void this.runner
.run(
{
goal: request.goal,
source: request.source,
...(request.context !== undefined ? { context: request.context } : {}),
...(request.expectedOutput !== undefined
? { expectedOutput: request.expectedOutput }
: {}),
parentThreadId: request.parentThreadId,
parentResourceId: request.parentResourceId,
parentSandboxPrincipalHash: request.parentSandboxPrincipalHash,
childThreadId,
taskPath,
},
{
projectId: context.projectId,
parentAgentId: context.parentAgentId,
credentialProvider: context.credentialProvider,
runType: context.runType,
workflowToolExecutionMode: context.workflowToolExecutionMode,
user: context.user,
instrumentation: context.instrumentation,
abortSignal: abortController.signal,
...(request.difficulty !== undefined
? { selfDelegationDifficulty: request.difficulty }
: {}),
...(context.parentWorkspaceHandle !== undefined
? { parentWorkspaceHandle: context.parentWorkspaceHandle }
: {}),
},
)
.then((result) => settlementFor(result))
.catch(
(error: unknown): AgentBackgroundJobSettlement => ({
status: 'failed',
error: error instanceof Error ? error.message : String(error),
}),
)
.then(async (settlement) => {
const settled = await this.jobService.settle(jobId, settlement);
if (settled && settlement.status === 'failed') {
this.logger.warn('Background sub-agent job failed', { jobId, error: settlement.error });
}
})
.catch((error: unknown) => {
// A rejection escaping this detached chain would surface as an
// unhandled rejection; the sweeper reconciles the row later.
this.logger.error('Failed to settle timed-out background job', { jobId, error });
// Only the settle write itself can land here: don't overwrite the
// outcome (a completed answer must not become 'failed' over a DB
// blip) and don't let the rejection escape the detached chain —
// the sweeper reconciles the still-running row later.
this.logger.error('Failed to settle background sub-agent job', { jobId, error });
})
.finally(() => abortController.abort());
}, SUB_AGENT_BACKGROUND_TIMEOUT_MS);
timeout.unref();
void this.runner
.run(
{
goal: request.goal,
source: request.source,
...(request.context !== undefined ? { context: request.context } : {}),
...(request.expectedOutput !== undefined
? { expectedOutput: request.expectedOutput }
: {}),
parentThreadId: request.parentThreadId,
parentResourceId: request.parentResourceId,
parentSandboxPrincipalHash: request.parentSandboxPrincipalHash,
childThreadId,
taskPath,
},
{
projectId: context.projectId,
parentAgentId: context.parentAgentId,
credentialProvider: context.credentialProvider,
runType: context.runType,
workflowToolExecutionMode: context.workflowToolExecutionMode,
user: context.user,
instrumentation: context.instrumentation,
abortSignal: abortController.signal,
...(request.difficulty !== undefined
? { selfDelegationDifficulty: request.difficulty }
: {}),
...(context.parentWorkspaceHandle !== undefined
? { parentWorkspaceHandle: context.parentWorkspaceHandle }
: {}),
},
)
.then((result) => settlementFor(result))
.catch(
(error: unknown): AgentBackgroundJobSettlement => ({
status: 'failed',
error: error instanceof Error ? error.message : String(error),
}),
)
.then(async (settlement) => {
const settled = await this.jobService.settle(jobId, settlement);
if (settled && settlement.status === 'failed') {
this.logger.warn('Background sub-agent job failed', { jobId, error: settlement.error });
}
})
.catch((error: unknown) => {
// Only the settle write itself can land here: don't overwrite the
// outcome (a completed answer must not become 'failed' over a DB
// blip) and don't let the rejection escape the detached chain —
// the sweeper reconciles the still-running row later.
this.logger.error('Failed to settle background sub-agent job', { jobId, error });
})
.finally(() => clearTimeout(timeout));
.finally(() => clearTimeout(timeout));
});
return receipt;
}

4
packages/cli/src/modules/agents/builder/agents-builder.service.ts
View File

@ -290,7 +290,7 @@ export class AgentsBuilderService {
};
const builderMemory = new Memory()
.storage(this.n8nMemory.getImplementation(agentId))
.storage(this.n8nMemory.getImplementation(agentId, false))
.observationalMemory({
observe: createObservationLogObserveFn(modelConfig, { onUsage: onMemoryUsage }),
reflect: createObservationLogReflectFn(modelConfig, { onUsage: onMemoryUsage }),
@ -301,7 +301,7 @@ export class AgentsBuilderService {
.instructions(finalInstructions)
.skills(runtimeSkills)
.memory(builderMemory)
.checkpoint(this.n8nCheckpointStorage.getStorage(agentId))
.checkpoint(this.n8nCheckpointStorage.getStorage(agentId, false))
.configuration({ maxIterations: 30 });
const promptCaching = resolveAIAPromptCaching(modelConfig);
if (promptCaching) {

33
packages/cli/src/modules/agents/entities/agent-conversation-lease.entity.ts Normal file
View File

@ -0,0 +1,33 @@
import { DateTimeColumn } from '@n8n/db';
import {
BaseEntity,
Column,
Entity,
Index,
JoinColumn,
ManyToOne,
PrimaryColumn,
type Relation,
} from '@n8n/typeorm';
import { Agent } from './agent.entity';
@Entity({ name: 'agent_conversation_lease' })
export class AgentConversationLease extends BaseEntity {
@PrimaryColumn({ type: 'varchar', length: 128 })
threadId: string;
@Index()
@Column({ type: 'varchar', length: 36 })
agentId: string;
@ManyToOne(() => Agent, { onDelete: 'CASCADE' })
@JoinColumn({ name: 'agentId' })
agent: Relation<Agent>;
@Column({ type: 'uuid' })
ownerToken: string;
@DateTimeColumn({ precision: 3 })
expiresAt: Date;
}

2
packages/cli/src/modules/agents/instance-ai-builder-delegate.adapter.ts
View File

@ -283,7 +283,7 @@ export class InstanceAiBuilderDelegateAdapterService {
});
for (const { id } of threads) {
// The target agent id is the suffix; memory impls are agent-scoped.
const memory = this.n8nMemory.getImplementation(id.slice(prefix.length));
const memory = this.n8nMemory.getImplementation(id.slice(prefix.length), false);
await memory.deleteMessagesByThread(id);
await memory.deleteThread(id);
}

17
packages/cli/src/modules/agents/integrations/__tests__/agent-chat-bridge.test.ts
View File

@ -1,7 +1,9 @@
import type { Mock } from 'vitest';
import type { SerializableAgentState, StreamChunk } from '@n8n/agents';
import { MAX_AGENT_CHAT_ATTACHMENT_FILENAME_LENGTH } from '@n8n/api-types';
import { LockAcquisitionTimeoutError, LockService } from '@n8n/backend-common';
import { LockService } from '@n8n/backend-common';
import { AgentConversationLeaseService } from '../../agent-conversation-lease.service';
import { mockConversationLeases } from '../../__tests__/mock-conversation-lease';
import type { HttpRequestClient } from '@n8n/backend-network';
import { Container } from '@n8n/di';
import type { Adapter, Author, Message, Thread } from 'chat';
@ -336,6 +338,7 @@ describe('AgentChatBridge — consumeStream', () => {
registry.register(new RestrictedTestIntegration());
registry.register(new SlackIntegration(mock<AgentRepository>()));
Container.set(ChatIntegrationRegistry, registry);
Container.set(AgentConversationLeaseService, mockConversationLeases());
});
afterEach(() => {
@ -1234,14 +1237,9 @@ describe('AgentChatBridge — consumeStream', () => {
sessionGenerationKey('agent-1:thread-1'),
{ generation: 0, lastActivityAt: Date.now() - 31 * 60_000 },
]);
const lockService = mock<LockService>();
lockService.withLease.mockImplementation(async (_namespace, key, callback) => {
if (key === 'agent-conversation:agent-1:thread-1') {
throw new LockAcquisitionTimeoutError('Conversation is busy');
}
return await callback(new AbortController().signal);
});
Container.set(LockService, lockService);
const leases = mockConversationLeases();
leases.isHeld.mockResolvedValue(true);
Container.set(AgentConversationLeaseService, leases);
const { bot, handlers } = makeBot();
const agentExecutor = makeAgentExecutor([finishChunk]);
const bridge = createBridge(
@ -3430,6 +3428,7 @@ describe('AgentChatBridge — Slack thread history', () => {
const registry = new ChatIntegrationRegistry();
registry.register(new SlackIntegration(mock<AgentRepository>()));
Container.set(ChatIntegrationRegistry, registry);
Container.set(AgentConversationLeaseService, mockConversationLeases());
});
afterEach(() => {

53
packages/cli/src/modules/agents/integrations/__tests__/n8n-checkpoint-storage.test.ts
View File

@ -1,3 +1,5 @@
import type { DataSource, EntityManager } from '@n8n/typeorm';
import { mockConversationLeases } from '../../__tests__/mock-conversation-lease';
import type { SerializableAgentState } from '@n8n/agents';
import { mockLogger } from '@n8n/backend-test-utils';
import type { AgentsConfig } from '@n8n/config';
@ -8,7 +10,7 @@ import {
hashAgentSandboxPrincipal,
} from '../../agent-sandbox-principal';
import type { AgentCheckpoint } from '../../entities/agent-checkpoint.entity';
import type { AgentCheckpointRepository } from '../../repositories/agent-checkpoint.repository';
import { AgentCheckpointRepository } from '../../repositories/agent-checkpoint.repository';
import {
CHECKPOINT_RECONCILIATION_OVERFLOW,
N8NCheckpointStorage,
@ -31,10 +33,22 @@ const principalHash = hashAgentSandboxPrincipal({ type: 'n8n-user', userId: 'use
function makeService() {
const repository = mock<AgentCheckpointRepository>();
const manager = mock<EntityManager>();
manager.findOneBy.mockImplementation(
async (_target, criteria) => await repository.findOneBy(criteria as never),
);
manager.create.mockImplementation((_target, values) => values as never);
manager.save.mockImplementation(
async (_target, values) => await repository.save(values as never),
);
const persistence = new AgentCheckpointRepository(mock<DataSource>({ manager }), mock());
repository.saveState.mockImplementation(persistence.saveState.bind(persistence));
const service = new N8NCheckpointStorage(
repository,
mockLogger(),
mock<AgentsConfig>({ checkpointTtlSeconds: 60 }),
mockConversationLeases(),
);
return { service, repository };
@ -43,24 +57,17 @@ function makeService() {
describe('N8NCheckpointStorage', () => {
it('creates new checkpoints with the storage agent as owner', async () => {
const { service, repository } = makeService();
const checkpoint = {
runId: 'run-1',
agentId: 'agent-1',
expired: false,
state: JSON.stringify(suspendedState),
} as AgentCheckpoint;
repository.findByRunId.mockResolvedValue(null);
repository.create.mockReturnValue(checkpoint);
repository.findOneBy.mockResolvedValue(null);
await service.getStorage('agent-1').save('run-1', suspendedState);
expect(repository.create).toHaveBeenCalledWith({
expect(repository.save).toHaveBeenCalledWith({
runId: 'run-1',
agentId: 'agent-1',
expired: false,
state: JSON.stringify(suspendedState),
});
expect(repository.save).toHaveBeenCalledWith(checkpoint);
});
it('updates a checkpoint only when the owner matches', async () => {
@ -71,23 +78,24 @@ describe('N8NCheckpointStorage', () => {
expired: true,
state: null,
} as AgentCheckpoint;
repository.findByRunId.mockResolvedValue(checkpoint);
repository.findOneBy.mockResolvedValue(checkpoint);
await service.getStorage('agent-1').save('run-1', suspendedState);
expect(checkpoint).toMatchObject({
agentId: 'agent-1',
expired: false,
state: JSON.stringify(suspendedState),
});
expect(repository.save).toHaveBeenCalledWith(checkpoint);
expect(repository.save).toHaveBeenCalledWith(
expect.objectContaining({
agentId: 'agent-1',
expired: false,
state: JSON.stringify(suspendedState),
}),
);
});
it.each(['agent-2', null])(
'rejects overwriting a checkpoint owned by %s',
async (existingAgentId) => {
const { service, repository } = makeService();
repository.findByRunId.mockResolvedValue({
repository.findOneBy.mockResolvedValue({
runId: 'run-1',
agentId: existingAgentId,
expired: false,
@ -127,6 +135,7 @@ describe('N8NCheckpointStorage', () => {
'agent-1',
JSON.stringify(suspendedState),
JSON.stringify({ ...suspendedState, status: 'running' }),
{},
);
await expect(storage.claimForResume?.('run-1', suspendedState)).resolves.toBe(false);
});
@ -141,6 +150,7 @@ describe('N8NCheckpointStorage', () => {
'run-1',
'agent-1',
JSON.stringify(suspendedState),
{},
);
});
@ -177,9 +187,12 @@ describe('N8NCheckpointStorage', () => {
it('expires only a checkpoint owned by the storage agent', async () => {
const { service, repository } = makeService();
repository.findByRunIdAndAgentId.mockResolvedValue(
mock<AgentCheckpoint>({ state: JSON.stringify(suspendedState) }),
);
await service.getStorage('agent-1').delete('run-1');
expect(repository.expireByRunIdAndAgentId).toHaveBeenCalledWith('run-1', 'agent-1');
expect(repository.expireByRunIdAndAgentId).toHaveBeenCalledWith('run-1', 'agent-1', {});
});
it('returns every persisted active run for the principal workspace', async () => {

67
packages/cli/src/modules/agents/integrations/__tests__/n8n-memory.test.ts
View File

@ -1,3 +1,5 @@
import type { DataSource, EntityManager } from '@n8n/typeorm';
import { mockConversationLeases } from '../../__tests__/mock-conversation-lease';
import { hashEpisodicMemoryEvidence, type NewObservationLogEntry } from '@n8n/agents';
import { Equal, In, IsNull, LessThan, Like, MoreThan } from '@n8n/typeorm';
import type { Mock, Mocked } from 'vitest';
@ -16,12 +18,12 @@ import type { AgentMemoryEntryCandidateRepository } from '../../repositories/age
import type { AgentMemoryEntryLockRepository } from '../../repositories/agent-memory-entry-lock.repository';
import type { AgentMemoryEntrySourceRepository } from '../../repositories/agent-memory-entry-source.repository';
import type { AgentMemoryEntryRepository } from '../../repositories/agent-memory-entry.repository';
import type { AgentMessageRepository } from '../../repositories/agent-message.repository';
import { AgentMessageRepository } from '../../repositories/agent-message.repository';
import type { AgentObservationCursorRepository } from '../../repositories/agent-observation-cursor.repository';
import type { AgentObservationLockRepository } from '../../repositories/agent-observation-lock.repository';
import type { AgentObservationRepository } from '../../repositories/agent-observation.repository';
import type { AgentResourceRepository } from '../../repositories/agent-resource.repository';
import type { AgentThreadRepository } from '../../repositories/agent-thread.repository';
import { AgentThreadRepository } from '../../repositories/agent-thread.repository';
import { N8nMemory } from '../n8n-memory';
type N8nMemoryImplementation = ReturnType<N8nMemory['getImplementation']>;
@ -68,6 +70,23 @@ describe('N8nMemory', () => {
vi.clearAllMocks();
messageRepository = mock<AgentMessageRepository>();
const messageManager = mock<EntityManager>();
messageManager.upsert.mockImplementation(
async (_target, values, options) => await messageRepository.upsert(values as never, options),
);
messageManager.delete.mockImplementation(
async (_target, criteria) => await messageRepository.delete(criteria as never),
);
const messagePersistence = new AgentMessageRepository(
mock<DataSource>({ manager: messageManager }),
mock(),
);
messageRepository.saveMessages.mockImplementation(
messagePersistence.saveMessages.bind(messagePersistence),
);
messageRepository.deleteMessagesByThread.mockImplementation(
messagePersistence.deleteMessagesByThread.bind(messagePersistence),
);
threadRepository = mock<AgentThreadRepository>();
resourceRepository = mock<AgentResourceRepository>();
observationRepository = mock<AgentObservationRepository>();
@ -137,27 +156,27 @@ describe('N8nMemory', () => {
updatedAt: entity.updatedAt ?? new Date('2026-05-12T10:00:00Z'),
})),
);
runInTransaction = vi.fn(
async (
callback: (trx: {
delete: typeof transactionDelete;
getRepository: Mock;
}) => Promise<void>,
) => {
await callback({
delete: transactionDelete,
getRepository: vi.fn((entity) => {
if (entity === AgentMemoryEntryEntity) {
return { update: transactionMemoryEntryUpdate };
}
return { find: transactionMemoryEntrySourceFind };
}),
});
},
runInTransaction = vi.fn(async (callback: () => Promise<unknown>) => await callback());
const threadManager = mock<EntityManager>();
threadManager.delete.mockImplementation(transactionDelete);
threadManager.update.mockImplementation(
async (_target, criteria, values) => await transactionMemoryEntryUpdate(criteria, values),
);
threadManager.getRepository.mockReturnValue({
find: transactionMemoryEntrySourceFind,
delete: async (criteria: unknown) =>
await transactionDelete(AgentMemoryEntrySourceEntity, criteria),
} as never);
const threadPersistence = new AgentThreadRepository(
mock<DataSource>({ manager: threadManager }),
mock(),
);
vi.spyOn(threadPersistence, 'runInTransaction').mockImplementation(
async (ctx, run) => await runInTransaction(async () => await run(threadManager, ctx)),
);
threadRepository.deleteWithMemory.mockImplementation(
threadPersistence.deleteWithMemory.bind(threadPersistence),
);
Object.defineProperty(threadRepository, 'manager', {
value: { transaction: runInTransaction },
});
memoryEntryRunInTransaction = vi.fn(
async (callback: (trx: { getRepository: Mock }) => Promise<unknown>) =>
await callback({
@ -195,8 +214,10 @@ describe('N8nMemory', () => {
memoryEntryCandidateRepository,
memoryEntryLockRepository,
memoryEntrySourceRepository,
mockConversationLeases(),
);
memory = memoryService.getImplementation('agent-1');
memory = memoryService.getImplementation('agent-1', false);
});
function makeMessageEntity(id: string, createdAt: Date, text: string): AgentMessageEntity {

17
packages/cli/src/modules/agents/integrations/agent-chat-bridge.ts
View File

@ -7,7 +7,7 @@ import {
type AgentIntegrationConfig,
type AgentMessageAuthor,
} from '@n8n/api-types';
import { LockAcquisitionTimeoutError, LockNamespace, LockService } from '@n8n/backend-common';
import { LockNamespace, LockService } from '@n8n/backend-common';
import { type HttpRequestClient, OutboundHttp } from '@n8n/backend-network';
import { Time } from '@n8n/constants';
import { Container } from '@n8n/di';
@ -25,9 +25,9 @@ import type {
ResumeForChatConfig,
} from '../agent-execution-orchestrator.service';
import { AgentExecutionService } from '../agent-execution.service';
import { AgentConversationLeaseService } from '../agent-conversation-lease.service';
import type { AgentMessageQueueService } from '../agent-message-queue.service';
import {
agentConversationLockKey,
type IntegrationMessageQueuePayload,
type IntegrationQueuePayload,
type QueueExecutionContext,
@ -564,18 +564,7 @@ export class AgentChatBridge {
) {
return true;
}
try {
await Container.get(LockService).withLease(
LockNamespace.KNOWN_LOCKS,
agentConversationLockKey(threadId),
async () => {},
{ waitTimeoutMs: 1 },
);
return false;
} catch (error) {
if (error instanceof LockAcquisitionTimeoutError) return true;
throw error;
}
return await Container.get(AgentConversationLeaseService).isHeld(threadId);
}
private async hasOpenSuspension(threadId: string): Promise<boolean> {

118
packages/cli/src/modules/agents/integrations/n8n-checkpoint-storage.ts
View File

@ -6,6 +6,7 @@ import {
import { Logger } from '@n8n/backend-common';
import { AgentsConfig } from '@n8n/config';
import { Time } from '@n8n/constants';
import type { OperationContext } from '@n8n/db';
import { Service } from '@n8n/di';
import { jsonParse, UnexpectedError, UserError } from 'n8n-workflow';
@ -14,6 +15,7 @@ import {
type AgentSandboxPrincipalHash,
} from '../agent-sandbox-principal';
import { AgentCheckpointRepository } from '../repositories/agent-checkpoint.repository';
import { AgentConversationLeaseService } from '../agent-conversation-lease.service';
/** File parts are checkpointed reference-only (a `Uint8Array` would not survive JSON round-tripping). */
function stripStateFileData(state: SerializableAgentState): SerializableAgentState {
@ -47,20 +49,36 @@ export class N8NCheckpointStorage {
private readonly agentCheckpointRepository: AgentCheckpointRepository,
private readonly logger: Logger,
private readonly agentsConfig: AgentsConfig,
private readonly leases: AgentConversationLeaseService,
) {
this.logger = this.logger.scoped('agents');
}
getStorage(agentId: string): CheckpointStore {
getStorage(agentId: string, requireOwnership = true): CheckpointStore {
return {
save: async (key, state) => await this.save(key, state, agentId),
save: async (key, state) => await this.save(key, state, agentId, requireOwnership),
load: async (key) => await this.load(key, agentId),
claimForResume: async (key: string, state: SerializableAgentState) =>
await this.claimForResume(key, state, agentId),
delete: async (key) => await this.delete(key, agentId),
await this.claimForResume(key, state, agentId, requireOwnership),
delete: async (key) => await this.delete(key, agentId, requireOwnership),
};
}
private async writeState<T>(
state: SerializableAgentState,
agentId: string,
requireOwnership: boolean,
write: (ctx: OperationContext) => Promise<T>,
): Promise<T> {
if (!requireOwnership) return await write({});
if (!state.persistence?.threadId)
throw new UnexpectedError('Agent checkpoint has no conversation');
return await this.leases.write(
this.leases.requireOwner(state.persistence.threadId, agentId),
write,
);
}
async getActiveRunIdsForSandbox(
agentId: string,
principalHash: AgentSandboxPrincipalHash,
@ -88,26 +106,20 @@ export class N8NCheckpointStorage {
return runIds;
}
async save(key: string, checkpointState: SerializableAgentState, agentId: string): Promise<void> {
async save(
key: string,
checkpointState: SerializableAgentState,
agentId: string,
requireOwnership = true,
): Promise<void> {
const state = stripStateFileData(checkpointState);
const existing = await this.agentCheckpointRepository.findByRunId(key);
if (existing) {
if (existing.agentId !== agentId) {
throw new UnexpectedError('Agent checkpoint is owned by a different agent');
}
existing.state = JSON.stringify(state);
existing.expired = false;
await this.agentCheckpointRepository.save(existing);
} else {
const checkpoint = this.agentCheckpointRepository.create({
runId: key,
agentId,
state: JSON.stringify(state),
expired: false,
});
await this.agentCheckpointRepository.save(checkpoint);
}
await this.writeState(
state,
agentId,
requireOwnership,
async (ctx) =>
await this.agentCheckpointRepository.saveState(key, agentId, JSON.stringify(state), ctx),
);
}
async load(key: string, agentId: string): Promise<SerializableAgentState | undefined> {
@ -131,13 +143,21 @@ export class N8NCheckpointStorage {
key: string,
checkpointState: SerializableAgentState,
agentId: string,
requireOwnership = true,
): Promise<boolean> {
const state = stripStateFileData(checkpointState);
return await this.agentCheckpointRepository.claimForResume(
key,
return await this.writeState(
state,
agentId,
JSON.stringify(state),
JSON.stringify({ ...state, status: 'running' }),
requireOwnership,
async (ctx) =>
await this.agentCheckpointRepository.claimForResume(
key,
agentId,
JSON.stringify(state),
JSON.stringify({ ...state, status: 'running' }),
ctx,
),
);
}
@ -147,10 +167,17 @@ export class N8NCheckpointStorage {
agentId: string,
): Promise<boolean> {
if (state.status !== 'suspended') return false;
return await this.agentCheckpointRepository.cancelSuspended(
key,
return await this.writeState(
state,
agentId,
JSON.stringify(state),
true,
async (ctx) =>
await this.agentCheckpointRepository.cancelSuspended(
key,
agentId,
JSON.stringify(state),
ctx,
),
);
}
@ -197,8 +224,35 @@ export class N8NCheckpointStorage {
return { status: 'active', checkpoint: state };
}
async delete(key: string, agentId: string): Promise<void> {
await this.agentCheckpointRepository.expireByRunIdAndAgentId(key, agentId);
async delete(key: string, agentId: string, requireOwnership = true): Promise<void> {
const checkpoint = await this.agentCheckpointRepository.findByRunIdAndAgentId(key, agentId);
if (!checkpoint?.state) return;
const state = jsonParse<SerializableAgentState>(checkpoint.state);
await this.writeState(
state,
agentId,
requireOwnership,
async (ctx) =>
await this.agentCheckpointRepository.expireByRunIdAndAgentId(key, agentId, ctx),
);
}
async deleteSuspended(key: string, agentId: string): Promise<void> {
const initial = await this.getStatus(key, agentId);
const threadId =
initial.status !== 'not-found' ? initial.checkpoint?.persistence?.threadId : undefined;
if (!threadId) return;
await this.leases.withLease(
agentId,
threadId,
async () => {
const current = await this.getStatus(key, agentId);
if (current.status !== 'not-found' && current.checkpoint?.status === 'suspended') {
await this.delete(key, agentId);
}
},
{ waitTimeoutMs: 250 },
);
}
/** Marks checkpoints past their TTL as expired. A failure propagates to the caller. */

128
packages/cli/src/modules/agents/integrations/n8n-memory.ts
View File

@ -40,11 +40,12 @@ import {
type ObservationLogTaskLockHandle,
type RetrievedEpisodicMemoryEntry,
type Thread,
stripHydratedFileData,
} from '@n8n/agents';
import { Service } from '@n8n/di';
import type { EntityManager, FindOperator, FindOptionsWhere } from '@n8n/typeorm';
import { Equal, In, IsNull, LessThan, Like, MoreThan } from '@n8n/typeorm';
import type { OperationContext } from '@n8n/db';
import { AgentConversationLeaseService } from '../agent-conversation-lease.service';
import type { FindOptionsWhere } from '@n8n/typeorm';
import { Equal, In, IsNull, LessThan, MoreThan } from '@n8n/typeorm';
import type { QueryDeepPartialEntity } from '@n8n/typeorm/query-builder/QueryPartialEntity';
import { UnexpectedError } from 'n8n-workflow';
@ -55,7 +56,6 @@ import { AgentMemoryEntryLockEntity } from '../entities/agent-memory-entry-lock.
import { AgentMemoryEntrySourceEntity } from '../entities/agent-memory-entry-source.entity';
import { AgentMemoryEntryEntity } from '../entities/agent-memory-entry.entity';
import type { AgentMessageEntity } from '../entities/agent-message.entity';
import { AgentObservationCursorEntity } from '../entities/agent-observation-cursor.entity';
import { AgentObservationLockEntity } from '../entities/agent-observation-lock.entity';
import { AgentObservationEntity } from '../entities/agent-observation.entity';
import { AgentResourceEntity } from '../entities/agent-resource.entity';
@ -92,9 +92,10 @@ export class N8nMemory {
private readonly memoryEntryCandidateRepository: AgentMemoryEntryCandidateRepository,
private readonly memoryEntryLockRepository: AgentMemoryEntryLockRepository,
private readonly memoryEntrySourceRepository: AgentMemoryEntrySourceRepository,
private readonly leases: AgentConversationLeaseService,
) {}
getImplementation(agentId: string) {
getImplementation(agentId: string, requireOwnership = true) {
return new N8nMemoryImpl(
agentId,
this.threadRepository,
@ -107,6 +108,8 @@ export class N8nMemory {
this.memoryEntryCandidateRepository,
this.memoryEntryLockRepository,
this.memoryEntrySourceRepository,
this.leases,
requireOwnership,
);
}
}
@ -131,6 +134,8 @@ export class N8nMemoryImpl
private readonly memoryEntryCandidateRepository: AgentMemoryEntryCandidateRepository,
private readonly memoryEntryLockRepository: AgentMemoryEntryLockRepository,
private readonly memoryEntrySourceRepository: AgentMemoryEntrySourceRepository,
private readonly leases: AgentConversationLeaseService,
private readonly requireOwnership = true,
) {}
readonly episodic: BuiltEpisodicMemoryCaptureStore['episodic'] = {
@ -204,59 +209,25 @@ export class N8nMemoryImpl
.execute();
}
private async writeConversation<T>(
threadId: string,
write: (ctx: OperationContext) => Promise<T>,
): Promise<T> {
if (!this.requireOwnership) return await write({});
return await this.leases.write(this.leases.requireOwner(threadId, this.agentId), write);
}
async deleteThread(threadId: string): Promise<void> {
await this.threadRepository.manager.transaction(async (trx) => {
await this.dropEpisodicEntriesWithoutSources(trx, threadId);
const observationScope = { agentId: this.agentId, observationScopeId: threadId };
await trx.delete(AgentObservationEntity, observationScope);
await trx.delete(AgentObservationCursorEntity, observationScope);
await trx.delete(AgentObservationLockEntity, observationScope);
await trx.delete(AgentThreadEntity, { id: threadId });
});
await this.writeConversation(
threadId,
async (ctx) => await this.threadRepository.deleteWithMemory(this.agentId, threadId, ctx),
);
}
async deleteThreadsByPrefix(threadIdPrefix: string): Promise<void> {
const observationScopeId = Like(`${threadIdPrefix}%`);
await this.threadRepository.manager.transaction(async (trx) => {
await this.dropEpisodicEntriesWithoutSources(trx, observationScopeId);
const observationScope = { agentId: this.agentId, observationScopeId };
await trx.delete(AgentObservationEntity, observationScope);
await trx.delete(AgentObservationCursorEntity, observationScope);
await trx.delete(AgentObservationLockEntity, observationScope);
await trx.delete(AgentThreadEntity, { id: observationScopeId });
});
}
private async dropEpisodicEntriesWithoutSources(
trx: EntityManager,
threadId: string | FindOperator<string>,
): Promise<void> {
const sourceRepo = trx.getRepository(AgentMemoryEntrySourceEntity);
const entryRepo = trx.getRepository(AgentMemoryEntryEntity);
const affectedSources = await sourceRepo.find({
select: { memoryEntryId: true },
where: { agentId: this.agentId, threadId },
});
const affectedEntryIds = uniqueStrings(affectedSources.map((source) => source.memoryEntryId));
if (affectedEntryIds.length === 0) return;
await trx.delete(AgentMemoryEntrySourceEntity, {
agentId: this.agentId,
threadId,
});
const remainingSources = await sourceRepo.find({
select: { memoryEntryId: true },
where: { agentId: this.agentId, memoryEntryId: In(affectedEntryIds) },
});
const entriesWithSources = new Set(remainingSources.map((source) => source.memoryEntryId));
const orphanedEntryIds = affectedEntryIds.filter((id) => !entriesWithSources.has(id));
if (orphanedEntryIds.length === 0) return;
await entryRepo.update(
{ agentId: this.agentId, id: In(orphanedEntryIds), status: 'active' },
droppedLifecycleState(),
);
if (this.requireOwnership)
throw new UnexpectedError('A conversation cannot delete other threads');
await this.threadRepository.deleteWithMemory(this.agentId, threadIdPrefix, {}, true);
}
// ── Message persistence ──────────────────────────────────────────────
@ -293,44 +264,33 @@ export class N8nMemoryImpl
messages: AgentDbMessage[];
}): Promise<void> {
if (args.messages.length === 0) return;
// Upsert by id — bulk INSERT … ON CONFLICT (id) DO UPDATE avoids the
// per-row SELECT that save() performs. createdAt is passed explicitly so
// the column is preserved on conflict; updatedAt is set manually because
// the @BeforeUpdate hook does not fire during upsert.
const now = new Date();
const entities = args.messages.map((message) => {
const dbMsg = stripHydratedFileData(message);
const role = 'role' in dbMsg ? (dbMsg.role as string) : 'custom';
const type = 'type' in dbMsg ? (dbMsg.type as string) : null;
return {
id: dbMsg.id,
threadId: args.threadId,
resourceId: args.resourceId,
role,
type: type ?? null,
content: dbMsg as unknown as Record<string, unknown>,
createdAt: dbMsg.createdAt,
updatedAt: now,
} as QueryDeepPartialEntity<AgentMessageEntity>;
});
await this.messageRepository.upsert(entities, ['id']);
await this.writeConversation(
args.threadId,
async (ctx) => await this.messageRepository.saveMessages(args, ctx),
);
}
async deleteMessages(messageIds: string[]): Promise<void> {
if (messageIds.length === 0) return;
await this.messageRepository.delete(messageIds);
if (!this.requireOwnership) {
await this.messageRepository.deleteMessages(messageIds, {});
return;
}
const owner = this.leases.currentOwner();
if (!owner || owner.lease.agentId !== this.agentId)
throw new UnexpectedError('Agent message deletion has no conversation owner');
await this.leases.write(
owner,
async (ctx) =>
await this.messageRepository.deleteMessages(messageIds, ctx, owner.lease.threadId),
);
}
async deleteMessagesByThread(threadId: string, resourceId?: string): Promise<void> {
// Mirrors `getMessages`: explicit `!== undefined` check so that a falsy
// (empty-string) `resourceId` cannot accidentally delete every user's
// messages on a shared thread.
await this.messageRepository.delete({
await this.writeConversation(
threadId,
...(resourceId !== undefined && { resourceId }),
});
async (ctx) => await this.messageRepository.deleteMessagesByThread(threadId, resourceId, ctx),
);
}
// ── Observation log ──────────────────────────────────────────────────

59
packages/cli/src/modules/agents/repositories/agent-checkpoint.repository.ts
View File

@ -1,16 +1,37 @@
import { BaseRepository, TransactionRunner, type OperationContext } from '@n8n/db';
import { Service } from '@n8n/di';
import { DataSource, Repository } from '@n8n/typeorm';
import { DataSource } from '@n8n/typeorm';
import { UnexpectedError } from 'n8n-workflow';
import { AgentCheckpoint } from '../entities/agent-checkpoint.entity';
@Service()
export class AgentCheckpointRepository extends Repository<AgentCheckpoint> {
constructor(dataSource: DataSource) {
super(AgentCheckpoint, dataSource.manager);
export class AgentCheckpointRepository extends BaseRepository<AgentCheckpoint> {
constructor(dataSource: DataSource, transactionRunner: TransactionRunner) {
super(AgentCheckpoint, dataSource.manager, transactionRunner);
}
async findByRunId(runId: string): Promise<AgentCheckpoint | null> {
return await this.findOneBy({ runId });
async saveState(
runId: string,
agentId: string,
state: string,
ctx: OperationContext,
): Promise<void> {
const manager = this.managerFor(ctx);
const existing = await manager.findOneBy(AgentCheckpoint, { runId });
if (existing && existing.agentId !== agentId) {
throw new UnexpectedError('Agent checkpoint is owned by a different agent');
}
await manager.save(
AgentCheckpoint,
manager.create(AgentCheckpoint, {
...existing,
runId,
agentId,
state,
expired: false,
}),
);
}
async findByRunIdAndAgentId(runId: string, agentId: string): Promise<AgentCheckpoint | null> {
@ -37,8 +58,10 @@ export class AgentCheckpointRepository extends Repository<AgentCheckpoint> {
agentId: string,
suspendedState: string,
runningState: string,
ctx: OperationContext,
): Promise<boolean> {
const result = await this.update(
const result = await this.managerFor(ctx).update(
AgentCheckpoint,
{ runId, agentId, expired: false, state: suspendedState },
{ state: runningState },
);
@ -46,8 +69,14 @@ export class AgentCheckpointRepository extends Repository<AgentCheckpoint> {
return (result.affected ?? 0) > 0;
}
async cancelSuspended(runId: string, agentId: string, suspendedState: string): Promise<boolean> {
const result = await this.update(
async cancelSuspended(
runId: string,
agentId: string,
suspendedState: string,
ctx: OperationContext,
): Promise<boolean> {
const result = await this.managerFor(ctx).update(
AgentCheckpoint,
{ runId, agentId, expired: false, state: suspendedState },
{ expired: true },
);
@ -55,8 +84,16 @@ export class AgentCheckpointRepository extends Repository<AgentCheckpoint> {
return (result.affected ?? 0) > 0;
}
async expireByRunIdAndAgentId(runId: string, agentId: string): Promise<void> {
await this.update({ runId, agentId }, { expired: true, state: null });
async expireByRunIdAndAgentId(
runId: string,
agentId: string,
ctx: OperationContext,
): Promise<void> {
await this.managerFor(ctx).update(
AgentCheckpoint,
{ runId, agentId },
{ expired: true, state: null },
);
}
async markExpired(olderThan: Date): Promise<number> {

103
packages/cli/src/modules/agents/repositories/agent-conversation-lease.repository.ts Normal file
View File

@ -0,0 +1,103 @@
import { BaseRepository, TransactionRunner, type OperationContext } from '@n8n/db';
import { Service } from '@n8n/di';
import { DataSource } from '@n8n/typeorm';
import { randomUUID } from 'node:crypto';
import {
AgentConversationLeaseLostError,
CONVERSATION_LEASE_TTL_MS,
type AgentConversationLeaseHandle,
} from '../agent-conversation-lease.types';
import { AgentConversationLease } from '../entities/agent-conversation-lease.entity';
/** Read the database wall clock, including after a transaction waits for a row. */
export function conversationDbTime(isPostgres: boolean, offsetMs = 0): string {
return isPostgres
? `clock_timestamp() + INTERVAL '${offsetMs} milliseconds'`
: `STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW', '${offsetMs / 1000} seconds')`;
}
@Service()
export class AgentConversationLeaseRepository extends BaseRepository<AgentConversationLease> {
constructor(dataSource: DataSource, transactionRunner: TransactionRunner) {
super(AgentConversationLease, dataSource.manager, transactionRunner);
}
private time(offsetMs = 0): string {
return conversationDbTime(this.manager.connection.options.type === 'postgres', offsetMs);
}
async acquire(agentId: string, threadId: string): Promise<AgentConversationLeaseHandle | null> {
const handle = { agentId, threadId, ownerToken: randomUUID() };
return await this.runInTransaction({}, async (manager) => {
await manager
.createQueryBuilder()
.update(AgentConversationLease)
.set({
ownerToken: handle.ownerToken,
expiresAt: () => this.time(CONVERSATION_LEASE_TTL_MS),
})
.where({ agentId, threadId })
.andWhere(`"expiresAt" <= ${this.time()}`)
.execute();
await manager
.createQueryBuilder()
.insert()
.into(AgentConversationLease)
.values({ ...handle, expiresAt: () => this.time(CONVERSATION_LEASE_TTL_MS) })
.orIgnore()
.execute();
return (await manager.existsBy(AgentConversationLease, handle)) ? handle : null;
});
}
async assertOwner(handle: AgentConversationLeaseHandle, ctx: OperationContext): Promise<void> {
const manager = this.managerFor(ctx);
// The write lock excludes takeover until the caller's protected mutation commits.
const locked = await manager.update(AgentConversationLease, handle, {
ownerToken: handle.ownerToken,
});
if (locked.affected !== 1) throw new AgentConversationLeaseLostError(handle.threadId);
const live = await manager
.createQueryBuilder(AgentConversationLease, 'lease')
.where(handle)
.andWhere(`lease.expiresAt > ${this.time()}`)
.getExists();
if (!live) throw new AgentConversationLeaseLostError(handle.threadId);
}
async renew(handle: AgentConversationLeaseHandle): Promise<boolean> {
try {
return await this.runInTransaction({}, async (manager, ctx) => {
await this.assertOwner(handle, ctx);
await manager.update(AgentConversationLease, handle, {
expiresAt: () => this.time(CONVERSATION_LEASE_TTL_MS),
});
return true;
});
} catch (error) {
if (error instanceof AgentConversationLeaseLostError) return false;
throw error;
}
}
async release(handle: AgentConversationLeaseHandle): Promise<boolean> {
try {
return await this.runInTransaction({}, async (manager, ctx) => {
await this.assertOwner(handle, ctx);
await manager.delete(AgentConversationLease, handle);
return true;
});
} catch (error) {
if (error instanceof AgentConversationLeaseLostError) return false;
throw error;
}
}
async isHeld(threadId: string): Promise<boolean> {
return await this.createQueryBuilder('lease')
.where({ threadId })
.andWhere(`lease.expiresAt > ${this.time()}`)
.getExists();
}
}

82
packages/cli/src/modules/agents/repositories/agent-execution.repository.ts
View File

@ -1,9 +1,11 @@
import { BaseRepository, TransactionRunner, type OperationContext } from '@n8n/db';
import { Service } from '@n8n/di';
import { DataSource, IsNull, LessThanOrEqual, Not, Repository } from '@n8n/typeorm';
import { DataSource, IsNull, Not } from '@n8n/typeorm';
import type { QueryDeepPartialEntity } from '@n8n/typeorm/query-builder/QueryPartialEntity';
import { AgentExecution, type AgentExecutionStatus } from '../entities/agent-execution.entity';
import type { ThreadFailureSummary } from '../utils/execution-failure-summary';
import { conversationDbTime } from './agent-conversation-lease.repository';
export type RunningAgentExecution = Pick<
AgentExecution,
@ -22,9 +24,28 @@ type AgentExecutionFinalizationValues = Pick<
>;
@Service()
export class AgentExecutionRepository extends Repository<AgentExecution> {
constructor(dataSource: DataSource) {
super(AgentExecution, dataSource.manager);
export class AgentExecutionRepository extends BaseRepository<AgentExecution> {
constructor(dataSource: DataSource, transactionRunner: TransactionRunner) {
super(AgentExecution, dataSource.manager, transactionRunner);
}
private time(offsetMs = 0): string {
return conversationDbTime(this.manager.connection.options.type === 'postgres', offsetMs);
}
async createExecution(
values: Partial<AgentExecution>,
ctx: OperationContext,
): Promise<AgentExecution> {
const manager = this.managerFor(ctx);
return await manager.save(AgentExecution, manager.create(AgentExecution, values));
}
async findStaleRunning(graceMs: number): Promise<RunningAgentExecution[]> {
return await this.createQueryBuilder('execution')
.where({ status: 'running' })
.andWhere(`execution.updatedAt <= ${this.time(-graceMs)}`)
.getMany();
}
/** All executions in a thread, oldest first — used by the timeline view. */
@ -32,13 +53,6 @@ export class AgentExecutionRepository extends Repository<AgentExecution> {
return await this.find({ where: { threadId }, order: { createdAt: 'ASC' } });
}
async findRunning(): Promise<RunningAgentExecution[]> {
return await this.find({
select: ['id', 'threadId', 'startedAt', 'updatedAt', 'timeline'],
where: { status: 'running' },
});
}
async existsRunningByThread(threadId: string): Promise<boolean> {
return await this.existsBy({ threadId, status: 'running' });
}
@ -50,26 +64,39 @@ export class AgentExecutionRepository extends Repository<AgentExecution> {
});
}
async touchRunning(executionId: string): Promise<void> {
await this.update({ id: executionId, status: 'running' }, { updatedAt: new Date() });
async touchRunning(executionId: string, ctx: OperationContext): Promise<void> {
await this.managerFor(ctx).update(
AgentExecution,
{ id: executionId, status: 'running' },
{
updatedAt: () => this.time(),
},
);
}
async updateTimelineIfRunning(
executionId: string,
timeline: AgentExecution['timeline'],
ctx: OperationContext,
): Promise<boolean> {
const result = await this.update({ id: executionId, status: 'running' }, {
timeline,
updatedAt: new Date(),
} as QueryDeepPartialEntity<AgentExecution>);
const result = await this.managerFor(ctx).update(
AgentExecution,
{ id: executionId, status: 'running' },
{
timeline,
updatedAt: () => this.time(),
} as QueryDeepPartialEntity<AgentExecution>,
);
return result.affected === 1;
}
async updateIfRunning(
executionId: string,
values: AgentExecutionFinalizationValues,
ctx: OperationContext,
): Promise<boolean> {
const result = await this.update(
const result = await this.managerFor(ctx).update(
AgentExecution,
{ id: executionId, status: 'running' },
values as QueryDeepPartialEntity<AgentExecution>,
);
@ -78,13 +105,17 @@ export class AgentExecutionRepository extends Repository<AgentExecution> {
async updateIfAbandoned(
executionId: string,
staleBefore: Date,
graceMs: number,
values: AgentExecutionFinalizationValues,
ctx: OperationContext,
): Promise<boolean> {
const result = await this.update(
{ id: executionId, status: 'running', updatedAt: LessThanOrEqual(staleBefore) },
values as QueryDeepPartialEntity<AgentExecution>,
);
const result = await this.managerFor(ctx)
.createQueryBuilder()
.update(AgentExecution)
.set(values)
.where({ id: executionId, status: 'running' })
.andWhere(`"updatedAt" <= ${this.time(-graceMs)}`)
.execute();
return result.affected === 1;
}
@ -235,9 +266,10 @@ export class AgentExecutionRepository extends Repository<AgentExecution> {
}
/** Backfill model on a set of executions in a single statement. */
async backfillModel(executionIds: string[], model: string): Promise<void> {
async backfillModel(executionIds: string[], model: string, ctx: OperationContext): Promise<void> {
if (executionIds.length === 0) return;
await this.createQueryBuilder()
await this.managerFor(ctx)
.createQueryBuilder()
.update(AgentExecution)
.set({ model })
.whereInIds(executionIds)

93
packages/cli/src/modules/agents/repositories/agent-message-queue.repository.ts
View File

@ -1,13 +1,24 @@
import { BaseRepository, TransactionRunner, type OperationContext } from '@n8n/db';
import { Service } from '@n8n/di';
import { DataSource, In, LessThan, Repository } from '@n8n/typeorm';
import { DataSource, In } from '@n8n/typeorm';
import { conversationDbTime } from './agent-conversation-lease.repository';
import type { AgentQueueInput } from '../agent-message-queue.types';
import { AgentMessageQueue } from '../entities/agent-message-queue.entity';
@Service()
export class AgentMessageQueueRepository extends Repository<AgentMessageQueue> {
constructor(dataSource: DataSource) {
super(AgentMessageQueue, dataSource.manager);
export class AgentMessageQueueRepository extends BaseRepository<AgentMessageQueue> {
constructor(dataSource: DataSource, transactionRunner: TransactionRunner) {
super(AgentMessageQueue, dataSource.manager, transactionRunner);
}
private time(offsetMs = 0): string {
return conversationDbTime(this.manager.connection.options.type === 'postgres', offsetMs);
}
async findById(id: string, ctx: OperationContext): Promise<AgentMessageQueue | null> {
return await this.managerFor(ctx).findOneBy(AgentMessageQueue, { id });
}
async enqueue(input: AgentQueueInput): Promise<AgentMessageQueue> {
@ -38,16 +49,42 @@ export class AgentMessageQueueRepository extends Repository<AgentMessageQueue> {
return await this.existsBy({ threadId });
}
async markProcessing(id: string): Promise<boolean> {
return (await this.update({ id, status: 'queued' }, { status: 'processing' })).affected === 1;
async markProcessing(id: string, ctx: OperationContext): Promise<boolean> {
return (
(
await this.managerFor(ctx).update(
AgentMessageQueue,
{ id, status: 'queued' },
{
status: 'processing',
updatedAt: () => this.time(),
},
)
).affected === 1
);
}
async linkExecution(id: string, executionId: string): Promise<void> {
await this.update({ id, status: 'processing' }, { executionId });
async linkExecution(id: string, executionId: string, ctx: OperationContext): Promise<void> {
await this.managerFor(ctx).update(
AgentMessageQueue,
{ id, status: 'processing' },
{ executionId },
);
}
async removeEntry(id: string): Promise<void> {
await this.delete({ id });
async removeEntry(id: string, ctx: OperationContext): Promise<void> {
await this.managerFor(ctx).delete(AgentMessageQueue, { id });
}
async removeStale(id: string, graceMs: number, ctx: OperationContext): Promise<boolean> {
const deleted = await this.managerFor(ctx)
.createQueryBuilder()
.delete()
.from(AgentMessageQueue)
.where({ id })
.andWhere(`"updatedAt" < ${this.time(-graceMs)}`)
.execute();
return deleted.affected === 1;
}
async cancelQueued(id: string): Promise<boolean> {
@ -60,29 +97,45 @@ export class AgentMessageQueueRepository extends Repository<AgentMessageQueue> {
async touchLiveEntries(ids: string[]): Promise<string[]> {
if (ids.length === 0) return [];
await this.update({ id: In(ids) }, { updatedAt: new Date() });
await this.update({ id: In(ids), status: 'queued' }, { updatedAt: () => this.time() });
return await this.findExistingIds(ids);
}
async touchProcessing(id: string, ctx: OperationContext): Promise<void> {
await this.managerFor(ctx).update(
AgentMessageQueue,
{ id, status: 'processing' },
{
updatedAt: () => this.time(),
},
);
}
async findExistingIds(ids: string[]): Promise<string[]> {
if (ids.length === 0) return [];
const rows = await this.find({ select: ['id'], where: { id: In(ids) } });
return rows.map(({ id }) => id);
}
async findStale(threadId: string, cutoff: Date): Promise<AgentMessageQueue[]> {
return await this.find({
where: [
{ threadId, status: 'processing', updatedAt: LessThan(cutoff) },
{ threadId, status: 'queued', source: 'preview', updatedAt: LessThan(cutoff) },
],
});
async findStale(threadId: string, graceMs: number): Promise<AgentMessageQueue[]> {
return await this.createQueryBuilder('item')
.where({ threadId })
.andWhere(`item.updatedAt < ${this.time(-graceMs)}`)
.andWhere(
'(item.status = :processing OR (item.status = :queued AND item.source = :preview))',
{
processing: 'processing',
queued: 'queued',
preview: 'preview',
},
)
.getMany();
}
async findStaleThreads(cutoff: Date): Promise<string[]> {
async findStaleThreads(graceMs: number): Promise<string[]> {
const rows = await this.createQueryBuilder('item')
.select('DISTINCT item.threadId', 'threadId')
.where('item.updatedAt < :cutoff', { cutoff })
.where(`item.updatedAt < ${this.time(-graceMs)}`)
.andWhere('(item.status = :processing OR item.source = :preview)', {
processing: 'processing',
preview: 'preview',

50
packages/cli/src/modules/agents/repositories/agent-message.repository.ts
View File

@ -1,12 +1,54 @@
import { stripHydratedFileData, type AgentDbMessage } from '@n8n/agents';
import { BaseRepository, TransactionRunner, type OperationContext } from '@n8n/db';
import { Service } from '@n8n/di';
import { DataSource, Repository } from '@n8n/typeorm';
import { DataSource, In } from '@n8n/typeorm';
import type { QueryDeepPartialEntity } from '@n8n/typeorm/query-builder/QueryPartialEntity';
import { AgentMessageEntity } from '../entities/agent-message.entity';
@Service()
export class AgentMessageRepository extends Repository<AgentMessageEntity> {
constructor(dataSource: DataSource) {
super(AgentMessageEntity, dataSource.manager);
export class AgentMessageRepository extends BaseRepository<AgentMessageEntity> {
constructor(dataSource: DataSource, transactionRunner: TransactionRunner) {
super(AgentMessageEntity, dataSource.manager, transactionRunner);
}
async saveMessages(
args: { threadId: string; resourceId: string; messages: AgentDbMessage[] },
ctx: OperationContext,
): Promise<void> {
const now = new Date();
const entities = args.messages.map((message) => {
const dbMsg = stripHydratedFileData(message);
return {
id: dbMsg.id,
threadId: args.threadId,
resourceId: args.resourceId,
role: 'role' in dbMsg ? dbMsg.role : 'custom',
type: 'type' in dbMsg ? dbMsg.type : null,
content: dbMsg as unknown as Record<string, unknown>,
createdAt: dbMsg.createdAt,
updatedAt: now,
} as QueryDeepPartialEntity<AgentMessageEntity>;
});
await this.managerFor(ctx).upsert(AgentMessageEntity, entities, ['id']);
}
async deleteMessages(ids: string[], ctx: OperationContext, threadId?: string): Promise<void> {
await this.managerFor(ctx).delete(AgentMessageEntity, {
id: In(ids),
...(threadId !== undefined ? { threadId } : {}),
});
}
async deleteMessagesByThread(
threadId: string,
resourceId: string | undefined,
ctx: OperationContext,
): Promise<void> {
await this.managerFor(ctx).delete(AgentMessageEntity, {
threadId,
...(resourceId !== undefined ? { resourceId } : {}),
});
}
/** Threads the resource has posted in, most recent activity first. */

53
packages/cli/src/modules/agents/repositories/agent-thread.repository.ts
View File

@ -1,11 +1,56 @@
import { droppedLifecycleState, uniqueStrings } from '@n8n/agents';
import { BaseRepository, TransactionRunner, type OperationContext } from '@n8n/db';
import { Service } from '@n8n/di';
import { DataSource, Repository } from '@n8n/typeorm';
import { DataSource, In, Like } from '@n8n/typeorm';
import { AgentThreadEntity } from '../entities/agent-thread.entity';
import { AgentMemoryEntrySourceEntity } from '../entities/agent-memory-entry-source.entity';
import { AgentMemoryEntryEntity } from '../entities/agent-memory-entry.entity';
import { AgentObservationEntity } from '../entities/agent-observation.entity';
import { AgentObservationCursorEntity } from '../entities/agent-observation-cursor.entity';
import { AgentObservationLockEntity } from '../entities/agent-observation-lock.entity';
@Service()
export class AgentThreadRepository extends Repository<AgentThreadEntity> {
constructor(dataSource: DataSource) {
super(AgentThreadEntity, dataSource.manager);
export class AgentThreadRepository extends BaseRepository<AgentThreadEntity> {
constructor(dataSource: DataSource, transactionRunner: TransactionRunner) {
super(AgentThreadEntity, dataSource.manager, transactionRunner);
}
async deleteWithMemory(
agentId: string,
threadId: string,
ctx: OperationContext,
prefix = false,
): Promise<void> {
await this.runInTransaction(ctx, async (manager) => {
const scope = prefix ? Like(`${threadId}%`) : threadId;
const sourceRepository = manager.getRepository(AgentMemoryEntrySourceEntity);
const sources = await sourceRepository.find({
select: { memoryEntryId: true },
where: { agentId, threadId: scope },
});
const entryIds = uniqueStrings(sources.map((source) => source.memoryEntryId));
if (entryIds.length > 0) {
await sourceRepository.delete({ agentId, threadId: scope });
const remaining = await sourceRepository.find({
select: { memoryEntryId: true },
where: { agentId, memoryEntryId: In(entryIds) },
});
const retained = new Set(remaining.map((source) => source.memoryEntryId));
const orphaned = entryIds.filter((id) => !retained.has(id));
if (orphaned.length > 0) {
await manager.update(
AgentMemoryEntryEntity,
{ agentId, id: In(orphaned), status: 'active' },
droppedLifecycleState(),
);
}
}
const observationScope = { agentId, observationScopeId: scope };
await manager.delete(AgentObservationEntity, observationScope);
await manager.delete(AgentObservationCursorEntity, observationScope);
await manager.delete(AgentObservationLockEntity, observationScope);
await manager.delete(AgentThreadEntity, { id: scope });
});
}
}

9
packages/cli/src/modules/agents/sub-agents/__tests__/sub-agent-runner.test.ts
View File

@ -1,3 +1,4 @@
import { mockConversationLeases } from '../../__tests__/mock-conversation-lease';
import {
INLINE_SUB_AGENT_ID,
type BuiltAgent,
@ -140,6 +141,8 @@ describe('SubAgentRunner', () => {
checkpointStorage,
logger,
aiConfigMock,
mockConversationLeases(),
);
childAgent = mock<BuiltAgent>();
@ -743,7 +746,7 @@ describe('SubAgentRunner', () => {
{ projectId, usePublishedVersion: true },
);
expect(result.threadId).toBe('child-thread-1');
expect(checkpointStorage.delete).toHaveBeenCalledWith('child-run-1', parentAgentId);
expect(checkpointStorage.deleteSuspended).toHaveBeenCalledWith('child-run-1', parentAgentId);
});
it('accepts a legacy pinned resume context', async () => {
@ -782,7 +785,7 @@ describe('SubAgentRunner', () => {
reason: 'Parent run aborted',
});
expect(checkpointStorage.delete).toHaveBeenCalledWith('child-run-1', 'agent-1');
expect(checkpointStorage.deleteSuspended).toHaveBeenCalledWith('child-run-1', 'agent-1');
expect(sourceResolver.resolveForRuntime).not.toHaveBeenCalled();
expect(reconstructionService.reconstructFromResolvedSource).not.toHaveBeenCalled();
expect(childAgent.resume).not.toHaveBeenCalled();
@ -803,7 +806,7 @@ describe('SubAgentRunner', () => {
reason: 'Parent run aborted',
}),
).rejects.toThrow('Configured sub-agent resume context is missing or invalid');
expect(checkpointStorage.delete).not.toHaveBeenCalled();
expect(checkpointStorage.deleteSuspended).not.toHaveBeenCalled();
});
it('marks the run as failed when the child result contains an error', async () => {

32
packages/cli/src/modules/agents/sub-agents/sub-agent-runner.ts
View File

@ -50,7 +50,12 @@ import { modelStreamStallOptions } from '../model-stream-stall-options';
import type { WorkflowToolExecutionMode } from '../tools/workflow-tool-factory';
import { streamAgentChunks } from '../utils/agent-stream';
import { createAttributionTracker } from '../utils/mcp-attribution';
import { SubAgentSourceResolver } from './sub-agent-source-resolver';
import {
SubAgentSourceResolver,
type ResolvedSubAgentRuntimeSource,
} from './sub-agent-source-resolver';
import { AgentConversationLeaseService } from '../agent-conversation-lease.service';
import { AgentConversationLeaseLostError } from '../agent-conversation-lease.types';
export interface SubAgentRunContext {
projectId: string;
@ -125,6 +130,7 @@ export class SubAgentRunner {
private readonly checkpointStorage: N8NCheckpointStorage,
private readonly logger: Logger,
private readonly aiConfig: AiConfig,
private readonly leases: AgentConversationLeaseService,
) {}
async run(
@ -170,7 +176,7 @@ export class SubAgentRunner {
throw new UserError('Configured sub-agent checkpoint metadata is missing or invalid');
}
const pinnedSource = parseResumeContext(request.resumeContext, expectedSourceAgentId);
await this.checkpointStorage.delete(request.childRunId, pinnedSource.agentId);
await this.checkpointStorage.deleteSuspended(request.childRunId, pinnedSource.agentId);
}
private async executeForeground(
@ -190,6 +196,26 @@ export class SubAgentRunner {
// can reference the child before the run starts.
const threadId =
operation.type === 'run' ? (operation.request.childThreadId ?? uuid()) : operation.threadId;
return await this.leases.withLease(
runtimeSource.source.sourceId,
threadId,
async (signal) =>
await this.executeOwnedForeground(
operation,
{ ...context, abortSignal: signal },
runtimeSource,
threadId,
),
{ signal: context.abortSignal },
);
}
private async executeOwnedForeground(
operation: ForegroundOperation,
context: SubAgentRunContext,
runtimeSource: ResolvedSubAgentRuntimeSource,
threadId: string,
): Promise<SubAgentRunResult> {
const resourceId =
operation.type === 'run' ? (operation.request.parentResourceId ?? threadId) : threadId;
const sandboxPrincipalHash = await this.resolveSandboxPrincipalHash(
@ -310,6 +336,7 @@ export class SubAgentRunner {
);
executionId = currentExecutionId;
} catch (error) {
if (error instanceof AgentConversationLeaseLostError) throw error;
this.logger.warn('Failed to start subagent execution recording', {
agentId: runtimeSource.source.sourceId,
taskPath: operation.taskPath,
@ -356,6 +383,7 @@ export class SubAgentRunner {
...(suspended ? { resumeContext: createResumeContext(runtimeSource.source) } : {}),
};
} catch (error) {
if (this.leases.requireOwner(threadId).signal.aborted) throw error;
if (!recorded) {
recorder.record({ type: 'error', error });
recorder.record({ type: 'finish', finishReason: 'error' });

399
packages/cli/test/integration/agents/agent-conversation-lease.test.ts Normal file
View File

@ -0,0 +1,399 @@
import { GlobalConfig } from '@n8n/config';
import type { AgentDbMessage, SerializableAgentState } from '@n8n/agents';
import { createDeferredPromise } from '@n8n/utils/promise/deferred-promise';
import { DataSource } from '@n8n/typeorm';
import { mock } from 'vitest-mock-extended';
import { AgentExecutionService } from '@/modules/agents/agent-execution.service';
import { N8NCheckpointStorage } from '@/modules/agents/integrations/n8n-checkpoint-storage';
import { N8nMemory } from '@/modules/agents/integrations/n8n-memory';
import { AgentExecutionRepository } from '@/modules/agents/repositories/agent-execution.repository';
import { AgentExecutionThreadRepository } from '@/modules/agents/repositories/agent-execution-thread.repository';
import { AgentMessageRepository } from '@/modules/agents/repositories/agent-message.repository';
import { AgentMessageQueueRepository } from '@/modules/agents/repositories/agent-message-queue.repository';
import { AgentConversationLease } from '@/modules/agents/entities/agent-conversation-lease.entity';
import { retryUntil } from '../shared/retry-until';
import { createTeamProject, testDb, testModules } from '@n8n/backend-test-utils';
import { Container } from '@n8n/di';
import { randomUUID } from 'node:crypto';
import { AgentConversationLeaseService } from '@/modules/agents/agent-conversation-lease.service';
import { AgentConversationLeaseLostError } from '@/modules/agents/agent-conversation-lease.types';
import { AgentConversationLeaseRepository } from '@/modules/agents/repositories/agent-conversation-lease.repository';
import { AgentRepository } from '@/modules/agents/repositories/agent.repository';
describe('agent conversation ownership', () => {
let repository: AgentConversationLeaseRepository;
let leases: AgentConversationLeaseService;
let agentId: string;
let projectId: string;
beforeAll(async () => {
// These races require independent database connections.
Container.get(GlobalConfig).database.postgresdb.poolSize = 4;
await testModules.loadModules(['agents']);
await testDb.init();
repository = Container.get(AgentConversationLeaseRepository);
leases = Container.get(AgentConversationLeaseService);
});
beforeEach(async () => {
const agents = Container.get(AgentRepository);
await agents.delete({});
const project = await createTeamProject();
projectId = project.id;
agentId = randomUUID();
await agents.save(
agents.create({ id: agentId, name: 'Agent', projectId: project.id, versionId: randomUUID() }),
);
});
afterAll(async () => await testDb.terminate());
it('grants one owner before history exists and permits independent conversations', async () => {
const claims = await Promise.all([
repository.acquire(agentId, 'first'),
repository.acquire(agentId, 'first'),
]);
expect(claims.filter(Boolean)).toHaveLength(1);
expect(await repository.acquire(agentId, 'second')).not.toBeNull();
expect(await repository.count()).toBe(2);
});
it('rejects expired renewal and release without changing a successor', async () => {
const former = await repository.acquire(agentId, 'conversation');
if (!former) throw new Error('Missing first owner');
await repository.update({ threadId: 'conversation' }, { expiresAt: new Date(0) });
expect(await repository.renew(former)).toBe(false);
expect(await repository.release(former)).toBe(false);
const successor = await repository.acquire(agentId, 'conversation');
expect(successor).not.toBeNull();
expect(successor?.ownerToken).not.toBe(former.ownerToken);
expect(await repository.renew(former)).toBe(false);
expect(await repository.release(former)).toBe(false);
expect(await repository.findOneByOrFail({ threadId: 'conversation' })).toMatchObject(
successor!,
);
});
it('rejects late writes even if the former runner ignores cancellation', async () => {
await leases.withLease(agentId, 'conversation', async () => {
const owner = leases.requireOwner('conversation');
await repository.update({ threadId: 'conversation' }, { expiresAt: new Date(0) });
const successor = await repository.acquire(agentId, 'conversation');
await expect(
leases.write(owner, async () => {
throw new Error('The former owner reached its protected write');
}),
).rejects.toBeInstanceOf(AgentConversationLeaseLostError);
expect(owner.signal.aborted).toBe(true);
expect(await repository.findOneByOrFail({ threadId: 'conversation' })).toMatchObject(
successor!,
);
});
expect(await repository.isHeld('conversation')).toBe(true);
});
it('keeps a captured owner invalid after release and acquisition by the same main', async () => {
const former = await leases.withLease(agentId, 'conversation', async () =>
leases.requireOwner('conversation'),
);
await leases.withLease(agentId, 'conversation', async () => {
const current = leases.requireOwner('conversation');
expect(current.lease.ownerToken).not.toBe(former.lease.ownerToken);
await expect(leases.write(former, async () => true)).rejects.toBeInstanceOf(
AgentConversationLeaseLostError,
);
expect(await leases.write(current, async () => true)).toBe(true);
});
expect(await repository.count()).toBe(0);
});
it('rejects former-owner execution, checkpoint, and saved-message mutations', async () => {
const memory = Container.get(N8nMemory).getImplementation(agentId);
const checkpoints = Container.get(N8NCheckpointStorage);
const executionRepository = Container.get(AgentExecutionRepository);
const executions = new AgentExecutionService(
mock(),
executionRepository,
Container.get(AgentExecutionThreadRepository),
Container.get(N8nMemory),
mock(),
mock(),
mock(),
mock({ modeTag: 'db' }),
mock(),
mock(),
Container.get(AgentMessageQueueRepository),
leases,
);
const params = {
agentId,
projectId,
threadId: 'protected',
agentName: 'Agent',
userMessage: 'Start',
};
const message: AgentDbMessage = {
id: randomUUID(),
role: 'user',
content: [{ type: 'text', text: 'Original' }],
createdAt: new Date(),
};
const state: SerializableAgentState = {
status: 'suspended',
persistence: { threadId: 'protected', resourceId: 'user' },
messageList: { messages: [], historyIds: [], inputIds: [], responseIds: [] },
pendingToolCalls: {},
};
await leases.withLease(agentId, 'protected', async () => {
await memory.saveThread({ id: 'protected', resourceId: 'user' });
await memory.saveMessages({ threadId: 'protected', resourceId: 'user', messages: [message] });
await checkpoints.save('run', state, agentId);
const executionId = await executions.startExecutionRecording(params, new Date());
await repository.update({ threadId: 'protected' }, { expiresAt: new Date(0) });
expect(await repository.acquire(agentId, 'protected')).not.toBeNull();
executions.recordTimelineSnapshot({
executionId,
agentId,
projectId,
threadId: 'protected',
timeline: [{ type: 'text', content: 'Late snapshot', timestamp: 1 }],
});
await Promise.all([...executions['timelineSnapshotWrites'].values()]);
const writes = [
async () => await executions.startExecutionRecording(params, new Date()),
async () =>
await executions.finalizeExecution(executionId, {
...params,
record: {
assistantResponse: 'Late',
model: null,
finishReason: 'stop',
usage: null,
totalCost: null,
timeline: [],
startTime: Date.now(),
duration: 1,
error: null,
},
}),
async () => await checkpoints.save('run', { ...state, status: 'running' }, agentId),
async () => await checkpoints.claimForResume('run', state, agentId),
async () => await checkpoints.cancelSuspended('run', state, agentId),
async () => await checkpoints.delete('run', agentId),
async () =>
await memory.saveMessages({
threadId: 'protected',
resourceId: 'user',
messages: [{ ...message, content: [{ type: 'text', text: 'Late replacement' }] }],
}),
async () => await memory.deleteMessages([message.id]),
async () => await memory.deleteMessagesByThread('protected'),
async () => await memory.deleteThread('protected'),
];
for (const write of writes)
await expect(write()).rejects.toBeInstanceOf(AgentConversationLeaseLostError);
expect(await executionRepository.findOneByOrFail({ id: executionId })).toMatchObject({
status: 'running',
timeline: null,
});
expect(await executionRepository.countBy({ threadId: 'protected' })).toBe(1);
expect(await checkpoints.getStatus('run', agentId)).toEqual({
status: 'active',
checkpoint: state,
});
expect((await memory.getMessages('protected'))[0]).toMatchObject({
content: message.content,
});
});
});
it('keeps stream creation, later chunks, and delayed callbacks in their creating context', async () => {
const memory = Container.get(N8nMemory).getImplementation(agentId);
const streams = ['one', 'two'].map((threadId) =>
leases.stream(agentId, threadId, async function* () {
const owner = leases.requireOwner(threadId);
await memory.saveThread({ id: threadId, resourceId: 'user' });
yield owner.lease.ownerToken;
await Promise.resolve();
expect(leases.requireOwner(threadId)).toBe(owner);
await memory.saveMessages({
threadId,
resourceId: 'user',
messages: [
{
id: randomUUID(),
role: 'user',
content: [{ type: 'text', text: threadId }],
createdAt: new Date(),
},
],
});
}),
);
const first = await Promise.all(streams.map(async (stream) => await stream.next()));
expect(first[0].value).not.toBe(first[1].value);
expect(leases.currentOwner()).toBeUndefined();
await Promise.all(streams.map(async (stream) => await stream.next()));
expect(await Container.get(AgentMessageRepository).count()).toBeGreaterThanOrEqual(2);
const releaseCallback = createDeferredPromise();
let delayed: Promise<void> | undefined;
const oldStream = await leases.withLease(agentId, 'one', async () => {
delayed = releaseCallback.promise.then(
async () => await memory.deleteMessagesByThread('one'),
);
return leases.stream(agentId, 'one', async function* () {
yield 'Must not run';
});
});
await leases.withLease(agentId, 'one', async () => {
releaseCallback.resolve();
await expect(delayed).rejects.toBeInstanceOf(AgentConversationLeaseLostError);
await expect(oldStream.next()).rejects.toBeInstanceOf(AgentConversationLeaseLostError);
expect(await memory.getMessages('one')).toHaveLength(1);
});
});
it('lets a detached child finish after parent ownership ends', async () => {
const started = createDeferredPromise();
const finish = createDeferredPromise();
let child: Promise<void> | undefined;
await leases.withLease(agentId, 'parent', async () => {
const parent = leases.requireOwner('parent');
child = leases.outsideConversation(async () => {
expect(leases.currentOwner()).toBeUndefined();
await leases.withLease(agentId, 'child', async (signal) => {
const owner = leases.requireOwner('child');
started.resolve();
await finish.promise;
expect(parent.signal.aborted).toBe(true);
expect(signal.aborted).toBe(false);
expect(await leases.write(owner, async () => 'saved')).toBe('saved');
});
});
await started.promise;
});
finish.resolve();
await child;
});
it('stops renewal and later writes after a renewal failure', async () => {
vi.useFakeTimers({ toFake: ['setInterval', 'clearInterval'] });
const renewal = vi
.spyOn(repository, 'renew')
.mockRejectedValueOnce(new Error('Database unavailable'));
try {
await leases.withLease(agentId, 'renewal', async (signal) => {
await vi.advanceTimersByTimeAsync(30_000);
expect(signal.aborted).toBe(true);
await expect(
leases.write(leases.requireOwner('renewal'), async () => true),
).rejects.toBeInstanceOf(AgentConversationLeaseLostError);
await vi.advanceTimersByTimeAsync(30_000);
expect(renewal).toHaveBeenCalledOnce();
});
expect(await repository.isHeld('renewal')).toBe(true);
} finally {
renewal.mockRestore();
vi.useRealTimers();
}
});
it('uses the database clock when process clocks differ', async () => {
const clock = vi.spyOn(Date, 'now').mockReturnValue(0);
try {
const owner = await repository.acquire(agentId, 'clock');
if (!owner) throw new Error('Missing owner');
clock.mockReturnValue(9_000_000_000_000);
expect(await repository.renew(owner)).toBe(true);
expect(await repository.isHeld('clock')).toBe(true);
expect(await repository.acquire(agentId, 'clock')).toBeNull();
} finally {
clock.mockRestore();
}
});
async function waitForLeaseLock(dataSource: DataSource) {
await retryUntil(async () => {
const [row] = await dataSource.query<Array<{ waiting: boolean }>>(
`SELECT EXISTS (SELECT 1 FROM pg_stat_activity WHERE datname = current_database()
AND wait_event_type = 'Lock' AND query LIKE '%agent_conversation_lease%') AS waiting`,
);
expect(row.waiting).toBe(true);
});
}
it.runIf(process.env.DB_TYPE === 'postgresdb')(
'checks expiry after waiting for the ownership row',
async () => {
const dataSource = Container.get(DataSource);
const blocker = dataSource.createQueryRunner();
try {
await leases.withLease(agentId, 'blocked', async () => {
const owner = leases.requireOwner('blocked');
await blocker.startTransaction();
try {
await blocker.manager.update(AgentConversationLease, owner.lease, {
ownerToken: owner.lease.ownerToken,
});
const mutation = vi.fn(async () => undefined);
const write = leases.write(owner, mutation).catch((error: unknown) => error);
await waitForLeaseLock(dataSource);
// This timestamp is later than the waiting transaction's start time.
await blocker.manager.update(AgentConversationLease, owner.lease, {
expiresAt: () => 'clock_timestamp()',
});
await blocker.commitTransaction();
expect(await write).toBeInstanceOf(AgentConversationLeaseLostError);
expect(mutation).not.toHaveBeenCalled();
} finally {
if (blocker.isTransactionActive) await blocker.rollbackTransaction();
}
});
} finally {
await blocker.release();
}
},
);
it.runIf(process.env.DB_TYPE === 'postgresdb')(
'holds ownership until the protected mutation commits',
async () => {
const entered = createDeferredPromise();
const finish = createDeferredPromise();
const queue = Container.get(AgentMessageQueueRepository);
const entry = await queue.enqueue({
agentId,
threadId: 'atomic',
payload: {
source: 'preview',
kind: 'message',
projectId,
userId: 'user',
resourceId: 'user',
message: 'Start',
},
});
const writing = leases.withLease(agentId, 'atomic', async () => {
await leases.write(leases.requireOwner('atomic'), async (ctx) => {
entered.resolve();
await finish.promise;
expect(await queue.markProcessing(entry.id, ctx)).toBe(true);
});
});
await entered.promise;
const expiring = repository.update({ threadId: 'atomic' }, { expiresAt: new Date(0) });
try {
await waitForLeaseLock(Container.get(DataSource));
} finally {
finish.resolve();
await Promise.all([writing, expiring]);
}
const successor = await repository.acquire(agentId, 'atomic');
expect(successor).not.toBeNull();
expect(await queue.findOneByOrFail({ id: entry.id })).toMatchObject({ status: 'processing' });
},
);
});

122
packages/cli/test/integration/agents/agent-message-queue.test.ts
View File

@ -1,5 +1,9 @@
import { AgentConversationLeaseLostError } from '@/modules/agents/agent-conversation-lease.types';
import { createTeamProject, testDb, testModules } from '@n8n/backend-test-utils';
import { LockService, type Logger } from '@n8n/backend-common';
import type { Logger } from '@n8n/backend-common';
import { TransactionRunner } from '@n8n/db';
import { AgentConversationLeaseService } from '@/modules/agents/agent-conversation-lease.service';
import { AgentConversationLeaseRepository } from '@/modules/agents/repositories/agent-conversation-lease.repository';
import { Container } from '@n8n/di';
import { GlobalConfig } from '@n8n/config';
import type { InstanceSettings } from 'n8n-core';
@ -24,6 +28,7 @@ import { AgentExecutionService } from '@/modules/agents/agent-execution.service'
import { AgentMessageQueueService } from '@/modules/agents/agent-message-queue.service';
import type {
AgentQueueInput,
QueueExecutionContext,
IntegrationMessageQueuePayload,
} from '@/modules/agents/agent-message-queue.types';
import type { AgentChatBridge } from '@/modules/agents/integrations/agent-chat-bridge';
@ -36,7 +41,6 @@ import { AgentRepository } from '@/modules/agents/repositories/agent.repository'
import { Publisher } from '@/scaling/pubsub/publisher.service';
import { Subscriber } from '@/scaling/pubsub/subscriber.service';
import { PubSubEventBus } from '@/scaling/pubsub/pubsub.eventbus';
import { RedisLockService } from '@/scaling/redis-lock.service';
import { RedisClientService } from '@/services/redis-client.service';
import { retryUntil } from '../shared/retry-until';
@ -47,7 +51,6 @@ describe('agent message queue', () => {
let agentId: string;
let projectId: string;
let mains: AgentMessageQueueService[];
let locks: LockService;
let redisStack: Awaited<ReturnType<typeof createServiceStack>> | undefined;
let redisConfig: GlobalConfig | undefined;
let closeRedis: Array<() => void>;
@ -113,13 +116,9 @@ describe('agent message queue', () => {
const logger = mock<Logger>();
logger.scoped.mockReturnValue(logger);
let publisher: Publisher;
let consumerLocks = locks;
let eventBus: PubSubEventBus | undefined;
if (redisConfig) {
const clients = new RedisClientService(logger, redisConfig);
const redisLocks = new RedisLockService(logger, redisConfig, clients);
consumerLocks = new LockService(mock<ConstructorParameters<typeof LockService>[0]>());
consumerLocks.setProvider(redisLocks);
const instance = mock<InstanceSettings>({ hostId: randomUUID(), isWorker: false });
publisher = new Publisher(logger, clients, instance, redisConfig.executions, redisConfig);
eventBus = new PubSubEventBus();
@ -135,7 +134,6 @@ describe('agent message queue', () => {
closeRedis.push(() => {
publisher.shutdown();
subscriber.shutdown();
redisLocks.destroy();
});
} else {
const localPublisher = mock<Publisher>();
@ -149,7 +147,11 @@ describe('agent message queue', () => {
const main = new AgentMessageQueueService(
logger,
repository,
consumerLocks,
new AgentConversationLeaseService(
Container.get(AgentConversationLeaseRepository),
Container.get(TransactionRunner),
logger,
),
checkpoints,
integrations,
agents,
@ -165,6 +167,8 @@ describe('agent message queue', () => {
};
beforeAll(async () => {
// These races require independent database connections.
Container.get(GlobalConfig).database.postgresdb.poolSize = 4;
await testModules.loadModules(['agents']);
await testDb.init();
repository = Container.get(AgentMessageQueueRepository);
@ -181,6 +185,7 @@ describe('agent message queue', () => {
mock(),
mock(),
repository,
Container.get(AgentConversationLeaseService),
);
if (process.env.DB_TYPE === 'postgresdb') {
redisStack = await createServiceStack({
@ -214,7 +219,6 @@ describe('agent message queue', () => {
);
mains = [];
closeRedis = [];
locks = Container.get(LockService);
received.length = 0;
checkpoints.findSuspendedForThread.mockResolvedValue(null);
integrations.getBridge.mockReturnValue(bridge);
@ -357,11 +361,11 @@ describe('agent message queue', () => {
mock(),
mock(),
mock(),
mainB['lockService'],
mainB['leases'],
mainB,
);
const jobs = mock<AgentBackgroundJobRepository>();
jobs.findWakeableUnconsumedSettled.mockResolvedValue([]).mockResolvedValueOnce([
jobs.findWakeableUnconsumedSettled.mockResolvedValue([
mock<AgentBackgroundJob>({
id: 'job',
parentAgentId: agentId,
@ -398,7 +402,7 @@ describe('agent message queue', () => {
checkpoints,
registry,
orchestrator,
mainB['lockService'],
mainB['leases'],
mock(),
mock(),
mock({ backgroundTasksEnabled: true }),
@ -598,13 +602,12 @@ describe('agent message queue', () => {
it('does not start an input removed after selection and before its claim', async () => {
const selected = createDeferredPromise();
const finishClaim = createDeferredPromise();
const claim = repository.markProcessing.bind(repository);
const delayedClaim = vi
.spyOn(repository, 'markProcessing')
.mockImplementationOnce(async (id) => {
.spyOn(checkpoints, 'findSuspendedForThread')
.mockImplementationOnce(async () => {
selected.resolve();
await finishClaim.promise;
return await claim(id);
return null;
});
const main = await makeMain();
try {
@ -621,6 +624,67 @@ describe('agent message queue', () => {
}
});
it('uses an edit saved after selection and before the fenced claim', async () => {
const selected = createDeferredPromise();
const finish = createDeferredPromise();
checkpoints.findSuspendedForThread.mockImplementationOnce(async () => {
selected.resolve();
await finish.promise;
return null;
});
const main = await makeMain();
const id = await main.enqueue(message('Original'));
try {
await selected.promise;
const saved = await repository.update(
{ id, status: 'queued' },
{ payload: message('Edited').payload as never },
);
expect(saved.affected).toBe(1);
finish.resolve();
await main.drain('conversation');
expect(received).toEqual(['Edited']);
} finally {
finish.resolve();
}
});
it('rejects a former processor link, heartbeat, and cleanup without touching a successor', async () => {
const started = createDeferredPromise<QueueExecutionContext>();
const finish = createDeferredPromise();
bridge.processQueuedInput.mockImplementation(async (_payload, _threadId, context) => {
started.resolve(context);
await finish.promise;
});
const main = await makeMain();
const id = await main.enqueue(message('Old turn'));
const context = await started.promise;
const oldDrain = main['drains'].get('conversation')?.catch((error: unknown) => error);
const leaseRepository = Container.get(AgentConversationLeaseRepository);
try {
await leaseRepository.update({ threadId: 'conversation' }, { expiresAt: new Date(0) });
const successor = await leaseRepository.acquire(agentId, 'conversation');
expect(successor).not.toBeNull();
await repository.update({ id }, { updatedAt: new Date(0) });
await expect(context.onExecutionStarted(randomUUID())).rejects.toBeInstanceOf(
AgentConversationLeaseLostError,
);
await main['heartbeat']();
finish.resolve();
expect(await oldDrain).toBeInstanceOf(AgentConversationLeaseLostError);
expect(await repository.findOneByOrFail({ id })).toMatchObject({
status: 'processing',
executionId: null,
updatedAt: new Date(0),
});
expect(await leaseRepository.findOneByOrFail({ threadId: 'conversation' })).toMatchObject(
successor!,
);
} finally {
finish.resolve();
}
});
it('heartbeats live preview waiters in one batch so another main does not cancel them', async () => {
vi.useFakeTimers({ toFake: ['setInterval', 'clearInterval'] });
const controller = new AbortController();
@ -636,9 +700,7 @@ describe('agent message queue', () => {
await repository.update({}, { updatedAt: new Date(Date.now() - 180_000) });
const heartbeat = vi.spyOn(repository, 'touchLiveEntries');
await vi.advanceTimersByTimeAsync(30_000);
await retryUntil(async () =>
expect(await repository.findStaleThreads(new Date(Date.now() - 120_000))).toEqual([]),
);
await retryUntil(async () => expect(await repository.findStaleThreads(120_000)).toEqual([]));
expect(heartbeat.mock.calls.filter(([ids]) => ids.length > 0)).toHaveLength(1);
expect(heartbeat.mock.calls.find(([ids]) => ids.length > 0)?.[0]).toHaveLength(2);
await peer.recover();
@ -678,8 +740,8 @@ describe('agent message queue', () => {
);
ids[threadId] = execution.id;
const entry = await repository.enqueue(message('do not replay', threadId));
await repository.markProcessing(entry.id);
await repository.linkExecution(entry.id, execution.id);
await repository.markProcessing(entry.id, {});
await repository.linkExecution(entry.id, execution.id, {});
await repository.update(entry.id, { updatedAt: old });
}
const suspension = mock<SerializableAgentState>({ status: 'suspended' });
@ -692,7 +754,7 @@ describe('agent message queue', () => {
vi.spyOn(executions, 'finalizeInterruptedExecution').mockImplementation(
async (execution, staleBefore) => {
if (execution.threadId === 'refreshed') {
await executionRepository.touchRunning(execution.id);
await executionRepository.touchRunning(execution.id, {});
}
return await finalize(execution, staleBefore);
},
@ -722,7 +784,6 @@ describe('agent message queue', () => {
});
it('stops recovery when its conversation lease is lost', async () => {
const lease = new AbortController();
const stale = await repository.enqueue(
preview('lost', 'lost-preview', [
{ id: 'unused-file', fileName: 'note.txt', mimeType: 'text/plain', sizeBytes: 1 },
@ -730,25 +791,26 @@ describe('agent message queue', () => {
);
await repository.update(stale.id, { updatedAt: new Date(Date.now() - 180_000) });
const findStale = repository.findStale.bind(repository);
vi.spyOn(repository, 'findStale').mockImplementationOnce(async (...args) => {
const staleRead = vi.spyOn(repository, 'findStale').mockImplementation(async (...args) => {
const entries = await findStale(...args);
lease.abort();
await Container.get(AgentConversationLeaseRepository).update(
{ threadId: 'lost-preview' },
{ expiresAt: new Date(0) },
);
return entries;
});
const main = await makeMain();
vi.spyOn(main['lockService'], 'withLease').mockImplementationOnce(
async (_namespace, _key, run) => await run(lease.signal),
);
await main.recover();
expect(await repository.existsBy({ id: stale.id })).toBe(true);
expect(attachments.deleteByIds).not.toHaveBeenCalled();
staleRead.mockRestore();
});
it('recovers waiting integrations, abandons interrupted inputs, and keeps live preview waiters', async () => {
const abandoned = await repository.enqueue(message('interrupted'));
await repository.markProcessing(abandoned.id);
await repository.markProcessing(abandoned.id, {});
await repository.update(abandoned.id, { updatedAt: new Date(Date.now() - 180_000) });
await repository.enqueue(message('waiting'));
const lostPreview = await repository.enqueue(

5
packages/cli/test/integration/database/repositories/agent-background-job.repository.test.ts
View File

@ -1,4 +1,5 @@
import { LockService, type Logger } from '@n8n/backend-common';
import { AgentConversationLeaseService } from '@/modules/agents/agent-conversation-lease.service';
import { type Logger } from '@n8n/backend-common';
import { createTeamProject, testDb, testModules } from '@n8n/backend-test-utils';
import type { AgentsConfig } from '@n8n/config';
import type { UserRepository } from '@n8n/db';
@ -220,7 +221,7 @@ describe('AgentBackgroundJobRepository', () => {
firstWakeStarted();
throw new Error('model unavailable');
});
const lockService = Container.get(LockService);
const lockService = Container.get(AgentConversationLeaseService);
const publisher = mock<Publisher>();
const agentsConfig = mock<AgentsConfig>({ backgroundTasksEnabled: true });
const logger = mock<Logger>();

14
packages/cli/test/migration/1789499956927-create-agent-message-queue.test.ts
View File

@ -45,6 +45,17 @@ describe('CreateAgentMessageQueue migration', () => {
VALUES (:agentId, 'Agent', :projectId, '[]', '{}', '{}', :now, :now)`,
{ agentId, projectId, now },
);
const leaseTable = escape.tableName('agent_conversation_lease');
await runQuery(
`INSERT INTO ${leaseTable} ("threadId", "agentId", "ownerToken", "expiresAt")
VALUES ('future-thread', :agentId, :ownerToken, :now)`,
{ agentId, ownerToken: randomUUID(), now },
);
const leaseSchema = await queryRunner.getTable(`${tablePrefix}agent_conversation_lease`);
expect(leaseSchema?.primaryColumns.map(({ name }) => name)).toEqual(['threadId']);
expect(
leaseSchema?.foreignKeys.map(({ columnNames, onDelete }) => ({ columnNames, onDelete })),
).toEqual([{ columnNames: ['agentId'], onDelete: 'CASCADE' }]);
const table = escape.tableName('agent_message_queue');
for (const text of ['one', 'two']) {
await runQuery(
@ -91,16 +102,19 @@ describe('CreateAgentMessageQueue migration', () => {
agentId,
});
expect(await runQuery(`SELECT "id" FROM ${table}`)).toEqual([]);
expect(await runQuery(`SELECT "threadId" FROM ${leaseTable}`)).toEqual([]);
});
await undoLastSingleMigration();
await withContext(async ({ queryRunner, tablePrefix }) => {
expect(await queryRunner.hasTable(`${tablePrefix}agent_message_queue`)).toBe(false);
expect(await queryRunner.hasTable(`${tablePrefix}agent_conversation_lease`)).toBe(false);
expect(await queryRunner.hasTable(`${tablePrefix}agents`)).toBe(true);
});
await runSingleMigration(MIGRATION);
await withContext(async ({ queryRunner, tablePrefix }) => {
expect(await queryRunner.hasTable(`${tablePrefix}agent_message_queue`)).toBe(true);
expect(await queryRunner.hasTable(`${tablePrefix}agent_conversation_lease`)).toBe(true);
});
});
});