mirror of
https://github.com/n8n-io/n8n.git
synced 2026-10-21 20:20:03 +02:00
Compare commits
8 Commits
master
...
n8n@2.44.3
| Author | SHA1 | Date | |
|---|---|---|---|
|
bfbeea9c35 | ||
|
|
c09e0a94f5 | ||
|
|
6525584709 | ||
|
|
c52f1817b3 | ||
|
|
16b49eb412 | ||
|
|
89f18b23b8 | ||
|
|
321b0203b7 | ||
|
|
c25417c88e |
|
|
@ -110,7 +110,6 @@ Run through this before requesting review. Each item is a real, recurring review
|
|||
- [ ] Identifiers go through **`escape.tableName(...)` / `escape.columnName(...)`**. Never hand-write `n8n_table` prefixes. — [Always escape identifiers](#always-escape-identifiers)
|
||||
- [ ] **Match column type to value semantics.** Native `uuid` for UUIDs, `timestampTimezone()` for timestamps, a numeric type for numbers, `bool` for booleans, `json` for structured data. Never `varchar` as a catch-all. — [Column types](#column-types)
|
||||
- [ ] **Pick the narrowest sane type within that category:** `int`/`smallint` not `bigint` when range allows; `text` not `varchar(255)` for unbounded strings; never `double` for version numbers. — [Column types](#column-types)
|
||||
- [ ] **A generated `id` key is an integer** (`int.primary.autoGenerate2`). A UUID key is a UUIDv7. For a UUID or string key, write the reason in the PR description. A join table uses its foreign-key columns as the key, not an `id` column. — [Primary Keys](#primary-keys)
|
||||
- [ ] **Default `notNull`**, relax only when justified. PK is implicitly NOT NULL. Migration's `notNull` matches the entity's nullability. — [NOT NULL and entity parity](#not-null-and-entity-parity)
|
||||
- [ ] **Enum-like columns** carry `.withEnumCheck([...])` AND `.comment('explains values')`. Opaque IDs / unix timestamps / JSON shapes also get `.comment()`. — [Constrain enum-like strings](#constrain-enum-like-strings), [Add comments on columns](#add-comments-on-columns)
|
||||
- [ ] **Every reference column has an explicit FK** with deliberate `onDelete`. Name FKs explicitly when SQLite recreate cycles risk duplicating them. Avoid polymorphic `(typeCol, idCol)` patterns. — [Foreign Key Constraints](#foreign-key-constraints), [General Design Guidance](#general-design-guidance)
|
||||
|
|
@ -440,32 +439,11 @@ Don't set the default of a `status` column to a terminal value — `"running"` m
|
|||
|
||||
### Primary Keys
|
||||
|
||||
Every table needs a primary key. **Use an integer key by default.** Read the options below in order. Use the first option that fits. Do not skip an option only because you like a later one better.
|
||||
Every table needs a primary key. Choose the type in this order:
|
||||
|
||||
1. **Integer (default)** — `column('id').int.primary.autoGenerate2`. Use this for every new table. Use a different option only when one of its conditions is true. Integer keys are small and make joins fast. `autoGenerate2` uses Postgres `IDENTITY`. Do not use `autoGenerate`. It uses `serial`, which is deprecated.
|
||||
2. **UUID** — `column('id').uuid.primary`. Use this only when one of these conditions is true:
|
||||
- The application must know the ID before the insert. For example, the client makes the ID.
|
||||
- The ID is in a URL or a public API. Users must not see how many rows exist or guess other IDs.
|
||||
|
||||
**You must use UUIDv7.** Make the ID in application code with the `uuid` package: `import { v7 as uuidv7 } from 'uuid'`.
|
||||
|
||||
Why: a UUIDv7 starts with a timestamp. Thus, new rows go near the end of the index. A UUIDv4 is random, for example `randomUUID()` from `node:crypto`. New rows go to random positions in the index. This splits index pages and makes the index larger.
|
||||
|
||||
A UUIDv7 does not give an exact creation order. Two IDs made in the same millisecond can sort in the wrong order. Clock differences between hosts can also cause this. If you need the exact order, sort by a timestamp column.
|
||||
|
||||
Do **not** chain `.autoGenerate2` on `.uuid`. The DSL throws an error. The database default makes a UUIDv4. Also, `uuid_generate_v4()` needs the `uuid-ossp` extension. Some managed Postgres services, such as Supabase, do not have it. Use `.uuid`, not `.varchar(36)`.
|
||||
3. **String** — `column('id').varchar(36).primary`. Use this only when one of these conditions is true:
|
||||
- The entity already uses IDs in a format that is not a UUID. For example, the public API or the UI already shows nanoid IDs for this entity.
|
||||
- The UI shows the ID to users. A UUID is too long to read, copy, or type.
|
||||
|
||||
Entity IDs are usually nanoids with 16 characters.
|
||||
|
||||
**Give a reason for every key that is not an integer.** Write the reason in the PR description. Reviewers will ask for it. These reasons are not sufficient:
|
||||
- "Other tables use string IDs."
|
||||
- "This module already does it this way."
|
||||
- "We might need it later."
|
||||
|
||||
**These options apply to a generated `id` column only.** A join table that only links other tables does not need an `id` column. Use its foreign-key columns as a composite key (see below). A foreign-key column always has the same type as the key that it refers to.
|
||||
1. **Integer** — `column('id').int.primary.autoGenerate2`. Preferred for new tables: compact, fast joins, no ordering surprises. `autoGenerate2` uses Postgres `IDENTITY` (preferred over the deprecated `serial`-based `autoGenerate`).
|
||||
2. **UUID** — `column('id').uuid.primary`. Use when IDs are generated client-side, exposed in URLs, or need to be unguessable. Generate UUIDs in application code via `randomUUID()` from `node:crypto`; do **not** chain `.autoGenerate2` on `.uuid` (the DSL throws — `DEFAULT uuid_generate_v4()` fails on managed Postgres like Supabase because it needs the `uuid-ossp` extension in `public`). Use `.uuid` instead of `.varchar(36)`.
|
||||
3. **String** — `column('id').varchar(36).primary` for IDs whose format isn't a UUID (e.g. nanoid-style IDs). Convention: nanoid length 16 for entity IDs.
|
||||
|
||||
**Keep ID-column types consistent across related tables.** Mixing `uuid` and `varchar(36)` for what is "the same kind of ID" creates JOIN footguns.
|
||||
|
||||
|
|
@ -718,7 +696,7 @@ If only Postgres needs the change, just put the file in `postgresdb/`; don't wri
|
|||
|
||||
### Postgres-version-aware UUID generation
|
||||
|
||||
`gen_random_uuid()` requires Postgres ≥ 13. n8n dropped Postgres 12 — prefer it over `uuid_generate_v4()` (which needs the `uuid-ossp` extension and breaks on managed services like Supabase). For UUID PKs, make a UUIDv7 in application code — see [Primary Keys](#primary-keys).
|
||||
`gen_random_uuid()` requires Postgres ≥ 13. n8n dropped Postgres 12 — prefer it over `uuid_generate_v4()` (which needs the `uuid-ossp` extension and breaks on managed services like Supabase). For UUID PKs, generate at the application level with `randomUUID()` — see [Primary Keys](#primary-keys).
|
||||
|
||||
### SQLite doesn't enforce `varchar(N)` length
|
||||
|
||||
|
|
|
|||
|
|
@ -1,158 +0,0 @@
|
|||
---
|
||||
name: n8n:env-var-docs
|
||||
description: >-
|
||||
Writes and reviews the JSDoc on `@Env(...)` config properties. The docs team
|
||||
extracts these comments from source into the public environment variable
|
||||
reference, so they are user documentation, not code comments. Use when adding
|
||||
or changing an `@Env` property, a config class, or reviewing a PR that touches
|
||||
one.
|
||||
---
|
||||
|
||||
# Environment variable docs
|
||||
|
||||
The JSDoc above every `@Env('...')` property is exported almost verbatim into
|
||||
the [environment variables reference](https://docs.n8n.io/deploy/host-n8n/configure-n8n/basic-configuration/use-environment-variables)
|
||||
on docs.n8n.io. The generated table has four columns: variable name,
|
||||
description, data type, default. Your JSDoc is the description column. The
|
||||
reader is a self-hosting operator, not an n8n developer.
|
||||
|
||||
Follow the [n8n docs style guide](https://docs.n8n.io/contribute/contribution-guide-for-n8n-docs/style-guide-for-n8n-docs)
|
||||
for these comments. It overrides the ASD-STE100 rule in `AGENTS.md` for this
|
||||
one case. The rules below are the parts of that guide that matter for a
|
||||
one-paragraph description.
|
||||
|
||||
## Rules
|
||||
|
||||
- **Every `@Env` property has a JSDoc.** No exceptions. A missing comment
|
||||
ships as an empty description cell.
|
||||
- **Write for the operator.** Say what the setting does, when to change it,
|
||||
and what happens when it's unset or wrong. Don't explain why the code is
|
||||
written the way it is.
|
||||
- **No code references.** No file names, class names, function names,
|
||||
constants, license keys, or other internals the reader can't open. Values
|
||||
the operator types into the variable are fine and often needed, for example
|
||||
a credential type ID like `googleOAuth2Api` or a mode like `postgresdb`.
|
||||
The test: would the operator paste this into a config file, or would they
|
||||
need the source to know what it means? Details for developers go in a
|
||||
regular `//` comment on the implementation, not in the JSDoc.
|
||||
- **No attack scenarios.** Say what the setting protects, not how an
|
||||
attacker would exploit it. This is a public page.
|
||||
- **Don't repeat the default.** The default renders in its own column. Don't
|
||||
write "Defaults to true" or "Default: 1 hour". Do say what the unset state
|
||||
means when it's not obvious, for example "When unset, n8n uses the Redis
|
||||
host".
|
||||
- **State units and formats.** Say "in milliseconds", "in bytes", "a
|
||||
comma-separated list", "a JSON object". The data type column only says
|
||||
`Number` or `String`.
|
||||
- **Use allowed values when the set is small.** "One of `sqlite`,
|
||||
`postgresdb`." For a list, show one inline example in the sentence.
|
||||
- **No JSDoc tags.** No `@example`, `@default`, `@see`, `@deprecated`. The
|
||||
exporter copies the tag name into the docs. Put the example or the
|
||||
deprecation notice in a sentence instead.
|
||||
- **Product names:** "Gateway credits", not "n8n Connect" or "AI Gateway".
|
||||
"Publish a workflow", not "activate". `n8n` is always lowercase.
|
||||
- **Style:** present tense, active voice, second person. Use contractions.
|
||||
No em dashes, use a comma or a new sentence. No "e.g." or "i.e.", write
|
||||
"for example" and "that is". Spell out zero to nine, numerals from 10.
|
||||
Values the operator types stay numerals, for example "Set to 0".
|
||||
Sentence case. One idea per sentence. End every sentence with a period.
|
||||
- **Plain words:** "use" not "utilize", "lets you" not "allows you to", cut
|
||||
"powerful", "seamless", "simply", "it's important to note".
|
||||
- **Length:** one to three sentences. If the setting needs more, the extra
|
||||
belongs in a docs page the comment can't link to yet; keep the comment to
|
||||
what fits a table cell.
|
||||
|
||||
## Examples
|
||||
|
||||
Bad, as found in the generated docs:
|
||||
|
||||
```ts
|
||||
/**
|
||||
* Instance-level enablement for n8n Connect (AI Gateway). Licensed instances
|
||||
* are enabled by default; false opts out. Budget / free allowance still comes
|
||||
* from the license (`quota:aiGatewayBudget`).
|
||||
*/
|
||||
```
|
||||
|
||||
Wrong product name, a restated default, and an internal license key. Good:
|
||||
|
||||
```ts
|
||||
/**
|
||||
* Whether Gateway credits are available on this instance. Set to false to
|
||||
* turn them off on a licensed instance. The credit allowance comes from your
|
||||
* license.
|
||||
*/
|
||||
```
|
||||
|
||||
Bad:
|
||||
|
||||
```ts
|
||||
/**
|
||||
* Time to live in milliseconds for Redis cache entries. Set to 0 to disable
|
||||
* expiry. Default: 1 hour.
|
||||
*/
|
||||
```
|
||||
|
||||
The default is already in the table. Good:
|
||||
|
||||
```ts
|
||||
/**
|
||||
* Time to live in milliseconds for Redis cache entries. Set to 0 to keep
|
||||
* entries until Redis evicts them.
|
||||
*/
|
||||
```
|
||||
|
||||
Bad:
|
||||
|
||||
```ts
|
||||
/**
|
||||
* Comma-separated list of credential types for which overwrites are skipped
|
||||
* when the credential has been customized.
|
||||
* @example `N8N_SKIP_CREDENTIAL_OVERWRITE=zohoOAuth2Api,salesforceOAuth2Api`
|
||||
*/
|
||||
```
|
||||
|
||||
The `@example` tag lands in the description as literal text. Good:
|
||||
|
||||
```ts
|
||||
/**
|
||||
* Comma-separated list of credential types to skip when applying credential
|
||||
* overwrites, for example `zohoOAuth2Api,salesforceOAuth2Api`. n8n skips an
|
||||
* overwrite only when a user has already changed one of its fields.
|
||||
*/
|
||||
```
|
||||
|
||||
Bad:
|
||||
|
||||
```ts
|
||||
/**
|
||||
* Bind OAuth credential callbacks to the browser that initiated the flow.
|
||||
* When enabled, a session-scoped `n8n-oauth-binding` cookie is set on flow
|
||||
* initiation and verified on callback. Defeats phishing attacks where an
|
||||
* attacker injects an OAuth authorize URL into a victim's browser to land
|
||||
* tokens on the attacker's credential.
|
||||
*/
|
||||
```
|
||||
|
||||
Attack narrative and an internal cookie name on a public page. Good:
|
||||
|
||||
```ts
|
||||
/**
|
||||
* Whether an OAuth credential connection must finish in the same browser
|
||||
* that started it. When enabled, n8n rejects the callback if it arrives
|
||||
* from a different browser.
|
||||
*/
|
||||
```
|
||||
|
||||
## Review checklist
|
||||
|
||||
When you review a PR that adds or changes an `@Env` property, check each
|
||||
comment against this list and quote the fix in the review:
|
||||
|
||||
1. JSDoc present.
|
||||
2. No code identifiers, file names, or internal keys.
|
||||
3. No default restated.
|
||||
4. Units, format, or allowed values stated.
|
||||
5. No JSDoc tags.
|
||||
6. No em dashes, Latin abbreviations, or filler words.
|
||||
7. Reads correctly as a sentence in a table cell with no surrounding context.
|
||||
|
|
@ -21,7 +21,8 @@ check the code, not license to drop a team default.
|
|||
## Non-negotiable rules
|
||||
|
||||
- New endpoints are `@PublicApiController` classes under `v1/controllers/`, one
|
||||
`*.public.controller.ts` per feature.
|
||||
`*.public.controller.ts` per feature. A controller is a class — never
|
||||
`export =` (the legacy tuple style; `require-public-api-controller` flags it).
|
||||
- Public API and internal REST are separate HTTP surfaces. A public controller
|
||||
never calls an internal controller/endpoint; both reuse the same service.
|
||||
- Controllers and handlers delegate to a service — never import a repository or
|
||||
|
|
@ -30,6 +31,7 @@ check the code, not license to drop a team default.
|
|||
`@ApiResponse(Dto)`.
|
||||
- Register each controller via a side-effect import in `v1/controllers/index.ts`
|
||||
(`public-api-controllers.test.ts` fails otherwise).
|
||||
- Don't add business logic to legacy `express-openapi-validator` (EOV) handlers.
|
||||
- Migrating a legacy endpoint must not change its public contract.
|
||||
|
||||
These are `n8n-local-rules` ESLint rules (see `packages/cli/eslint.config.mjs`)
|
||||
|
|
@ -198,5 +200,6 @@ existing tests.
|
|||
- [Test-before-save endpoints](reference.md#test-before-save-endpoints)
|
||||
- [Errors](reference.md#errors)
|
||||
- [Testing matrix](reference.md#testing-matrix)
|
||||
- [Migrating legacy EOV endpoints](reference.md#migrating-legacy-eov-endpoints)
|
||||
- [Verifying a migration](reference.md#verifying-a-migration)
|
||||
- [CI and merging](reference.md#ci-and-merging)
|
||||
|
|
|
|||
|
|
@ -122,10 +122,11 @@ app-wide `bodyParser` before the registry sees it. Declaring
|
|||
- An unknown field (one the DTO doesn't declare, on a `{ strict: true }` DTO)
|
||||
fails with `Unexpected form field "<name>"` — not the default "unrecognized
|
||||
keys" wording.
|
||||
- Multer parsing errors map to these statuses: `413` for a size or count limit,
|
||||
`500` for `LIMIT_UNEXPECTED_FILE`, `400` for any other multer error, and `500`
|
||||
for anything else (a malformed body). The `500` is not masked, because it is a
|
||||
`ResponseError`. A missing boundary gives `400` with `multipart file(s) required`.
|
||||
- Multer's own parsing errors map to the same statuses express-openapi-validator
|
||||
used: `413` for a size/count limit, `400` for any other multer error, `500`
|
||||
— unmasked, since this is a `ResponseError` — for anything else (a
|
||||
malformed body; a missing boundary gives `400` with `multipart file(s)
|
||||
required`).
|
||||
- The body is parsed **after** every auth/scope/license/quota gate and
|
||||
**before** controller/route middlewares — a caller those gates would reject
|
||||
never has their (possibly huge) body read off the socket, and a middleware
|
||||
|
|
@ -174,6 +175,58 @@ example an `application/gzip` stream. The framework treats options whose
|
|||
The runtime part is `sendBinaryResponse` in
|
||||
`packages/cli/src/public-api/media-types/binary-response.ts`.
|
||||
|
||||
## Migrating legacy EOV endpoints
|
||||
|
||||
Legacy `express-openapi-validator` endpoints live under
|
||||
`v1/handlers/`, wired through `openapi.yml` with `x-eov-operation-*` and request
|
||||
types in `packages/cli/src/public-api/types.ts`. Treat these as migration targets,
|
||||
not templates.
|
||||
|
||||
- Prefer migrating to `@PublicApiController` over extending the handler.
|
||||
- Preserve the public contract: path, method, scopes, status codes, response
|
||||
shape, and pagination.
|
||||
- Move HTTP concerns into the controller and business orchestration into the
|
||||
shared service; keep the controller a thin HTTP boundary. If the legacy
|
||||
handler was itself already just a thin wrapper around an internal
|
||||
`@RestController` (calling its methods directly, e.g.
|
||||
`Container.get(SomeController).createThing(req, res, payload)`), the new
|
||||
public controller can call that same internal controller directly — no need
|
||||
to duplicate its validation/business logic.
|
||||
- A route must be served by either the EOV handler or a controller, not both —
|
||||
the build's `mergeDecoratorDocument` (`v1/openapi-gen/generate.ts`) throws on
|
||||
a path+method declared by both sides. Remove the legacy wiring (its path's
|
||||
`$ref` entry in `openapi.yml`, the `x-eov-operation-*` handler, and its
|
||||
`handler.ts`) only after the new controller is registered, `pnpm build`
|
||||
regenerates the spec cleanly, and tests are updated.
|
||||
- Fully delete the migrated legacy files: the handler's `.ts`, its
|
||||
`spec/paths/*.yml` and `spec/schemas/*.yml`, and any now-dead request type
|
||||
in `packages/cli/src/public-api/types.ts`. Then check
|
||||
`v1/shared/spec/schemas/_index.yml` and `v1/shared/spec/parameters/_index.yml`
|
||||
for entries that `$ref` one of the deleted schema/parameter files — those are
|
||||
separate from the path's own `$ref` in `openapi.yml` and are easy to miss;
|
||||
left dangling, the next bundle fails on a broken `$ref`.
|
||||
- If the legacy handler gated on a license (`isLicensed('feat:x')` middleware),
|
||||
`@Licensed('feat:x')` now replicates that for a controller route (see the
|
||||
decorator table in [SKILL.md](SKILL.md#declaring-a-controller)) — but only for
|
||||
a single feature. If the legacy check was an any-of/all-of over several flags
|
||||
(e.g. `LicenseState.isProvisioningLicensed()`), `@Licensed` can't express
|
||||
that; replicate it manually in the controller instead, don't drop it - this
|
||||
is exactly what the internal `provisioning.controller.ee.ts` and
|
||||
`role-mapping-rule.controller.ee.ts` already do, since neither uses
|
||||
`@Licensed` for that reason.
|
||||
- As a legacy file drops repository access / the `export =` tuple, remove its
|
||||
entry from the `off` allowlists for `no-repository-in-public-api-handler` and
|
||||
`require-public-api-controller` in `packages/cli/eslint.config.mjs` (shrink-only
|
||||
— never extend them).
|
||||
- A `multipart/form-data` legacy endpoint migrates onto `@Body({ mediaType:
|
||||
'multipart/form-data', uploadLimits })` — see
|
||||
[Request body media types](#request-body-media-types). For any other
|
||||
non-standard endpoint, study the nearest existing handler first.
|
||||
- Keep each field in its original position when you extract a request shape shared
|
||||
by two routes, and destructure out the ones a route doesn't take. The generator
|
||||
emits properties in shape order, so a moved field rewrites the `*.generated.yml`
|
||||
of a route the PR wasn't changing.
|
||||
|
||||
## Verifying a migration
|
||||
|
||||
Tests are written against the new code, so they can't show that the old behavior
|
||||
|
|
|
|||
|
|
@ -1,464 +0,0 @@
|
|||
---
|
||||
name: n8n:testmax-assistant-ui
|
||||
description: >-
|
||||
Tests an n8n Assistant (Instance AI) UI feature end to end. Drafts the test
|
||||
paths from the feature and the diff, writes one Playwright script for each
|
||||
path, runs the paths in parallel on fresh n8n instances with a real LLM, and
|
||||
compares the main Assistant screens with master to find visual
|
||||
regressions. Shows live screenshots during the run, and builds one HTML
|
||||
report with video replays, step screenshots, LLM steps, tokens, cost and
|
||||
visual diffs. Use when the user wants to test an Instance AI or Assistant
|
||||
UI feature end to end, check that a UI change does not break existing
|
||||
screens, or says /testmax-assistant-ui.
|
||||
---
|
||||
|
||||
# Test an Instance AI feature
|
||||
|
||||
Use this skill to test a UI feature of the n8n Assistant end to end. In code,
|
||||
the Assistant is "Instance AI". You draft the test paths, the user approves
|
||||
them, and the skill runs them all at the same time. A local page shows each run
|
||||
live. At the end, you judge each run and build one report.
|
||||
|
||||
The skill uses the scripts of `testmax-n8n-ai` for the n8n API, the
|
||||
LangSmith usage, the live page server, and the report template. Do not copy
|
||||
those scripts.
|
||||
|
||||
## What the skill does
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[Feature + diff] --> B[Draft scenarios.json<br/>and scripts]
|
||||
B --> T[--try and --estimate]
|
||||
T --> C{User approves<br/>plan + cost}
|
||||
C --> D[run.py]
|
||||
D --> E[One fresh branch instance<br/>for each feature + regression]
|
||||
D --> F[Fresh branch instance<br/>visual]
|
||||
D --> G[Fresh master instance<br/>visual, or a kept run]
|
||||
E & F & G --> H[Live page]
|
||||
F & G --> I[Pixel + accessibility tree diff]
|
||||
E & F & G --> J[summary.txt with triage]
|
||||
J --> K[You judge: verdicts.json]
|
||||
K --> L[Report rebuilds itself]
|
||||
```
|
||||
|
||||
- Each run is one scenario: a Playwright script that drives the real
|
||||
Assistant UI in Chrome, with a real LLM.
|
||||
- There are three kinds of scenario:
|
||||
- `feature`: tests a path of the new feature.
|
||||
- `regression`: tests an existing flow that the feature touches.
|
||||
- `visual`: captures screens. The skill runs it on the branch and on master,
|
||||
then compares the captures.
|
||||
- All instances are fresh: a new data folder, a new owner, and no workflows or
|
||||
threads. Thus the screens of the branch and master are the same when the
|
||||
code is the same. A fresh instance starts in about 5 seconds, because it
|
||||
copies a data folder that is already migrated and seeded.
|
||||
- Each feature and regression scenario gets its own fresh instance, so the
|
||||
scenarios do not see each other's data. The visual scenarios share one
|
||||
branch instance and one master instance.
|
||||
- The feature flags are the same on all instances. The instances do not ask
|
||||
PostHog for flags, so no instance gets a random experiment variant. Set the
|
||||
flags that the feature needs in the plan, with `featureFlags`.
|
||||
- `run.py` builds this checkout first (with turbo, so a build that is up to
|
||||
date takes about 1 second). The master instance starts from a git worktree
|
||||
at `~/.cache/n8n-feature-test/master`, at the merge base of `HEAD` and
|
||||
`origin/master`. Thus the visual check shows only the changes of this
|
||||
branch, not the changes that master got after the branch started.
|
||||
- A master visual run that passed is kept. The next run with the same merge
|
||||
base, the same script and the same flags uses it again, and does not start
|
||||
master.
|
||||
|
||||
## Requirements
|
||||
|
||||
Run `python3 .agents/skills/testmax-assistant-ui/scripts/run.py --check`
|
||||
before the first run. The commands below use `$S` for this scripts folder.
|
||||
`--check` checks the tools,
|
||||
Chrome, the settings, the disk space and the master build, and prints the
|
||||
plan folder for this branch.
|
||||
|
||||
- Google Chrome. The scripts use the Playwright package of the repository.
|
||||
- The Instance AI model settings in `.env.local` or `.env.ai-surfaces`, as for
|
||||
your own instance. Fresh instances get the same settings, without the
|
||||
license, the database, the port and the encryption key.
|
||||
- `LANGSMITH_API_KEY` and LangSmith tracing, for the tokens and cost. Without
|
||||
them, the report shows no tokens or cost. Do not set
|
||||
`N8N_DIAGNOSTICS_ENABLED=false`: then Instance AI sends no traces.
|
||||
- About 3 GB of disk space for the master worktree and its build.
|
||||
|
||||
Never put a key on the command line. Read settings from the env files and the
|
||||
environment only.
|
||||
|
||||
## 1. Understand the feature
|
||||
|
||||
Find what changed and what the user wants to test:
|
||||
|
||||
```bash
|
||||
git diff origin/master...HEAD --stat -- packages/frontend packages/@n8n/instance-ai packages/cli/src/modules/instance-ai
|
||||
git diff origin/master...HEAD -- <the files of the feature>
|
||||
```
|
||||
|
||||
Read the changed Vue components, the stores, and the backend events that they
|
||||
use. Find the `data-test-id` values of the new elements. If the user did not
|
||||
describe the feature, ask for one or two sentences about it.
|
||||
|
||||
If the plan has visual scenarios, start the master baseline now, in the
|
||||
background. The first build takes 15 to 20 minutes. Later builds take less
|
||||
than a minute when the turbo cache has the files. You can draft the plan while
|
||||
it builds:
|
||||
|
||||
```bash
|
||||
S=.agents/skills/testmax-assistant-ui/scripts
|
||||
python3 $S/run.py --check # the setup, and the plan folder
|
||||
python3 $S/baseline.py prepare # in the background
|
||||
python3 $S/baseline.py status # later: is it ready?
|
||||
```
|
||||
|
||||
## 2. Draft the test paths
|
||||
|
||||
Write the plan folder that `--check` prints:
|
||||
`~/.cache/n8n-feature-test/plans/<branch>/<feature>/`. The folder stays after
|
||||
a reboot. It holds `scenarios.json` and one script for each scenario. The
|
||||
folder name is the name of the run folders.
|
||||
|
||||
Draft these paths:
|
||||
|
||||
- **Feature paths.** The main path, then each branch of it: other user
|
||||
choices, empty and long inputs, cancel and stop, a reload in the middle, a
|
||||
second thread, errors that the user can see. One path for each scenario.
|
||||
- **Regression paths.** The existing flows near the change: a plain chat
|
||||
reply, approvals, the plan review, the preview panel, the thread list. The
|
||||
library has `smoke-chat` for the plain chat reply.
|
||||
- **Visual screens.** The screens that the change can affect, on the branch
|
||||
and on master. Capture each screen in a stable state. Mask all text that
|
||||
comes from the LLM, and all data that changes between runs. The library has
|
||||
`visual-assistant-home` and `visual-thread-reply`.
|
||||
|
||||
Keep each scenario short and focused. A scenario that fails must show where it
|
||||
failed. Use more scenarios, not longer ones. A visual scenario must not need
|
||||
the new feature, because it also runs on master.
|
||||
|
||||
Some UI states show only for a moment, for example a status line that folds
|
||||
away when the run ends. Check them while the run is active, not after it.
|
||||
|
||||
If the feature is behind a feature flag or an experiment, set the flag in
|
||||
`featureFlags`. Without it, the instances use the default value of each flag.
|
||||
|
||||
Before you show the plan to the user, check the new scripts and get the
|
||||
estimate (see [Check the scripts](#check-the-scripts)). Then show the user the
|
||||
list of paths: the id, the kind, the title and the expected result of each.
|
||||
Also show the estimate: the number of LLM runs, the cost and the time. Change
|
||||
the plan until the user approves it.
|
||||
|
||||
### scenarios.json
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "Stop button in the Assistant",
|
||||
"feature": "The user can stop a run while the Assistant works.",
|
||||
"scenarios": [
|
||||
{
|
||||
"id": "stop-mid-run",
|
||||
"title": "Stop a run in the middle",
|
||||
"kind": "feature",
|
||||
"script": "stop-mid-run.mjs",
|
||||
"expected": "The stop button shows during the run. A click ends the run without an error message.",
|
||||
"timeoutMin": 6
|
||||
},
|
||||
{
|
||||
"id": "smoke-chat",
|
||||
"title": "A prompt gets a reply",
|
||||
"kind": "regression",
|
||||
"script": "lib:smoke-chat.mjs",
|
||||
"expected": "A short prompt gets the reply OK."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The plan has these fields: `title` (required), `feature` (the feature, for the
|
||||
judge), and `featureFlags` (optional). `featureFlags` maps a flag name to
|
||||
`true`, `false` or a variant name, for example
|
||||
`{ "<flag name>": "variant" }`. Each instance gets these flags, on the
|
||||
branch and on master. They go into `N8N_FEATURE_FLAG_OVERRIDES` and replace
|
||||
the values in your env files.
|
||||
|
||||
Each scenario has these fields:
|
||||
|
||||
| Field | Required | Description |
|
||||
|---|---|---|
|
||||
| `id` | yes | Lowercase letters, digits and `-`. No `--`. Unique. |
|
||||
| `title` | yes | A short name for the report. |
|
||||
| `kind` | yes | `feature`, `regression` or `visual`. |
|
||||
| `script` | yes | A path relative to `scenarios.json`, or `lib:<file>` for a library script. |
|
||||
| `expected` | yes | The expected result, for the judge. |
|
||||
| `autoConfirm` | no | Answer the Assistant's confirmations automatically. Default `true`. Credential and setup cards get "Later". Set `false` when the script answers a confirmation or a card. |
|
||||
| `isolated` | no | Run the scenario on its own fresh instance. Default `true`. Instances start ahead of their scenario, so this costs almost no time. Set `false` to run the scenario on one branch instance that it shares with the other `false` scenarios. Visual scenarios ignore this field. |
|
||||
| `timeoutMin` | no | The time limit of the scenario. Default 10. |
|
||||
| `viewport` | no | The window size, for example `1600x1000` (the default). |
|
||||
|
||||
`examples/stop-button/` is a full example plan.
|
||||
|
||||
### Scenario scripts
|
||||
|
||||
A script exports one async function. It gets the kit `t`:
|
||||
|
||||
```js
|
||||
export default async function (t) {
|
||||
await t.step('Open the Assistant', () => t.goto());
|
||||
await t.step('Send a prompt', () => t.send('Reply with exactly the word OK and nothing else.'));
|
||||
await t.step('Wait until the run ends', () => t.waitForIdle({ timeout: 180_000 }));
|
||||
await t.check('The reply says OK', () => t.expect(t.ai.assistantMessages().last()).toContainText('OK'));
|
||||
await t.capture('thread', { mask: [t.ai.assistantMessages()] });
|
||||
}
|
||||
```
|
||||
|
||||
| Kit member | What it does |
|
||||
|---|---|
|
||||
| `t.step(title, fn)` | Runs an action. Records a screenshot and the time. A failure stops the scenario. |
|
||||
| `t.check(title, fn)` | Runs an assertion, with a screenshot. A failure is recorded, and the scenario continues. |
|
||||
| `t.note(title)` | Adds a step with a screenshot and no action. |
|
||||
| `t.goto(path = '/assistant')` | Opens a page and waits for the chat input. Closes the "Try new n8n Assistant" dialog. |
|
||||
| `t.send(text)` | Types into the chat input, sends, and waits for the thread URL. Records the thread for the trace. |
|
||||
| `t.waitForIdle({timeout, settle})` | Waits until the thread has no active run. Returns `'idle'`, or `'suspended'` when the run waits for the user. |
|
||||
| `t.capture(name, {mask, maskText, root, fullPage})` | Saves a screen and its accessibility tree for the visual check. `mask` takes locators, and `maskText` takes texts: all elements with that text. Masked areas show as one color. In the tree, the subtree of a `mask` element and each masked text show as `[masked]`. |
|
||||
| `t.api(method, path, body)` | Calls the n8n REST API as the signed-in user. Returns `data`. |
|
||||
| `t.threadStatus(threadId)` | Returns `{active, suspended, background}`: the thread has a run, the run waits for the user, and the number of running background tasks. |
|
||||
| `t.ai.*` | Locators for the Assistant: `chatInput`, `sendButton`, `stopButton`, `userMessages`, `assistantMessages`, `assistantText(text)`, `panelText(text)`, `sidebarToggle`, `threadList`, `newThreadButton`, `approveButton`, `denyButton`, `planApproveButton`, `previewPanel`, `previewCanvasNodes`, and more in `scripts/kit.mjs`. |
|
||||
| `t.expect`, `t.page`, `t.baseUrl`, `t.sleep(ms)` | Playwright `expect`, the page, the instance URL, and a pause. |
|
||||
|
||||
Rules for scripts:
|
||||
|
||||
- Use `data-test-id` locators (`page.getByTestId`). Use text only for text that
|
||||
the test controls.
|
||||
- Use `t.expect(...).toBeVisible()` and other assertions that wait. Do not use
|
||||
`t.sleep` to wait for the UI.
|
||||
- LLM answers change from run to run. Check the UI state, not the exact words.
|
||||
Ask for a fixed reply, such as "Reply with exactly the word OK", when you
|
||||
need a known answer.
|
||||
- To test an approval, set `"autoConfirm": false`, wait with
|
||||
`t.waitForIdle()` until it returns `'suspended'`, then click the button.
|
||||
|
||||
### Check the scripts
|
||||
|
||||
Check the new scripts on fresh instances before you show the plan:
|
||||
|
||||
```bash
|
||||
python3 $S/run.py --plan <plan>/scenarios.json --try --only id1,id2
|
||||
```
|
||||
|
||||
`--try` does not run master, does not start the live page, and builds no
|
||||
report. It prints a table with the status, the steps and the first error of
|
||||
each run. The runs are in `/tmp/feature-test/try/`. Fix the scripts that fail
|
||||
because of the script, then try them again. Each LLM scenario in a try costs
|
||||
money too, so try only the new scripts.
|
||||
|
||||
Then get the estimate for the full run:
|
||||
|
||||
```bash
|
||||
python3 $S/run.py --plan <plan>/scenarios.json --estimate
|
||||
```
|
||||
|
||||
The estimate uses the cost and time of earlier runs of the same scenario ids.
|
||||
A scenario without an earlier run shows `*` and the median of all earlier
|
||||
runs. A master run that is kept shows `(kept)` and costs nothing.
|
||||
|
||||
## 3. Run
|
||||
|
||||
Open the live page for the user on each run. Do not skip this step. Do this:
|
||||
|
||||
1. Find out if your harness has a browser tool that opens a tab in the
|
||||
editor. In Cursor, this is `browser_navigate` of the `cursor-ide-browser`
|
||||
MCP server.
|
||||
2. If it has one, start the run in the background with `--no-open`:
|
||||
|
||||
```bash
|
||||
python3 $S/run.py --plan <plan>/scenarios.json --no-open
|
||||
```
|
||||
|
||||
Wait for the `live page: <url>` line in the output, which comes in a few
|
||||
seconds. Then open that URL in a new editor tab that the user can see (in
|
||||
Cursor: `newTab: true` and `position: "active"`). If the tab does not open,
|
||||
run `open <url>` (macOS) or `xdg-open <url>` (Linux).
|
||||
3. If it has none, start the run in the background without `--no-open`.
|
||||
`run.py` opens the live page in the default browser:
|
||||
|
||||
```bash
|
||||
python3 $S/run.py --plan <plan>/scenarios.json
|
||||
```
|
||||
|
||||
4. Give the user the URL in both cases.
|
||||
|
||||
`run.py` does not read `$BROWSER`, so do not set it.
|
||||
|
||||
`run.py` does these steps:
|
||||
|
||||
1. Prints the estimate and checks the setup. It stops when the setup has an
|
||||
error, for example a missing Chrome.
|
||||
2. Copies the plan into a new run folder,
|
||||
`/tmp/feature-test/<time>-<plan folder name>/`.
|
||||
3. Starts the live page and prints its URL. Without `--no-open`, it opens the
|
||||
page in the default browser.
|
||||
4. Builds this checkout with turbo. A build error stops the run, and the
|
||||
output shows the end of `build.log`.
|
||||
5. Starts the instances: one for each feature and regression scenario, one
|
||||
branch instance for the visual scenarios, and one master instance. It
|
||||
starts the instances ahead of their scenario. Master starts only for the
|
||||
master runs that it does not keep from an earlier run. It prepares the
|
||||
master build first, if it is not ready.
|
||||
6. Runs the scenarios, 4 at a time. Each run records a video, the steps with
|
||||
screenshots, the browser errors, the thread events, and the LangSmith ids.
|
||||
7. Stops the instances, compares the visual captures, writes `summary.txt`,
|
||||
and builds the report without verdicts.
|
||||
8. Prints a result table, the command to run the plan again, and one JSON
|
||||
object on several lines: `runDir`, `report`, `live`, `summary`, `results`
|
||||
(the status of each run), `visual` (the number of captures, the changed
|
||||
ones, and the error when the visual comparison failed), `rerun` and
|
||||
`next`.
|
||||
9. Gets the tokens and cost from LangSmith, then builds the report again.
|
||||
LangSmith receives traces with a delay, so this can take up to 90 seconds.
|
||||
You can start to judge while it runs.
|
||||
|
||||
| Option | Use |
|
||||
|---|---|
|
||||
| `--check` | Check the setup and print the plan folder, then stop. |
|
||||
| `--estimate` | Print the expected cost and time, then stop. |
|
||||
| `--try` | Check new scripts: no master, no live page, no report. See [Check the scripts](#check-the-scripts). |
|
||||
| `--only id1,id2` | Run only these scenarios. |
|
||||
| `--rerun last` | Run the plan of the last run again, in a new folder. |
|
||||
| `--rerun <run-dir>` | Run the plan of this earlier run again, in a new folder. If the plan folder of that run still exists, it uses the current plan and scripts from there. Thus a fixed script runs in its fixed version. |
|
||||
| `--concurrency N` | Runs at the same time. Default 4, or `FEATURE_TEST_CONCURRENCY`. |
|
||||
| `--branch-url URL` | Use a running instance for all branch runs, not fresh ones. The threads stay on that instance. The scenarios share its data, and the visual check is not reliable. |
|
||||
| `--no-build` | Do not build this checkout. The output warns when source files are newer than their build. |
|
||||
| `--no-baseline` | Do not run master. There is no visual check. |
|
||||
| `--no-cache` | Run master again, also when a kept master run exists. |
|
||||
| `--no-fetch` | Do not fetch `origin/master`. Use the merge base with the `origin/master` that git has. |
|
||||
| `--no-open` | Do not open the live page in the default browser. Use this when you open it in the editor tab. |
|
||||
| `--no-live` | Do not start the live page. Use this only when the user asks for it. |
|
||||
| `--keep-instances` | Keep the fresh instances running after the run, for a manual check. The output gives `instancePorts`. Each instance is at `http://localhost:<port>`. Stop them yourself. |
|
||||
| `--run-dir DIR` | Use this run folder. |
|
||||
|
||||
| Setting | Default |
|
||||
|---|---|
|
||||
| `FEATURE_TEST_RUNS_DIR` | `/tmp/feature-test` |
|
||||
| `FEATURE_TEST_CACHE` | `~/.cache/n8n-feature-test`: the plans, the master worktree, the kept master runs, and the instance data templates |
|
||||
| `FEATURE_TEST_CONCURRENCY` | `4` |
|
||||
|
||||
### Watch the runs
|
||||
|
||||
The live page shows each run as a row, with its status and its last
|
||||
screenshot. The screenshot changes every few seconds. When the run is done,
|
||||
the same address shows the report. The page stops 30 minutes after its last
|
||||
request.
|
||||
|
||||
## 4. Judge
|
||||
|
||||
Read `<runDir>/summary.txt`. It starts with a triage: each run is `clean`, or
|
||||
`look closer` with the reasons (a failed step or check, a browser error, a
|
||||
failed tool call, or a visual change). Then it has, for each run: the expected
|
||||
result, the steps and their status with the path of each screenshot, the
|
||||
prompts, the last reply, the browser errors, and the visual results.
|
||||
|
||||
For a clean run, look at the last screenshot and check it against the expected
|
||||
result. For a run to look at closer, look at the screenshots of the steps in
|
||||
the reasons, and the diff images in `visual/<id>/`.
|
||||
|
||||
A script that passed is not always a pass. Check that the run shows the
|
||||
expected result, not only that the script ended. Check for these problems:
|
||||
|
||||
- a browser error or a failed request that comes from the feature
|
||||
- a step that passed, but the screenshot shows the wrong state
|
||||
- a visual change that the feature did not intend
|
||||
- a script that failed because of the script, not the feature. Say so, and
|
||||
fix the script
|
||||
|
||||
Write `<runDir>/verdicts.json`. Use the run names as keys. A master run has
|
||||
the name `<id>--master`, and does not need a verdict:
|
||||
|
||||
```json
|
||||
{
|
||||
"stop-mid-run": {
|
||||
"verdict": "pass",
|
||||
"summary": "One sentence about the result.",
|
||||
"checks": [
|
||||
{ "ok": true, "text": "The stop button ends the run" },
|
||||
{ "ok": false, "text": "The send button comes back after 9 seconds" },
|
||||
{ "ok": null, "text": "Could not confirm the stopped label" }
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Use `pass` when all checks pass, `fail` when the main goal is not met, and
|
||||
`partial` for all other results. For a visual scenario, judge each change:
|
||||
intended by the feature, a regression, or noise (for example, a timestamp
|
||||
that is not masked). Be strict and factual.
|
||||
|
||||
Write each check as one short sentence with one fact. Put selectors, test
|
||||
IDs and UI text in backticks. The page shows them as code. The page shows
|
||||
the failed checks first.
|
||||
|
||||
## 5. The report
|
||||
|
||||
While the live page runs, it builds the report again each time
|
||||
`verdicts.json` changes. The open tab loads the new report by itself, and
|
||||
keeps the selected run and the scroll position. Thus you do not need to load
|
||||
the page again. When the live page does not run, build the report yourself:
|
||||
|
||||
```bash
|
||||
python3 $S/report.py <runDir>
|
||||
```
|
||||
|
||||
The report is `<runDir>/report/index.html`. It is one folder with all its
|
||||
files, so you can copy or share it. It has, from top to bottom:
|
||||
|
||||
- the video replays of all runs
|
||||
- a table of all runs: kind, result, steps passed, time, LLM calls, tool calls,
|
||||
tokens, cost and browser errors. The result is your verdict. A master run
|
||||
shows `Baseline`, and a run without a verdict shows `Not judged`, or
|
||||
`Failed` when its script failed
|
||||
- the visual check: the branch screen, the master screen, and the diff image
|
||||
of each capture, with the accessibility tree diff
|
||||
- for each run: the video of the run, the expected result and the verdict,
|
||||
the steps with their screenshots, the browser errors, and the LLM steps
|
||||
from the trace. A click on a row of the table shows the panel and the
|
||||
video of that run, and pauses the video of the other runs
|
||||
|
||||
Each box shows closed. It opens and closes when you click its title. On the
|
||||
live page, a box that you open or close stays that way when the page
|
||||
updates.
|
||||
|
||||
A master run that is kept from an earlier run shows "Reused" in its run box.
|
||||
|
||||
Give the user the path and the `live` URL. Tell the user the result of each
|
||||
scenario, the visual changes, and the problems that you found.
|
||||
|
||||
## Single steps
|
||||
|
||||
| Step | Command |
|
||||
|---|---|
|
||||
| Prepare the master baseline | `python3 $S/baseline.py prepare` |
|
||||
| Show the baseline state | `python3 $S/baseline.py status` |
|
||||
| Run one script on any instance | `N8N_BASE_URL=<url> node $S/run_scenario.mjs --out <dir> --script <file>` |
|
||||
| Compare the visual captures again | `node $S/visual_diff.mjs <runDir>` |
|
||||
| Print the judge summary | `python3 $S/report.py <runDir> --summary` |
|
||||
| Serve the live page of a run | `python3 $S/live_page.py --run-dir <runDir>` |
|
||||
|
||||
## Limits
|
||||
|
||||
- LLM answers change from run to run, and a run can fail because of the LLM.
|
||||
Run a failed scenario again with `--rerun last --only <id>` before you call
|
||||
it a bug.
|
||||
- A stopped LLM call does not show in LangSmith. The cost of a run with a stop
|
||||
includes only the calls that ended.
|
||||
- Each LLM scenario costs money. The tokens and cost show in the report.
|
||||
- Parallel runs share the CPU and the LLM rate limits. The times include the
|
||||
shared load.
|
||||
- The visual check compares pixels with a small tolerance. Fonts and
|
||||
animations can cause small changes. Masks hide the parts that change between
|
||||
runs. The accessibility tree diff shows changes in structure and text.
|
||||
- The worktree shows in `git worktree list`. To remove it, run
|
||||
`git worktree remove ~/.cache/n8n-feature-test/master`.
|
||||
- Kept master runs older than 14 days are deleted. The instance data
|
||||
templates in `~/.cache/n8n-feature-test/templates` change when the
|
||||
migrations change. You can delete old templates at any time.
|
||||
- A run folder is about 35 MB. The run folders are in `/tmp`, so the system
|
||||
deletes them on reboot.
|
||||
|
|
@ -1,38 +0,0 @@
|
|||
{
|
||||
"title": "Stop button in the Assistant",
|
||||
"feature": "The user can stop a run while the Assistant works. After the stop, the composer is ready and a new message gets a reply. The Assistant screens look the same as on master.",
|
||||
"scenarios": [
|
||||
{
|
||||
"id": "stop-mid-run",
|
||||
"title": "Stop a run in the middle",
|
||||
"kind": "feature",
|
||||
"script": "stop-mid-run.mjs",
|
||||
"expected": "The stop button shows during the run. A click ends the run without an error message. The send button comes back, and a follow-up message gets a reply.",
|
||||
"timeoutMin": 6
|
||||
},
|
||||
{
|
||||
"id": "smoke-chat",
|
||||
"title": "A prompt gets a reply",
|
||||
"kind": "regression",
|
||||
"script": "lib:smoke-chat.mjs",
|
||||
"expected": "A short prompt opens a new thread and gets the reply OK. The composer is ready after the reply.",
|
||||
"timeoutMin": 5
|
||||
},
|
||||
{
|
||||
"id": "assistant-home",
|
||||
"title": "Assistant home and chat history",
|
||||
"kind": "visual",
|
||||
"script": "lib:visual-assistant-home.mjs",
|
||||
"expected": "The home screen and the chat history popover look the same as on master.",
|
||||
"timeoutMin": 3
|
||||
},
|
||||
{
|
||||
"id": "thread-reply",
|
||||
"title": "Thread with one reply",
|
||||
"kind": "visual",
|
||||
"script": "lib:visual-thread-reply.mjs",
|
||||
"expected": "A thread with one short exchange looks the same as on master.",
|
||||
"timeoutMin": 5
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
// Feature: the user stops a run while the Assistant works, and can send a new message after it.
|
||||
export default async function (t) {
|
||||
await t.step('Open the Assistant', () => t.goto());
|
||||
await t.step('Send a prompt that takes a while', () =>
|
||||
t.send(
|
||||
'Explain step by step, in at least 30 numbered points, how to design a workflow that syncs ' +
|
||||
'contacts between two CRMs. Do not build anything.',
|
||||
),
|
||||
);
|
||||
await t.step('The stop button shows while the run is active', () =>
|
||||
t.expect(t.ai.stopButton()).toBeVisible({ timeout: 30_000 }),
|
||||
);
|
||||
await t.step('Click stop', () => t.ai.stopButton().click());
|
||||
await t.check('The stop button goes away', () =>
|
||||
t.expect(t.ai.stopButton()).toBeHidden({ timeout: 15_000 }),
|
||||
);
|
||||
await t.check('The send button comes back', () =>
|
||||
t.expect(t.ai.sendButton()).toBeVisible({ timeout: 15_000 }),
|
||||
);
|
||||
await t.step('Wait until the thread is idle', () => t.waitForIdle({ timeout: 60_000 }));
|
||||
await t.check('The thread shows no error message', async () => {
|
||||
await t.expect(t.ai.panelText(/something went wrong/i)).toHaveCount(0);
|
||||
});
|
||||
await t.step('Send a follow-up message', () =>
|
||||
t.send('Reply with exactly the word OK and nothing else.'),
|
||||
);
|
||||
await t.step('Wait for the follow-up reply', () => t.waitForIdle({ timeout: 120_000 }));
|
||||
await t.check('The follow-up gets a reply', () =>
|
||||
t.expect(t.ai.assistantMessages().last()).toContainText('OK'),
|
||||
);
|
||||
}
|
||||
|
|
@ -1,17 +0,0 @@
|
|||
// Regression: a prompt gets a reply in a new thread.
|
||||
export default async function (t) {
|
||||
await t.step('Open the Assistant', () => t.goto());
|
||||
await t.step('Send a short prompt', () =>
|
||||
t.send('Reply with exactly the word OK and nothing else.'),
|
||||
);
|
||||
await t.step('Wait until the run ends', () => t.waitForIdle({ timeout: 180_000 }));
|
||||
await t.check('The user message shows', () =>
|
||||
t.expect(t.ai.userMessages().first()).toBeVisible(),
|
||||
);
|
||||
await t.check('The reply says OK', () =>
|
||||
t.expect(t.ai.assistantMessages().last()).toContainText('OK'),
|
||||
);
|
||||
await t.check('The composer is ready for the next message', () =>
|
||||
t.expect(t.ai.sendButton()).toBeVisible(),
|
||||
);
|
||||
}
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
// Visual: the Assistant home screen, and the chat history popover.
|
||||
export default async function (t) {
|
||||
await t.step('Open the Assistant', () => t.goto());
|
||||
await t.capture('assistant-home');
|
||||
await t.step('Open the chat history', async () => {
|
||||
await t.ai.sidebarToggle().click();
|
||||
await t.expect(t.ai.threadList()).toBeVisible();
|
||||
});
|
||||
// Other visual scenarios create threads at the same time, so the list content changes between runs.
|
||||
await t.capture('chat-history', { mask: [t.ai.threadList()] });
|
||||
}
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
// Visual: a thread with one short exchange, and the composer after the reply.
|
||||
export default async function (t) {
|
||||
await t.step('Open the Assistant', () => t.goto());
|
||||
await t.step('Send a short prompt', () =>
|
||||
t.send('Reply with exactly the word OK and nothing else.'),
|
||||
);
|
||||
await t.step('Wait until the run ends', () => t.waitForIdle({ timeout: 180_000 }));
|
||||
// The reply text and the thread title come from the LLM. The title also shows in the main sidebar.
|
||||
const title = (await t.ai.sidebarToggle().innerText()).trim();
|
||||
await t.capture('thread-reply', { mask: [t.ai.assistantMessages()], maskText: [title] });
|
||||
}
|
||||
|
|
@ -1,117 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Keep a built copy of master for the visual check.
|
||||
|
||||
baseline.py status JSON: the worktree, the built commit, and the commit to build
|
||||
baseline.py prepare [--no-fetch] fetch origin/master, update the worktree, install, and build
|
||||
|
||||
The commit to build is the merge base of HEAD and origin/master: the master commit that this branch
|
||||
starts from. Then a visual change comes from the branch, not from newer commits on master. The
|
||||
build runs only when the merge base moves, for example after a rebase.
|
||||
|
||||
The worktree is in <FEATURE_TEST_CACHE>/master (default ~/.cache/n8n-feature-test/master). It is a
|
||||
detached git worktree of this repository, so `git worktree list` shows it. The first build takes 15
|
||||
to 20 minutes. The output goes to <FEATURE_TEST_CACHE>/build.log.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import fcntl
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from common import CACHE, git
|
||||
|
||||
WORKTREE = CACHE / 'master'
|
||||
STATE = CACHE / 'built.json'
|
||||
BUILD_LOG = CACHE / 'build.log'
|
||||
|
||||
|
||||
def built():
|
||||
try:
|
||||
state = json.loads(STATE.read_text())
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
cli = WORKTREE / 'packages' / 'cli' / 'dist'
|
||||
editor = WORKTREE / 'packages' / 'frontend' / 'editor-ui' / 'dist' / 'index.html'
|
||||
return state if cli.exists() and editor.exists() else None
|
||||
|
||||
|
||||
def target(fetch=False):
|
||||
"""The master commit to build: the merge base of HEAD and origin/master."""
|
||||
if fetch:
|
||||
git('fetch', '--quiet', 'origin', 'master')
|
||||
try:
|
||||
return git('merge-base', 'HEAD', 'origin/master')
|
||||
except subprocess.CalledProcessError:
|
||||
return git('rev-parse', 'origin/master')
|
||||
|
||||
|
||||
def status():
|
||||
state = built() or {}
|
||||
try:
|
||||
commit = target()
|
||||
except subprocess.CalledProcessError:
|
||||
commit = None
|
||||
return {'worktree': str(WORKTREE), 'exists': WORKTREE.exists(), 'builtCommit': state.get('commit'),
|
||||
'builtAt': state.get('builtAt'), 'baseCommit': commit,
|
||||
'upToDate': bool(commit) and state.get('commit') == commit, 'log': str(BUILD_LOG)}
|
||||
|
||||
|
||||
def run_logged(cmd, log, env=None):
|
||||
log.write(f'\n$ {" ".join(cmd)}\n')
|
||||
log.flush()
|
||||
code = subprocess.run(cmd, cwd=WORKTREE, stdout=log, stderr=subprocess.STDOUT, env=env).returncode
|
||||
if code != 0:
|
||||
raise RuntimeError(f'{" ".join(cmd)} failed with exit code {code}. See {BUILD_LOG}')
|
||||
|
||||
|
||||
def prepare(fetch=True, say=print):
|
||||
"""Return {"root", "commit"} of a built merge base. Builds only when the merge base moved."""
|
||||
CACHE.mkdir(parents=True, exist_ok=True)
|
||||
with (CACHE / 'build.lock').open('w') as lock:
|
||||
# Two runs at the same time must not build into the same worktree.
|
||||
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||
if fetch:
|
||||
say('fetch origin/master')
|
||||
commit = target(fetch)
|
||||
if not (WORKTREE / '.git').exists():
|
||||
say(f'create the master worktree in {WORKTREE}')
|
||||
git('worktree', 'add', '--detach', str(WORKTREE), commit)
|
||||
elif git('rev-parse', 'HEAD', cwd=WORKTREE) != commit:
|
||||
git('checkout', '--detach', '--force', commit, cwd=WORKTREE)
|
||||
state = built()
|
||||
if state and state.get('commit') == commit:
|
||||
say(f'master {commit[:11]} is built')
|
||||
return {'root': WORKTREE, 'commit': commit}
|
||||
started = time.time()
|
||||
say(f'build master {commit[:11]} (the merge base). The first build takes 15 to 20 minutes. Log: {BUILD_LOG}')
|
||||
with BUILD_LOG.open('w') as log:
|
||||
# DOCKER_BUILD skips the git hook install of the prepare script, so the hooks of the
|
||||
# main checkout stay as they are.
|
||||
run_logged(['pnpm', 'install', '--frozen-lockfile'], log, env={**os.environ, 'DOCKER_BUILD': 'true'})
|
||||
run_logged(['pnpm', 'turbo', 'run', 'build', '--filter=n8n...'], log)
|
||||
STATE.write_text(json.dumps({'commit': commit, 'builtAt': datetime.now(timezone.utc).isoformat()}, indent=2))
|
||||
say(f'master {commit[:11]} built in {time.time() - started:.0f}s')
|
||||
return {'root': WORKTREE, 'commit': commit}
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('action', choices=['status', 'prepare'])
|
||||
ap.add_argument('--no-fetch', action='store_true')
|
||||
args = ap.parse_args()
|
||||
if args.action == 'status':
|
||||
print(json.dumps(status(), indent=2))
|
||||
return
|
||||
try:
|
||||
result = prepare(fetch=not args.no_fetch, say=lambda m: print(m, flush=True))
|
||||
except (RuntimeError, subprocess.CalledProcessError) as e:
|
||||
sys.exit(str(e))
|
||||
print(json.dumps({'root': str(result['root']), 'commit': result['commit']}, indent=2))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -1,175 +0,0 @@
|
|||
"""Shared helpers: paths, settings, and fresh n8n instances for the scenario runs. Standard library only.
|
||||
|
||||
The skill reuses the scripts of testmax-n8n-ai (settings, REST client, report parts,
|
||||
live server). This module puts them on the import path.
|
||||
"""
|
||||
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import signal
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
SHARED = HERE.parents[1] / 'testmax-n8n-ai' / 'scripts'
|
||||
sys.path.insert(0, str(SHARED))
|
||||
|
||||
from n8n_api import REPO_ROOT, N8nClient, config, setting # noqa: E402
|
||||
|
||||
CACHE = Path(setting('FEATURE_TEST_CACHE', default=str(Path.home() / '.cache' / 'n8n-feature-test'))).expanduser().resolve()
|
||||
RUNS_ROOT = Path(setting('FEATURE_TEST_RUNS_DIR', default='/tmp/feature-test')).expanduser().resolve()
|
||||
PLANS_ROOT = CACHE / 'plans'
|
||||
TEMPLATE_VERSION = '1'
|
||||
# These settings belong to the instance that the user runs. A fresh instance must not reuse them.
|
||||
INSTANCE_ENV_SKIP = ('N8N_LICENSE_', 'DB_', 'N8N_PORT', 'N8N_USER_FOLDER', 'N8N_EDITOR_BASE_URL', 'WEBHOOK_URL',
|
||||
'N8N_RUNNERS_BROKER_PORT', 'N8N_HOST', 'N8N_PROTOCOL', 'N8N_ENCRYPTION_KEY', 'N8N_BASE_URL')
|
||||
|
||||
|
||||
def free_port():
|
||||
with socket.socket() as s:
|
||||
s.bind(('127.0.0.1', 0))
|
||||
return s.getsockname()[1]
|
||||
|
||||
|
||||
def instance_env(port, home, feature_flags=None):
|
||||
"""The settings of the user's instance (.env.local and .env.ai-surfaces), with a new port and data folder.
|
||||
|
||||
Feature flags do not come from PostHog: PostHog gives each new instance its own experiment variants,
|
||||
so the screens of two instances with the same code can differ. The flags are their defaults, plus
|
||||
N8N_FEATURE_FLAG_OVERRIDES, plus feature_flags.
|
||||
"""
|
||||
env = {k: v for k, v in os.environ.items() if not k.startswith(INSTANCE_ENV_SKIP)}
|
||||
env.update({k: v for k, v in config().items() if not k.startswith(INSTANCE_ENV_SKIP)})
|
||||
# Keep N8N_DIAGNOSTICS_ENABLED as the user set it: when it is false, Instance AI sends no LangSmith traces.
|
||||
env.update(
|
||||
N8N_PORT=str(port), N8N_RUNNERS_BROKER_PORT=str(free_port()), N8N_USER_FOLDER=str(home),
|
||||
N8N_EDITOR_BASE_URL=f'http://localhost:{port}', WEBHOOK_URL=f'http://localhost:{port}/',
|
||||
N8N_PERSONALIZATION_ENABLED='false', N8N_INSTANCE_AI_EVAL_INSTANCE='true',
|
||||
)
|
||||
if feature_flags:
|
||||
try:
|
||||
overrides = json.loads(env.get('N8N_FEATURE_FLAG_OVERRIDES') or '{}')
|
||||
except ValueError:
|
||||
overrides = {}
|
||||
env['N8N_FEATURE_FLAG_OVERRIDES'] = json.dumps({**overrides, **feature_flags})
|
||||
return env
|
||||
|
||||
|
||||
def template_dir(root):
|
||||
"""The data folder template of a checkout. The key has the database migrations and the owner, so a build
|
||||
with other migrations or another N8N_EVAL_EMAIL gets a new template. Change TEMPLATE_VERSION when seed()
|
||||
changes."""
|
||||
migrations = Path(root) / 'packages' / '@n8n' / 'db' / 'dist' / 'migrations'
|
||||
names = sorted(str(p.relative_to(migrations)) for p in migrations.rglob('*.js')) if migrations.exists() else []
|
||||
owner = client('http://localhost')
|
||||
parts = [TEMPLATE_VERSION, str(Path(root).resolve()), owner.email, owner.password, *names]
|
||||
return CACHE / 'templates' / hashlib.sha256('\n'.join(parts).encode()).hexdigest()[:16]
|
||||
|
||||
|
||||
class Instance:
|
||||
"""A fresh n8n instance with its own port and data folder, started from a checkout that is built.
|
||||
|
||||
The first instance of a checkout runs the database migrations and the owner setup, then saves its
|
||||
data folder as a template. Later instances start from a copy of the template, which is faster.
|
||||
"""
|
||||
|
||||
def __init__(self, name, root, home, log, feature_flags=None):
|
||||
self.name, self.root, self.home, self.log_path = name, Path(root), Path(home), Path(log)
|
||||
self.feature_flags = feature_flags
|
||||
self.port = free_port()
|
||||
self.url = f'http://localhost:{self.port}'
|
||||
self.proc = None
|
||||
|
||||
def start(self, timeout=180):
|
||||
template = template_dir(self.root)
|
||||
if (template / '.ready').exists():
|
||||
shutil.copytree(template / '.n8n', self.home / '.n8n', dirs_exist_ok=True)
|
||||
self._boot(timeout)
|
||||
return self
|
||||
template.parent.mkdir(parents=True, exist_ok=True)
|
||||
with (template.parent / f'{template.name}.lock').open('w') as lock:
|
||||
# Only one instance makes the template. The others start without it and do not wait.
|
||||
try:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
owner = True
|
||||
except BlockingIOError:
|
||||
owner = False
|
||||
self._boot(timeout)
|
||||
seed(self.url)
|
||||
if owner and not (template / '.ready').exists():
|
||||
self._save_template(template)
|
||||
return self
|
||||
|
||||
def _save_template(self, template):
|
||||
"""Stop, copy the data folder, and start again on the same port."""
|
||||
self.stop()
|
||||
shutil.rmtree(template, ignore_errors=True)
|
||||
shutil.copytree(self.home / '.n8n', template / '.n8n', ignore=shutil.ignore_patterns('n8nEventLog*', '*.log'))
|
||||
(template / '.ready').touch()
|
||||
self._boot(60)
|
||||
|
||||
def _boot(self, timeout):
|
||||
self.home.mkdir(parents=True, exist_ok=True)
|
||||
log = self.log_path.open('a')
|
||||
# A new session, so stop() can end n8n and its task runners together.
|
||||
self.proc = subprocess.Popen(['node', str(self.root / 'packages' / 'cli' / 'bin' / 'n8n'), 'start'],
|
||||
cwd=self.home, env=instance_env(self.port, self.home, self.feature_flags),
|
||||
stdin=subprocess.DEVNULL, stdout=log, stderr=log, start_new_session=True)
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
if self.proc.poll() is not None:
|
||||
raise RuntimeError(f'{self.name} instance stopped at start (exit {self.proc.returncode}), see {self.log_path}')
|
||||
try:
|
||||
with urllib.request.urlopen(f'{self.url}/healthz/readiness', timeout=2) as res:
|
||||
if res.status == 200:
|
||||
return
|
||||
except OSError:
|
||||
pass
|
||||
time.sleep(0.25)
|
||||
raise RuntimeError(f'{self.name} instance did not become ready in {timeout}s, see {self.log_path}')
|
||||
|
||||
def stop(self, clean=False):
|
||||
"""Stop n8n. clean also deletes the static file cache (about 80 MB), which n8n writes again at each start."""
|
||||
if self.proc and self.proc.poll() is None:
|
||||
try:
|
||||
os.killpg(self.proc.pid, signal.SIGTERM)
|
||||
self.proc.wait(timeout=30)
|
||||
except (ProcessLookupError, subprocess.TimeoutExpired):
|
||||
try:
|
||||
os.killpg(self.proc.pid, signal.SIGKILL)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
if clean:
|
||||
shutil.rmtree(self.home / '.cache', ignore_errors=True)
|
||||
|
||||
|
||||
def client(base_url):
|
||||
c = N8nClient()
|
||||
c.base_url = base_url.rstrip('/')
|
||||
return c
|
||||
|
||||
|
||||
def seed(base_url):
|
||||
"""Create the owner (N8N_EVAL_EMAIL and N8N_EVAL_PASSWORD) and finish the Assistant setup."""
|
||||
c = client(base_url)
|
||||
c.request('POST', '/rest/owner/setup',
|
||||
{'email': c.email, 'firstName': 'Feature', 'lastName': 'Test', 'password': c.password})
|
||||
c.login()
|
||||
# Web search is optional. Without a choice, the Assistant shows its setup screen instead of the chat.
|
||||
# A search provider from the environment makes the choice, and the API rejects searchDisabled then.
|
||||
current = c.request('GET', '/rest/instance-ai/settings') or {}
|
||||
update = {'enabled': True}
|
||||
if not current.get('searchEnvConfigured'):
|
||||
update['searchDisabled'] = True
|
||||
c.request('PUT', '/rest/instance-ai/settings', update)
|
||||
|
||||
|
||||
def git(*args, cwd=REPO_ROOT):
|
||||
return subprocess.run(['git', *args], cwd=cwd, capture_output=True, text=True, check=True).stdout.strip()
|
||||
|
|
@ -1,351 +0,0 @@
|
|||
// The helper object `t` that a scenario script receives. See SKILL.md for the full API.
|
||||
//
|
||||
// export default async function (t) {
|
||||
// await t.step('Open the Assistant', () => t.goto());
|
||||
// await t.step('Send a prompt', () => t.send('Reply with exactly: OK'));
|
||||
// await t.step('Wait for the reply', () => t.waitForIdle());
|
||||
// await t.check('The reply shows', () => t.expect(t.ai.assistantMessages().first()).toBeVisible());
|
||||
// await t.capture('thread', { mask: [t.ai.assistantMessages()] });
|
||||
// }
|
||||
//
|
||||
// A step that throws ends the scenario. A check that fails is recorded and the scenario goes on.
|
||||
// Each step and check writes a screenshot. The runner shows them in the report.
|
||||
import { mkdirSync, renameSync, writeFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
|
||||
import { expect } from '../../testmax-n8n-ai/scripts/n8n_browser.mjs';
|
||||
|
||||
const THREAD_URL = /\/assistant\/([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})/;
|
||||
const RUN_GRACE_MS = 10_000;
|
||||
|
||||
/**
|
||||
* Replace the accessibility subtree of each masked element with one line. Masked texts alone keep
|
||||
* the structure, so a list with a different number of rows still shows as a change. Each subtree
|
||||
* replaces only its first unmasked match, so an unmasked element with the same content stays.
|
||||
*/
|
||||
function maskSubtrees(tree, subtrees) {
|
||||
const lines = tree.split('\n');
|
||||
for (const subtree of subtrees) {
|
||||
const sub = subtree.split('\n').filter((l) => l.trim());
|
||||
if (!sub.length || sub[0].startsWith('#')) continue;
|
||||
for (let i = 0; i + sub.length <= lines.length; i++) {
|
||||
if (!lines[i].endsWith(sub[0])) continue;
|
||||
const indent = lines[i].slice(0, lines[i].length - sub[0].length);
|
||||
if (indent.trim() || !sub.every((l, j) => lines[i + j] === indent + l)) continue;
|
||||
lines.splice(i, sub.length, `${indent}- text: "[masked]"`);
|
||||
break;
|
||||
}
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
/** The accessibility tree without the masked texts and the instance port, so branch and master compare. */
|
||||
function maskTree(tree, texts) {
|
||||
let out = tree.replace(/localhost(:|%3A)\d+/g, 'localhost$1PORT');
|
||||
for (const text of [...texts].sort((a, b) => b.length - a.length))
|
||||
out = out.split(text).join('[masked]');
|
||||
return out;
|
||||
}
|
||||
|
||||
export class StepFailed extends Error {}
|
||||
|
||||
/** Locators of the Instance AI page. They mirror pages/InstanceAiPage.ts in the Playwright tests. */
|
||||
function locators(page) {
|
||||
const container = () => page.getByTestId('instance-ai-container');
|
||||
return {
|
||||
container,
|
||||
chatInput: () => container().getByRole('textbox').or(container().getByRole('combobox')),
|
||||
composer: () => container().getByTestId('instance-ai-composer'),
|
||||
sendButton: () => container().getByTestId('instance-ai-send-button'),
|
||||
stopButton: () => container().getByTestId('instance-ai-stop-button'),
|
||||
userMessages: () => container().getByTestId('instance-ai-user-message'),
|
||||
assistantMessages: () => container().getByTestId('instance-ai-assistant-message'),
|
||||
assistantText: (text) =>
|
||||
container().getByTestId('instance-ai-assistant-message').getByText(text),
|
||||
panelText: (text) => container().getByText(text),
|
||||
emptyState: () => container().getByTestId('instance-ai-empty-state'),
|
||||
statusBar: () => container().getByTestId('instance-ai-status-bar'),
|
||||
sidebarToggle: () => container().getByTestId('instance-ai-sidebar-toggle'),
|
||||
threadList: () => page.getByTestId('instance-ai-thread-list'),
|
||||
newThreadButton: () =>
|
||||
page
|
||||
.getByTestId('project-instance-ai-menu-item')
|
||||
.getByRole('menuitem', { name: 'Assistant', exact: true }),
|
||||
mentionButton: () => container().getByTestId('instance-ai-mention-button'),
|
||||
mentionMenu: () => page.getByTestId('instance-ai-mention-menu-content'),
|
||||
fileInput: () => container().locator('input[type="file"]'),
|
||||
approveButton: () => container().getByTestId('approval-card-allow-once'),
|
||||
alwaysAllowButton: () => container().getByTestId('approval-card-always-allow'),
|
||||
denyButton: () => container().getByTestId('approval-card-deny'),
|
||||
domainAllowOnce: () => container().getByTestId('domain-access-allow-once'),
|
||||
planApproveButton: () => container().getByTestId('instance-ai-plan-approve'),
|
||||
credentialContinue: () => container().getByTestId('instance-ai-credential-continue-button'),
|
||||
previewPanel: () => container().getByTestId('instance-ai-preview-panel'),
|
||||
previewCanvasNodes: () =>
|
||||
container().getByTestId('workflow-canvas-host').locator('[data-test-id="canvas-node"]'),
|
||||
previewToggle: () => container().getByTestId('instance-ai-artifacts-preview-toggle'),
|
||||
artifactsSidebar: () => container().getByTestId('instance-ai-artifacts-sidebar'),
|
||||
outOfCredits: () => container().getByTestId('instance-ai-out-of-credits'),
|
||||
};
|
||||
}
|
||||
|
||||
export function createKit({ page, out, baseUrl, autoConfirm, emit }) {
|
||||
const started = Date.now();
|
||||
const steps = [];
|
||||
const threads = [];
|
||||
const prompts = [];
|
||||
const ai = locators(page);
|
||||
let stopped = false;
|
||||
mkdirSync(join(out, 'captures'), { recursive: true });
|
||||
|
||||
const seconds = () => (Date.now() - started) / 1000;
|
||||
function save() {
|
||||
const partial = join(out, 'steps.tmp.json');
|
||||
writeFileSync(partial, JSON.stringify(steps, null, 1));
|
||||
renameSync(partial, join(out, 'steps.json'));
|
||||
}
|
||||
async function shot(file) {
|
||||
try {
|
||||
await page.screenshot({ path: join(out, file) });
|
||||
return file;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
function message(error) {
|
||||
return String(error?.message ?? error)
|
||||
.replace(/\u001b\[[0-9;]*m/g, '')
|
||||
.slice(0, 2000);
|
||||
}
|
||||
|
||||
async function record(kind, title, fn) {
|
||||
if (stopped) throw new StepFailed('The scenario has stopped.');
|
||||
const entry = { i: steps.length + 1, kind, title, status: 'running', t: seconds() };
|
||||
steps.push(entry);
|
||||
save();
|
||||
emit(`${kind} ${title}`);
|
||||
let value;
|
||||
let failed = false;
|
||||
try {
|
||||
value = typeof fn === 'function' ? await fn() : fn;
|
||||
if (value === false) throw new Error('The check returned false.');
|
||||
} catch (error) {
|
||||
failed = true;
|
||||
entry.error = message(error);
|
||||
}
|
||||
// After a stop, run_scenario.mjs owns the steps. A late result must not change them.
|
||||
if (stopped) throw new StepFailed('The scenario has stopped.');
|
||||
entry.status = failed ? 'failed' : 'passed';
|
||||
if (failed) emit(`${kind} failed: ${entry.error.split('\n')[0]}`);
|
||||
entry.duration = seconds() - entry.t;
|
||||
entry.shot = await shot(`${kind}-${String(entry.i).padStart(2, '0')}.png`);
|
||||
if (!stopped) save();
|
||||
if (failed && kind === 'step') throw new StepFailed(entry.error);
|
||||
return failed ? undefined : value;
|
||||
}
|
||||
|
||||
async function api(method, path, body) {
|
||||
return await page.evaluate(
|
||||
async ({ method, path, body }) => {
|
||||
const res = await fetch(path, {
|
||||
method,
|
||||
headers: {
|
||||
'content-type': 'application/json',
|
||||
'browser-id': localStorage.getItem('n8n-browserId') ?? '',
|
||||
},
|
||||
body: body === undefined ? undefined : JSON.stringify(body),
|
||||
});
|
||||
const text = await res.text();
|
||||
if (!res.ok) throw new Error(`${method} ${path} -> ${res.status}: ${text.slice(0, 300)}`);
|
||||
const json = text ? JSON.parse(text) : null;
|
||||
return json && typeof json === 'object' && Object.keys(json).length === 1 && 'data' in json
|
||||
? json.data
|
||||
: json;
|
||||
},
|
||||
{ method, path, body },
|
||||
);
|
||||
}
|
||||
|
||||
function currentThread() {
|
||||
return page.url().match(THREAD_URL)?.[1] ?? null;
|
||||
}
|
||||
|
||||
function noteThread(id) {
|
||||
if (id && !threads.includes(id)) {
|
||||
threads.push(id);
|
||||
emit(`thread ${id}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function threadStatus(id) {
|
||||
const s = await api('GET', `/rest/instance-ai/threads/${id}/status`);
|
||||
const background = (s?.backgroundTasks ?? []).filter((x) => x.status === 'running').length;
|
||||
return { active: Boolean(s?.hasActiveRun), suspended: Boolean(s?.isSuspended), background };
|
||||
}
|
||||
|
||||
const t = {
|
||||
page,
|
||||
ai,
|
||||
expect,
|
||||
baseUrl,
|
||||
threads,
|
||||
prompts,
|
||||
|
||||
/** A step. When it throws, the scenario ends as failed. */
|
||||
step: (title, fn) => record('step', title, fn),
|
||||
/** A soft check. A failure is recorded and the scenario goes on. fn can return false to fail. */
|
||||
check: (title, fn) => record('check', title, fn),
|
||||
/** A line in the step list, with a screenshot. */
|
||||
note: (title) => record('note', title, () => true),
|
||||
|
||||
/** Open a page of the instance. Default: the Assistant, with the composer ready. */
|
||||
async goto(path = '/assistant') {
|
||||
await page.goto(baseUrl + path, { waitUntil: 'domcontentloaded' });
|
||||
if (path.startsWith('/assistant')) {
|
||||
const enable = page.getByRole('dialog').filter({ hasText: 'Try new n8n Assistant' });
|
||||
if (await enable.isVisible({ timeout: 1500 }).catch(() => false)) {
|
||||
await enable
|
||||
.getByRole('button', { name: /Enable n8n Assistant on this instance/ })
|
||||
.click();
|
||||
await enable.getByRole('button', { name: /^(Continue|Enable)$/ }).click();
|
||||
}
|
||||
await expect(ai.chatInput()).toBeVisible({ timeout: 30_000 });
|
||||
}
|
||||
noteThread(currentThread());
|
||||
},
|
||||
|
||||
/** Type a message in the composer and send it. Returns the thread id. */
|
||||
async send(text) {
|
||||
const before = await ai.assistantMessages().count();
|
||||
await expect(ai.chatInput()).toBeVisible({ timeout: 30_000 });
|
||||
await ai.chatInput().fill(text);
|
||||
await expect(ai.sendButton()).toBeEnabled({ timeout: 30_000 });
|
||||
await ai.sendButton().click();
|
||||
await page.waitForURL(THREAD_URL, { timeout: 30_000 });
|
||||
const id = currentThread();
|
||||
noteThread(id);
|
||||
prompts.push({
|
||||
t: seconds(),
|
||||
text,
|
||||
thread: id,
|
||||
assistantMessagesBefore: before,
|
||||
sentAt: Date.now(),
|
||||
});
|
||||
return id;
|
||||
},
|
||||
|
||||
/**
|
||||
* Wait until the thread has no active run, no open confirmation, and no running background task.
|
||||
* Returns 'idle', or 'suspended' when a confirmation card waits for the scenario
|
||||
* (autoConfirm is off, or the card stays open for 20 seconds).
|
||||
*/
|
||||
async waitForIdle({ timeout = 600_000, settle = 3000 } = {}) {
|
||||
const id = currentThread() ?? threads.at(-1);
|
||||
if (!id) throw new Error('No thread is open. Call t.send() first.');
|
||||
const last = prompts.filter((p) => p.thread === id).at(-1);
|
||||
const deadline = Date.now() + timeout;
|
||||
let seenRun = false;
|
||||
let idleSince = null;
|
||||
let suspendedSince = null;
|
||||
while (!stopped && Date.now() < deadline) {
|
||||
let s;
|
||||
try {
|
||||
s = await threadStatus(id);
|
||||
} catch {
|
||||
await page.waitForTimeout(1000);
|
||||
continue;
|
||||
}
|
||||
// The status can show no run for a moment after the message is sent. A run that the user
|
||||
// stops at once can end before it shows, so after a grace period "no run" means "done".
|
||||
if (!seenRun && Date.now() - (last?.sentAt ?? 0) > RUN_GRACE_MS) {
|
||||
seenRun = true;
|
||||
}
|
||||
if (!seenRun) {
|
||||
const replies = await ai
|
||||
.assistantMessages()
|
||||
.count()
|
||||
.catch(() => 0);
|
||||
seenRun =
|
||||
s.active ||
|
||||
s.suspended ||
|
||||
s.background > 0 ||
|
||||
replies > (last?.assistantMessagesBefore ?? 0);
|
||||
}
|
||||
if (s.suspended) {
|
||||
suspendedSince ??= Date.now();
|
||||
if (!autoConfirm || Date.now() - suspendedSince > 20_000) return 'suspended';
|
||||
} else {
|
||||
suspendedSince = null;
|
||||
}
|
||||
const idle = seenRun && !s.active && !s.suspended && s.background === 0;
|
||||
if (!idle) {
|
||||
idleSince = null;
|
||||
} else {
|
||||
idleSince ??= Date.now();
|
||||
if (Date.now() - idleSince >= settle) {
|
||||
await ai
|
||||
.sendButton()
|
||||
.waitFor({ state: 'visible', timeout: 10_000 })
|
||||
.catch(() => {});
|
||||
return 'idle';
|
||||
}
|
||||
}
|
||||
await page.waitForTimeout(1000);
|
||||
}
|
||||
throw new Error(`The thread did not go idle in ${Math.round(timeout / 1000)}s.`);
|
||||
},
|
||||
|
||||
/**
|
||||
* Save a screen for the visual check. Visual scenarios run on this branch and on master, and the
|
||||
* report compares the screens with the same name. mask hides changing parts, such as LLM text.
|
||||
* root is the part of the page for the accessibility tree diff (default: the whole page).
|
||||
*/
|
||||
async capture(name, { mask = [], maskText = [], root, fullPage = false } = {}) {
|
||||
if (!/^[a-z0-9][a-z0-9-]*$/.test(name))
|
||||
throw new Error(`Capture name "${name}": use a-z, 0-9 and "-".`);
|
||||
return await record('capture', `Capture ${name}`, async () => {
|
||||
await page.waitForLoadState('networkidle', { timeout: 5000 }).catch(() => {});
|
||||
await page.waitForTimeout(500);
|
||||
const texts = new Set(maskText.filter(Boolean));
|
||||
for (const locator of mask) {
|
||||
for (const inner of await locator.allInnerTexts().catch(() => [])) {
|
||||
for (const line of inner.split('\n'))
|
||||
if (line.trim().length > 1) texts.add(line.trim());
|
||||
}
|
||||
}
|
||||
const textMasks = maskText.filter(Boolean).map((s) => page.getByText(s, { exact: true }));
|
||||
await page.screenshot({
|
||||
path: join(out, 'captures', `${name}.png`),
|
||||
mask: [...mask, ...textMasks],
|
||||
maskColor: '#FF00FF',
|
||||
animations: 'disabled',
|
||||
caret: 'hide',
|
||||
fullPage,
|
||||
});
|
||||
const tree = await (root ?? page.locator('body'))
|
||||
.ariaSnapshot({ timeout: 10_000 })
|
||||
.catch((e) => `# ${message(e)}`);
|
||||
const subtrees = [];
|
||||
for (const locator of mask) {
|
||||
for (const element of await locator.all().catch(() => [])) {
|
||||
subtrees.push(await element.ariaSnapshot({ timeout: 5_000 }).catch(() => ''));
|
||||
}
|
||||
}
|
||||
writeFileSync(
|
||||
join(out, 'captures', `${name}.aria.yml`),
|
||||
maskTree(maskSubtrees(tree, subtrees), texts) + '\n',
|
||||
);
|
||||
});
|
||||
},
|
||||
|
||||
/** A REST call as the signed-in user. Returns the `data` field of the answer when it has one. */
|
||||
api,
|
||||
threadStatus,
|
||||
sleep: (ms) => page.waitForTimeout(ms),
|
||||
};
|
||||
/** End the scenario: later steps throw, and a running step does not change the steps. */
|
||||
const stop = () => {
|
||||
stopped = true;
|
||||
};
|
||||
return { t, steps, threads, prompts, stop };
|
||||
}
|
||||
|
|
@ -1,55 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Serve the live page of a feature test run, then its report.
|
||||
|
||||
live_page.py --run-dir <run-dir> [--parent-pid <pid>] [--port <port>]
|
||||
|
||||
run.py starts it in the background. It uses the server of testmax-n8n-ai
|
||||
(live_server.py): 127.0.0.1 only, the address in <run-dir>/live-url.txt, and it stops
|
||||
30 minutes after the last request when the run is done. When verdicts.json changes after the
|
||||
run, the page builds the report again, and the open report reloads.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import common # noqa: F401 (puts the shared scripts on the import path)
|
||||
import report
|
||||
from live_server import read_status, serve
|
||||
from report import ScenarioApp
|
||||
|
||||
|
||||
def rebuild_on_verdicts(run_dir):
|
||||
verdicts, built = run_dir / 'verdicts.json', None
|
||||
while True:
|
||||
time.sleep(2)
|
||||
if read_status(run_dir).get('phase') != 'done' or not verdicts.exists():
|
||||
continue
|
||||
mtime = verdicts.stat().st_mtime
|
||||
if mtime == built:
|
||||
continue
|
||||
try:
|
||||
report.build(run_dir)
|
||||
built = mtime
|
||||
except ValueError:
|
||||
# The judge is still writing the file. The next check reads it again.
|
||||
pass
|
||||
except Exception as e:
|
||||
print(f'cannot build the report: {e}', flush=True)
|
||||
built = mtime
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--run-dir', required=True)
|
||||
ap.add_argument('--parent-pid', type=int)
|
||||
ap.add_argument('--port', type=int, default=0)
|
||||
args = ap.parse_args()
|
||||
run_dir = Path(args.run_dir).resolve()
|
||||
threading.Thread(target=rebuild_on_verdicts, args=(run_dir,), daemon=True).start()
|
||||
serve(ScenarioApp(run_dir), args.parent_pid, args.port)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -1,690 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Build the report of a feature test run, and the live page while it runs.
|
||||
|
||||
report.py <run-dir> build <run-dir>/report/index.html (run it again after verdicts.json)
|
||||
report.py <run-dir> --summary print the summary for the judge
|
||||
|
||||
Inputs in <run-dir>:
|
||||
plan/scenarios.json the plan (run.py copies it)
|
||||
status.json the status of each run (run.py writes it)
|
||||
runs/<name>/ the files of one run (run_scenario.mjs and run.py write them)
|
||||
visual.json the visual check (visual_diff.mjs and run.py write it)
|
||||
verdicts.json optional: {"<name>": {"verdict": "pass|partial|fail", "summary": "...",
|
||||
"checks": [{"ok": true|false|null, "text": "..."}]}}
|
||||
|
||||
The page uses the template and the LLM steps view of testmax-n8n-ai.
|
||||
"""
|
||||
|
||||
import fcntl
|
||||
import json
|
||||
import shutil
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
import common # noqa: F401 (puts the shared scripts on the import path)
|
||||
import build_report as br
|
||||
import live_server as ls
|
||||
|
||||
STATUS = {'waiting': ('st-waiting', 'Waiting'), 'starting': ('st-waiting', 'Starting'), 'running': ('st-running', 'Running'),
|
||||
'passed': ('st-done', 'Passed'), 'failed': ('st-failed', 'Failed'), 'error': ('st-failed', 'Error'),
|
||||
'skipped': ('st-skipped', 'Skipped')}
|
||||
FINISHED = ('passed', 'failed', 'error', 'skipped')
|
||||
KIND_LABELS = {'feature': 'Feature', 'regression': 'Regression', 'visual': 'Visual'}
|
||||
VISUAL = {'changed': ('vis-changed', 'Changed'), 'unchanged': ('vis-same', 'Unchanged'),
|
||||
'branch-only': ('vis-only', 'Only on branch'), 'master-only': ('vis-only', 'Only on master')}
|
||||
THEAD = ('<tr><th>Scenario</th><th>Kind</th><th>Result</th><th>Steps</th><th>Time</th><th>LLM calls</th>'
|
||||
'<th>Tool calls</th><th>Input tokens</th><th>Output tokens</th><th>Cost</th><th>Browser errors</th></tr>')
|
||||
|
||||
CSS = """<style>
|
||||
.run-video video{display:block;width:100%;max-height:70vh;aspect-ratio:16/10;object-fit:contain;background:var(--black);border-radius:var(--r8)}
|
||||
.badge.vis-changed{background:var(--pink-70);color:var(--bg)}
|
||||
.badge.vis-same{background:var(--bg-3);color:var(--black)}
|
||||
.badge.vis-only{box-shadow:inset 0 0 0 1px var(--line-strong);color:var(--dim)}
|
||||
.facts{display:grid;grid-template-columns:max-content minmax(0,1fr);gap:var(--s4) var(--s16);margin:0}
|
||||
.facts dt{font:10px/18px var(--mono);text-transform:uppercase;letter-spacing:.02em;color:var(--dim)}
|
||||
.facts dd{margin:0;word-break:break-word}
|
||||
.run-error{margin:var(--s12) 0 0;background:var(--pink-10);color:var(--pink-70);border-radius:var(--r8);padding:var(--s12);white-space:pre-wrap;word-break:break-word;font:12px/16px var(--mono);max-height:240px;overflow:auto}
|
||||
.tsteps{list-style:none;margin:0;padding:0;display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:var(--s16)}
|
||||
@media (max-width:1000px){.tsteps{grid-template-columns:repeat(2,minmax(0,1fr))}}
|
||||
@media (max-width:640px){.tsteps{grid-template-columns:minmax(0,1fr)}}
|
||||
.tstep{display:flex;flex-direction:column;min-width:0;background:var(--white);border:1px solid var(--line);border-radius:var(--r8);overflow:hidden}
|
||||
.tstep.failed{background:var(--pink-10);border-color:var(--pink-70);box-shadow:0 0 0 1px var(--pink-70)}
|
||||
.tstep-shot{display:block;width:100%;aspect-ratio:16/10;padding:0;border:0;border-bottom:1px solid var(--line);background:var(--bg-3);cursor:zoom-in;overflow:hidden}
|
||||
.tstep-shot img{display:block;width:100%;height:100%;object-fit:cover;object-position:top center;transition:transform .15s}
|
||||
.tstep-shot:hover img{transform:scale(1.03)}
|
||||
.tstep-shot:focus-visible{outline:2px solid var(--pink);outline-offset:-2px}
|
||||
@media (prefers-reduced-motion:reduce){.tstep-shot img{transition:none}}
|
||||
.tstep-noshot{display:grid;place-items:center;aspect-ratio:16/10;background:var(--bg-2);border-bottom:1px solid var(--line);color:var(--gray-50);font:10px/14px var(--mono);text-transform:uppercase;letter-spacing:.02em}
|
||||
.tstep-info{display:flex;gap:var(--s12);align-items:flex-start;padding:var(--s12);min-width:0}
|
||||
.tstep-info > div{flex:1;min-width:0;display:flex;flex-direction:column;gap:var(--s4)}
|
||||
.tstep-info.running .num{background:var(--pink-10);color:var(--pink-70)}
|
||||
.tstep-info.passed .num{background:var(--pink)}
|
||||
.tstep-info.failed .num{background:var(--pink-70);color:var(--bg)}
|
||||
.tstep-title{font-weight:500;padding-top:3px;word-break:break-word}
|
||||
.tstep-title .mono-label{font-weight:400;margin-right:var(--s8)}
|
||||
.tstep-info pre{margin:var(--s4) 0 0;white-space:pre-wrap;word-break:break-word;font:12px/16px var(--mono);color:var(--pink-70);max-height:200px;overflow:auto}
|
||||
.shot-viewer .theater-bar{flex-wrap:nowrap;align-items:flex-start}
|
||||
.shot-viewer-info{flex:1;min-width:0;padding-top:3px}
|
||||
.shot-viewer-actions{flex:none;display:flex;align-items:center;gap:var(--s16)}
|
||||
.shot-viewer-info .tstep-info{padding:0}
|
||||
.shot-viewer-info .tstep-title{font-size:18px;line-height:22px}
|
||||
.shot-viewer-img{flex:1;min-height:0;display:flex;align-items:center;justify-content:center;cursor:zoom-out}
|
||||
.shot-viewer-img img{display:block;max-width:100%;max-height:100%;border-radius:var(--r8);box-shadow:0 0 0 1px var(--line);cursor:default}
|
||||
.err-list{margin:0;padding-left:var(--s16);font:12px/16px var(--mono)}
|
||||
.err-list li{margin-bottom:var(--s4);word-break:break-word}
|
||||
.vis{background:var(--white);border-radius:var(--r8);padding:var(--s12);box-shadow:inset 0 0 0 1px var(--line);margin-top:var(--s12)}
|
||||
.vis.changed{box-shadow:inset 4px 0 0 var(--pink-70),inset 0 0 0 1px var(--line)}
|
||||
.vis-grid{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:var(--s12)}
|
||||
.vis-grid figure{margin:0}
|
||||
.vis-grid figcaption{margin-bottom:var(--s4)}
|
||||
.vis-grid img{display:block;width:100%;height:auto;border-radius:var(--r4);box-shadow:0 0 0 1px var(--line)}
|
||||
.vis details summary{cursor:pointer;color:var(--dim);margin-top:var(--s8);font:10px/14px var(--mono);text-transform:uppercase;letter-spacing:.02em}
|
||||
.vis pre{background:var(--bg);border-radius:var(--r4);padding:var(--s12);margin:var(--s8) 0 0;overflow:auto;max-height:320px;font-size:12px;line-height:16px;white-space:pre}
|
||||
</style>"""
|
||||
|
||||
|
||||
def read_json(path, default):
|
||||
try:
|
||||
return json.loads(path.read_text())
|
||||
except (OSError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def seconds(value):
|
||||
return f'{value:.0f}s'
|
||||
|
||||
|
||||
def load_trace(sub, result):
|
||||
"""LLM steps from the thread events, and per-call tokens from usage.json when LangSmith had them."""
|
||||
data = ls.load_instance_ai_live(sub)
|
||||
events_file = sub / 'events.jsonl'
|
||||
rows = br.read_jsonl(events_file) if events_file.exists() else []
|
||||
starts = [r['ts'] / 1000 for r in rows if r.get('type') == 'run-start' and r.get('ts')]
|
||||
usage = read_json(sub / 'usage.json', None)
|
||||
if usage and starts:
|
||||
start = starts[0]
|
||||
data['calls'] = sorted((
|
||||
{'t': br.ts(u['start']) - start, 'start': br.ts(u['start']) - start,
|
||||
'end': br.ts(u['end']) - start if u.get('end') else None,
|
||||
'name': (u.get('name') or '').removeprefix('llm: ') or None,
|
||||
'prompt': u['in'], 'cache_read': u['cache_read'], 'cache_write': u['cache_write'],
|
||||
'out': u['out'], 'cost': u['cost']} for u in usage), key=lambda c: c['t'])
|
||||
data['model'] = ', '.join(sorted({u['model'] for u in usage if u.get('model')})) or None
|
||||
data['cost'] = f'${sum(c["cost"] for c in data["calls"]):.2f}'
|
||||
data['costSource'] = 'langsmith'
|
||||
if result.get('startedAt') and result.get('finishedAt'):
|
||||
data['duration'] = seconds(br.ts(result['finishedAt']) - br.ts(result['startedAt']))
|
||||
return data
|
||||
|
||||
|
||||
def load(run_dir):
|
||||
"""The plan, the status, and one dict for each run in plan order."""
|
||||
plan = read_json(run_dir / 'plan' / 'scenarios.json', {'scenarios': []})
|
||||
status = ls.read_status(run_dir)
|
||||
entries = status.get('runs') or {}
|
||||
runs = []
|
||||
for s in plan['scenarios']:
|
||||
for name in (s['id'], f'{s["id"]}--master'):
|
||||
if name not in entries:
|
||||
continue
|
||||
entry = entries[name]
|
||||
sub = run_dir / 'runs' / name
|
||||
result = read_json(sub / 'result.json', {})
|
||||
data = load_trace(sub, result)
|
||||
if entry.get('status') in ('waiting', 'running'):
|
||||
data['cost'] = '-'
|
||||
if entry.get('startedAt'):
|
||||
data['duration'] = seconds((entry.get('finishedAt') or time.time()) - entry['startedAt'])
|
||||
runs.append({
|
||||
'name': name, 'scenario': s, 'entry': entry, 'master': name.endswith('--master'),
|
||||
'steps': read_json(sub / 'steps.json', []), 'result': result,
|
||||
'console': read_json(sub / 'console.json', {}), 'meta': read_json(sub / 'run.json', {}),
|
||||
'data': data,
|
||||
})
|
||||
return plan, status, runs
|
||||
|
||||
|
||||
def status_badge(state):
|
||||
cls, label = STATUS.get(state, ('st-waiting', state))
|
||||
return f'<span class="badge {cls}">{br.esc(label)}</span>'
|
||||
|
||||
|
||||
def run_title(run):
|
||||
return run['scenario']['title'] + (' (master)' if run['master'] else '')
|
||||
|
||||
|
||||
def check_count(steps):
|
||||
checked = [s for s in steps if s['kind'] in ('step', 'check', 'capture')]
|
||||
return f'{sum(s["status"] == "passed" for s in checked)}/{len(checked)}' if checked else '-'
|
||||
|
||||
|
||||
def row_cells(run, result_html):
|
||||
data, result = run['data'], run['result']
|
||||
calls, tools = data['calls'], [e for e in data['events'] if e['kind'] == 'tool']
|
||||
errors = sum(e['error'] for e in tools)
|
||||
browser = (result.get('consoleErrors') or 0) + (result.get('requestErrors') or 0)
|
||||
cells = [
|
||||
f'<span class="run-name">{br.esc(run_title(run))}</span>',
|
||||
KIND_LABELS.get(run['scenario']['kind'], run['scenario']['kind']),
|
||||
result_html, check_count(run['steps']), data['duration'], str(len(calls)) if calls else '-',
|
||||
f'{len(tools)}' + (f' <span class="err">({errors} err)</span>' if errors else ''),
|
||||
br.kilo(sum(c['prompt'] for c in calls)) if calls else '-',
|
||||
br.kilo(sum(c['out'] for c in calls)) if calls else '-',
|
||||
f'<strong>{br.esc(data["cost"])}</strong>',
|
||||
f'<span class="err">{browser}</span>' if browser else '0',
|
||||
]
|
||||
return ''.join(f'<td>{c}</td>' for c in cells)
|
||||
|
||||
|
||||
def result_html(run, verdict):
|
||||
"""The judge's verdict. The script status shows only while it runs or when it fails, because the Steps
|
||||
column already counts the steps that passed."""
|
||||
if verdict and verdict.get('verdict'):
|
||||
return br.badge(verdict['verdict'])
|
||||
state = run['entry'].get('status', 'waiting')
|
||||
if state != 'passed':
|
||||
return status_badge(state)
|
||||
if run['master']:
|
||||
return '<span class="badge st-done">Baseline</span>'
|
||||
return br.badge(None)
|
||||
|
||||
|
||||
def render_video(screen):
|
||||
"""The video of the run, above the expected outcome. Only the built report has it."""
|
||||
if not screen:
|
||||
return ''
|
||||
poster = f' poster="{br.esc(screen["poster"])}"' if screen.get('poster') else ''
|
||||
return br.fold('Video', f'<span class="mono-label">n8n UI · {br.esc(screen["duration"])}</span>',
|
||||
f'<div class="run-video"><video controls muted playsinline preload="metadata" '
|
||||
f'src="{br.esc(screen["src"])}"{poster}></video></div>', cls='brief-card prompt')
|
||||
|
||||
|
||||
def render_expected(run, verdict):
|
||||
head = br.verdict_head(verdict) if verdict else ''
|
||||
body = f'<div class="msg">{br.esc(run["scenario"]["expected"])}</div>'
|
||||
if verdict:
|
||||
body += f'<div class="verdict-body">{br.render_checks(verdict)}</div>'
|
||||
elif run['entry'].get('status') in FINISHED:
|
||||
body += '<p class="dim">Not judged yet.</p>'
|
||||
return br.fold('Expected outcome', head, body)
|
||||
|
||||
|
||||
def render_run_box(run):
|
||||
entry, result, meta = run['entry'], run['result'], run['meta']
|
||||
facts = [('Instance', f'{"master" if run["master"] else "branch"} · {entry.get("baseUrl") or "-"}')]
|
||||
if entry.get('startedAt'):
|
||||
facts.append(('Started', datetime.fromtimestamp(entry['startedAt']).strftime('%H:%M:%S')))
|
||||
if result.get('threads'):
|
||||
facts.append(('Threads', ', '.join(result['threads'])))
|
||||
if meta.get('cached'):
|
||||
facts.append(('Reused', 'This master run is from an earlier run of the same master commit and script.'))
|
||||
if meta.get('decisions'):
|
||||
facts.append(('Confirmations', f'{len(meta["decisions"])} answered by the auto-approve policy'))
|
||||
if result:
|
||||
facts.append(('Browser errors', f'{result.get("consoleErrors", 0)} console, {result.get("requestErrors", 0)} requests'))
|
||||
dl = ''.join(f'<dt>{br.esc(k)}</dt><dd>{br.esc(v)}</dd>' for k, v in facts)
|
||||
error = result.get('error') or (entry.get('reason') if entry.get('status') in ('error', 'skipped') else None)
|
||||
error_html = f'<pre class="run-error">{br.esc(error)}</pre>' if error else ''
|
||||
return br.fold('Run', status_badge(entry.get('status', 'waiting')), f'<dl class="facts">{dl}</dl>{error_html}')
|
||||
|
||||
|
||||
def render_scenario_steps(steps, files):
|
||||
if not steps:
|
||||
return br.fold('Scenario steps', '', '<p class="dim">No steps yet.</p>')
|
||||
items = []
|
||||
for s in steps:
|
||||
status, title = br.esc(s['status']), br.esc(s['title'])
|
||||
duration = f' · took {s["duration"]:.1f}s' if s.get('duration') is not None else ''
|
||||
if s.get('shot'):
|
||||
src = f'{files}{br.esc(s["shot"])}'
|
||||
thumb = (f'<button type="button" class="tstep-shot" data-src="{src}" aria-label="Show step {s["i"]} '
|
||||
f'full screen"><img src="{src}" loading="lazy" alt="Screen after: {title}"></button>')
|
||||
else:
|
||||
thumb = '<div class="tstep-noshot">No screenshot</div>'
|
||||
error = f'<pre>{br.esc(s["error"])}</pre>' if s.get('error') else ''
|
||||
items.append(
|
||||
f'<li class="tstep {status}">{thumb}<div class="tstep-info {status}"><span class="num">{s["i"]}</span><div>'
|
||||
f'<div class="tstep-title"><span class="mono-label">{br.esc(s["kind"])}</span><span>{title}</span></div>'
|
||||
f'<div class="mono-label">at {s["t"]:.1f}s{duration}</div>{error}</div></div></li>')
|
||||
passed = sum(s['status'] == 'passed' for s in steps)
|
||||
return br.fold('Scenario steps', f'<span class="mono-label">{passed} of {len(steps)} passed</span>',
|
||||
f'<ol class="tsteps">{"".join(items)}</ol>')
|
||||
|
||||
|
||||
def render_errors(console):
|
||||
console_errors, request_errors = console.get('consoleErrors') or [], console.get('requestErrors') or []
|
||||
if not console_errors and not request_errors:
|
||||
return ''
|
||||
items = [f'<li>{br.esc(e.get("text", ""))}</li>' for e in console_errors[:50]]
|
||||
items += [f'<li>{e.get("status") or "failed"} {br.esc(e.get("method", ""))} {br.esc(e.get("url", ""))}'
|
||||
f'{" · " + br.esc(e["error"]) if e.get("error") else ""}</li>' for e in request_errors[:50]]
|
||||
return br.fold('Browser errors', f'<span class="mono-label">{len(console_errors)} console · '
|
||||
f'{len(request_errors)} requests</span>', f'<ul class="err-list">{"".join(items)}</ul>')
|
||||
|
||||
|
||||
def render_panel(run, verdict, files, scale, screen=None):
|
||||
data = run['data']
|
||||
prompt = (run['result'].get('prompts') or [{}])[0].get('text', '')
|
||||
load_error = f'<div class="box err">Could not read the trace: {br.esc(data["loadError"])}</div>' if data.get('loadError') else ''
|
||||
return (f'{load_error}{render_video(screen)}'
|
||||
f'{render_expected(run, verdict)}{render_run_box(run)}'
|
||||
f'{render_scenario_steps(run["steps"], files)}'
|
||||
f'{br.render_steps(data["calls"], data["events"], *scale, prompt)}'
|
||||
f'{render_errors(run["console"])}')
|
||||
|
||||
|
||||
def render_case(plan, status):
|
||||
branch, master = status.get('branch') or {}, status.get('master')
|
||||
kinds = [s['kind'] for s in plan['scenarios']]
|
||||
lines = [f'Branch {branch.get("commit") or "?"}' + (' with local changes' if branch.get('dirty') else '')
|
||||
+ (f' on {branch["url"]}' if branch.get('url') else ' on a fresh instance')]
|
||||
if master is not None:
|
||||
lines.append(f'Master {(master.get("commit") or "(building)")[:11]} (the merge base) on a fresh instance')
|
||||
lines.append(f'{len(kinds)} scenarios: ' + ', '.join(f'{kinds.count(k)} {k}' for k in KIND_LABELS if kinds.count(k)))
|
||||
lines.append(f'{status.get("concurrency", 1)} at a time')
|
||||
return (f'<div class="case"><div class="box"><div class="mono-label">Feature</div><div class="msg">'
|
||||
f'{br.esc(plan.get("feature") or "-")}</div></div><div class="box"><div class="mono-label">Run</div>'
|
||||
f'<div class="msg">{br.esc(chr(10).join(lines))}</div></div></div>')
|
||||
|
||||
|
||||
def notes(status, runs, verdicts):
|
||||
parts = []
|
||||
if not verdicts and status.get('phase') == 'done':
|
||||
parts.append('The runs are not judged yet. This page reloads when the verdicts and the token usage are ready.')
|
||||
parts.append('Times are seconds from the start of each scenario. Several scenarios ran at the same time on one '
|
||||
'machine, so their times include the shared load. Tokens and cost come from LangSmith, matched by the '
|
||||
'traces of the threads of each scenario. An asterisk on a time marks text placed at the previous tool call.')
|
||||
if any(r['master'] for r in runs):
|
||||
parts.append('Visual scenarios ran on a fresh branch instance and on a fresh master instance, so both start '
|
||||
'from the same empty state. Master is the merge base of the branch.')
|
||||
if status.get('stale'):
|
||||
parts.append('Warning: these packages had source changes newer than their build, so the branch instance ran '
|
||||
f'an old build: {", ".join(status["stale"])}.')
|
||||
return br.esc(' '.join(parts))
|
||||
|
||||
|
||||
def render_visual(results, runs, files):
|
||||
if not results:
|
||||
return ''
|
||||
titles = {r['scenario']['id']: r['scenario']['title'] for r in runs}
|
||||
cards = []
|
||||
for r in results:
|
||||
cls, label = VISUAL.get(r['status'], ('vis-only', r['status']))
|
||||
ratio = f'{100 * r["ratio"]:.2f}% of pixels' if r.get('ratio') is not None else ''
|
||||
sizes = (f' · size {r["masterSize"][0]}×{r["masterSize"][1]} → {r["branchSize"][0]}×{r["branchSize"][1]}'
|
||||
if r.get('branchSize') and r['branchSize'] != r.get('masterSize') else '')
|
||||
figures = ''
|
||||
for caption, key in (('Master', 'master'), ('Branch', 'branch'), ('Difference', 'diff')):
|
||||
src = files(r, key)
|
||||
figures += (f'<figure><figcaption class="mono-label">{caption}</figcaption>'
|
||||
+ (f'<a href="{src}" target="_blank"><img src="{src}" loading="lazy" alt="{caption}: {br.esc(r["name"])}"></a>'
|
||||
if src else '<p class="dim">No screen.</p>') + '</figure>')
|
||||
tree = (f'<details><summary>Accessibility tree diff</summary><pre>{br.esc(r["ariaDiff"])}</pre></details>'
|
||||
if r.get('ariaDiff') else '')
|
||||
cards.append(
|
||||
f'<div class="vis {"changed" if r["status"] != "unchanged" else ""}"><div class="head">'
|
||||
f'<h3>{br.esc(titles.get(r["scenario"], r["scenario"]))} › {br.esc(r["name"])}</h3>'
|
||||
f'<span class="mono-label">{ratio}{sizes}</span><span class="badge {cls}">{label}</span></div>'
|
||||
f'<div class="vis-grid">{figures}</div>{tree}</div>')
|
||||
changed = sum(r['status'] != 'unchanged' for r in results)
|
||||
return br.fold('Visual check against master',
|
||||
f'<span class="mono-label">{len(results)} captures · {changed} changed</span>',
|
||||
'<p class="fineprint">Pink pixels in the difference changed by more than a small color threshold. '
|
||||
'Magenta boxes are masked parts, such as LLM text. The accessibility tree diff shows changes in '
|
||||
'structure, roles and labels.</p>' + ''.join(cards))
|
||||
|
||||
|
||||
STEP_VIEWER_JS = """
|
||||
// A step screenshot opens full screen. The listeners are delegated, because the live page replaces the panels.
|
||||
(()=>{
|
||||
const viewer = document.createElement('div');
|
||||
viewer.className = 'theater shot-viewer';
|
||||
viewer.hidden = true;
|
||||
viewer.setAttribute('role', 'dialog');
|
||||
viewer.setAttribute('aria-modal', 'true');
|
||||
viewer.setAttribute('aria-label', 'Step screenshot');
|
||||
viewer.innerHTML = '<div class="theater-bar"><div class="shot-viewer-info"></div><div class="shot-viewer-actions">'
|
||||
+ '<span class="mono-label" data-shot-count></span>'
|
||||
+ '<button type="button" class="btn" data-shot-move="-1">Previous</button>'
|
||||
+ '<button type="button" class="btn" data-shot-move="1">Next</button>'
|
||||
+ '<button type="button" class="btn accent" data-shot-close>Close</button></div></div>'
|
||||
+ '<div class="shot-viewer-img"><img alt=""></div>'
|
||||
+ '<p class="fineprint">Press the left or right arrow key to change the step. Press Esc to close.</p>';
|
||||
document.body.append(viewer);
|
||||
let run = null, index = 0, opener = null;
|
||||
const shots = () => [...(document.getElementById(run)?.querySelectorAll('.tstep-shot') || [])];
|
||||
function render(){
|
||||
const items = shots();
|
||||
if (!items.length) return close();
|
||||
index = Math.min(index, items.length - 1);
|
||||
const button = items[index], img = viewer.querySelector('.shot-viewer-img img');
|
||||
img.src = button.dataset.src;
|
||||
img.alt = button.querySelector('img').alt;
|
||||
viewer.querySelector('.shot-viewer-info').replaceChildren(button.closest('.tstep').querySelector('.tstep-info').cloneNode(true));
|
||||
viewer.querySelector('[data-shot-count]').textContent = `${index + 1} of ${items.length}`;
|
||||
}
|
||||
function open(button){
|
||||
run = button.closest('.run').id;
|
||||
index = Math.max(0, shots().indexOf(button));
|
||||
opener = button;
|
||||
viewer.hidden = false;
|
||||
document.body.classList.add('theater-open');
|
||||
render();
|
||||
viewer.querySelector('[data-shot-close]').focus();
|
||||
}
|
||||
function close(){
|
||||
viewer.hidden = true;
|
||||
run = null;
|
||||
document.body.classList.remove('theater-open');
|
||||
if (opener?.isConnected) opener.focus();
|
||||
}
|
||||
function move(step){
|
||||
const count = shots().length;
|
||||
if (!count) return close();
|
||||
index = (index + step + count) % count;
|
||||
render();
|
||||
}
|
||||
document.addEventListener('click', e=>{
|
||||
const button = e.target.closest('.tstep-shot');
|
||||
if (button) open(button);
|
||||
});
|
||||
viewer.addEventListener('click', e=>{
|
||||
const step = e.target.closest('[data-shot-move]');
|
||||
if (step) move(Number(step.dataset.shotMove));
|
||||
else if (e.target.closest('[data-shot-close]') || e.target.classList.contains('shot-viewer-img')) close();
|
||||
});
|
||||
document.addEventListener('keydown', e=>{
|
||||
if (viewer.hidden) return;
|
||||
if (e.key === 'Escape') close();
|
||||
else if (e.key === 'ArrowLeft' || e.key === 'ArrowRight') { e.preventDefault(); move(e.key === 'ArrowRight' ? 1 : -1); }
|
||||
});
|
||||
})();
|
||||
"""
|
||||
|
||||
VIDEO_JS = """
|
||||
// The video of a run plays only while its panel shows.
|
||||
const showPanel = show;
|
||||
show = (id)=>{
|
||||
showPanel(id);
|
||||
document.querySelectorAll('.run:not(.active) .run-video video').forEach(v=>v.pause());
|
||||
};
|
||||
"""
|
||||
|
||||
RELOAD_JS = """
|
||||
// Served by the live page: reload when the report is built again, for example after the verdicts.
|
||||
(()=>{
|
||||
if (!location.protocol.startsWith('http')) return;
|
||||
const saved = JSON.parse(sessionStorage.getItem('report-view') || 'null');
|
||||
if (saved) {
|
||||
sessionStorage.removeItem('report-view');
|
||||
if (document.getElementById(saved.run)) show(saved.run);
|
||||
window.scrollTo(0, saved.y);
|
||||
}
|
||||
setInterval(async ()=>{
|
||||
try {
|
||||
const version = (await (await fetch('version.txt', {cache: 'no-store'})).text()).trim();
|
||||
if (version && version !== REPORT_VERSION) {
|
||||
const run = document.querySelector('.run.active')?.id;
|
||||
sessionStorage.setItem('report-view', JSON.stringify({run, y: window.scrollY}));
|
||||
location.reload();
|
||||
}
|
||||
} catch {}
|
||||
}, 3000);
|
||||
})();
|
||||
"""
|
||||
|
||||
|
||||
def build(run_dir):
|
||||
"""Write report/index.html with its images and videos. Returns its path. Two builds of one run can
|
||||
happen at the same time (run.py, report.py, the live page), so they take turns."""
|
||||
run_dir.mkdir(parents=True, exist_ok=True)
|
||||
with (run_dir / 'report.lock').open('w') as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||
return _build(run_dir)
|
||||
|
||||
|
||||
def _build(run_dir):
|
||||
plan, status, runs = load(run_dir)
|
||||
verdicts = br.load_verdicts(run_dir)
|
||||
report = run_dir / 'report'
|
||||
for sub in ('runs', 'screens', 'visual'):
|
||||
shutil.rmtree(report / sub, ignore_errors=True)
|
||||
report.mkdir(parents=True, exist_ok=True)
|
||||
for sub in ('fonts', 'brand'):
|
||||
if (br.ASSET_CACHE / sub).exists():
|
||||
shutil.copytree(br.ASSET_CACHE / sub, report / sub, dirs_exist_ok=True)
|
||||
|
||||
scale = br.token_scale([(r['name'], {}, r['data']) for r in runs])
|
||||
rows, panels, screens = [], [], []
|
||||
for i, run in enumerate(runs):
|
||||
name, sub = run['name'], run_dir / 'runs' / run['name']
|
||||
shots = report / 'runs' / name
|
||||
for png in sub.glob('*.png'):
|
||||
if png.name not in ('live.png', 'screen-poster.png'):
|
||||
shots.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy(png, shots / png.name)
|
||||
verdict = verdicts.get(name)
|
||||
screen = None
|
||||
if (sub / 'screen.webm').exists():
|
||||
(report / 'screens').mkdir(exist_ok=True)
|
||||
shutil.copy(sub / 'screen.webm', report / 'screens' / f'{name}.webm')
|
||||
screen = {'name': name, 'title': run_title(run), 'duration': run['data']['duration'], 'type': 'video',
|
||||
'src': f'screens/{name}.webm', 'verdict': (verdict or {}).get('verdict'),
|
||||
'label': KIND_LABELS.get(run['scenario']['kind'], '')}
|
||||
if (sub / 'screen-poster.png').exists():
|
||||
shutil.copy(sub / 'screen-poster.png', report / 'screens' / f'{name}-poster.png')
|
||||
screen['poster'] = f'screens/{name}-poster.png'
|
||||
screens.append(screen)
|
||||
active = ' active' if i == 0 else ''
|
||||
rows.append(f'<tr class="row{active}" data-run="{name}" onclick="show(\'{name}\')">'
|
||||
+ row_cells(run, result_html(run, verdict)) + '</tr>')
|
||||
panels.append(f'<section class="run{active}" id="{name}">'
|
||||
+ render_panel(run, verdict, f'runs/{name}/', scale, screen) + '</section>')
|
||||
|
||||
results = read_json(run_dir / 'visual.json', [])
|
||||
for r in results:
|
||||
target = report / 'visual' / r['scenario']
|
||||
target.mkdir(parents=True, exist_ok=True)
|
||||
for key, source in (('branch', run_dir / 'runs' / r['scenario'] / 'captures' / f'{r["name"]}.png'),
|
||||
('master', run_dir / 'runs' / f'{r["scenario"]}--master' / 'captures' / f'{r["name"]}.png'),
|
||||
('diff', run_dir / (r.get('diff') or 'missing'))):
|
||||
if source.is_file():
|
||||
shutil.copy(source, target / f'{r["name"]}.{key}.png')
|
||||
|
||||
def visual_file(r, key):
|
||||
return (f'visual/{r["scenario"]}/{r["name"]}.{key}.png'
|
||||
if (report / 'visual' / r['scenario'] / f'{r["name"]}.{key}.png').exists() else None)
|
||||
|
||||
replay = br.render_replay(screens, 'Each video shows one scenario in the n8n UI. Press Esc to close.')
|
||||
version = str(time.time_ns())
|
||||
page = br.fill_template(
|
||||
br.esc(plan.get('title') or 'Instance AI feature test'), ''.join(rows), notes(status, runs, verdicts),
|
||||
''.join(panels), replay=replay, after_table=render_visual(results, runs, visual_file), head=CSS,
|
||||
scripts=(f'<script>const SCREENS = {json.dumps(screens)}; const REPORT_VERSION = "{version}";</script>'
|
||||
f'<script>window.addEventListener("DOMContentLoaded", ()=>{{{VIDEO_JS}{RELOAD_JS}}});</script>'
|
||||
f'<script>{STEP_VIEWER_JS}</script>'),
|
||||
case=render_case(plan, status), thead=THEAD, kicker='Assistant UI test')
|
||||
(report / 'index.html').write_text(page)
|
||||
(report / 'version.txt').write_text(version)
|
||||
return report / 'index.html'
|
||||
|
||||
|
||||
def clock(status):
|
||||
started = status.get('startedAt')
|
||||
elapsed = seconds((status.get('finishedAt') or time.time()) - started) if started else ''
|
||||
return ' · '.join(x for x in (f'{status.get("concurrency", 1)} at a time', elapsed, status.get('step')) if x)
|
||||
|
||||
|
||||
def live_box(status, runs):
|
||||
cards = []
|
||||
for run in runs:
|
||||
name, title = run['name'], br.esc(run_title(run))
|
||||
cards.append(
|
||||
f'<div class="live-card"><div class="live-screen"><button type="button" class="live-zoom" '
|
||||
f'data-zoom="{name}" aria-label="Show {title} full screen"><img data-shot="{name}" '
|
||||
f'alt="{title} in the n8n UI" hidden></button>'
|
||||
f'<div class="live-empty" data-shot-empty="{name}">Waiting for the first screenshot</div></div>'
|
||||
f'<div class="reel-meta"><span class="run-name">{title}</span>'
|
||||
f'<span data-live-badge="{name}">{status_badge(run["entry"].get("status", "waiting"))}</span></div></div>')
|
||||
return (f'<div class="box"><div class="head"><h2>Live</h2><span class="mono-label" id="live-clock">'
|
||||
f'{br.esc(clock(status))}</span></div><div class="live-grid">{"".join(cards)}</div></div>')
|
||||
|
||||
|
||||
def render_live(run_dir):
|
||||
plan, status, runs = load(run_dir)
|
||||
verdicts = br.load_verdicts(run_dir)
|
||||
scale = br.token_scale([(r['name'], {}, r['data']) for r in runs])
|
||||
rows, panels = [], []
|
||||
for i, run in enumerate(runs):
|
||||
name, active = run['name'], ' active' if i == 0 else ''
|
||||
rows.append(f'<tr class="row{active}" data-run="{name}" onclick="show(\'{name}\')">'
|
||||
+ row_cells(run, result_html(run, verdicts.get(name))) + '</tr>')
|
||||
panels.append(f'<section class="run{active}" id="{name}">'
|
||||
+ render_panel(run, verdicts.get(name), f'/files/runs/{name}/', scale) + '</section>')
|
||||
text = ('This page updates while the scenarios run, and changes to the full report when the run is done. '
|
||||
'Tokens, cost, the visual check and verdicts come after the run.')
|
||||
return br.fill_template(
|
||||
br.esc(plan.get('title') or 'Instance AI feature test'), ''.join(rows), br.esc(text), ''.join(panels),
|
||||
replay=live_box(status, runs), head=CSS,
|
||||
scripts=f'<script>const SCREENS = [];</script><script>{ls.LIVE_JS}</script><script>{STEP_VIEWER_JS}</script>',
|
||||
case=render_case(plan, status), thead=THEAD, kicker='Assistant UI test')
|
||||
|
||||
|
||||
class ScenarioApp:
|
||||
"""The live page for live_server.serve(): one card, table row and panel for each scenario run."""
|
||||
|
||||
def __init__(self, run_dir):
|
||||
self.run_dir = run_dir
|
||||
|
||||
def entry(self, name):
|
||||
return ls.read_status(self.run_dir).get('runs', {}).get(name)
|
||||
|
||||
def finished(self, name):
|
||||
return (self.entry(name) or {}).get('status') in FINISHED
|
||||
|
||||
def done(self):
|
||||
return ls.read_status(self.run_dir).get('phase') == 'done' and (self.run_dir / 'report' / 'index.html').exists()
|
||||
|
||||
def page(self):
|
||||
return render_live(self.run_dir)
|
||||
|
||||
def state(self):
|
||||
plan, status, runs = load(self.run_dir)
|
||||
verdicts = br.load_verdicts(self.run_dir)
|
||||
scale = br.token_scale([(r['name'], {}, r['data']) for r in runs])
|
||||
state = {'phase': status.get('phase', 'running'), 'clock': clock(status), 'runs': {}}
|
||||
if self.done():
|
||||
state['report'] = '/report/index.html'
|
||||
for run in runs:
|
||||
name = run['name']
|
||||
shot = self.shot(name)
|
||||
state['runs'][name] = {
|
||||
'row': row_cells(run, result_html(run, verdicts.get(name))),
|
||||
'panel': render_panel(run, verdicts.get(name), f'/files/runs/{name}/', scale),
|
||||
'badge': status_badge(run['entry'].get('status', 'waiting')),
|
||||
'shot': shot.stat().st_mtime if shot.exists() else None,
|
||||
}
|
||||
return state
|
||||
|
||||
def shot(self, name):
|
||||
return self.run_dir / 'runs' / name / 'live.png'
|
||||
|
||||
def casts(self, name):
|
||||
return ()
|
||||
|
||||
def file(self, path):
|
||||
"""The step screenshots of a run while it runs: /files/runs/<name>/<file>.png."""
|
||||
if not path.startswith('/files/runs/') or not path.endswith('.png'):
|
||||
return None
|
||||
base = (self.run_dir / 'runs').resolve()
|
||||
candidate = (self.run_dir / path.removeprefix('/files/')).resolve()
|
||||
return candidate if candidate.is_relative_to(base) and candidate.is_file() else None
|
||||
|
||||
|
||||
def triage(run, visual):
|
||||
"""The reasons to look closer at a run. An empty list means that the run looks clean: every step and check
|
||||
passed, with no browser error, no failed tool call, and no visual change."""
|
||||
reasons = []
|
||||
status = run['entry'].get('status')
|
||||
if status != 'passed':
|
||||
reasons.append(f'the run {status}')
|
||||
failed = [s for s in run['steps'] if s['status'] == 'failed']
|
||||
if failed:
|
||||
reasons.append(f'{len(failed)} failed step(s) or check(s)')
|
||||
console = run['console']
|
||||
errors = len(console.get('consoleErrors') or []) + len(console.get('requestErrors') or [])
|
||||
if errors:
|
||||
reasons.append(f'{errors} browser error(s)')
|
||||
tool_errors = sum(e['error'] for e in run['data']['events'] if e['kind'] == 'tool')
|
||||
if tool_errors:
|
||||
reasons.append(f'{tool_errors} failed tool call(s)')
|
||||
changed = [r['name'] for r in visual if r['scenario'] == run['scenario']['id'] and r['status'] != 'unchanged']
|
||||
if changed and not run['master']:
|
||||
reasons.append(f'visual change in {", ".join(changed)}')
|
||||
return reasons
|
||||
|
||||
|
||||
def summarize(run_dir):
|
||||
"""Plain text for the judge: a triage list, then each run with its steps, final reply, errors and visual
|
||||
results."""
|
||||
plan, status, runs = load(run_dir)
|
||||
visual = read_json(run_dir / 'visual.json', [])
|
||||
out = [f'# {plan.get("title", "")}', '', 'Feature:', plan.get('feature') or '-', '', '## Triage', '',
|
||||
'Clean runs passed every step and check, with no browser error, failed tool call or visual change.',
|
||||
'For these, check that the last reply and the final screenshot match the expected result.',
|
||||
'Look closely at the other runs: their screenshots, errors and diffs.', '']
|
||||
judged = [r for r in runs if not r['master']]
|
||||
for run in judged:
|
||||
reasons = triage(run, visual)
|
||||
out.append(f'- {run["name"]}: ' + ('clean' if not reasons else 'look closer: ' + '; '.join(reasons)))
|
||||
out.append('')
|
||||
for run in runs:
|
||||
s, result, data = run['scenario'], run['result'], run['data']
|
||||
out += [f'## {run["name"]}: {run_title(run)} ({s["kind"]}, {"master" if run["master"] else "branch"})',
|
||||
f'Expected: {s["expected"]}',
|
||||
f'Status: {run["entry"].get("status")}' + (f' ({result["error"]})' if result.get('error') else ''),
|
||||
'Steps:']
|
||||
for st in run['steps']:
|
||||
shot = f' (runs/{run["name"]}/{st["shot"]})' if st.get('shot') else ''
|
||||
out.append(f' {st["i"]}. [{st["status"]}] {st["kind"]}: {st["title"]}{shot}')
|
||||
if st.get('error'):
|
||||
out.append(' ' + st['error'].replace('\n', '\n ')[:1500])
|
||||
for p in result.get('prompts') or []:
|
||||
out.append(f'Prompt (thread {p.get("thread")}): {p["text"]}')
|
||||
tools = [e for e in data['events'] if e['kind'] == 'tool']
|
||||
if tools:
|
||||
out.append('Tool calls: ' + ', '.join(e['name'] + (' (error)' if e['error'] else '') for e in tools))
|
||||
texts = [e['text'] for e in data['events'] if e['kind'] == 'text']
|
||||
if texts:
|
||||
out.append('Last reply: ' + texts[-1].strip()[:2000])
|
||||
for d in run['meta'].get('decisions') or []:
|
||||
out.append(f'Confirmation: {d["decision"]} [{d.get("toolName")}]')
|
||||
console = run['console']
|
||||
for e in (console.get('consoleErrors') or [])[:8]:
|
||||
out.append(f'Console error: {e.get("text", "")[:300]}')
|
||||
for e in (console.get('requestErrors') or [])[:8]:
|
||||
out.append(f'Request error: {e.get("status")} {e.get("method")} {e.get("url")}')
|
||||
if s['kind'] == 'visual' and not run['master']:
|
||||
for r in (v for v in visual if v['scenario'] == s['id']):
|
||||
ratio = f', {100 * r["ratio"]:.2f}% of pixels' if r.get('ratio') is not None else ''
|
||||
out.append(f'Visual {r["name"]}: {r["status"]}{ratio}')
|
||||
if r.get('ariaDiff'):
|
||||
out.append(' Accessibility tree diff:\n ' + '\n '.join(r['ariaDiff'].splitlines()[:40]))
|
||||
out.append('')
|
||||
return '\n'.join(out)
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
sys.exit(__doc__)
|
||||
run_dir = Path(sys.argv[1]).resolve()
|
||||
if '--summary' in sys.argv:
|
||||
print(summarize(run_dir))
|
||||
else:
|
||||
print(build(run_dir))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -1,931 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Run a test plan of Instance AI scenarios and build one report. The judge step is separate.
|
||||
|
||||
run.py --plan <dir>/scenarios.json [--only id1,id2] [--concurrency 4]
|
||||
run.py --rerun last|<run-dir> the plan of an earlier run, again in a new folder
|
||||
run.py --plan ... --try a quick check of new scripts: no master, no live page, no report
|
||||
run.py --plan ... --estimate print the expected cost and time, then stop
|
||||
run.py --check check the setup, then stop
|
||||
|
||||
Steps:
|
||||
1. Check the setup, and build this checkout with turbo (a few seconds when nothing changed).
|
||||
2. Start fresh n8n instances from the build. Each feature and regression scenario gets its own
|
||||
instance, unless it sets "isolated": false. Instances start ahead of their scenario, so a
|
||||
scenario does not wait for one.
|
||||
3. Run the scenarios, several at the same time.
|
||||
4. At the same time: build the master merge base (baseline.py), start a fresh master instance and
|
||||
a fresh branch instance, and run each visual scenario on both. A master run that passed is kept
|
||||
and used again while the merge base and the script stay the same.
|
||||
5. Compare the visual captures, write summary.txt for the judge, and build the report.
|
||||
6. Fetch the token usage from LangSmith and build the report again.
|
||||
A local page shows the runs live (live_page.py). It opens in the default browser, unless --no-open.
|
||||
Settings come from .env.local and .env.ai-surfaces.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import difflib
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import signal
|
||||
import socket
|
||||
import statistics
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import webbrowser
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import baseline
|
||||
import report
|
||||
from common import CACHE, HERE, PLANS_ROOT, REPO_ROOT, RUNS_ROOT, SHARED, Instance, client, git, setting
|
||||
from langsmith_usage import fetch_calls, totals
|
||||
from live import enabled as live_enabled
|
||||
from run_instance_ai import decide
|
||||
|
||||
KINDS = ('feature', 'regression', 'visual')
|
||||
LIB = HERE.parent / 'scenarios'
|
||||
ID = re.compile(r'^[a-z0-9][a-z0-9-]{0,60}$')
|
||||
MASTER_CACHE = CACHE / 'master-runs'
|
||||
MASTER_CACHE_DAYS = 14
|
||||
PRINT_LOCK = threading.Lock()
|
||||
RUNNERS = set()
|
||||
S = '.agents/skills/testmax-assistant-ui/scripts'
|
||||
|
||||
|
||||
def say(line, prefix=None):
|
||||
with PRINT_LOCK:
|
||||
print(f'[{prefix}] {line}' if prefix else line, flush=True)
|
||||
|
||||
|
||||
def step(title):
|
||||
say(f'\n=== {title} ===')
|
||||
|
||||
|
||||
def now_iso():
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def slug(text, words=8):
|
||||
return '-'.join(re.findall(r'[a-z0-9]+', text.lower())[:words]) or 'run'
|
||||
|
||||
|
||||
def script_path(spec, base):
|
||||
return LIB / spec[4:] if spec.startswith('lib:') else (base / spec).resolve()
|
||||
|
||||
|
||||
def load_plan(path):
|
||||
"""Read and check scenarios.json. Returns the plan with each script path resolved."""
|
||||
path = Path(path).resolve()
|
||||
plan = json.loads(path.read_text())
|
||||
errors, seen = [], set()
|
||||
if not plan.get('scenarios'):
|
||||
errors.append('The plan has no scenarios.')
|
||||
if not isinstance(plan.get('featureFlags', {}), dict):
|
||||
errors.append('featureFlags must be an object of flag names and values.')
|
||||
for s in plan.get('scenarios', []):
|
||||
sid = s.get('id', '')
|
||||
if not ID.match(sid) or '--' in sid:
|
||||
errors.append(f'Scenario id "{sid}": use a-z, 0-9 and single "-" characters.')
|
||||
if sid in seen:
|
||||
errors.append(f'Scenario id "{sid}" is used two times.')
|
||||
seen.add(sid)
|
||||
if s.get('kind') not in KINDS:
|
||||
errors.append(f'Scenario "{sid}": kind must be one of {", ".join(KINDS)}.')
|
||||
if not s.get('script') or not script_path(s['script'], path.parent).is_file():
|
||||
errors.append(f'Scenario "{sid}": script {s.get("script")} not found.')
|
||||
if not s.get('title') or not s.get('expected'):
|
||||
errors.append(f'Scenario "{sid}": give a title and an expected outcome.')
|
||||
if errors:
|
||||
sys.exit('The plan is not valid:\n ' + '\n '.join(errors))
|
||||
for s in plan['scenarios']:
|
||||
s['scriptPath'] = str(script_path(s['script'], path.parent))
|
||||
plan.setdefault('source', str(path))
|
||||
plan['name'] = path.parent.name if path.parent.name != 'plan' else slug(plan.get('title') or 'run')
|
||||
return plan
|
||||
|
||||
|
||||
def run_dirs():
|
||||
"""Earlier run folders, newest first."""
|
||||
if not RUNS_ROOT.exists():
|
||||
return []
|
||||
found = [d for d in RUNS_ROOT.iterdir() if (d / 'plan' / 'scenarios.json').is_file()]
|
||||
return sorted(found, key=lambda d: (d / 'plan' / 'scenarios.json').stat().st_mtime, reverse=True)
|
||||
|
||||
|
||||
def rerun_plan(arg):
|
||||
"""The plan to run again. The source plan wins over the copy, so a fixed script is used."""
|
||||
if arg == 'last':
|
||||
found = run_dirs()
|
||||
if not found:
|
||||
sys.exit(f'No earlier run in {RUNS_ROOT}.')
|
||||
run_dir = found[0]
|
||||
else:
|
||||
run_dir = Path(arg).expanduser().resolve()
|
||||
copy = run_dir / 'plan' / 'scenarios.json'
|
||||
if not copy.is_file():
|
||||
sys.exit(f'{run_dir} has no plan/scenarios.json.')
|
||||
source = json.loads(copy.read_text()).get('source')
|
||||
if source and Path(source).is_file():
|
||||
say(f'rerun of {run_dir.name}, with the current plan and scripts in {Path(source).parent}')
|
||||
return source
|
||||
say(f'rerun of {run_dir.name}, with the scripts copied into that run')
|
||||
return copy
|
||||
|
||||
|
||||
def new_run_dir(root, name):
|
||||
base = root / f'{datetime.now():%Y%m%d-%H%M%S}-{name}'
|
||||
run_dir, n = base, 2
|
||||
while run_dir.exists():
|
||||
run_dir, n = base.with_name(f'{base.name}-{n}'), n + 1
|
||||
run_dir.mkdir(parents=True)
|
||||
return run_dir
|
||||
|
||||
|
||||
def copy_plan(plan, run_dir):
|
||||
"""Keep the plan and its scripts in the run folder, so --rerun can repeat the run."""
|
||||
plan_dir = run_dir / 'plan'
|
||||
(plan_dir / 'scripts').mkdir(parents=True, exist_ok=True)
|
||||
saved = json.loads(json.dumps(plan))
|
||||
saved.pop('name', None)
|
||||
for s in saved['scenarios']:
|
||||
target = plan_dir / 'scripts' / f'{s["id"]}.mjs'
|
||||
shutil.copy(s.pop('scriptPath'), target)
|
||||
s['script'] = f'scripts/{target.name}'
|
||||
(plan_dir / 'scenarios.json').write_text(json.dumps(saved, indent=2))
|
||||
|
||||
|
||||
class Progress:
|
||||
"""Writes status.json, which the live page reads."""
|
||||
|
||||
def __init__(self, run_dir, data):
|
||||
self.path, self.lock, self.data = run_dir / 'status.json', threading.Lock(), data
|
||||
self.save()
|
||||
|
||||
def save(self):
|
||||
tmp = self.path.with_suffix('.tmp')
|
||||
tmp.write_text(json.dumps(self.data, indent=2))
|
||||
os.replace(tmp, self.path)
|
||||
|
||||
def update(self, **fields):
|
||||
with self.lock:
|
||||
self.data.update(fields)
|
||||
self.save()
|
||||
|
||||
def run(self, name, **fields):
|
||||
with self.lock:
|
||||
self.data['runs'][name].update(fields)
|
||||
self.save()
|
||||
|
||||
def finish(self):
|
||||
with self.lock:
|
||||
for entry in self.data['runs'].values():
|
||||
if entry['status'] in ('waiting', 'starting'):
|
||||
entry.update(status='skipped', reason=entry.get('reason') or 'The run did not start.')
|
||||
elif entry['status'] == 'running':
|
||||
entry['status'] = 'error'
|
||||
if self.data.get('phase') != 'done':
|
||||
self.data.update(phase='done', step='Done', finishedAt=time.time())
|
||||
self.save()
|
||||
|
||||
|
||||
def close_stream(res):
|
||||
"""Close an event stream that another thread reads. close() alone waits for the next server
|
||||
keep-alive (about 13 seconds), because the reader holds the buffer lock. A socket shutdown ends the
|
||||
read at once."""
|
||||
try:
|
||||
with socket.socket(fileno=os.dup(res.fileno())) as sock:
|
||||
sock.shutdown(socket.SHUT_RDWR)
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
try:
|
||||
res.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
class ThreadEvents:
|
||||
"""Reads the event stream of each thread that a scenario opens. The stream starts at the first event."""
|
||||
|
||||
def __init__(self, base_url, out, auto_confirm, prefix):
|
||||
self.client = client(base_url).login()
|
||||
self.out, self.auto_confirm, self.prefix = out, auto_confirm, prefix
|
||||
self.file, self.lock = (out / 'events.jsonl').open('a'), threading.Lock()
|
||||
self.run_ids, self.decisions, self.streams, self.closed = [], [], [], False
|
||||
|
||||
def follow(self, thread_id):
|
||||
threading.Thread(target=self._read, args=(thread_id,), daemon=True).start()
|
||||
|
||||
def _read(self, thread_id):
|
||||
try:
|
||||
res = self.client.open_stream(f'/rest/instance-ai/events/{thread_id}')
|
||||
except Exception as e:
|
||||
say(f'event stream of {thread_id} did not open: {e}', self.prefix)
|
||||
return
|
||||
self.streams.append(res)
|
||||
data, name = [], None
|
||||
try:
|
||||
for raw in res:
|
||||
line = raw.decode(errors='replace').rstrip('\r\n')
|
||||
if line.startswith('data:'):
|
||||
data.append(line[5:].lstrip())
|
||||
elif line.startswith('event:'):
|
||||
name = line[6:].strip()
|
||||
elif line == '' and data:
|
||||
try:
|
||||
self._event(json.loads('\n'.join(data)))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
data, name = [], None
|
||||
except Exception:
|
||||
# close() ends the stream when the scenario stops.
|
||||
pass
|
||||
|
||||
def _event(self, event):
|
||||
with self.lock:
|
||||
if self.closed:
|
||||
return
|
||||
self.file.write(json.dumps(event) + '\n')
|
||||
self.file.flush()
|
||||
payload = event.get('payload') or {}
|
||||
if event.get('type') == 'run-start' and payload.get('langsmithRunId'):
|
||||
self.run_ids.append(payload['langsmithRunId'])
|
||||
if event.get('type') == 'confirmation-request' and self.auto_confirm:
|
||||
body, label = decide(payload)
|
||||
try:
|
||||
self.client.request('POST', f'/rest/instance-ai/confirm/{payload["requestId"]}', body)
|
||||
ok = True
|
||||
except RuntimeError as e:
|
||||
ok, label = False, f'{label} (failed: {str(e)[:200]})'
|
||||
self.decisions.append({'at': now_iso(), 'toolName': payload.get('toolName'), 'decision': label, 'ok': ok})
|
||||
say(f'confirm: {label} [{payload.get("toolName")}]', self.prefix)
|
||||
|
||||
def close(self):
|
||||
with self.lock:
|
||||
self.closed = True
|
||||
self.file.close()
|
||||
for res in self.streams:
|
||||
close_stream(res)
|
||||
|
||||
|
||||
def run_job(name, scenario, url, instance, run_dir, progress, live):
|
||||
sub = run_dir / 'runs' / name
|
||||
sub.mkdir(parents=True, exist_ok=True)
|
||||
progress.run(name, status='running', startedAt=time.time(), baseUrl=url)
|
||||
auto_confirm = scenario.get('autoConfirm', True)
|
||||
timeout_min = float(scenario.get('timeoutMin', 10))
|
||||
started = now_iso()
|
||||
try:
|
||||
events = ThreadEvents(url, sub, auto_confirm, name)
|
||||
except Exception as e:
|
||||
progress.run(name, status='error', finishedAt=time.time(), reason=f'Cannot sign in to {url}: {e}')
|
||||
return
|
||||
cmd = ['node', str(HERE / 'run_scenario.mjs'), '--out', str(sub), '--script', scenario['scriptPath'],
|
||||
'--timeout-min', str(timeout_min), '--viewport', scenario.get('viewport', '1600x1000')]
|
||||
cmd += ['--live'] if live else []
|
||||
cmd += ['--auto-confirm'] if auto_confirm else []
|
||||
proc = subprocess.Popen(cmd, cwd=HERE, env={**os.environ, 'N8N_BASE_URL': url},
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
|
||||
RUNNERS.add(proc)
|
||||
# The scenario has its own time limit. This one covers a runner that hangs outside the scenario.
|
||||
watchdog = threading.Timer(timeout_min * 60 + 180, proc.kill)
|
||||
watchdog.start()
|
||||
with (sub / 'runner.log').open('w') as log:
|
||||
for line in proc.stdout:
|
||||
log.write(line)
|
||||
say(line.rstrip(), name)
|
||||
match = re.match(r'thread (\S+)', line)
|
||||
if match:
|
||||
events.follow(match[1])
|
||||
proc.wait()
|
||||
RUNNERS.discard(proc)
|
||||
watchdog.cancel()
|
||||
# The last events of a run can arrive just after the scenario ends.
|
||||
time.sleep(1)
|
||||
events.close()
|
||||
try:
|
||||
result = json.loads((sub / 'result.json').read_text())
|
||||
except (OSError, ValueError):
|
||||
result = {'status': 'error', 'error': f'The runner stopped without a result (exit {proc.returncode}). See runner.log.'}
|
||||
(sub / 'result.json').write_text(json.dumps(result, indent=2))
|
||||
meta = {
|
||||
'kind': 'scenario', 'scenario': scenario['id'], 'instance': instance, 'baseUrl': url,
|
||||
'threadIds': result.get('threads', []), 'langsmithRunIds': events.run_ids, 'decisions': events.decisions,
|
||||
'startedAt': started, 'finishedAt': now_iso(),
|
||||
'screen': 'screen.webm' if (sub / 'screen.webm').exists() else None,
|
||||
}
|
||||
(sub / 'run.json').write_text(json.dumps(meta, indent=2))
|
||||
progress.run(name, status=result['status'], finishedAt=time.time(), reason=result.get('error'))
|
||||
|
||||
|
||||
def stale_packages():
|
||||
"""Packages with a source change in this branch that is newer than their build output."""
|
||||
try:
|
||||
base = subprocess.run(['git', 'merge-base', 'HEAD', 'origin/master'], cwd=REPO_ROOT, capture_output=True,
|
||||
text=True, check=True).stdout.strip()
|
||||
changed = subprocess.run(['git', 'diff', '--name-only', base], cwd=REPO_ROOT, capture_output=True,
|
||||
text=True, check=True).stdout.split()
|
||||
changed += [line[3:] for line in subprocess.run(['git', 'status', '--porcelain'], cwd=REPO_ROOT,
|
||||
capture_output=True, text=True, check=True).stdout.splitlines()]
|
||||
except subprocess.CalledProcessError:
|
||||
return []
|
||||
newest_dist, stale = {}, set()
|
||||
for rel in changed:
|
||||
path = REPO_ROOT / rel
|
||||
if not rel.startswith('packages/') or '/src/' not in rel or re.search(r'(\.test\.|__tests__|\.md$)', rel):
|
||||
continue
|
||||
if not path.is_file():
|
||||
continue
|
||||
pkg = REPO_ROOT / rel.split('/src/', 1)[0]
|
||||
dist = pkg / 'dist'
|
||||
if not dist.exists():
|
||||
continue
|
||||
if pkg not in newest_dist:
|
||||
newest_dist[pkg] = max((f.stat().st_mtime for f in dist.rglob('*') if f.is_file()), default=0)
|
||||
if path.stat().st_mtime > newest_dist[pkg]:
|
||||
stale.add(str(pkg.relative_to(REPO_ROOT)))
|
||||
return sorted(stale)
|
||||
|
||||
|
||||
def build_branch(log_path):
|
||||
"""Build n8n and its packages with turbo. Unchanged packages come from the turbo cache in seconds."""
|
||||
started = time.time()
|
||||
with log_path.open('w') as log:
|
||||
code = subprocess.run(['pnpm', 'turbo', 'run', 'build', '--filter=n8n...', '--output-logs=errors-only'],
|
||||
cwd=REPO_ROOT, stdout=log, stderr=subprocess.STDOUT).returncode
|
||||
text = log_path.read_text()
|
||||
if code != 0:
|
||||
sys.exit(f'The build failed. The end of {log_path}:\n' + '\n'.join(text.splitlines()[-30:]))
|
||||
cached = re.search(r'Cached:\s+(\d+) cached, (\d+) total', text)
|
||||
if cached and cached[1] == cached[2]:
|
||||
say(f'build: up to date ({time.time() - started:.0f}s)')
|
||||
else:
|
||||
say(f'build: done in {time.time() - started:.0f}s')
|
||||
|
||||
|
||||
def preflight(build, need_master, local_instances=True):
|
||||
"""Problems that stop a run (errors) or make the report less useful (warnings).
|
||||
local_instances is False with --branch-url: the branch runs then need no local build."""
|
||||
errors, warnings = [], []
|
||||
for tool in ('node', 'git') + (('pnpm',) if build or need_master else ()):
|
||||
if not shutil.which(tool):
|
||||
errors.append(f'{tool} is not on the PATH.')
|
||||
if shutil.which('node'):
|
||||
probe = ("import(process.argv[1]).then(async (m) => { const b = await m.launchBrowser(); await b.close(); })"
|
||||
".catch((e) => { console.error(e.message.split('\\n')[0]); process.exit(1); })")
|
||||
res = subprocess.run(['node', '-e', probe, str(SHARED / 'n8n_browser.mjs')], capture_output=True, text=True,
|
||||
timeout=60)
|
||||
if res.returncode != 0:
|
||||
errors.append('Playwright cannot start a browser: '
|
||||
f'{(res.stderr or res.stdout).strip()[:300]}. Run pnpm install, and install Google Chrome.')
|
||||
if not build and local_instances:
|
||||
for path in ('packages/cli/dist', 'packages/frontend/editor-ui/dist/index.html'):
|
||||
if not (REPO_ROOT / path).exists():
|
||||
errors.append(f'{path} does not exist. Build first, or run without --no-build.')
|
||||
if not setting('N8N_INSTANCE_AI_MODEL'):
|
||||
warnings.append('N8N_INSTANCE_AI_MODEL is not set in .env.local, .env.ai-surfaces or the environment. '
|
||||
'Scenarios that send a prompt can fail.')
|
||||
if not setting('LANGSMITH_API_KEY'):
|
||||
warnings.append('LANGSMITH_API_KEY is not set. The report has no tokens or cost.')
|
||||
elif (setting('LANGSMITH_TRACING') or '').lower() != 'true':
|
||||
warnings.append('LANGSMITH_TRACING is not true. The instances send no traces, so the report has no tokens or cost.')
|
||||
if (setting('N8N_DIAGNOSTICS_ENABLED') or '').lower() == 'false':
|
||||
warnings.append('N8N_DIAGNOSTICS_ENABLED is false. Instance AI sends no traces, so the report has no tokens or cost.')
|
||||
RUNS_ROOT.mkdir(parents=True, exist_ok=True)
|
||||
free = shutil.disk_usage(RUNS_ROOT).free / 1e9
|
||||
if free < 2:
|
||||
warnings.append(f'Only {free:.1f} GB free in {RUNS_ROOT}. A run needs about 0.5 GB.')
|
||||
if need_master and not baseline.built():
|
||||
CACHE.mkdir(parents=True, exist_ok=True)
|
||||
free = shutil.disk_usage(CACHE).free / 1e9
|
||||
warnings.append('The master build does not exist yet. The first build takes 15 to 20 minutes'
|
||||
+ (f', and needs about 3 GB (free: {free:.1f} GB).' if free < 5 else '.'))
|
||||
return errors, warnings
|
||||
|
||||
|
||||
def history():
|
||||
"""Cost and time of earlier runs, by scenario id. Master runs and runs without a result are left out."""
|
||||
by_id, every = {}, []
|
||||
for run_dir in run_dirs():
|
||||
try:
|
||||
runs = json.loads((run_dir / 'status.json').read_text()).get('runs', {})
|
||||
except (OSError, ValueError):
|
||||
continue
|
||||
for name, entry in runs.items():
|
||||
if name.endswith('--master') or entry.get('status') not in ('passed', 'failed'):
|
||||
continue
|
||||
if not (entry.get('startedAt') and entry.get('finishedAt')):
|
||||
continue
|
||||
usage = run_dir / 'runs' / name / 'usage.json'
|
||||
try:
|
||||
if usage.exists():
|
||||
cost = sum(c['cost'] for c in json.loads(usage.read_text()))
|
||||
else:
|
||||
meta = json.loads((run_dir / 'runs' / name / 'run.json').read_text())
|
||||
cost = None if meta.get('langsmithRunIds') else 0.0
|
||||
except (OSError, ValueError):
|
||||
cost = None
|
||||
sample = (cost, entry['finishedAt'] - entry['startedAt'])
|
||||
by_id.setdefault(entry.get('scenario', name), []).append(sample)
|
||||
every.append(sample)
|
||||
return by_id, every
|
||||
|
||||
|
||||
def estimate(jobs, concurrency, compare_master, flags):
|
||||
"""Expected cost and time of the jobs, from the earlier runs of the same scenario ids. A master run that
|
||||
can be reused costs nothing."""
|
||||
by_id, every = history()
|
||||
try:
|
||||
base = baseline.target() if compare_master else None
|
||||
except subprocess.CalledProcessError:
|
||||
base = None
|
||||
|
||||
def median(samples, index, default):
|
||||
values = [x[index] for x in samples if x[index] is not None]
|
||||
return statistics.median(values) if values else default
|
||||
|
||||
any_cost, any_time = median(every, 0, None), median(every, 1, 60)
|
||||
rows, cost_total, known = [], 0.0, True
|
||||
for name, s, role in jobs:
|
||||
if role == 'master' and base and (MASTER_CACHE / master_key(s, base, flags) / 'result.json').is_file():
|
||||
rows.append((name + ' (kept)', 0.0, 0, True))
|
||||
continue
|
||||
samples = by_id.get(s['id'], [])
|
||||
cost = median(samples, 0, any_cost)
|
||||
seconds = median(samples, 1, any_time)
|
||||
if cost is None:
|
||||
known = False
|
||||
else:
|
||||
cost_total += cost
|
||||
rows.append((name, cost, seconds, bool(samples)))
|
||||
times = sorted((r[2] for r in rows), reverse=True)
|
||||
wall = max(times[0] if times else 0, sum(times) / max(1, concurrency)) + 15
|
||||
return {'rows': rows, 'cost': cost_total if known else None, 'seconds': wall,
|
||||
'masterBuild': compare_master and not baseline.built()}
|
||||
|
||||
|
||||
def print_estimate(est):
|
||||
say('Estimate (median of earlier runs; * means no earlier run of this scenario):')
|
||||
for name, cost, seconds, seen in est['rows']:
|
||||
price = f'${cost:.2f}' if cost is not None else ' ? '
|
||||
say(f' {name:44} {price:>7} {seconds:5.0f}s{"" if seen else " *"}')
|
||||
total = f'about ${est["cost"]:.2f}' if est['cost'] is not None else 'unknown cost'
|
||||
extra = ', plus 15 to 20 minutes for the first master build' if est['masterBuild'] else ''
|
||||
say(f' {len(est["rows"])} runs, {total}, about {est["seconds"] / 60:.1f} minutes{extra}')
|
||||
|
||||
|
||||
def fetch_usage(run_dir, names, timeout=90):
|
||||
"""Token usage and cost of each run from LangSmith, in one query for all runs. LangSmith receives traces
|
||||
with a delay, so the query repeats until each run has calls and the count stops changing."""
|
||||
if not setting('LANGSMITH_API_KEY'):
|
||||
return
|
||||
metas = {}
|
||||
for name in names:
|
||||
file = run_dir / 'runs' / name / 'run.json'
|
||||
meta = json.loads(file.read_text()) if file.exists() else {}
|
||||
if meta.get('langsmithRunIds') and not meta.get('cached'):
|
||||
metas[name] = meta
|
||||
if not metas:
|
||||
return
|
||||
start = min(m['startedAt'] for m in metas.values())
|
||||
end = max(m['finishedAt'] for m in metas.values())
|
||||
traces = {t for m in metas.values() for t in m['langsmithRunIds']}
|
||||
deadline, previous, calls = time.time() + timeout, None, []
|
||||
while True:
|
||||
try:
|
||||
calls = fetch_calls(start, end, traces)
|
||||
except (SystemExit, OSError, ValueError) as e:
|
||||
say(f'usage failed: {str(e)[:200]}')
|
||||
return
|
||||
seen = {c['trace'] for c in calls}
|
||||
complete = all(set(m['langsmithRunIds']) & seen for m in metas.values())
|
||||
if (complete and len(calls) == previous) or time.time() > deadline:
|
||||
break
|
||||
previous = len(calls)
|
||||
time.sleep(10)
|
||||
for name, meta in metas.items():
|
||||
own = [c for c in calls if c['trace'] in set(meta['langsmithRunIds'])]
|
||||
(run_dir / 'runs' / name / 'usage.json').write_text(json.dumps(own, indent=1))
|
||||
usage = totals(own)
|
||||
say(f'{usage["calls"]} LLM calls, ${usage["cost"]:.2f}', name)
|
||||
|
||||
|
||||
def compare_trees(run_dir):
|
||||
"""Add the accessibility tree diff of each capture to visual.json."""
|
||||
path = run_dir / 'visual.json'
|
||||
if not path.exists():
|
||||
return []
|
||||
results = json.loads(path.read_text())
|
||||
for r in results:
|
||||
trees = []
|
||||
for name in (f'{r["scenario"]}--master', r['scenario']):
|
||||
file = run_dir / 'runs' / name / 'captures' / f'{r["name"]}.aria.yml'
|
||||
trees.append(file.read_text().splitlines() if file.exists() else [])
|
||||
diff = '\n'.join(difflib.unified_diff(*trees, 'master', 'branch', lineterm='', n=2))
|
||||
r['ariaDiff'] = diff
|
||||
r['pixelStatus'] = r['status']
|
||||
if diff and r['status'] == 'unchanged':
|
||||
r['status'] = 'changed'
|
||||
path.write_text(json.dumps(results, indent=2))
|
||||
return results
|
||||
|
||||
|
||||
def master_key(scenario, commit, flags):
|
||||
"""A master run gives the same screens while the commit, the flags, the script and the runner stay the same."""
|
||||
h = hashlib.sha256()
|
||||
overrides = setting('N8N_FEATURE_FLAG_OVERRIDES') or ''
|
||||
for part in (commit, json.dumps(flags or {}, sort_keys=True), overrides, scenario.get('viewport', '1600x1000'),
|
||||
str(scenario.get('autoConfirm', True))):
|
||||
h.update(part.encode() + b'\0')
|
||||
for file in (Path(scenario['scriptPath']), HERE / 'kit.mjs', HERE / 'run_scenario.mjs'):
|
||||
h.update(file.read_bytes())
|
||||
return h.hexdigest()[:20]
|
||||
|
||||
|
||||
def save_master_run(sub, key):
|
||||
if json.loads((sub / 'result.json').read_text()).get('status') != 'passed':
|
||||
return
|
||||
target = MASTER_CACHE / key
|
||||
shutil.rmtree(target, ignore_errors=True)
|
||||
shutil.copytree(sub, target, ignore=shutil.ignore_patterns('live.png', 'usage.json'))
|
||||
cutoff = time.time() - MASTER_CACHE_DAYS * 86400
|
||||
for old in MASTER_CACHE.iterdir():
|
||||
if old.is_dir() and old.stat().st_mtime < cutoff:
|
||||
shutil.rmtree(old, ignore_errors=True)
|
||||
|
||||
|
||||
def reuse_master_run(sub, key):
|
||||
"""Copy a kept master run into this run. Returns False when there is none."""
|
||||
source = MASTER_CACHE / key
|
||||
if not (source / 'result.json').is_file():
|
||||
return False
|
||||
shutil.copytree(source, sub, dirs_exist_ok=True)
|
||||
meta = json.loads((sub / 'run.json').read_text())
|
||||
meta['cached'] = True
|
||||
(sub / 'run.json').write_text(json.dumps(meta, indent=2))
|
||||
return True
|
||||
|
||||
|
||||
def start_live_server(run_dir):
|
||||
url_file = run_dir / 'live-url.txt'
|
||||
url_file.unlink(missing_ok=True)
|
||||
log = (run_dir / 'live-server.log').open('w')
|
||||
# A new session, so the page still serves the report after this script ends.
|
||||
subprocess.Popen([sys.executable, str(HERE / 'live_page.py'), '--run-dir', str(run_dir),
|
||||
'--parent-pid', str(os.getpid())],
|
||||
cwd=HERE, stdin=subprocess.DEVNULL, stdout=log, stderr=log, start_new_session=True)
|
||||
for _ in range(50):
|
||||
if url_file.exists() and url_file.read_text().strip():
|
||||
return url_file.read_text().strip()
|
||||
time.sleep(0.1)
|
||||
return None
|
||||
|
||||
|
||||
def open_in_default_browser(url):
|
||||
"""Opens url in the default browser. Returns False if no browser opened.
|
||||
|
||||
$BROWSER is not used: agents often set it to a command that does nothing.
|
||||
"""
|
||||
command = {'darwin': ['open'], 'linux': ['xdg-open']}.get(sys.platform)
|
||||
if command and shutil.which(command[0]):
|
||||
try:
|
||||
return subprocess.run(command + [url], capture_output=True, timeout=15).returncode == 0
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return False
|
||||
try:
|
||||
return webbrowser.open(url)
|
||||
except webbrowser.Error:
|
||||
return False
|
||||
|
||||
|
||||
def result_table(run_dir, runs):
|
||||
"""A plain text table of the results, with the first problem of each run that did not pass."""
|
||||
lines = [f' {"run":44} {"status":15} {"steps":>7} {"time":>6} {"errors":>6}']
|
||||
for name, entry in runs.items():
|
||||
sub = run_dir / 'runs' / name
|
||||
steps = report.read_json(sub / 'steps.json', [])
|
||||
result = report.read_json(sub / 'result.json', {})
|
||||
checked = [s for s in steps if s['kind'] in ('step', 'check', 'capture')]
|
||||
passed = f'{sum(s["status"] == "passed" for s in checked)}/{len(checked)}' if checked else '-'
|
||||
took = (entry.get('finishedAt') or 0) - (entry.get('startedAt') or 0)
|
||||
errors = (result.get('consoleErrors') or 0) + (result.get('requestErrors') or 0)
|
||||
status = entry['status'] + (' (kept)' if report.read_json(sub / 'run.json', {}).get('cached') else '')
|
||||
lines.append(f' {name:44} {status:15} {passed:>7} {took:5.0f}s {errors:>6}')
|
||||
failed = next((s for s in steps if s['status'] == 'failed'), None)
|
||||
if failed:
|
||||
problem = f'{failed["kind"]} "{failed["title"]}": {failed.get("error") or ""}'
|
||||
else:
|
||||
problem = entry.get('reason') if entry['status'] != 'passed' else ''
|
||||
if (problem or '').strip():
|
||||
lines.append(f' {problem.strip().splitlines()[0][:160]}')
|
||||
return '\n'.join(lines)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--plan', help='scenarios.json')
|
||||
ap.add_argument('--rerun', help='"last", or an earlier run folder. Uses the current plan and scripts when they exist.')
|
||||
ap.add_argument('--only', help='Comma-separated scenario ids.')
|
||||
ap.add_argument('--run-dir')
|
||||
ap.add_argument('--concurrency', type=int, default=int(setting('FEATURE_TEST_CONCURRENCY', default='4')))
|
||||
ap.add_argument('--branch-url', help='Use this running instance for the branch runs instead of fresh ones.')
|
||||
ap.add_argument('--try', dest='try_mode', action='store_true',
|
||||
help='A quick check of new scripts: no master, no live page, no usage, no report.')
|
||||
ap.add_argument('--estimate', action='store_true', help='Print the expected cost and time, then stop.')
|
||||
ap.add_argument('--check', action='store_true', help='Check the setup, then stop.')
|
||||
ap.add_argument('--no-build', action='store_true', help='Do not build this checkout first.')
|
||||
ap.add_argument('--no-baseline', action='store_true', help='Skip master. Visual scenarios run on the branch only.')
|
||||
ap.add_argument('--no-cache', action='store_true', help='Run master again, also when a kept master run exists.')
|
||||
ap.add_argument('--no-fetch', action='store_true', help='Use origin/master as it is, without git fetch.')
|
||||
ap.add_argument('--no-live', action='store_true', help='Do not start the live page.')
|
||||
ap.add_argument('--no-open', action='store_true',
|
||||
help='Do not open the live page in the default browser. Use this when you open it in the editor browser.')
|
||||
ap.add_argument('--keep-instances', action='store_true', help='Keep the fresh instances running after the run.')
|
||||
args = ap.parse_args()
|
||||
if args.try_mode:
|
||||
args.no_baseline = args.no_live = True
|
||||
build = not args.no_build and not args.branch_url
|
||||
|
||||
if args.check and not (args.plan or args.rerun):
|
||||
errors, warnings = preflight(build, need_master=True)
|
||||
for w in warnings:
|
||||
say(f'warning: {w}')
|
||||
for e in errors:
|
||||
say(f'error: {e}')
|
||||
branch = git('rev-parse', '--abbrev-ref', 'HEAD').replace('/', '-')
|
||||
say(f'plan folder for this branch: {PLANS_ROOT / branch}/<feature>/')
|
||||
say('The setup is ready.' if not errors else 'Fix the errors, then run again.')
|
||||
sys.exit(1 if errors else 0)
|
||||
if not (args.plan or args.rerun):
|
||||
sys.exit('Give --plan <scenarios.json>, --rerun last|<run-dir>, or --check.')
|
||||
# The fresh instances run in their own sessions. Exit through the cleanup, so they stop too.
|
||||
for sig in (signal.SIGTERM, signal.SIGHUP):
|
||||
signal.signal(sig, lambda signum, _frame: sys.exit(128 + signum))
|
||||
|
||||
plan = load_plan(rerun_plan(args.rerun) if args.rerun else args.plan)
|
||||
if args.only:
|
||||
wanted = {x.strip() for x in args.only.split(',')}
|
||||
plan['scenarios'] = [s for s in plan['scenarios'] if s['id'] in wanted]
|
||||
if not plan['scenarios']:
|
||||
sys.exit(f'No scenario matches --only {args.only}.')
|
||||
|
||||
visual = [s for s in plan['scenarios'] if s['kind'] == 'visual']
|
||||
compare_master = bool(visual) and not args.no_baseline
|
||||
jobs = [] # (name, scenario, instance role)
|
||||
for s in plan['scenarios']:
|
||||
if args.branch_url:
|
||||
role = 'branch'
|
||||
elif s['kind'] == 'visual':
|
||||
role = 'branch-visual'
|
||||
else:
|
||||
role = 'isolated' if s.get('isolated', True) else 'branch'
|
||||
jobs.append((s['id'], s, role))
|
||||
if s['kind'] == 'visual' and compare_master:
|
||||
jobs.append((f'{s["id"]}--master', s, 'master'))
|
||||
|
||||
est = estimate(jobs, args.concurrency, compare_master, plan.get('featureFlags'))
|
||||
print_estimate(est)
|
||||
if args.estimate:
|
||||
return
|
||||
errors, warnings = preflight(build, compare_master, local_instances=not args.branch_url)
|
||||
for w in warnings:
|
||||
say(f'warning: {w}')
|
||||
if errors:
|
||||
sys.exit('The setup is not ready:\n ' + '\n '.join(errors))
|
||||
|
||||
if args.run_dir:
|
||||
run_dir = Path(args.run_dir).expanduser().resolve()
|
||||
run_dir.mkdir(parents=True, exist_ok=True)
|
||||
else:
|
||||
run_dir = new_run_dir(RUNS_ROOT / 'try' if args.try_mode else RUNS_ROOT, plan['name'])
|
||||
copy_plan(plan, run_dir)
|
||||
say(f'run directory: {run_dir}')
|
||||
|
||||
branch_commit = subprocess.run(['git', 'rev-parse', '--short', 'HEAD'], cwd=REPO_ROOT, capture_output=True,
|
||||
text=True).stdout.strip()
|
||||
dirty = bool(subprocess.run(['git', 'status', '--porcelain', '--', 'packages'], cwd=REPO_ROOT,
|
||||
capture_output=True, text=True).stdout.strip())
|
||||
live = live_enabled() and not args.no_live
|
||||
progress = Progress(run_dir, {
|
||||
'phase': 'running', 'step': 'Building the branch' if build else 'Starting the instances',
|
||||
'startedAt': time.time(), 'concurrency': args.concurrency,
|
||||
'branch': {'commit': branch_commit, 'dirty': dirty, 'url': args.branch_url},
|
||||
'master': {'commit': None} if compare_master else None,
|
||||
'runs': {name: {'status': 'waiting', 'scenario': s['id'], 'instance': role.split('-')[0]}
|
||||
for name, s, role in jobs},
|
||||
})
|
||||
|
||||
live_url = None
|
||||
if live:
|
||||
live_url = start_live_server(run_dir)
|
||||
if live_url:
|
||||
say(f'live page: {live_url}/report/live')
|
||||
if not args.no_open and not open_in_default_browser(f'{live_url}/report/live'):
|
||||
say(f'no browser opened. Open {live_url}/report/live to watch the runs.')
|
||||
else:
|
||||
say(f'the live page did not start, see {run_dir / "live-server.log"}')
|
||||
|
||||
instances, lock = [], threading.Lock()
|
||||
keep = args.keep_instances
|
||||
pool = ThreadPoolExecutor(max_workers=max(1, args.concurrency))
|
||||
# Shared instances start at once. Isolated instances start ahead of their scenario, with at most one
|
||||
# more alive than the number of scenarios that run at the same time.
|
||||
shared_pool = ThreadPoolExecutor(max_workers=3)
|
||||
boot_pool = ThreadPoolExecutor(max_workers=args.concurrency + 1)
|
||||
alive = threading.BoundedSemaphore(args.concurrency + 1)
|
||||
futures, shared = [], {}
|
||||
|
||||
def start_instance(name, root):
|
||||
inst = Instance(name, root, run_dir / 'instances' / name, run_dir / 'instances' / f'{name}.log',
|
||||
plan.get('featureFlags'))
|
||||
with lock:
|
||||
instances.append(inst)
|
||||
started = time.time()
|
||||
inst.start()
|
||||
# The port only: Cursor opens an editor tab for each new localhost URL that a command prints.
|
||||
say(f'{name} instance ready in {time.time() - started:.0f}s on port {inst.port}')
|
||||
return inst
|
||||
|
||||
def boot_isolated(name):
|
||||
alive.acquire()
|
||||
try:
|
||||
return start_instance(name, REPO_ROOT)
|
||||
except Exception:
|
||||
alive.release()
|
||||
raise
|
||||
|
||||
def run_on(name, scenario, role, instance):
|
||||
"""Run one job when its instance is ready. instance is a future of an Instance, or a URL."""
|
||||
progress.run(name, status='starting')
|
||||
try:
|
||||
inst = instance.result() if hasattr(instance, 'result') else None
|
||||
except Exception as e:
|
||||
progress.run(name, status='error', finishedAt=time.time(), reason=f'The instance did not start: {e}')
|
||||
return
|
||||
url = inst.url if inst else instance
|
||||
try:
|
||||
run_job(name, scenario, url, role.split('-')[0], run_dir, progress, live)
|
||||
if role == 'master' and not args.no_cache:
|
||||
save_master_run(run_dir / 'runs' / name,
|
||||
master_key(scenario, progress.data['master']['commit'], plan.get('featureFlags')))
|
||||
finally:
|
||||
if role == 'isolated':
|
||||
if not keep:
|
||||
inst.stop(clean=True)
|
||||
alive.release()
|
||||
|
||||
def submit(name, scenario, role, instance):
|
||||
futures.append(pool.submit(run_on, name, scenario, role, instance))
|
||||
|
||||
def skip(role, reason):
|
||||
for name, _, r in jobs:
|
||||
if r == role and progress.data['runs'][name]['status'] == 'waiting':
|
||||
progress.run(name, status='skipped', reason=reason)
|
||||
|
||||
master_jobs = [(name, s) for name, s, r in jobs if r == 'master']
|
||||
base_commit = None
|
||||
if compare_master:
|
||||
try:
|
||||
base_commit = baseline.target(fetch=not args.no_fetch)
|
||||
except subprocess.CalledProcessError as e:
|
||||
say(f'cannot find the master merge base: {e}', 'master')
|
||||
if base_commit:
|
||||
progress.update(master={'commit': base_commit})
|
||||
if not args.no_cache:
|
||||
left = []
|
||||
for name, s in master_jobs:
|
||||
if reuse_master_run(run_dir / 'runs' / name, master_key(s, base_commit, plan.get('featureFlags'))):
|
||||
now = time.time()
|
||||
progress.run(name, status='passed', startedAt=now, finishedAt=now)
|
||||
say(f'master run kept from an earlier run of {base_commit[:11]}', name)
|
||||
else:
|
||||
left.append((name, s))
|
||||
master_jobs = left
|
||||
|
||||
def master_side():
|
||||
try:
|
||||
built = baseline.prepare(fetch=False, say=lambda m: say(m, 'master'))
|
||||
progress.update(master={'commit': built['commit']})
|
||||
inst = start_instance('master', built['root'])
|
||||
for name, s in master_jobs:
|
||||
submit(name, s, 'master', inst.url)
|
||||
except Exception as e:
|
||||
say(f'master is not available: {e}', 'master')
|
||||
skip('master', f'Master is not available: {e}')
|
||||
|
||||
visual_error = None
|
||||
try:
|
||||
if build:
|
||||
step('Build this checkout')
|
||||
build_branch(run_dir / 'build.log')
|
||||
elif not args.branch_url:
|
||||
stale = stale_packages()
|
||||
if stale:
|
||||
say('warning: these packages have source changes that are newer than their build: '
|
||||
f'{", ".join(stale)}. The fresh instances run the old build.')
|
||||
progress.update(stale=stale)
|
||||
|
||||
step(f'Run {len(jobs)} scenario runs, {args.concurrency} at a time')
|
||||
progress.update(step='Running the scenarios')
|
||||
for role in ('branch', 'branch-visual'):
|
||||
if any(r == role for _, _, r in jobs):
|
||||
shared[role] = args.branch_url or shared_pool.submit(start_instance, role, REPO_ROOT)
|
||||
boots = {name: boot_pool.submit(boot_isolated, name) for name, _, r in jobs if r == 'isolated'}
|
||||
master_thread = None
|
||||
if compare_master and master_jobs and base_commit:
|
||||
master_thread = threading.Thread(target=master_side, daemon=True)
|
||||
master_thread.start()
|
||||
elif compare_master and not base_commit:
|
||||
skip('master', 'The master merge base is not known.')
|
||||
for name, s, role in jobs:
|
||||
if role == 'isolated':
|
||||
submit(name, s, role, boots[name])
|
||||
elif role in shared:
|
||||
submit(name, s, role, shared[role])
|
||||
if master_thread:
|
||||
master_thread.join()
|
||||
# Jobs can be added while the pool runs, so wait until the list stops growing.
|
||||
while any(not f.done() for f in list(futures)):
|
||||
time.sleep(1)
|
||||
for f in futures:
|
||||
if f.exception():
|
||||
say(f'a run failed: {f.exception()}')
|
||||
pool.shutdown()
|
||||
if not keep:
|
||||
for inst in instances:
|
||||
inst.stop(clean=True)
|
||||
|
||||
names = [name for name, _, _ in jobs]
|
||||
if compare_master:
|
||||
step('Compare the visual captures with master')
|
||||
progress.update(step='Comparing the captures with master')
|
||||
diff = subprocess.run(['node', 'visual_diff.mjs', str(run_dir)], cwd=HERE)
|
||||
if diff.returncode != 0:
|
||||
visual_error = f'visual_diff.mjs failed with exit code {diff.returncode}. The visual check is not complete.'
|
||||
say(f'error: {visual_error}')
|
||||
progress.update(visualError=visual_error)
|
||||
trees = [r for r in compare_trees(run_dir) if r.get('ariaDiff')]
|
||||
if trees:
|
||||
say(f'{len(trees)} captures with a difference in the accessibility tree: '
|
||||
+ ', '.join(f'{r["scenario"]}/{r["name"]}' for r in trees))
|
||||
|
||||
step('Summarize the runs for the judge')
|
||||
summary = report.summarize(run_dir)
|
||||
if visual_error:
|
||||
summary = f'VISUAL CHECK FAILED: {visual_error}\n\n{summary}'
|
||||
(run_dir / 'summary.txt').write_text(summary)
|
||||
if not args.try_mode:
|
||||
step('Build the report')
|
||||
progress.update(step='Building the report')
|
||||
report.build(run_dir)
|
||||
progress.finish()
|
||||
finally:
|
||||
pool.shutdown(wait=False, cancel_futures=True)
|
||||
boot_pool.shutdown(wait=False, cancel_futures=True)
|
||||
shared_pool.shutdown(wait=False, cancel_futures=True)
|
||||
for proc in list(RUNNERS):
|
||||
proc.kill()
|
||||
if not keep:
|
||||
for inst in instances:
|
||||
inst.stop(clean=True)
|
||||
progress.finish()
|
||||
|
||||
runs = progress.data['runs']
|
||||
visual_results = json.loads((run_dir / 'visual.json').read_text()) if (run_dir / 'visual.json').exists() else []
|
||||
not_passed = [n for n, e in runs.items() if e['status'] != 'passed' and not n.endswith('--master')]
|
||||
rerun = f'python3 {S}/run.py --rerun {run_dir}' + (f' --only {",".join(not_passed)}' if not_passed else '')
|
||||
usage_later = not args.try_mode and bool(setting('LANGSMITH_API_KEY'))
|
||||
say('\n=== Result ===')
|
||||
say(result_table(run_dir, runs))
|
||||
if not_passed:
|
||||
say(f'Run the ones that did not pass again: {rerun}')
|
||||
print(json.dumps({
|
||||
'runDir': str(run_dir),
|
||||
'report': None if args.try_mode else str(run_dir / 'report' / 'index.html'),
|
||||
'live': f'{live_url}/report/index.html' if live_url else None,
|
||||
'summary': str(run_dir / 'summary.txt'),
|
||||
'results': {name: runs[name]['status'] for name in runs},
|
||||
'visual': {'captures': len(visual_results),
|
||||
'changed': [f'{r["scenario"]}/{r["name"]}' for r in visual_results if r['status'] != 'unchanged'],
|
||||
'error': visual_error},
|
||||
'instancePorts': {i.name: i.port for i in instances} if keep else None,
|
||||
'rerun': rerun,
|
||||
'usage': 'The token usage comes in the next lines. The report updates when it is done.' if usage_later else None,
|
||||
'next': ('Fix the scripts that failed, then run --try again, or run the full plan.' if args.try_mode else
|
||||
'Judge each run from summary.txt and write verdicts.json. The live page rebuilds the report.'),
|
||||
}, indent=2), flush=True)
|
||||
|
||||
if usage_later:
|
||||
step('Fetch the token usage from LangSmith')
|
||||
fetch_usage(run_dir, [n for n in runs])
|
||||
report.build(run_dir)
|
||||
say('usage: done, the report is updated')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -1,203 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
// Run one scenario script in a browser that is signed in to an n8n instance.
|
||||
//
|
||||
// node run_scenario.mjs --out <dir> --script <file.mjs> [--live] [--auto-confirm]
|
||||
// [--timeout-min 10] [--viewport 1600x1000]
|
||||
//
|
||||
// N8N_BASE_URL selects the instance. Writes to <dir>:
|
||||
// steps.json the steps, checks and captures with their screenshots (updated while it runs)
|
||||
// step-NN.png ... one screenshot for each step, check and capture
|
||||
// captures/<name>.png screens for the visual check, with <name>.aria.yml
|
||||
// live.png the screen every 2.5 seconds, for the live page (with --live)
|
||||
// screen.webm the video, with screen-poster.png
|
||||
// console.json browser errors and failed requests to the instance
|
||||
// result.json {status: passed|failed|error, error, threads, prompts, startedAt, finishedAt, ...}
|
||||
// Prints "thread <id>" for each thread that the scenario opens, so run.py can read its events.
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { copyFileSync, mkdirSync, mkdtempSync, renameSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join, resolve } from 'node:path';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
import { parseArgs } from 'node:util';
|
||||
|
||||
import {
|
||||
baseUrl,
|
||||
launchBrowser,
|
||||
selectors,
|
||||
setting,
|
||||
signIn,
|
||||
} from '../../testmax-n8n-ai/scripts/n8n_browser.mjs';
|
||||
import { StepFailed, createKit } from './kit.mjs';
|
||||
|
||||
const { values: args } = parseArgs({
|
||||
options: {
|
||||
out: { type: 'string' },
|
||||
script: { type: 'string' },
|
||||
live: { type: 'boolean', default: false },
|
||||
'auto-confirm': { type: 'boolean', default: false },
|
||||
'timeout-min': { type: 'string', default: '10' },
|
||||
viewport: { type: 'string', default: '1600x1000' },
|
||||
},
|
||||
});
|
||||
const out = resolve(args.out);
|
||||
mkdirSync(out, { recursive: true });
|
||||
const [width, height] = args.viewport.split('x').map(Number);
|
||||
const size = { width, height };
|
||||
const LIVE_SHOT_MS = 2500;
|
||||
const emit = (line) => console.log(line.replace(/\s+/g, ' ').slice(0, 300));
|
||||
|
||||
/**
|
||||
* Sign in through the REST API, which is about 1.3 seconds faster than the sign-in page. The auth cookie
|
||||
* is bound to the browser-id header of the login, so the page gets the same id in localStorage.
|
||||
* Falls back to the sign-in page.
|
||||
*/
|
||||
async function signedInState(browser) {
|
||||
const login = await browser.newContext();
|
||||
try {
|
||||
const browserId = randomUUID();
|
||||
const res = await login.request.post(`${baseUrl}/rest/login`, {
|
||||
headers: { 'browser-id': browserId },
|
||||
data: {
|
||||
emailOrLdapLoginId: setting('N8N_EVAL_EMAIL', 'nathan@n8n.io'),
|
||||
password: setting('N8N_EVAL_PASSWORD', 'PlaywrightTest123'),
|
||||
},
|
||||
});
|
||||
if (res.ok()) {
|
||||
const state = await login.storageState();
|
||||
state.origins = [
|
||||
{
|
||||
origin: new URL(baseUrl).origin,
|
||||
localStorage: [{ name: 'n8n-browserId', value: browserId }],
|
||||
},
|
||||
];
|
||||
return state;
|
||||
}
|
||||
await signIn(await login.newPage());
|
||||
return await login.storageState();
|
||||
} finally {
|
||||
await login.close();
|
||||
}
|
||||
}
|
||||
|
||||
selectors.setTestIdAttribute('data-test-id');
|
||||
const browser = await launchBrowser();
|
||||
const storageState = await signedInState(browser);
|
||||
|
||||
// The video starts after the sign-in, at the first page of the scenario.
|
||||
const videoDir = mkdtempSync(join(tmpdir(), 'feature-test-video-'));
|
||||
const context = await browser.newContext({
|
||||
viewport: size,
|
||||
storageState,
|
||||
recordVideo: { dir: videoDir, size },
|
||||
});
|
||||
const page = await context.newPage();
|
||||
|
||||
const origin = new URL(baseUrl).origin;
|
||||
const consoleErrors = [];
|
||||
const requestErrors = [];
|
||||
page.on('console', (m) => {
|
||||
if (m.type() === 'error')
|
||||
consoleErrors.push({ text: m.text().slice(0, 1000), url: m.location()?.url });
|
||||
});
|
||||
page.on('pageerror', (e) => consoleErrors.push({ text: `Uncaught: ${e.message}`.slice(0, 1000) }));
|
||||
page.on('response', (r) => {
|
||||
if (r.status() >= 400 && r.url().startsWith(origin)) {
|
||||
requestErrors.push({
|
||||
status: r.status(),
|
||||
method: r.request().method(),
|
||||
url: r.url().slice(origin.length),
|
||||
});
|
||||
}
|
||||
});
|
||||
page.on('requestfailed', (r) => {
|
||||
const failure = r.failure()?.errorText ?? '';
|
||||
// Closed event streams and aborted navigations are normal.
|
||||
if (r.url().startsWith(origin) && !/ERR_ABORTED|NS_BINDING_ABORTED/.test(failure)) {
|
||||
requestErrors.push({
|
||||
status: 0,
|
||||
method: r.method(),
|
||||
url: r.url().slice(origin.length),
|
||||
error: failure,
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
let shooting = false;
|
||||
async function liveShot() {
|
||||
if (shooting) return;
|
||||
shooting = true;
|
||||
const partial = join(out, 'live.tmp.png');
|
||||
try {
|
||||
await page.screenshot({ path: partial });
|
||||
renameSync(partial, join(out, 'live.png'));
|
||||
} catch {
|
||||
// The page can be in the middle of a navigation. The next interval tries again.
|
||||
} finally {
|
||||
shooting = false;
|
||||
}
|
||||
}
|
||||
const liveTimer = args.live ? setInterval(liveShot, LIVE_SHOT_MS) : null;
|
||||
|
||||
const { t, threads, prompts, steps, stop } = createKit({
|
||||
page,
|
||||
out,
|
||||
baseUrl,
|
||||
autoConfirm: args['auto-confirm'],
|
||||
emit,
|
||||
});
|
||||
const result = { status: 'passed', startedAt: new Date().toISOString(), baseUrl };
|
||||
const timeoutMs = Number(args['timeout-min']) * 60_000;
|
||||
let timer;
|
||||
try {
|
||||
const scenario = (await import(pathToFileURL(resolve(args.script)).href)).default;
|
||||
if (typeof scenario !== 'function')
|
||||
throw new Error('The script must export a default async function (t).');
|
||||
await Promise.race([
|
||||
scenario(t),
|
||||
new Promise((_, reject) => {
|
||||
timer = setTimeout(() => {
|
||||
// The race does not end the scenario function. stop() makes its next step throw.
|
||||
stop();
|
||||
reject(new Error(`The scenario did not finish in ${args['timeout-min']} minutes.`));
|
||||
}, timeoutMs);
|
||||
}),
|
||||
]);
|
||||
if (steps.some((s) => s.status === 'failed')) result.status = 'failed';
|
||||
} catch (error) {
|
||||
result.status = error instanceof StepFailed || steps.length ? 'failed' : 'error';
|
||||
result.error = String(error?.message ?? error)
|
||||
.replace(/\u001b\[[0-9;]*m/g, '')
|
||||
.slice(0, 2000);
|
||||
if (!(error instanceof StepFailed)) emit(`error: ${result.error.split('\n')[0]}`);
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
|
||||
stop();
|
||||
clearInterval(liveTimer);
|
||||
for (const s of steps.filter((x) => x.status === 'running')) {
|
||||
s.status = 'failed';
|
||||
s.error = 'The scenario stopped during this step.';
|
||||
}
|
||||
writeFileSync(join(out, 'steps.json'), JSON.stringify(steps, null, 1));
|
||||
await page.waitForTimeout(1500);
|
||||
if (args.live) await liveShot();
|
||||
await page.screenshot({ path: join(out, 'screen-poster.png') }).catch(() => {});
|
||||
const video = page.video();
|
||||
await context.close();
|
||||
if (video) copyFileSync(await video.path(), join(out, 'screen.webm'));
|
||||
await browser.close();
|
||||
rmSync(videoDir, { recursive: true, force: true });
|
||||
|
||||
Object.assign(result, {
|
||||
finishedAt: new Date().toISOString(),
|
||||
threads,
|
||||
prompts,
|
||||
consoleErrors: consoleErrors.length,
|
||||
requestErrors: requestErrors.length,
|
||||
});
|
||||
writeFileSync(join(out, 'console.json'), JSON.stringify({ consoleErrors, requestErrors }, null, 1));
|
||||
writeFileSync(join(out, 'result.json'), JSON.stringify(result, null, 2));
|
||||
emit(`result ${result.status}`);
|
||||
// A scenario that timed out can still wait on its own timers. The results are written, so end here.
|
||||
process.exit(0);
|
||||
|
|
@ -1,96 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
// Compare the captures of each visual scenario: this branch against master.
|
||||
//
|
||||
// node visual_diff.mjs <run-dir>
|
||||
//
|
||||
// Reads runs/<id>/captures/*.png (branch) and runs/<id>--master/captures/*.png (master) for each
|
||||
// visual scenario in plan/scenarios.json. Writes visual/<id>/<name>.diff.png and visual.json:
|
||||
// [{scenario, name, status: unchanged|changed|branch-only|master-only, changedPixels, ratio,
|
||||
// branchSize, masterSize, diff}]
|
||||
// A pixel counts as changed when one color channel differs by more than PIXEL_THRESHOLD. A capture
|
||||
// counts as changed when more than RATIO_THRESHOLD of its pixels changed, or its size changed.
|
||||
import { existsSync, mkdirSync, readFileSync, readdirSync, writeFileSync } from 'node:fs';
|
||||
import { join, resolve } from 'node:path';
|
||||
|
||||
import { loadPng } from '../../testmax-n8n-ai/scripts/n8n_browser.mjs';
|
||||
|
||||
const PIXEL_THRESHOLD = 32;
|
||||
const RATIO_THRESHOLD = 0.0005;
|
||||
const PNG = loadPng();
|
||||
const runDir = resolve(process.argv[2] ?? '.');
|
||||
const plan = JSON.parse(readFileSync(join(runDir, 'plan', 'scenarios.json'), 'utf8'));
|
||||
|
||||
const read = (file) => PNG.sync.read(readFileSync(file));
|
||||
const names = (dir) =>
|
||||
existsSync(dir)
|
||||
? readdirSync(dir)
|
||||
.filter((f) => f.endsWith('.png'))
|
||||
.map((f) => f.slice(0, -4))
|
||||
: [];
|
||||
|
||||
function compare(branch, master) {
|
||||
const width = Math.max(branch.width, master.width);
|
||||
const height = Math.max(branch.height, master.height);
|
||||
const diff = new PNG({ width, height });
|
||||
let changed = 0;
|
||||
for (let y = 0; y < height; y++) {
|
||||
for (let x = 0; x < width; x++) {
|
||||
const o = (y * width + x) * 4;
|
||||
const inB = x < branch.width && y < branch.height;
|
||||
const inM = x < master.width && y < master.height;
|
||||
const b = inB ? (y * branch.width + x) * 4 : -1;
|
||||
const m = inM ? (y * master.width + x) * 4 : -1;
|
||||
let differs = !inB || !inM;
|
||||
if (!differs) {
|
||||
for (let c = 0; c < 4 && !differs; c++)
|
||||
differs = Math.abs(branch.data[b + c] - master.data[m + c]) > PIXEL_THRESHOLD;
|
||||
}
|
||||
if (differs) {
|
||||
changed++;
|
||||
diff.data.set([224, 17, 95, 255], o);
|
||||
} else {
|
||||
// The unchanged page shows as a faint gray copy of the branch screen.
|
||||
const gray =
|
||||
0.299 * branch.data[b] + 0.587 * branch.data[b + 1] + 0.114 * branch.data[b + 2];
|
||||
const v = Math.round(255 - (255 - gray) * 0.25);
|
||||
diff.data.set([v, v, v, 255], o);
|
||||
}
|
||||
}
|
||||
}
|
||||
return { diff, changed, ratio: changed / (width * height) };
|
||||
}
|
||||
|
||||
const results = [];
|
||||
for (const s of plan.scenarios.filter((x) => x.kind === 'visual')) {
|
||||
const branchDir = join(runDir, 'runs', s.id, 'captures');
|
||||
const masterDir = join(runDir, 'runs', `${s.id}--master`, 'captures');
|
||||
const all = [...new Set([...names(branchDir), ...names(masterDir)])].sort();
|
||||
for (const name of all) {
|
||||
const entry = { scenario: s.id, name };
|
||||
const hasB = existsSync(join(branchDir, `${name}.png`));
|
||||
const hasM = existsSync(join(masterDir, `${name}.png`));
|
||||
if (!hasB || !hasM) {
|
||||
results.push({ ...entry, status: hasB ? 'branch-only' : 'master-only' });
|
||||
continue;
|
||||
}
|
||||
const branch = read(join(branchDir, `${name}.png`));
|
||||
const master = read(join(masterDir, `${name}.png`));
|
||||
const { diff, changed, ratio } = compare(branch, master);
|
||||
const out = join(runDir, 'visual', s.id);
|
||||
mkdirSync(out, { recursive: true });
|
||||
writeFileSync(join(out, `${name}.diff.png`), PNG.sync.write(diff));
|
||||
const resized = branch.width !== master.width || branch.height !== master.height;
|
||||
results.push({
|
||||
...entry,
|
||||
status: resized || ratio > RATIO_THRESHOLD ? 'changed' : 'unchanged',
|
||||
changedPixels: changed,
|
||||
ratio,
|
||||
branchSize: [branch.width, branch.height],
|
||||
masterSize: [master.width, master.height],
|
||||
diff: `visual/${s.id}/${name}.diff.png`,
|
||||
});
|
||||
}
|
||||
}
|
||||
writeFileSync(join(runDir, 'visual.json'), JSON.stringify(results, null, 2));
|
||||
const changed = results.filter((r) => r.status !== 'unchanged').length;
|
||||
console.log(`${results.length} captures compared, ${changed} with a difference`);
|
||||
|
|
@ -1,745 +0,0 @@
|
|||
---
|
||||
name: n8n:testmax-n8n-ai
|
||||
description: >-
|
||||
Runs one workflow prompt through Instance AI, Claude Code with the n8n MCP
|
||||
server, and Codex with the n8n MCP server on a local n8n instance. Collects
|
||||
the traces, per-call tokens, cost, and canvas screenshots, judges each run
|
||||
against the expected outcome, and builds one HTML comparison page. Claude
|
||||
Code and Codex run in a sandbox that sees only the n8n MCP server. Use when
|
||||
the user wants to compare builders, test a prompt across Instance AI and
|
||||
MCP, or says /testmax-n8n-ai.
|
||||
---
|
||||
|
||||
# Compare workflow builders
|
||||
|
||||
This skill sends the same prompt to up to three builders on the local n8n
|
||||
instance. The builders run at the same time, and a local page shows them
|
||||
live. It then builds one page that shows, for each run:
|
||||
|
||||
- the result against the expected outcome (pass, partial or fail, with reasons
|
||||
and the evidence that a panel of blind judges cited)
|
||||
- deterministic checks of each workflow: the node definitions, the node
|
||||
issues in the n8n editor, the graph, the expression references, and a run
|
||||
of each branch with pinned data
|
||||
- a separate n8n standards score, from the rules in
|
||||
[n8n-io/skills](https://github.com/n8n-io/skills) and the workflow-sdk best
|
||||
practices. This score does not change the verdict
|
||||
- the canvas of the workflow that the builder made
|
||||
- a replay of the run: the exact terminal screen, or a video of the n8n UI
|
||||
- the cost
|
||||
- the LLM steps: one entry for each LLM call, with its tokens (cache read,
|
||||
cache write, uncached input, output) and the agent text, reasoning and tool
|
||||
calls (with input and output) that the call made
|
||||
|
||||
The page uses the light mode of the
|
||||
[n8n brand guidelines](https://guidelines.n8n.design/).
|
||||
|
||||
|
||||
|
||||
All scripts are in `scripts/` next to this file. They use only the Python
|
||||
standard library and the Playwright copy in the repository.
|
||||
|
||||
```bash
|
||||
S="$(git rev-parse --show-toplevel)/.agents/skills/testmax-n8n-ai/scripts"
|
||||
```
|
||||
|
||||
## Settings
|
||||
|
||||
All settings are in `.env.ai-surfaces` in the repository root. Git ignores
|
||||
this file. The template is `.env.ai-surfaces.example`, also in the
|
||||
repository root. The scripts read,
|
||||
from low to high priority: `.env.local`, `.env.ai-surfaces`, the shell
|
||||
environment. Empty values are ignored. Thus keys that are already in
|
||||
`.env.local` work without a change.
|
||||
|
||||
If `.env.ai-surfaces` does not exist, copy `.env.ai-surfaces.example` to it. Tell the user
|
||||
which values to fill in. Never read the values back to the user, never print
|
||||
them, and never put a secret on a command line.
|
||||
|
||||
The most important settings:
|
||||
|
||||
| Setting | Use |
|
||||
|---|---|
|
||||
| `COMPARE_BUILDERS` | Default builders, for example `iai,claude,codex` |
|
||||
| `COMPARE_PARALLEL` | `1` (default) runs the builders at the same time. `0` runs them one at a time |
|
||||
| `COMPARE_LIVE` | `1` (default) opens a live page of the runs. `0` turns it off |
|
||||
| `COMPARE_LIVE_WINDOWS` | `1` also opens the Instance AI thread and a terminal window for each CLI. Default `0` |
|
||||
| `COMPARE_RECORD` | `1` (default) records the Instance AI video for the replay. `0` turns it off |
|
||||
| `COMPARE_CLI_MODE` | `interactive` (default) records the real CLI screen. `headless` uses the JSON event stream |
|
||||
| `COMPARE_RECORD_REQUESTS` | `1` (default) records the Codex request bodies, without headers, for the tool list. `0` turns it off |
|
||||
| `COMPARE_SCENARIOS` | `1` (default) runs each workflow with pinned data through the n8n MCP server. `0` turns it off |
|
||||
| `COMPARE_WIRE` | `1` (default) runs each workflow in process with recorded API fixtures (`run_wire.mjs`). `0` turns it off |
|
||||
| `COMPARE_BINARY_CHECKS` | `1` (default) runs the deterministic binary checks of the Instance AI eval suite. `0` turns it off |
|
||||
| `GOOGLE_GENERATIVE_AI_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `MISTRAL_API_KEY`, `GROQ_API_KEY` | Model keys for the real AI runs. A workflow model without a key skips those runs |
|
||||
| `CLAUDE_MODEL`, `CODEX_MODEL` | Model ids. Empty uses the CLI default. A prefix such as `anthropic/` is removed |
|
||||
| `COMPARE_CLAUDE_TOOLS` | `none` (default) gives Claude Code only the n8n tools. `default` also gives it its built-in tools, all allowed (see "How the builders run") |
|
||||
| `CLAUDE_API_KEY` | Anthropic key for Claude Code. Falls back to `ANTHROPIC_API_KEY` |
|
||||
| `CODEX_API_KEY` | OpenAI key for Codex. Falls back to `OPENAI_API_KEY` |
|
||||
| `N8N_MCP_TOKEN` | MCP key of the n8n user. Optional |
|
||||
| `LANGSMITH_*` | Instance AI cost and tokens |
|
||||
|
||||
## 1. Ask the user
|
||||
|
||||
Ask for these two items. If the harness has a structured question tool, use
|
||||
it. If not, ask in chat. Do not guess them.
|
||||
|
||||
1. **Prompt.** The exact text to send to each builder.
|
||||
2. **Expected outcome.** What a correct result does. Ask for concrete checks,
|
||||
for example "WhatsApp trigger", "Gemini compares the question to the FAQ
|
||||
sheet", "the false branch makes a Notion ticket and a Slack message".
|
||||
|
||||
Use the builders and models from the settings. Ask about them only if the
|
||||
user wants a change for this run.
|
||||
|
||||
Write one expected outcome item on each line, numbered or bulleted. The
|
||||
number of an item is its id: the judges cite it, and the page groups the
|
||||
checks by it.
|
||||
|
||||
Then write the scenarios for the pinned-data runs, from the expected
|
||||
outcome. You do not need to ask the user for them. A scenario sets the data
|
||||
of the trigger, the service nodes and the AI nodes, and says which nodes
|
||||
must or must not run. Logic nodes (IF, Switch, Set, Code, Aggregate and
|
||||
others) run for real. Make one scenario for each branch of the expected
|
||||
outcome:
|
||||
|
||||
```json
|
||||
{
|
||||
"pinByType": {
|
||||
"googleSheets": [{ "Question": "What are your opening hours?", "Answer": "9:00 to 17:00" }]
|
||||
},
|
||||
"scenarios": [
|
||||
{ "name": "The FAQ answers the question", "ai": { "booleans": true },
|
||||
"expect": { "reach": ["whatsApp"], "skip": ["notion", "slack"] } },
|
||||
{ "name": "The FAQ does not answer the question", "ai": { "booleans": false },
|
||||
"expect": { "reach": ["notion", "slack"] } }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- `pinByType`: output items for every node of a type, in all scenarios. A
|
||||
type is the full name or the part after the last dot. A scenario can have
|
||||
its own `pinByType`.
|
||||
- `trigger`: output items of the trigger. Without it, the trigger gets a
|
||||
fixture in the shape that the trigger code emits (WhatsApp, Webhook, Chat,
|
||||
Schedule, Telegram, Form and Manual triggers have one).
|
||||
- `ai`: the output of each AI node. `booleans` sets every boolean in the
|
||||
output, `set` sets fields, and `text` sets the whole text of a node
|
||||
without a structured output parser. The output follows the schema of the
|
||||
structured output parser, or the JSON example in the prompt of the node.
|
||||
- `expect`: `reach` lists node types that must run without an error,
|
||||
`skip` lists node types that must not run.
|
||||
- `triggerNode`: the name of the trigger that the scenario starts from, for a
|
||||
workflow with more than one trigger. Without it, the scenario starts from
|
||||
the trigger that reaches the most `reach` types.
|
||||
|
||||
Without a scenarios file, `run_scenarios.py` makes one scenario for each
|
||||
value of the AI booleans, with no expectations.
|
||||
|
||||
`run_wire.mjs` runs the same scenarios in process. There, service nodes
|
||||
send real requests that recorded fixtures answer, so a scenario can also
|
||||
check the API contract. Add these keys to the scenarios file:
|
||||
|
||||
```json
|
||||
{
|
||||
"scenarios": [
|
||||
{ "name": "The FAQ does not answer the question", "ai": { "booleans": false },
|
||||
"expect": { "reach": ["notion", "slack"], "requests": [
|
||||
{ "type": "notion", "method": "POST", "contains": ["\"title\"", "@trigger.messages[0].from"] },
|
||||
{ "type": "slack", "method": "POST",
|
||||
"contains": ["@trigger.messages[0].text.body", ["@response.notion.id", "@response.notion.url"]] } ] } }
|
||||
],
|
||||
"edge": [
|
||||
{ "name": "The FAQ sheet is empty", "pinByType": { "googleSheets": [] }, "ai": { "booleans": false },
|
||||
"expect": { "reach": ["notion", "slack"] } }
|
||||
],
|
||||
"questions": [
|
||||
{ "name": "Opening hours, which the FAQ answers", "message": "Hi! What time do you open?",
|
||||
"expect": { "skip": ["notion"], "requests": [
|
||||
{ "type": "whatsApp", "method": "POST", "contains": ["@trigger.messages[0].from", "9:00"] } ] } }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- `expect.requests`: each entry needs a request from a node of `type`,
|
||||
optionally with `method` and an exact `count`, whose URL, query or body
|
||||
contains every value in `contains`. An array in `contains` means any one
|
||||
of its values. `@trigger.<path>` is a value of the trigger item.
|
||||
`@response.<type>.<path>` is a value of the response to the last write
|
||||
request of that node type, for example the id of the Notion page.
|
||||
- `expect.noWrites`: the scenario must send no write request.
|
||||
- `edge`: inputs that the prompt does not describe. The trigger kits in
|
||||
`sdk.mjs` add standard edge cases, for example two WhatsApp messages in one
|
||||
webhook call (each sender must get a request) and a status update (no
|
||||
writes). Edge cases are a score, not part of the verdict.
|
||||
- `questions`: the AI nodes call the real model with this customer
|
||||
`message`. The trigger fixture carries the message. A real-AI run needs a
|
||||
model key, and is a score, not part of the verdict.
|
||||
|
||||
## 2. Run the builders
|
||||
|
||||
```bash
|
||||
python3 $S/compare.py --title "<short title>" \
|
||||
--prompt-file <file with the prompt> --expected-file <file with the outcome> \
|
||||
--scenarios-file <file with the scenarios>
|
||||
```
|
||||
|
||||
Write the prompt, the expected outcome and the scenarios to files with the
|
||||
file tool first. Do not put them on the command line, because they can
|
||||
contain quotes. Use `--builders iai,claude` to change the builders for one
|
||||
run. Use `--rerun <old run dir>` to run the same case again. It also copies
|
||||
`scenarios.json`.
|
||||
|
||||
The command takes 5 to 30 minutes. Start it in the background if the harness
|
||||
supports it, and read its output until it ends. It prints one line for each
|
||||
tool call, with the builder name in front, so the user can follow the run.
|
||||
It does these steps:
|
||||
|
||||
1. Checks the setup and skips a builder that cannot run. It prints the
|
||||
reason.
|
||||
2. Starts the live page (see below).
|
||||
3. Runs the builders at the same time. `--sequential` or
|
||||
`COMPARE_PARALLEL=0` runs them one at a time.
|
||||
4. Fetches the Instance AI cost from LangSmith.
|
||||
5. Takes a screenshot of each canvas, and saves the node issues that the
|
||||
n8n editor shows (`canvas-issues.json`).
|
||||
6. Runs each workflow once for each scenario, with pinned data, through the
|
||||
`test_workflow` tool of the n8n MCP server (`exec.json`). It turns on
|
||||
"Available in MCP" for a workflow that does not have it.
|
||||
7. Runs each workflow in process with recorded API fixtures, for the
|
||||
scenarios, the edge cases and the real AI questions (`wire.json`).
|
||||
8. Runs the deterministic eval binary checks (`binary-checks.json`) and
|
||||
checks the final-message claims (`claims.json`).
|
||||
9. Derives the deterministic checks (`auto-checks.json`). See "Checks on
|
||||
the page".
|
||||
10. Writes `summary.txt`, and `judge/summary-blind.txt` for the judges.
|
||||
11. Builds the page without verdicts.
|
||||
|
||||
The last output is JSON with `runDir`, `report`, `live`, `parallel`,
|
||||
`summary`, `blindSummary`, `models` and `skipped`.
|
||||
|
||||
Parallel runs share one n8n instance and one machine. Thus the times of the
|
||||
builders include the shared load. The page says "run in parallel" under
|
||||
the title. Use
|
||||
`--sequential` when the user wants to compare times exactly.
|
||||
|
||||
### Watch the runs
|
||||
|
||||
`compare.py` starts `live_server.py` and opens its page in the default
|
||||
browser. It prints `live page: <url>`. Give the user this URL. The page
|
||||
shows each builder while it runs, and updates every 2 seconds without a
|
||||
reload:
|
||||
|
||||
- the table: status, time, LLM calls, tool calls and tokens of each builder
|
||||
- **Live:** the real terminal screen of Claude Code and Codex, streamed from
|
||||
the tmux recording, and a screenshot of the Instance AI thread every
|
||||
2.5 seconds
|
||||
- the LLM steps of each builder. Claude Code and Codex steps come from the
|
||||
session files of the CLIs, with tokens. Instance AI steps come from its
|
||||
event stream, without tokens until the run ends
|
||||
|
||||
The cost, the canvases and the Instance AI tokens come after the run. Until
|
||||
then, the Expected outcome box shows "Not verified yet", and the
|
||||
Correctness, Execution and n8n standards boxes show "Not run yet". When
|
||||
the report is ready, the page changes to it. The server listens on
|
||||
`127.0.0.1` only. It stops 30 minutes after the last request, when the run
|
||||
is done.
|
||||
|
||||
The report shows the price and judge steps (steps 3 and 4) as queued until
|
||||
you start them. The Result column shows "Judge queued" or "Judging", and
|
||||
the Cost column of a CLI builder without a price shows "Price queued" or
|
||||
"Finding price". Each `build_report.py --step` call builds the page again.
|
||||
While a step is queued or running, the page on the live server reloads
|
||||
itself when the page changes. A page that you open from the file needs a
|
||||
manual reload.
|
||||
|
||||
Headless mode and clean isolation have no terminal screen. For them, the
|
||||
page shows only the table and the LLM steps. `COMPARE_RECORD=0` removes the
|
||||
Instance AI screenshot.
|
||||
|
||||
`COMPARE_LIVE_WINDOWS=1` also opens the old live views:
|
||||
|
||||
- **Instance AI:** the thread opens in the default browser. The browser must
|
||||
be logged in as `N8N_EVAL_EMAIL`, because a thread is visible only to its
|
||||
owner. The script answers the confirmation cards, so the user must not
|
||||
click them.
|
||||
- **Claude Code and Codex:** a terminal window shows the real screen of the
|
||||
CLI, read-only (`docker exec … tmux attach -r`). In headless mode, the
|
||||
window follows `<builder>/live.log` instead. On macOS the window opens in
|
||||
Terminal. The first time, macOS asks if the agent's app can control
|
||||
Terminal.
|
||||
|
||||
Use `--no-live` or `COMPARE_LIVE=0` for runs without a display, for example
|
||||
in CI.
|
||||
|
||||
### Replay on the page
|
||||
|
||||
The Replay section shows the last screen of each builder. The play button
|
||||
on a screen plays that run in place on the page. **Expand** opens a
|
||||
full-screen player. In that player, the user sees all builders side by
|
||||
side, or picks one builder, and plays the runs at 1× to 8×:
|
||||
|
||||
- **Claude Code and Codex:** `<builder>/screen.cast` is an asciinema
|
||||
recording of the exact terminal screen. The page plays it with the
|
||||
asciinema player.
|
||||
- **Instance AI:** `iai/screen.webm` is a video of the thread in the n8n UI.
|
||||
A headless browser records it while the run continues.
|
||||
|
||||
`COMPARE_RECORD=0` skips the Instance AI video. The terminal recordings come
|
||||
with interactive mode. `COMPARE_CLI_MODE=headless` has no terminal
|
||||
recording.
|
||||
|
||||
### Canvas on the page
|
||||
|
||||
`capture_canvas.mjs` expands the node groups, zooms to fit, and crops the
|
||||
screenshot to the nodes, their labels and their subtitles. Empty click areas
|
||||
and hidden hover toolbars do not change the crop. The page shows the canvas
|
||||
next to the expected outcome, at its real size and at most 360 px high.
|
||||
|
||||
A click on the canvas opens a full-screen viewer. It has a tab for each run
|
||||
that has a canvas, the verdict and the n8n link of the run, and a zoom
|
||||
button. The left and right arrow keys switch runs. A click on the image
|
||||
zooms in to that point. Esc closes the viewer, and the page then shows the
|
||||
last run that the user looked at. Cmd-click or Ctrl-click still opens the
|
||||
image in a new tab.
|
||||
|
||||
### Checks on the page
|
||||
|
||||
Under the expected outcome and the canvas, each run has sections from
|
||||
`derive_checks.mjs`. No LLM makes these checks, except the model calls of
|
||||
the real AI runs. Correctness, Execution with pinned data and Service calls
|
||||
change the verdict. The other sections show a score only.
|
||||
|
||||
**Correctness.** A failed check changes the verdict.
|
||||
|
||||
| Id | Check | Source |
|
||||
|---|---|---|
|
||||
| C1 | Node parameters match the node definitions | `validateWorkflow` of `@n8n/workflow-sdk`, with the built node definitions |
|
||||
| C2 | The n8n editor shows no node errors | The issues of each node in the editor. Missing credentials and empty required pickers are setup for the user (C2s, information only) |
|
||||
| C3 | Every node can be reached from a trigger | The connections |
|
||||
| C4 | Each `$('Node')` names a node that exists and runs before the node that reads it | The expressions |
|
||||
| C5 | Each expression field is in the output of the node that it reads | The real output of the pinned-data runs |
|
||||
| C6 | No keys or tokens in node parameters | The parameters |
|
||||
|
||||
C5 fails only when the output is real: a node that ran, the trigger
|
||||
fixture, the case data, or the schema of the structured output parser. A
|
||||
field that is missing only from a node schema sample or a prompt example
|
||||
shows as "Unconfirmed".
|
||||
|
||||
**Execution with pinned data.** One check for each scenario that ran without
|
||||
an error, and one for each `reach` or `skip` expectation. A failed check
|
||||
changes the verdict.
|
||||
|
||||
**Service calls.** `run_wire.mjs` loads the n8n execution engine from the
|
||||
built `packages/core` and runs each workflow in process. Triggers and AI
|
||||
nodes get pinned data. Service nodes run their real code, and the core
|
||||
eval hook answers each HTTP request with a recorded fixture before any
|
||||
credential is used. The process has no sandbox, and the hook sees only the
|
||||
requests that go through the n8n request helpers. Thus each node that runs
|
||||
must be on an allow-list (`runsInProcess` in `run_wire.mjs`):
|
||||
|
||||
- logic nodes from `LOGIC_TYPES`, HTTP Request, Wait and Telegram
|
||||
- declarative nodes
|
||||
- nodes whose credentials all authenticate HTTP requests (an `authenticate`
|
||||
block or OAuth)
|
||||
- a fixed set of AI nodes, with models that have a key in `MODEL_CREDENTIALS`
|
||||
|
||||
Code, Function and AI Transform nodes, and nodes with database, SSH, FTP,
|
||||
SMTP, file or shell access, are not on the list. Each scenario checks the
|
||||
nodes that run from its trigger. Pinned nodes do not run, so a node under
|
||||
a pinned AI node counts only in the real AI runs. If a node outside the list
|
||||
would run, the scenario is skipped. The report shows it as skipped, not
|
||||
failed, with the names of the nodes.
|
||||
The pinned-data runs still run Code nodes in the n8n task runner. No request
|
||||
leaves the machine, except the model calls of the real AI runs. Empty pickers get
|
||||
`mock-<parameter>` values for the run. Each scenario row lists the nodes
|
||||
that ran with their item counts, and each request. A failed check changes
|
||||
the verdict.
|
||||
|
||||
**Edge cases** and **Real AI runs.** The same in-process runs for the
|
||||
`edge` and `questions` entries of the scenarios file. A score.
|
||||
|
||||
**Final message claims.** `claims.py` compares the final message with the
|
||||
trace and the workflow. A score.
|
||||
|
||||
| Id | Claim | Source |
|
||||
|---|---|---|
|
||||
| M1 | The workflow id or URL is a workflow of this run | `run.json` |
|
||||
| M2 | The publish state (draft or active) | `workflow.json` `active` |
|
||||
| M3 | The workflow passed validation | The last validation tool call |
|
||||
| M4 | The workflow was tested, or was not run | Test and execution tool calls. Scripts that report passed tests count, with a note that they ran outside n8n |
|
||||
| M5 | Values in `code spans` | The workflow, the prompt and the tool calls |
|
||||
| M6 | A claim that the workflow was checked agrees with the service-call runs | `wire.json` |
|
||||
|
||||
**Eval binary checks.** `binary_checks.mts` runs the deterministic binary
|
||||
checks of the Instance AI eval suite
|
||||
(`packages/@n8n/instance-ai/evaluations/binaryChecks`), for example
|
||||
`no_empty_set_nodes` and `no_unnecessary_code_nodes`. Checks that do not
|
||||
apply are left out. A score.
|
||||
|
||||
**Eval quality checks.** The judges answer the LLM binary checks of the
|
||||
same suite: `valid_data_flow`, `correct_node_operations`,
|
||||
`handles_multiple_items`, `descriptive_node_names` and
|
||||
`response_describes_changes_accurately`. `summarize_runs.py` reads their
|
||||
rules from the check sources. A score.
|
||||
|
||||
**n8n standards.** A score, not part of the verdict. The builders do not get
|
||||
these rules: Claude Code and Codex run without the n8n skills, and Instance
|
||||
AI uses its own prompts. Only the rules that apply to the workflow count.
|
||||
|
||||
| Id | Rule | Source |
|
||||
|---|---|---|
|
||||
| S1 | An unattended workflow has an error workflow | `n8n-error-handling-official` |
|
||||
| S12 | A failed node goes to a wired error output, not on as data | `n8n-error-handling-official` |
|
||||
| S2 | Nodes that call a service have `retryOnFail` | `n8n-error-handling-official` |
|
||||
| S3 | Expressions read data by node name, not `$json` | `n8n-expressions-official` |
|
||||
| S4 | Structured AI output uses a manual schema with `autoFix` | `n8n-ai-nodes-official` STRUCTURED_OUTPUT.md |
|
||||
| S5 | Messages do not carry the n8n attribution | `n8n-node-configuration-official` COMMS_NODES.md, and the node definitions |
|
||||
| S6 | References do not read only the first item of a node that can output several | `n8n-expressions-official`, and the trigger code |
|
||||
| S7 | A model that decides a branch has a temperature of 0.2 or lower | workflow-sdk best practices: triage |
|
||||
| S8 | AI output that decides a branch has a reasoning field | workflow-sdk best practices: triage |
|
||||
| S9 | Every branch after an AI decision, including the fallback, goes somewhere | workflow-sdk best practices: triage |
|
||||
| S10 | A chatbot agent has memory | workflow-sdk best practices: chatbot |
|
||||
| S11 | The memory session key names a node and identifies the conversation | workflow-sdk best practices: chatbot, and `n8n-ai-nodes-official` MEMORY.md |
|
||||
|
||||
The skill rules come from commit `27b6b9f` of n8n-io/skills. Each check
|
||||
links to its source.
|
||||
|
||||
**The verdict** comes from the checks, not from the judges alone:
|
||||
|
||||
- `fail` when there is no workflow, or the judges say that the workflow does
|
||||
not do what the prompt asks (`goalMet: false`)
|
||||
- `pass` when every expected outcome item is met, and no correctness,
|
||||
execution or service-call check failed
|
||||
- `partial` for all other results
|
||||
|
||||
A met item must cite evidence that the workflow contains. Without evidence,
|
||||
or with evidence that is not in the workflow, the item shows as not
|
||||
confirmed. The Result column shows the verdict and the standards score
|
||||
(`Std 5/10`).
|
||||
|
||||
An old `verdicts.json` without ids or evidence keeps the verdict of the
|
||||
judge. A failed correctness or execution check changes its `pass` to
|
||||
`partial`.
|
||||
|
||||
### LLM steps on the page
|
||||
|
||||
Each run has a list of LLM steps, like the LLM steps view in the Instance AI
|
||||
debug panel. The box is collapsed by default. Click its title to open it. On
|
||||
the live page, a box that you open stays open when the page updates. The list
|
||||
shows the tools or text of each call, its time and a
|
||||
token bar. Select a step, or use the arrow keys, to see the token counts and
|
||||
what the call did. Step 1 starts with what the model gets before the first
|
||||
call, then the user message. Tool output is collapsed, except when the tool
|
||||
failed.
|
||||
|
||||
- **Claude Code:** the system prompt and the tool definitions, from the
|
||||
request shape in the session file. The tools show as chips in groups:
|
||||
System (the built-in tools of the CLI), one group for each MCP server, and
|
||||
Other (for example tools that the model provider runs). System and MCP
|
||||
show "None" when the model had no such tools. The tools that the run
|
||||
called are highlighted, with a call count and a dot for failures. Select a chip to see the description and a table of
|
||||
the parameters. A filter box searches names and descriptions.
|
||||
- **Codex:** the system prompt (base instructions), the developer messages
|
||||
and the environment context, from the session file. Codex does not save
|
||||
its tool definitions, so the runner records them (see "Isolation"). The
|
||||
tools use the same chips. Codex declares most system tools inside the
|
||||
description of its `exec` tool, so they show as "System · in exec" with
|
||||
their TypeScript declaration. Codex does not put MCP tools in the prompt:
|
||||
its scripts look them up in `ALL_TOOLS` at run time. The MCP group thus
|
||||
shows only the MCP tools that the run called, with a dashed border and no
|
||||
schema. Runs from before the recorder show no Codex tools.
|
||||
- **Instance AI:** the system prompt, the context that n8n puts before the
|
||||
prompt (for example `<thread-context>`), and the tools, from the first
|
||||
LangSmith LLM call of each agent. `langsmith_usage.py` saves them in
|
||||
`context.json`. Tools that the agent loads later with `load_tool` show as
|
||||
"System · loaded with load_tool". A sub-agent shows its own system prompt,
|
||||
briefing and tools at its first step. Runs from before `context.json` show
|
||||
no Instance AI context: run `langsmith_usage.py` again for them.
|
||||
|
||||
- **Claude Code:** the session file links each event to its LLM call.
|
||||
- **Codex:** text goes to the response that wrote it. A tool call goes to the
|
||||
response that asked for it. Codex calls the n8n tools from scripts that it
|
||||
runs with its `exec` tool. The page shows these scripts too, but does not
|
||||
count them as tool calls.
|
||||
- **Instance AI:** the LangSmith call times place each event. Text times are
|
||||
approximate. An asterisk on a time marks text that the page puts at the
|
||||
previous tool call.
|
||||
|
||||
### When a check fails
|
||||
|
||||
- Cannot log in: n8n is not running, or `N8N_BASE_URL`, `N8N_EVAL_EMAIL` or
|
||||
`N8N_EVAL_PASSWORD` is wrong.
|
||||
- The MCP token does not work: n8n shows the full MCP key only once. Ask the
|
||||
user to put the key in `N8N_MCP_TOKEN`, or ask before you run
|
||||
`python3 $S/prepare.py --rotate`. Rotation makes the old key stop working
|
||||
in every other MCP client of this user.
|
||||
- Docker is not running: Codex is skipped, and Claude runs as a clean
|
||||
process.
|
||||
|
||||
## 3. Find the prices
|
||||
|
||||
Codex reports no cost, and the estimate of Claude Code can be out of date
|
||||
for new models. Thus, find the published price of each model that a CLI
|
||||
builder used. The models are in `models` in the output of `compare.py`, and
|
||||
in `<runDir>/models.json`. Instance AI cost comes from LangSmith, so do not
|
||||
look up its model.
|
||||
|
||||
Do not skip this step. In interactive mode neither CLI reports a cost, so
|
||||
without prices the page shows "not reported" for both. `build_report.py`
|
||||
prints `models without a published price in case.json: …` while a model has
|
||||
no price.
|
||||
|
||||
First, tell the page that you look for the prices:
|
||||
|
||||
```bash
|
||||
python3 $S/build_report.py <runDir> --step prices=running
|
||||
```
|
||||
|
||||
For each model:
|
||||
|
||||
1. Search the web for the official price page of the provider, for example
|
||||
the Anthropic pricing page or the OpenAI pricing page. Do not use
|
||||
third-party price lists if the provider page has the model.
|
||||
2. Read four prices in USD per million tokens: input, cache read, cache
|
||||
write, output. If the provider has no cache-write price, use the input
|
||||
price. Anthropic has two cache-write prices. Use the 5-minute price, and
|
||||
tell the user that the 1-hour price is higher.
|
||||
3. If you cannot find the model, write no price for it. The page then shows
|
||||
the estimate of Claude Code, or "not reported" for Codex. Tell the user.
|
||||
|
||||
Add the prices to `<runDir>/case.json`, with the URL that you used:
|
||||
|
||||
```json
|
||||
"prices": {
|
||||
"<model id as in models.json>": {
|
||||
"input": 5, "cached": 0.5, "cacheWrite": 6.25, "output": 25,
|
||||
"source": "https://..."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The page calculates the cost of each call from its tokens. The tooltip of a
|
||||
Cost cell shows the source of the cost, the prices and the URL.
|
||||
|
||||
## 4. Judge each run
|
||||
|
||||
Show the costs and tell the page that you judge the runs:
|
||||
|
||||
```bash
|
||||
python3 $S/build_report.py <runDir> --step prices=done --step judge=running
|
||||
```
|
||||
|
||||
A panel of three judges judges the runs blind. Each judge reads only
|
||||
`<runDir>/judge/summary-blind.txt`. In that file the builders are called A,
|
||||
B and C in a random order, and the final messages have no builder or model
|
||||
names. `judge/blind-map.json` maps the letters back. Do not give a judge
|
||||
`summary.txt`, the map, the trace files or this conversation.
|
||||
|
||||
If the harness can start sub-agents, start three sub-agents with a fresh
|
||||
context, one for each judge. Use a model that is not one of the builder
|
||||
models when the harness lets you choose. If the harness cannot start
|
||||
sub-agents, judge once yourself from the blind file only, and write
|
||||
`verdicts-1.json`.
|
||||
|
||||
Give each judge these instructions:
|
||||
|
||||
> Read `<runDir>/judge/summary-blind.txt`. It lists the expected outcome
|
||||
> criteria with their ids, and for each builder the final message, every
|
||||
> node with all its parameters, the connections, and deterministic facts
|
||||
> (AUTO-CHECK FACTS). Trust the facts: they come from the node definitions,
|
||||
> the n8n editor and real pinned-data runs. For each builder, make one check
|
||||
> for each criterion id. A met check must cite evidence: the node name, the
|
||||
> parameter path, and the exact value in the workflow. Without evidence,
|
||||
> set `ok` to `null`. Set `goalMet` to false when the workflow does not do
|
||||
> what the prompt asks, and give the reason in `goalReason`. Then answer
|
||||
> each QUALITY CHECK in `quality`, with the check name as the `id`. Write
|
||||
> the result to `<runDir>/judge/verdicts-<n>.json`.
|
||||
|
||||
The format of each judge file, with the builder letters as keys:
|
||||
|
||||
```json
|
||||
{
|
||||
"A": {
|
||||
"summary": "One or two sentences about the result.",
|
||||
"goalMet": true,
|
||||
"goalReason": "",
|
||||
"checks": [
|
||||
{ "id": "1", "ok": true, "text": "A WhatsApp trigger starts the workflow on `messages`",
|
||||
"evidence": [{ "node": "WhatsApp Message Received", "path": "parameters.updates", "value": "messages" }] },
|
||||
{ "id": "9", "ok": false, "text": "Send Reply reads `$json.from`, which does not exist" },
|
||||
{ "id": "16", "ok": null, "text": "Could not confirm the sheet columns" },
|
||||
{ "ok": false, "text": "The Slack node is not connected" }
|
||||
],
|
||||
"quality": [
|
||||
{ "id": "valid_data_flow", "ok": true, "text": "Every expression reads a field that its source outputs" },
|
||||
{ "id": "descriptive_node_names", "ok": false, "text": "Two nodes keep the default names `Code` and `IF`" }
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `id` is the id of the criterion in brackets, as a string. Leave out `id`
|
||||
for a problem that is not a criterion. The page shows it under "Other
|
||||
findings".
|
||||
- `evidence` lists `{node, path, value}` entries. `path` is a dotted path in
|
||||
the node, for example `parameters.textBody` or
|
||||
`parameters.conditions.conditions[0].leftValue`. The page checks that the
|
||||
node has the path and that the path contains the value. An entry with
|
||||
`"source": "message"` and a `value` quotes the final message and is not
|
||||
checked.
|
||||
- Write each check as one short sentence with one fact. Put expressions,
|
||||
field paths and parameter names in backticks. The page shows them as
|
||||
code. Do not repeat the failed checks in the summary.
|
||||
|
||||
Then merge the panel into `verdicts.json`:
|
||||
|
||||
```bash
|
||||
python3 $S/merge_verdicts.py <runDir>
|
||||
```
|
||||
|
||||
For each criterion, the merged check takes the value that all judges or a
|
||||
majority gave. A split shows on the page ("3 judges · 1 split", and "2 of 3
|
||||
judges" on the check). Without a majority, the check is not confirmed. The
|
||||
summary and the other findings come from judge 1. The page computes the
|
||||
verdict (see "Checks on the page").
|
||||
|
||||
## 5. Build the page again
|
||||
|
||||
```bash
|
||||
python3 $S/build_report.py <runDir> --step judge=done
|
||||
```
|
||||
|
||||
If the command prints `models without a published price`, do step 3 for
|
||||
those models and build the page again.
|
||||
|
||||
The page is `<runDir>/report/index.html`. It works offline. Give the user the
|
||||
path, and the `live` URL from the output of `compare.py` while the server
|
||||
runs. Then, in a few sentences, report:
|
||||
|
||||
- the verdict of each builder, with the most important reason
|
||||
- failed correctness, execution or service-call checks, and judge splits
|
||||
- the standards, edge-case, real-AI, claims and eval scores of each
|
||||
builder, as separate results
|
||||
- the cost, time and number of LLM calls of each builder
|
||||
- builders that were skipped, and data that is missing
|
||||
|
||||
## How the builders run
|
||||
|
||||
**Instance AI** runs through the REST API of the instance, with its model.
|
||||
The runner answers every confirmation card with one fixed policy, so runs
|
||||
can be compared. It approves actions and plans, picks the first option of a
|
||||
question, allows a domain once, defers workflow and credential setup cards,
|
||||
and denies access to local files and MCP connections. `iai/run.json` lists
|
||||
each decision.
|
||||
|
||||
**Claude Code and Codex** run in a Docker container (`COMPARE_ISOLATION=container`):
|
||||
|
||||
- The container mounts only the run folder of the builder. The builder
|
||||
cannot see host files, skills, memory, `CLAUDE.md`, `AGENTS.md` or user
|
||||
settings.
|
||||
- The container has no Linux capabilities and a read-only root file system.
|
||||
- Claude starts with a new config folder, no built-in tools, and only the
|
||||
`n8n` MCP server. It does not use `--bare`, because `--bare` replaces the
|
||||
normal system prompt with a 150-byte one. The container already keeps out
|
||||
host settings, so Claude gets its normal system prompt, like Codex does.
|
||||
`CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1` stops update checks and
|
||||
telemetry.
|
||||
- `COMPARE_CLAUDE_TOOLS=default` gives Claude the built-in tools of a normal
|
||||
Claude Code session, such as Bash, Read, Write, Edit and WebFetch, with
|
||||
`--permission-mode bypassPermissions`. Claude then uses every tool without
|
||||
a question, as Codex does. The tools can change files in the run folder
|
||||
and read the environment, which holds the model key and the MCP token.
|
||||
The secret check after the run finds a key that a tool wrote to a file.
|
||||
Claude Code also turns on tool search: the model gets only the names of
|
||||
the n8n tools and of some built-in tools, and loads a schema with
|
||||
`ToolSearch`. The page shows these deferred tools with a dashed border.
|
||||
The default `none` keeps runs comparable with older runs, because the
|
||||
extra tools change the tokens and the cost.
|
||||
- Codex gets a new `CODEX_HOME` that holds only the n8n MCP server.
|
||||
Its plugins, apps, browser use, computer use and image generation are off.
|
||||
- The container can still reach the internet and the host network. It needs
|
||||
them for the model API and for n8n.
|
||||
- Codex sends its model requests through a local recorder
|
||||
(`request_recorder.py`). The recorder sends each request to OpenAI without
|
||||
changes and writes the request body, but not the headers, to
|
||||
`codex/requests.jsonl`. The key is in a header, so it stays out of the file.
|
||||
The file does not hold the conversation input. It holds the tools and the
|
||||
instructions only when they change. The recorder listens on `127.0.0.1`,
|
||||
which Docker Desktop forwards to `host.docker.internal`. On Linux, set
|
||||
`COMPARE_RECORDER_BIND` to the address of the Docker bridge. Set
|
||||
`COMPARE_RECORD_REQUESTS=0` to send Codex straight to OpenAI.
|
||||
|
||||
By default the CLIs run in interactive mode (`COMPARE_CLI_MODE=interactive`):
|
||||
|
||||
- The normal terminal screen of the CLI runs in tmux, at 120 × 36
|
||||
characters. tmux answers the terminal queries of the CLI, and its output
|
||||
goes to `screen.cast`.
|
||||
- The CLIs connect MCP servers in the background. Thus the script pastes the
|
||||
prompt when the input box shows, plus a short wait, not as an argument.
|
||||
- Claude uses `--permission-mode dontAsk`: the n8n tools are pre-approved,
|
||||
and all other tools are denied without a question.
|
||||
- The model key reaches the CLI through a variable that only the CLI reads
|
||||
(`apiKeyHelper` for Claude, a provider `env_key` for Codex). Thus no screen
|
||||
asks to confirm the key or shows part of it, and no key is in a file.
|
||||
- The script reads the session file of the CLI for the progress lines, the
|
||||
end of the turn, the trace and the tokens. Claude Code then shows no cost
|
||||
estimate, so the cost needs the published prices.
|
||||
- After each run, the script searches the run folder for the model key and
|
||||
the MCP token. It prints a warning with the file name if it finds one.
|
||||
|
||||
`COMPARE_CLI_MODE=headless` uses `claude -p` and `codex exec` with a JSON
|
||||
event stream instead.
|
||||
|
||||
The first run builds the image `n8n-builder-sandbox` from
|
||||
`sandbox/Dockerfile`. A change to the Dockerfile rebuilds the image on the
|
||||
next run. The image has `python3`, `jq` and `ripgrep`, because Codex and
|
||||
Claude Code with `COMPARE_CLAUDE_TOOLS=default` use them in their shell. To get new CLI
|
||||
versions, run `docker rmi n8n-builder-sandbox`.
|
||||
|
||||
`COMPARE_ISOLATION=clean` runs Claude as a clean process on the host: an
|
||||
empty environment, a new config folder, no built-in tools, and only the n8n
|
||||
MCP server. It ignores `COMPARE_CLAUDE_TOOLS`. It uses `--bare` to keep out host settings, so it gets the
|
||||
minimal system prompt. It needs an API key, because `--bare` does not read
|
||||
the subscription login. Codex has no clean mode, because its shell tool can
|
||||
read the host disk.
|
||||
|
||||
## Single steps
|
||||
|
||||
`compare.py` calls these scripts. Use them to repeat one step.
|
||||
|
||||
| Script | Step |
|
||||
|---|---|
|
||||
| `prepare.py` | Check the setup. `--rotate` makes a new MCP key |
|
||||
| `run_instance_ai.py --out <dir>/iai --prompt-file <file>` | Run Instance AI |
|
||||
| `run_cli_builder.py --builder claude\|codex --out <dir>/<name> --prompt-file <file>` | Run a CLI builder. `--mode headless` skips the screen |
|
||||
| `node record_iai.mjs <dir>/iai` | Record the Instance AI video. `run_instance_ai.py` starts it |
|
||||
| `live_server.py --run-dir <dir>` | Serve the live page and the report of a run folder. Prints nothing. The URL is in `<dir>/live-url.txt` |
|
||||
| `langsmith_usage.py --run-dir <dir>/iai` | Fetch Instance AI usage and model context |
|
||||
| `node capture_canvas.mjs <dir>` | Save `workflow.json`, `canvas.png` and `canvas-issues.json` for each run |
|
||||
| `run_scenarios.py <dir>` | Run each workflow with pinned data. Writes `<name>/exec.json` |
|
||||
| `node sdk.mjs pin-plan <workflow.json> [scenarios.json]` | Print the pin data of each scenario, without a run |
|
||||
| `node run_wire.mjs <dir>` | Run each workflow in process with recorded API fixtures. Writes `<name>/wire.json` |
|
||||
| `pnpm --filter @n8n/instance-ai exec tsx $S/binary_checks.mts <dir>` | Run the deterministic eval binary checks. Writes `<name>/binary-checks.json` |
|
||||
| `claims.py <dir>` | Check the final-message claims. Writes `<name>/claims.json` |
|
||||
| `node derive_checks.mjs <dir>` | Write `<name>/auto-checks.json` from all the files above |
|
||||
| `summarize_runs.py <dir> [--blind]` | Print the judge summary. `--blind` writes `judge/summary-blind.txt` and `judge/blind-map.json` |
|
||||
| `merge_verdicts.py <dir>` | Merge `judge/verdicts-*.json` into `verdicts.json` |
|
||||
| `build_report.py <dir> [--step judge\|prices=queued\|running\|done]` | Build the page. `--step` first sets the state of a step after the run |
|
||||
|
||||
If a runner did not find a workflow id, read its trace for the id. Set
|
||||
`workflowId` in that `run.json`, then run `capture_canvas.mjs`,
|
||||
`run_scenarios.py`, `run_wire.mjs`, `binary_checks.mts`, `claims.py`,
|
||||
`derive_checks.mjs` and `summarize_runs.py` again.
|
||||
|
||||
## Notes
|
||||
|
||||
- Cost sources: Instance AI cost comes from LangSmith. For Claude Code and
|
||||
Codex, the cost is tokens × the published prices in `case.json`. Without a
|
||||
price, Claude Code in headless mode shows its own estimate. All other runs
|
||||
show "not reported".
|
||||
The tooltip of each Cost cell shows the source of the cost.
|
||||
- The workflows stay in the n8n instance. Archive them in n8n when they are
|
||||
no longer necessary. The pinned-data runs add test executions to them.
|
||||
- The n8n skills stay out of the sandboxes, so the standards score shows
|
||||
what each builder does without them. Do not add the skills to the
|
||||
sandboxes to raise the score.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 102 KiB |
|
|
@ -1,21 +0,0 @@
|
|||
# Sandbox for CLI workflow builders. The container sees only the run directory
|
||||
# that is mounted at /out. It reaches n8n through host.docker.internal.
|
||||
FROM node:26-slim
|
||||
|
||||
ARG CLAUDE_CODE_VERSION=latest
|
||||
ARG CODEX_VERSION=latest
|
||||
|
||||
# Codex uses the system trust store. Claude Code uses the CA list in Node.
|
||||
# tmux runs the interactive CLI screens so that they can be recorded and watched.
|
||||
# Claude Code (with built-in tools) and Codex use python3, jq and ripgrep in their shell to read large tool results.
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends ca-certificates tmux python3 jq ripgrep \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" "@openai/codex@${CODEX_VERSION}" \
|
||||
&& npm cache clean --force
|
||||
|
||||
ENV DISABLE_AUTOUPDATER=1 \
|
||||
DISABLE_TELEMETRY=1 \
|
||||
DISABLE_ERROR_REPORTING=1 \
|
||||
CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
|
||||
|
|
@ -1,100 +0,0 @@
|
|||
// Run an interactive CLI inside tmux and record its screen as an asciinema v2 file.
|
||||
//
|
||||
// node screen.mjs run <dir> start tmux, record, and wait for <dir>/stop or the CLI exit
|
||||
// node screen.mjs launch <dir> (inside tmux) start the command in <dir>/command.json
|
||||
// node screen.mjs record <dir> (tmux pipe-pane) write the pane output to <dir>/screen.cast
|
||||
//
|
||||
// <dir>/command.json is {"argv": [...], "cols": 120, "rows": 36, "input": "prompt",
|
||||
// "readyText": "text on the screen when the CLI accepts input", "settleMs": 0}. tmux answers the terminal
|
||||
// queries of the CLI, so the CLI behaves as in a real terminal. The recording has the raw
|
||||
// output of the CLI, so a player shows the exact screen.
|
||||
import { spawn, spawnSync } from 'node:child_process';
|
||||
import { createWriteStream, existsSync, readFileSync, writeFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
|
||||
const [mode, dir] = process.argv.slice(2);
|
||||
const command = JSON.parse(readFileSync(join(dir, 'command.json'), 'utf8'));
|
||||
const READY = '/tmp/screen-ready';
|
||||
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
|
||||
const tmux = (...args) => spawnSync('tmux', args, { stdio: 'inherit' }).status;
|
||||
const screenText = () =>
|
||||
spawnSync('tmux', ['capture-pane', '-p', '-t', 'run'], { encoding: 'utf8' }).stdout ?? '';
|
||||
|
||||
// The CLIs connect MCP servers in the background. A prompt on the command line can start
|
||||
// before the n8n tools exist, so the prompt is pasted when the input box is ready.
|
||||
async function typePrompt({ input, readyText, settleMs = 0 }) {
|
||||
const deadline = Date.now() + 60_000;
|
||||
while (readyText && !screenText().includes(readyText) && Date.now() < deadline) await sleep(250);
|
||||
await sleep(settleMs);
|
||||
tmux('set-buffer', '-b', 'prompt', '--', input);
|
||||
tmux('paste-buffer', '-p', '-d', '-b', 'prompt', '-t', 'run');
|
||||
await sleep(500);
|
||||
tmux('send-keys', '-t', 'run', 'Enter');
|
||||
}
|
||||
|
||||
if (mode === 'record') {
|
||||
const out = createWriteStream(join(dir, 'screen.cast'));
|
||||
out.write(
|
||||
JSON.stringify({
|
||||
version: 2,
|
||||
width: command.cols,
|
||||
height: command.rows,
|
||||
timestamp: Math.floor(Date.now() / 1000),
|
||||
env: { TERM: 'xterm-256color' },
|
||||
}) + '\n',
|
||||
);
|
||||
const start = process.hrtime.bigint();
|
||||
const decoder = new TextDecoder('utf-8');
|
||||
process.stdin.on('data', (chunk) => {
|
||||
const text = decoder.decode(chunk, { stream: true });
|
||||
if (!text) return;
|
||||
const t = Number(process.hrtime.bigint() - start) / 1e9;
|
||||
out.write(JSON.stringify([Number(t.toFixed(6)), 'o', text]) + '\n');
|
||||
});
|
||||
process.stdin.on('end', () => out.end());
|
||||
} else if (mode === 'launch') {
|
||||
while (!existsSync(READY)) await sleep(50);
|
||||
const [bin, ...args] = command.argv;
|
||||
const child = spawn(bin, args, { stdio: 'inherit' });
|
||||
child.on('exit', (code) => {
|
||||
writeFileSync(join(dir, 'exited'), String(code ?? 1));
|
||||
process.exit(code ?? 1);
|
||||
});
|
||||
// A CLI that cannot start emits error, not exit. Without the marker, the run waits for ever.
|
||||
child.on('error', (error) => {
|
||||
console.error(`Cannot start ${bin}: ${error.message}`);
|
||||
writeFileSync(join(dir, 'exited'), '127');
|
||||
process.exit(127);
|
||||
});
|
||||
} else if (mode === 'run') {
|
||||
const self = new URL(import.meta.url).pathname;
|
||||
// mouse and focus-events stop the tmux tips that Claude Code shows in the recorded screen.
|
||||
writeFileSync(
|
||||
'/tmp/tmux.conf',
|
||||
'set -g status off\nset -g default-terminal xterm-256color\nset -g mouse on\nset -g focus-events on\n',
|
||||
);
|
||||
tmux(
|
||||
'-f',
|
||||
'/tmp/tmux.conf',
|
||||
'new-session',
|
||||
'-d',
|
||||
'-s',
|
||||
'run',
|
||||
'-x',
|
||||
String(command.cols),
|
||||
'-y',
|
||||
String(command.rows),
|
||||
`node ${self} launch ${dir}`,
|
||||
);
|
||||
// A watcher in another terminal size must not resize the recorded screen. tmux 3.5 stops
|
||||
// when this option is in the startup config, so it is set after the session starts.
|
||||
tmux('set-option', '-g', 'window-size', 'manual');
|
||||
tmux('pipe-pane', '-O', '-t', 'run', `node ${self} record ${dir}`);
|
||||
writeFileSync(READY, '');
|
||||
if (command.input) await typePrompt(command);
|
||||
while (!existsSync(join(dir, 'stop')) && !existsSync(join(dir, 'exited'))) await sleep(500);
|
||||
// Keep the final screen in the recording for a moment.
|
||||
await sleep(2500);
|
||||
tmux('kill-server');
|
||||
await sleep(500);
|
||||
}
|
||||
|
|
@ -1,50 +0,0 @@
|
|||
// Runs the deterministic binary checks of the Instance AI eval suite on each workflow of a run.
|
||||
// Run it with tsx from the instance-ai package, because the checks are TypeScript sources:
|
||||
//
|
||||
// pnpm --filter @n8n/instance-ai exec tsx <skill>/scripts/binary_checks.mts <run-dir>
|
||||
//
|
||||
// Writes <builder>/binary-checks.json with {source, outcomes: [{name, description, dimension, status, comment}]}.
|
||||
import { existsSync, readdirSync, readFileSync, writeFileSync } from 'node:fs';
|
||||
import { dirname, join, resolve } from 'node:path';
|
||||
import { fileURLToPath, pathToFileURL } from 'node:url';
|
||||
|
||||
const repo = resolve(dirname(fileURLToPath(import.meta.url)), '../../../..');
|
||||
const SOURCE = 'packages/@n8n/instance-ai/evaluations/binaryChecks/checks';
|
||||
|
||||
type Check = {
|
||||
name: string;
|
||||
description: string;
|
||||
kind: 'deterministic' | 'llm';
|
||||
dimension: string;
|
||||
run(workflow: unknown, ctx: Record<string, unknown>): unknown;
|
||||
};
|
||||
type Result = { pass: boolean; comment?: string; applicable?: boolean; errored?: boolean };
|
||||
|
||||
const readJson = (file: string): unknown => (existsSync(file) ? JSON.parse(readFileSync(file, 'utf8')) : null);
|
||||
|
||||
const runDir = resolve(process.argv[2] ?? '.');
|
||||
const { DETERMINISTIC_CHECKS } = (await import(pathToFileURL(join(repo, SOURCE, 'index.ts')).href)) as { DETERMINISTIC_CHECKS: Check[] };
|
||||
const prompt = existsSync(join(runDir, 'prompt.txt')) ? readFileSync(join(runDir, 'prompt.txt'), 'utf8') : '';
|
||||
|
||||
for (const entry of readdirSync(runDir, { withFileTypes: true })) {
|
||||
const dir = join(runDir, entry.name);
|
||||
const workflow = entry.isDirectory() ? readJson(join(dir, 'workflow.json')) : null;
|
||||
if (!workflow) continue;
|
||||
const outcomes = [];
|
||||
for (const check of DETERMINISTIC_CHECKS) {
|
||||
let status: string;
|
||||
let comment: string | undefined;
|
||||
try {
|
||||
const r = (await check.run(workflow, { prompt })) as Result;
|
||||
status = r.errored ? 'error' : r.applicable === false ? 'n_a' : r.pass ? 'pass' : 'fail';
|
||||
comment = r.comment;
|
||||
} catch (error) {
|
||||
status = 'error';
|
||||
comment = error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
outcomes.push({ name: check.name, description: check.description, dimension: check.dimension, status, ...(comment ? { comment } : {}) });
|
||||
}
|
||||
writeFileSync(join(dir, 'binary-checks.json'), JSON.stringify({ source: SOURCE, outcomes }, null, 1));
|
||||
const scored = outcomes.filter((o) => o.status === 'pass' || o.status === 'fail');
|
||||
console.log(`${entry.name}: binary-checks.json written (${scored.filter((o) => o.status === 'pass').length}/${scored.length})`);
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -1,162 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
// Take a cropped canvas screenshot of each workflow that a run created.
|
||||
//
|
||||
// node capture_canvas.mjs <run-dir>
|
||||
//
|
||||
// Reads <run-dir>/<builder>/run.json and writes <run-dir>/<builder>/canvas.png, workflow.json and
|
||||
// canvas-issues.json.
|
||||
import { existsSync, readdirSync, readFileSync, writeFileSync } from 'node:fs';
|
||||
import { join, resolve } from 'node:path';
|
||||
|
||||
import { baseUrl, launchBrowser, signIn } from './n8n_browser.mjs';
|
||||
|
||||
const runDir = resolve(process.argv[2] ?? '.');
|
||||
|
||||
// The status icons that CanvasNodeStatusIcons.vue draws on a node. n8n computes them from the node
|
||||
// type description, so they are the verdict of n8n itself on each node.
|
||||
const ISSUE_MARKERS = ['node-issues', 'node-setup-required', 'node-not-installed', 'node-restricted'];
|
||||
|
||||
/**
|
||||
* The status icon of each node, with the issues that the editor computed for it. The tooltips
|
||||
* are teleported and show after a delay, so the issues come from the workflow document store.
|
||||
*/
|
||||
async function readIssues(page, workflowId) {
|
||||
return await page.evaluate(
|
||||
({ markers, workflowId }) => {
|
||||
const icons = {};
|
||||
for (const el of document.querySelectorAll(
|
||||
markers.map((m) => `[data-node-name] [data-test-id="${m}"]`).join(','),
|
||||
)) {
|
||||
icons[el.closest('[data-node-name]').dataset.nodeName] = el.dataset.testId;
|
||||
}
|
||||
const pinia = document.querySelector('#app')?.__vue_app__?.config.globalProperties.$pinia;
|
||||
const key = [...(pinia?._s.keys() ?? [])].find((k) =>
|
||||
k.startsWith(`workflowDocuments/${workflowId}@`),
|
||||
);
|
||||
const storeNodes = key ? (pinia._s.get(key).allNodes ?? []) : [];
|
||||
const nodes = storeNodes
|
||||
.filter((n) => icons[n.name] || (n.issues && Object.keys(n.issues).length))
|
||||
.map((n) => ({
|
||||
name: n.name,
|
||||
type: n.type,
|
||||
marker: icons[n.name] ?? null,
|
||||
issues: n.issues ?? {},
|
||||
}));
|
||||
for (const [name, marker] of Object.entries(icons)) {
|
||||
if (!nodes.some((n) => n.name === name)) nodes.push({ name, marker, issues: {} });
|
||||
}
|
||||
return { markers, store: Boolean(key), nodes };
|
||||
},
|
||||
{ markers: ISSUE_MARKERS, workflowId },
|
||||
);
|
||||
}
|
||||
|
||||
const runs = readdirSync(runDir, { withFileTypes: true })
|
||||
.filter((d) => d.isDirectory() && existsSync(join(runDir, d.name, 'run.json')))
|
||||
.map((d) => ({
|
||||
dir: join(runDir, d.name),
|
||||
meta: JSON.parse(readFileSync(join(runDir, d.name, 'run.json'), 'utf8')),
|
||||
}))
|
||||
.filter((r) => r.meta.workflowId);
|
||||
|
||||
if (runs.length === 0) {
|
||||
console.log('No run has a workflowId. Nothing to capture.');
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const browser = await launchBrowser();
|
||||
const VIEWPORT = { width: 1800, height: 1000 };
|
||||
const context = await browser.newContext({ viewport: VIEWPORT, deviceScaleFactor: 2 });
|
||||
const page = await context.newPage();
|
||||
await signIn(page);
|
||||
|
||||
for (const { dir, meta } of runs) {
|
||||
const id = meta.workflowId;
|
||||
const res = await page.request.get(`${baseUrl}/rest/workflows/${id}`, {
|
||||
headers: {
|
||||
'browser-id': await page.evaluate(() => localStorage.getItem('n8n-browserId') ?? ''),
|
||||
},
|
||||
});
|
||||
if (res.ok())
|
||||
writeFileSync(join(dir, 'workflow.json'), JSON.stringify((await res.json()).data, null, 1));
|
||||
else console.log(`${id}: workflow fetch failed with ${res.status()}`);
|
||||
|
||||
await page.goto(`${baseUrl}/workflow/${id}`);
|
||||
try {
|
||||
await page.locator('[data-test-id="canvas-node"]').first().waitFor({ timeout: 30_000 });
|
||||
} catch {
|
||||
console.log(`${id}: no canvas nodes, skipped`);
|
||||
continue;
|
||||
}
|
||||
// Expanding one group re-renders the others, so repeat until no collapsed group is left.
|
||||
for (let round = 0; round < 5; round++) {
|
||||
const clicked = await page.evaluate(() => {
|
||||
const buttons = [...document.querySelectorAll('button[aria-label="Expand"]')];
|
||||
buttons.forEach((b) => b.click());
|
||||
return buttons.length;
|
||||
});
|
||||
if (clicked === 0) break;
|
||||
await page.waitForTimeout(400);
|
||||
}
|
||||
// The layout moves after groups expand. Fit twice so the second fit sees the final size.
|
||||
for (let i = 0; i < 2; i++) {
|
||||
await page.waitForTimeout(1000);
|
||||
await page
|
||||
.locator('[data-test-id="zoom-to-fit"]')
|
||||
.click({ timeout: 5_000 })
|
||||
.catch(() => {});
|
||||
}
|
||||
await page.waitForTimeout(800);
|
||||
|
||||
const box = await page.evaluate(() => {
|
||||
const rects = [...document.querySelectorAll('.vue-flow__node')].map((n) =>
|
||||
n.getBoundingClientRect(),
|
||||
);
|
||||
if (rects.length === 0) return null;
|
||||
// Labels, subtitles and expanded groups overflow their node element, so use the descendants too.
|
||||
// Only count what draws: hit areas and hover toolbars are empty boxes around the node.
|
||||
const draws = (el) =>
|
||||
el instanceof SVGSVGElement ||
|
||||
el instanceof HTMLImageElement ||
|
||||
[...el.childNodes].some((c) => c.nodeType === Node.TEXT_NODE && c.textContent.trim());
|
||||
const visible = (el) => {
|
||||
for (let e = el; e && !e.classList.contains('vue-flow__node'); e = e.parentElement) {
|
||||
const style = getComputedStyle(e);
|
||||
if (
|
||||
style.visibility === 'hidden' ||
|
||||
style.display === 'none' ||
|
||||
Number(style.opacity) === 0
|
||||
)
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
};
|
||||
const all = rects.concat(
|
||||
[...document.querySelectorAll('.vue-flow__node *')]
|
||||
.filter((el) => draws(el) && visible(el))
|
||||
.map((n) => n.getBoundingClientRect())
|
||||
.filter((r) => r.width > 0 && r.height > 0),
|
||||
);
|
||||
const left = Math.min(...all.map((r) => r.left));
|
||||
const top = Math.min(...all.map((r) => r.top));
|
||||
const right = Math.max(...all.map((r) => r.right));
|
||||
const bottom = Math.max(...all.map((r) => r.bottom));
|
||||
return { left, top, right, bottom };
|
||||
});
|
||||
const pad = 40;
|
||||
const clip = box
|
||||
? {
|
||||
x: Math.max(0, box.left - pad),
|
||||
y: Math.max(0, box.top - pad),
|
||||
width: Math.min(VIEWPORT.width, box.right + pad) - Math.max(0, box.left - pad),
|
||||
height: Math.min(VIEWPORT.height, box.bottom + pad) - Math.max(0, box.top - pad),
|
||||
}
|
||||
: undefined;
|
||||
await page.screenshot({ path: join(dir, 'canvas.png'), clip });
|
||||
console.log(`${id}: canvas.png written`);
|
||||
const issues = await readIssues(page, id);
|
||||
writeFileSync(join(dir, 'canvas-issues.json'), JSON.stringify(issues, null, 1));
|
||||
console.log(`${id}: canvas-issues.json written (${issues.nodes.length} nodes with a status icon)`);
|
||||
}
|
||||
|
||||
await browser.close();
|
||||
|
|
@ -1,180 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Check what the final message of each builder says against the trace, the workflow and the runs.
|
||||
|
||||
claims.py <run-dir>
|
||||
|
||||
Writes <builder>/claims.json with {message, checks: [{id, ok, text, details}]}. No LLM is involved.
|
||||
A check is about one kind of claim: the workflow id, the publish state, validation, testing, the
|
||||
values in `code spans`, and if a claim that the workflow works agrees with the service-call runs.
|
||||
A message that does not make a kind of claim gets no check for it.
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from build_report import load_run # noqa: E402
|
||||
from summarize_runs import final_message # noqa: E402
|
||||
|
||||
TEST_TOOL = re.compile(r'test|execut|verify|run[-_]?workflow|simulat', re.I)
|
||||
SHELL_TOOL = re.compile(r'command|shell|bash', re.I)
|
||||
SCRIPT_PASSED = re.compile(r'"passed"\s*:\s*true|\b\d+ (?:tests? )?passed\b|\ball tests passed\b', re.I)
|
||||
VALIDATE_TOOL = re.compile(r'validat', re.I)
|
||||
NEGATION = re.compile(r"\b(not|never|no|nothing|without)\b|n't\b", re.I)
|
||||
# A clause about what can or will happen later is not a claim about the run.
|
||||
MODAL = re.compile(r"\b(can|can't|cannot|will|won't|could|would|should|once|until|after that|let me know|need)\b", re.I)
|
||||
TESTED = re.compile(r'\b(tested|test run|ran|run|runs|executed|simulat\w*|routing checks?|checks? passed)\b', re.I)
|
||||
VALIDATED = re.compile(r'\b(validat\w*|valid)\b', re.I)
|
||||
PASSED = re.compile(r'\b(pass\w*|succeed\w*|valid|ok|clean)\b', re.I)
|
||||
DRAFT = re.compile(r"\b(draft|inactive|unpublished|not (?:yet )?(?:live|active|published)|isn't (?:yet )?(?:live|active)|not turned on)\b", re.I)
|
||||
LIVE = re.compile(r'\b(?:is|now) (?:live|active|published)\b', re.I)
|
||||
WORKFLOW_ID = re.compile(r'/workflow/([A-Za-z0-9]{8,})|\bID `?([A-Za-z0-9]{16})`?')
|
||||
CODE_SPAN = re.compile(r'`([^`\n]{2,80})`')
|
||||
INVALID = re.compile(r'"valid"\s*:\s*false|"success"\s*:\s*false', re.I)
|
||||
# A test tool can answer without a transport error and still report a failed execution.
|
||||
RUN_FAILED = re.compile(r'"status"\s*:\s*"(?:error|failed|crashed|canceled|cancelled)"|"success"\s*:\s*false', re.I)
|
||||
|
||||
|
||||
def clauses(message):
|
||||
"""The message split into clauses, with Markdown emphasis removed."""
|
||||
text = re.sub(r'[*_]{1,2}([^*_]+)[*_]{1,2}', r'\1', message or '')
|
||||
# Streamed messages can join sentences with no space, as in "done.Next".
|
||||
return [c.strip() for c in re.split(r'(?<=[.!?:])\s+|(?<=[a-z][.!?])(?=[A-Z])|\n+|;\s+|,\s+but\s+|\s+but\s+', text) if c.strip()]
|
||||
|
||||
|
||||
def claim(message, pattern, extra=None):
|
||||
"""(positive, negative) clauses that match pattern. extra must also match a positive clause."""
|
||||
pos, neg = [], []
|
||||
for c in clauses(message):
|
||||
if not pattern.search(c) or MODAL.search(c):
|
||||
continue
|
||||
if NEGATION.search(c):
|
||||
neg.append(c)
|
||||
elif extra is None or extra.search(c):
|
||||
pos.append(c)
|
||||
return pos, neg
|
||||
|
||||
|
||||
def tool_text(e):
|
||||
return ' '.join(x if isinstance(x, str) else json.dumps(x, ensure_ascii=False) for x in (e.get('args'), e.get('code'), e.get('result')) if x)
|
||||
|
||||
|
||||
def check(id_, ok, text, details):
|
||||
return {'id': id_, 'ok': ok, 'text': text, 'details': details}
|
||||
|
||||
|
||||
def quote(clauses_):
|
||||
return [f'Message: "{c}"' for c in clauses_[:3]]
|
||||
|
||||
|
||||
def id_check(message, meta, workflow):
|
||||
ids = {a or b for a, b in WORKFLOW_ID.findall(message or '')}
|
||||
if not ids:
|
||||
return None
|
||||
known = set(meta.get('workflowIds') or []) | {meta.get('workflowId'), (workflow or {}).get('id')}
|
||||
wrong = sorted(ids - known)
|
||||
return check('M1', not wrong, 'The workflow id in the message is the workflow that the run created',
|
||||
[f'{i} is not a workflow of this run' for i in wrong] or [f'{", ".join(sorted(ids))} matches.'])
|
||||
|
||||
|
||||
def state_check(message, workflow):
|
||||
draft, live = DRAFT.findall(message or ''), LIVE.findall(message or '')
|
||||
if workflow is None or not (draft or live):
|
||||
return None
|
||||
active = bool(workflow.get('active'))
|
||||
says = 'active' if live and not draft else 'a draft'
|
||||
ok = active == (says == 'active')
|
||||
return check('M2', ok, f'The message says the workflow is {says}', [f'workflow.json active is {str(active).lower()}.'])
|
||||
|
||||
|
||||
def validation_check(message, events):
|
||||
pos, _ = claim(message, VALIDATED, PASSED)
|
||||
if not pos:
|
||||
return None
|
||||
calls = [e for e in events if (e['kind'] == 'tool' and VALIDATE_TOOL.search(e.get('name') or ''))
|
||||
or (e['kind'] == 'code' and VALIDATE_TOOL.search(e.get('code') or ''))]
|
||||
if not calls:
|
||||
return check('M3', False, 'The message says the workflow passed validation', quote(pos) + ['No validation tool call in the trace.'])
|
||||
last = calls[-1]
|
||||
failed = last.get('error') or INVALID.search(str(last.get('result') or ''))
|
||||
return check('M3', not failed, 'The message says the workflow passed validation', quote(pos) + [
|
||||
f'{len(calls)} validation call(s). The last one {"failed" if failed else "passed"}: {last.get("name")}.'])
|
||||
|
||||
|
||||
def test_check(message, events):
|
||||
pos, neg = claim(message, TESTED)
|
||||
runs = [e for e in events if e['kind'] == 'tool' and TEST_TOOL.search(e.get('name') or '')
|
||||
and not VALIDATE_TOOL.search(e.get('name') or '') and not SHELL_TOOL.search(e.get('name') or '')]
|
||||
scripts = [e for e in events if e['kind'] == 'code' and SCRIPT_PASSED.search(str(e.get('result') or ''))]
|
||||
if pos:
|
||||
ok_runs = [e for e in runs if not e.get('error') and not RUN_FAILED.search(tool_text({'result': e.get('result')}))]
|
||||
if ok_runs:
|
||||
names = sorted({e['name'] for e in ok_runs})
|
||||
return check('M4', True, 'The message says the workflow was tested', quote(pos) + [f'{len(ok_runs)} successful test call(s): {", ".join(names)}.'])
|
||||
if scripts:
|
||||
return check('M4', True, 'The message says the workflow was tested', quote(pos) + [
|
||||
f'{len(scripts)} script(s) in the trace report passed tests. These ran outside n8n, not as a workflow execution.'])
|
||||
return check('M4', False, 'The message says the workflow was tested', quote(pos) + ['No test or execution call in the trace.'])
|
||||
if neg and runs:
|
||||
return check('M4', False, 'The message says the workflow was not run', quote(neg) + [f'The trace has {len(runs)} test call(s).'])
|
||||
if neg:
|
||||
return check('M4', True, 'The message says the workflow was not run', quote(neg) + ['No test or execution call in the trace.'])
|
||||
return None
|
||||
|
||||
|
||||
def span_check(message, events, workflow, prompt):
|
||||
spans = sorted({s.strip() for s in CODE_SPAN.findall(message or '')})
|
||||
if not spans:
|
||||
return None
|
||||
haystack = '\n'.join([json.dumps(workflow or {}, ensure_ascii=False), prompt] + [tool_text(e) for e in events])
|
||||
loose = lambda s: re.sub(r'\s+', ' ', s).lower()
|
||||
hay = loose(haystack.replace('\\"', '"'))
|
||||
missing = [s for s in spans if loose(s) not in hay]
|
||||
return check('M5', not missing, f'Values in code spans are in the workflow, the prompt or a tool result ({len(spans) - len(missing)} of {len(spans)})',
|
||||
[f'`{s}` is not in the workflow, the prompt or any tool call' for s in missing])
|
||||
|
||||
|
||||
def works_check(message, wire):
|
||||
pos, _ = claim(message, re.compile(TESTED.pattern + '|' + VALIDATED.pattern, re.I), None)
|
||||
if not pos or not wire or wire.get('skipped'):
|
||||
return None
|
||||
failed = [s for s in wire.get('scenarios', []) if s.get('kind') == 'contract'
|
||||
and (s.get('status') != 'success' or any(c.get('ok') is False for c in s.get('checks') or []))]
|
||||
details = quote(pos) + [f'"{s["name"]}" fails in the service-call run: {s.get("error") or "a check fails"}.' for s in failed]
|
||||
return check('M6', not failed, 'A claim that the workflow was checked agrees with the service-call runs', details)
|
||||
|
||||
|
||||
def claims_for(sub, meta, prompt):
|
||||
workflow = json.loads((sub / 'workflow.json').read_text()) if (sub / 'workflow.json').exists() else None
|
||||
wire = json.loads((sub / 'wire.json').read_text()) if (sub / 'wire.json').exists() else None
|
||||
try:
|
||||
message = final_message(sub, meta)
|
||||
except Exception as e:
|
||||
return {'message': None, 'error': f'The final message is not readable: {e}', 'checks': []}
|
||||
data = load_run(sub, meta, {})
|
||||
events = data.get('events') or []
|
||||
checks = [id_check(message, meta, workflow), state_check(message, workflow), validation_check(message, events),
|
||||
test_check(message, events), span_check(message, events, workflow, prompt), works_check(message, wire)]
|
||||
out = {'message': message, 'checks': [c for c in checks if c]}
|
||||
if data.get('loadError'):
|
||||
out['error'] = f'The trace is not readable: {data["loadError"]}'
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
run_dir = Path(sys.argv[1] if len(sys.argv) > 1 else '.')
|
||||
prompt = (run_dir / 'prompt.txt').read_text() if (run_dir / 'prompt.txt').exists() else ''
|
||||
for f in sorted(run_dir.glob('*/run.json')):
|
||||
sub = f.parent
|
||||
if sub.name == 'report':
|
||||
continue
|
||||
result = claims_for(sub, json.loads(f.read_text()), prompt)
|
||||
(sub / 'claims.json').write_text(json.dumps(result, indent=1, ensure_ascii=False))
|
||||
ok = sum(1 for c in result['checks'] if c['ok'])
|
||||
print(f'{sub.name}: claims.json written ({ok}/{len(result["checks"])})')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -1,328 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Run the full comparison in one command, except the judge step.
|
||||
|
||||
compare.py --prompt "..." --expected "..." [--title "..."] [--builders iai,claude,codex]
|
||||
compare.py --prompt-file p.txt --expected-file e.txt [--scenarios-file s.json]
|
||||
compare.py --rerun /tmp/builder-compare/<old-run> same prompt, expected outcome and scenarios, new run
|
||||
|
||||
Steps: check the setup, run the builders at the same time (--sequential: one at a time),
|
||||
fetch Instance AI usage, capture the canvases and their node issues, run each workflow with
|
||||
pinned data (run_scenarios.py), run each workflow in process with recorded API requests
|
||||
(run_wire.mjs), run the eval binary checks (binary_checks.mts), check the final-message claims
|
||||
(claims.py), derive the deterministic checks (derive_checks.mjs), write
|
||||
summary.txt and judge/summary-blind.txt for the judges, and build the report. A local page
|
||||
shows the runs live (see live_server.py).
|
||||
Settings come from .env.ai-surfaces (template: .env.ai-surfaces.example in the repository root).
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import webbrowser
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from live import enabled as live_enabled
|
||||
from live import is_on, recording
|
||||
from n8n_api import setting
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
RUNNERS = {'iai', 'claude', 'codex'}
|
||||
PRINT_LOCK = threading.Lock()
|
||||
|
||||
|
||||
def slug(text):
|
||||
words = re.findall(r'[a-z0-9]+', text.lower())[:5]
|
||||
return '-'.join(words) or 'run'
|
||||
|
||||
|
||||
def step(title):
|
||||
with PRINT_LOCK:
|
||||
print(f'\n=== {title} ===', flush=True)
|
||||
|
||||
|
||||
def run(cmd, log=None, prefix=None):
|
||||
"""Run a script, show its output, and keep a copy in log. Returns the exit code."""
|
||||
with subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, cwd=HERE) as proc:
|
||||
lines = []
|
||||
for line in proc.stdout:
|
||||
shown = line if not prefix or line.startswith(f'[{prefix}]') else f'[{prefix}] {line}'
|
||||
with PRINT_LOCK:
|
||||
print(shown, end='', flush=True)
|
||||
lines.append(line)
|
||||
if log:
|
||||
Path(log).write_text(''.join(lines))
|
||||
return proc.returncode
|
||||
|
||||
|
||||
class Progress:
|
||||
"""Writes status.json, which the live page reads."""
|
||||
|
||||
def __init__(self, run_dir, mode, builders, skipped, screens):
|
||||
self.path, self.lock = run_dir / 'status.json', threading.Lock()
|
||||
self.data = {
|
||||
'mode': mode, 'phase': 'running', 'step': 'Running the builders', 'startedAt': time.time(),
|
||||
'screens': screens,
|
||||
'builders': {**{b: {'status': 'waiting'} for b in builders},
|
||||
**{b: {'status': 'skipped', 'reason': r} for b, r in skipped.items()}},
|
||||
}
|
||||
self.save()
|
||||
|
||||
def save(self):
|
||||
tmp = self.path.with_suffix('.tmp')
|
||||
tmp.write_text(json.dumps(self.data, indent=2))
|
||||
os.replace(tmp, self.path)
|
||||
|
||||
def update(self, **fields):
|
||||
with self.lock:
|
||||
self.data.update(fields)
|
||||
self.save()
|
||||
|
||||
def builder(self, name, **fields):
|
||||
with self.lock:
|
||||
self.data['builders'][name].update(fields)
|
||||
self.save()
|
||||
|
||||
def finish(self):
|
||||
with self.lock:
|
||||
for entry in self.data['builders'].values():
|
||||
if entry['status'] in ('waiting', 'running'):
|
||||
entry['status'] = 'failed'
|
||||
self.data.update(phase='done', step='Done', finishedAt=time.time())
|
||||
self.save()
|
||||
|
||||
|
||||
def start_live_server(run_dir):
|
||||
"""Start live_server.py in the background. Returns its URL, or None."""
|
||||
url_file = run_dir / 'live-url.txt'
|
||||
url_file.unlink(missing_ok=True)
|
||||
log = (run_dir / 'live-server.log').open('w')
|
||||
# A new session, so the page still serves the final report after this script ends.
|
||||
subprocess.Popen([sys.executable, str(HERE / 'live_server.py'), '--run-dir', str(run_dir),
|
||||
'--parent-pid', str(os.getpid())],
|
||||
cwd=HERE, stdin=subprocess.DEVNULL, stdout=log, stderr=log, start_new_session=True)
|
||||
for _ in range(50):
|
||||
if url_file.exists() and url_file.read_text().strip():
|
||||
return url_file.read_text().strip()
|
||||
time.sleep(0.1)
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--prompt')
|
||||
ap.add_argument('--prompt-file')
|
||||
ap.add_argument('--expected')
|
||||
ap.add_argument('--expected-file')
|
||||
ap.add_argument('--title')
|
||||
ap.add_argument('--builders', default=setting('COMPARE_BUILDERS', default='iai,claude,codex'))
|
||||
ap.add_argument('--run-dir')
|
||||
ap.add_argument('--scenarios-file', help='Pinned-data scenarios for run_scenarios.py (see SKILL.md).')
|
||||
ap.add_argument('--rerun', help='Copy prompt.txt, case.json and scenarios.json from an earlier run directory.')
|
||||
ap.add_argument('--no-live', action='store_true', help='Do not start the live page.')
|
||||
ap.add_argument('--sequential', action='store_true', default=not is_on('COMPARE_PARALLEL'),
|
||||
help='Run the builders one at a time. Default: COMPARE_PARALLEL=1 runs them at the same time.')
|
||||
args = ap.parse_args()
|
||||
if args.no_live:
|
||||
os.environ['COMPARE_LIVE'] = '0'
|
||||
|
||||
if args.rerun:
|
||||
old = Path(args.rerun)
|
||||
prompt = (old / 'prompt.txt').read_text().strip()
|
||||
case = json.loads((old / 'case.json').read_text())
|
||||
else:
|
||||
prompt = args.prompt or (Path(args.prompt_file).read_text().strip() if args.prompt_file else None)
|
||||
expected = args.expected or (Path(args.expected_file).read_text().strip() if args.expected_file else None)
|
||||
if not prompt or not expected:
|
||||
sys.exit('Give a prompt and an expected outcome (--prompt/--prompt-file and --expected/--expected-file).')
|
||||
case = {'title': args.title or prompt[:60], 'expected': expected}
|
||||
builders = [b.strip() for b in args.builders.split(',') if b.strip()]
|
||||
unknown = set(builders) - RUNNERS
|
||||
if unknown:
|
||||
sys.exit(f'Unknown builders: {", ".join(sorted(unknown))}. Use iai, claude, codex.')
|
||||
|
||||
root = Path(setting('COMPARE_RUNS_DIR', default='/tmp/builder-compare'))
|
||||
# The child scripts run from HERE, so a relative path must not reach them.
|
||||
run_dir = (Path(args.run_dir) if args.run_dir else root / f'{datetime.now():%Y%m%d-%H%M}-{slug(case["title"])}').resolve()
|
||||
run_dir.mkdir(parents=True, exist_ok=True)
|
||||
(run_dir / 'prompt.txt').write_text(prompt + '\n')
|
||||
(run_dir / 'case.json').write_text(json.dumps(case, indent=2))
|
||||
scenarios = Path(args.scenarios_file) if args.scenarios_file else (Path(args.rerun) / 'scenarios.json' if args.rerun else None)
|
||||
if scenarios and scenarios.exists():
|
||||
json.loads(scenarios.read_text())
|
||||
shutil.copy(scenarios, run_dir / 'scenarios.json')
|
||||
elif args.scenarios_file:
|
||||
sys.exit(f'No scenarios file at {scenarios}.')
|
||||
print(f'run directory: {run_dir}', flush=True)
|
||||
|
||||
step('Check the setup')
|
||||
check = subprocess.run([sys.executable, 'prepare.py'], cwd=HERE, capture_output=True, text=True)
|
||||
print(check.stdout, end='')
|
||||
try:
|
||||
status = json.loads(check.stdout)
|
||||
except json.JSONDecodeError:
|
||||
sys.exit(check.stderr or 'prepare.py failed')
|
||||
if not status.get('login'):
|
||||
sys.exit('Cannot log in to n8n. Is it running? Check N8N_BASE_URL, N8N_EVAL_EMAIL and N8N_EVAL_PASSWORD.')
|
||||
isolation = setting('COMPARE_ISOLATION', default='container')
|
||||
skipped = {}
|
||||
if 'iai' in builders and not status.get('instanceAiEnabled'):
|
||||
skipped['iai'] = 'Instance AI is not enabled on this instance.'
|
||||
for cli in ('claude', 'codex'):
|
||||
if cli in builders and not status.get('mcpTokenWorks'):
|
||||
skipped[cli] = status.get('mcpTokenHelp', 'The MCP token does not work.')
|
||||
elif cli in builders and not status['keys'].get(cli):
|
||||
skipped[cli] = f'No API key. Set {cli.upper()}_API_KEY in .env.ai-surfaces.'
|
||||
if isolation == 'container' and not status.get('docker'):
|
||||
skipped.setdefault('codex', 'Docker is not running. Codex runs only in a container.')
|
||||
if 'claude' in builders and 'claude' not in skipped:
|
||||
print('Docker is not running. Claude falls back to clean-process isolation.', flush=True)
|
||||
isolation = 'clean'
|
||||
if isolation == 'clean':
|
||||
skipped.setdefault('codex', 'Codex runs only in a container. Clean isolation is for Claude only.')
|
||||
if not status.get('claudeOnHost'):
|
||||
skipped.setdefault('claude', 'Clean isolation runs claude on the host, and claude is not on PATH.')
|
||||
skipped = {name: reason for name, reason in skipped.items() if name in builders}
|
||||
for name, reason in skipped.items():
|
||||
print(f'skip {name}: {reason}', flush=True)
|
||||
builders = [b for b in builders if b not in skipped]
|
||||
parallel = not args.sequential and len(builders) > 1
|
||||
|
||||
cli_mode = setting('COMPARE_CLI_MODE', default='interactive')
|
||||
screens = {b: ('shot' if recording() else None) if b == 'iai'
|
||||
else ('cast' if isolation == 'container' and cli_mode == 'interactive' else None) for b in builders}
|
||||
progress = Progress(run_dir, 'parallel' if parallel else 'sequential', builders, skipped, screens)
|
||||
live_url = None
|
||||
if live_enabled() and not args.no_live:
|
||||
live_url = start_live_server(run_dir)
|
||||
if live_url:
|
||||
print(f'live page: {live_url}/report/live', flush=True)
|
||||
webbrowser.open(f'{live_url}/report/live')
|
||||
else:
|
||||
print(f'the live page did not start, see {run_dir / "live-server.log"}', flush=True)
|
||||
|
||||
results = {}
|
||||
try:
|
||||
if parallel and isolation == 'container' and {'claude', 'codex'} & set(builders):
|
||||
# Both CLI runners need the image. Build it once before they start.
|
||||
from run_cli_builder import ensure_image
|
||||
ensure_image(False)
|
||||
|
||||
def run_builder(builder):
|
||||
progress.builder(builder, status='running', startedAt=time.time())
|
||||
started = time.time()
|
||||
if builder == 'iai':
|
||||
cmd = [sys.executable, 'run_instance_ai.py', '--out', str(run_dir / 'iai'),
|
||||
'--prompt-file', str(run_dir / 'prompt.txt')]
|
||||
else:
|
||||
cmd = [sys.executable, 'run_cli_builder.py', '--builder', builder, '--out', str(run_dir / builder),
|
||||
'--prompt-file', str(run_dir / 'prompt.txt'), '--isolation', isolation]
|
||||
code = run(cmd, run_dir / f'{builder}.log', prefix=builder if parallel else None)
|
||||
# run_cli_builder.py exits 0 when the CLI fails. run.json has the exit code of the CLI.
|
||||
meta_file = run_dir / builder / 'run.json'
|
||||
if code == 0 and meta_file.exists():
|
||||
code = json.loads(meta_file.read_text()).get('exitCode') or 0
|
||||
results[builder] = {'exitCode': code, 'seconds': round(time.time() - started)}
|
||||
progress.builder(builder, status='done' if code == 0 else 'failed', finishedAt=time.time(), exitCode=code)
|
||||
|
||||
if parallel:
|
||||
step(f'Run {", ".join(builders)} at the same time')
|
||||
threads = [threading.Thread(target=run_builder, args=(b,)) for b in builders]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
else:
|
||||
for builder in builders:
|
||||
step(f'Run {builder}')
|
||||
run_builder(builder)
|
||||
|
||||
if 'iai' in builders and (run_dir / 'iai' / 'run.json').exists():
|
||||
step('Fetch Instance AI usage from LangSmith')
|
||||
progress.update(step='Fetching Instance AI usage')
|
||||
if status['keys'].get('langsmith'):
|
||||
# LangSmith receives traces with a delay.
|
||||
finished = progress.data['builders']['iai'].get('finishedAt') or time.time()
|
||||
time.sleep(max(0.0, 30 - (time.time() - finished)))
|
||||
run([sys.executable, 'langsmith_usage.py', '--run-dir', str(run_dir / 'iai')])
|
||||
else:
|
||||
print('LANGSMITH_API_KEY is not set. The report has no Instance AI cost or tokens.', flush=True)
|
||||
|
||||
step('Capture the canvases')
|
||||
progress.update(step='Capturing the canvases')
|
||||
node = shutil.which('node')
|
||||
if node:
|
||||
run([node, 'capture_canvas.mjs', str(run_dir)])
|
||||
else:
|
||||
print('node is not on PATH. No canvases.', flush=True)
|
||||
|
||||
if not is_on('COMPARE_SCENARIOS'):
|
||||
print('COMPARE_SCENARIOS=0. The workflows do not run with pinned data.', flush=True)
|
||||
elif not node or not status.get('mcpTokenWorks'):
|
||||
print('Pinned-data runs need node and a working MCP token. Skipped.', flush=True)
|
||||
else:
|
||||
step('Run each workflow with pinned data')
|
||||
progress.update(step='Running the workflows with pinned data')
|
||||
run([sys.executable, 'run_scenarios.py', str(run_dir)], run_dir / 'scenarios.log')
|
||||
|
||||
if node and is_on('COMPARE_WIRE'):
|
||||
step('Run each workflow in process with recorded API requests')
|
||||
progress.update(step='Running the workflows with recorded API requests')
|
||||
run([node, 'run_wire.mjs', str(run_dir)], run_dir / 'wire.log')
|
||||
elif node:
|
||||
print('COMPARE_WIRE=0. No in-process runs with recorded API requests.', flush=True)
|
||||
|
||||
pnpm = shutil.which('pnpm')
|
||||
if pnpm and is_on('COMPARE_BINARY_CHECKS'):
|
||||
step('Run the eval binary checks')
|
||||
progress.update(step='Running the eval binary checks')
|
||||
run([pnpm, '--filter', '@n8n/instance-ai', 'exec', 'tsx', str(HERE / 'binary_checks.mts'), str(run_dir)],
|
||||
run_dir / 'binary-checks.log')
|
||||
|
||||
step('Check the final-message claims')
|
||||
run([sys.executable, 'claims.py', str(run_dir)])
|
||||
|
||||
if node:
|
||||
step('Derive the deterministic checks')
|
||||
progress.update(step='Deriving the deterministic checks')
|
||||
run([node, 'derive_checks.mjs', str(run_dir)])
|
||||
|
||||
step('Summarize the runs for the judges')
|
||||
summary = subprocess.run([sys.executable, 'summarize_runs.py', str(run_dir)], cwd=HERE,
|
||||
capture_output=True, text=True)
|
||||
(run_dir / 'summary.txt').write_text(summary.stdout)
|
||||
run([sys.executable, 'summarize_runs.py', str(run_dir), '--blind'])
|
||||
|
||||
step('Build the report')
|
||||
# The agent does the price and judge steps after this script. The page shows them as queued until then.
|
||||
progress.update(step='Building the report', postRun={'prices': 'queued', 'judge': 'queued'})
|
||||
run([sys.executable, 'build_report.py', str(run_dir)])
|
||||
finally:
|
||||
progress.finish()
|
||||
|
||||
models_file = run_dir / 'models.json'
|
||||
print(json.dumps({
|
||||
'runDir': str(run_dir),
|
||||
'report': str(run_dir / 'report' / 'index.html'),
|
||||
'live': f'{live_url}/report/index.html' if live_url else None,
|
||||
'parallel': parallel,
|
||||
'summary': str(run_dir / 'summary.txt'),
|
||||
'blindSummary': str(run_dir / 'judge' / 'summary-blind.txt'),
|
||||
'models': json.loads(models_file.read_text()) if models_file.exists() else [],
|
||||
'results': results,
|
||||
'skipped': skipped,
|
||||
'next': 'Run build_report.py --step prices=running, add published prices for the CLI models to case.json, '
|
||||
'run build_report.py --step prices=done --step judge=running, give judge/summary-blind.txt to '
|
||||
'3 judges with a fresh context, write judge/verdicts-1.json to verdicts-3.json, run '
|
||||
'merge_verdicts.py, then run build_report.py --step judge=done.',
|
||||
}, indent=2))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -1,743 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
// Deterministic checks for each workflow that a run created. No LLM is involved.
|
||||
//
|
||||
// node derive_checks.mjs <run-dir>
|
||||
//
|
||||
// Reads <builder>/workflow.json, canvas-issues.json and exec.json, and the optional wire.json,
|
||||
// claims.json and binary-checks.json. Writes <builder>/auto-checks.json. A group with gating true
|
||||
// changes the verdict. The other groups show a separate score.
|
||||
// correctness node definitions, canvas issues, graph and expression references (gating)
|
||||
// execution the pinned-data scenarios of run_scenarios.py (gating)
|
||||
// contract service-call requests of run_wire.mjs (gating)
|
||||
// edge, ai edge-case and real-AI scenarios of run_wire.mjs
|
||||
// claims the final message against the trace, from claims.py
|
||||
// standards rules from n8n-io/skills and the workflow-sdk best practices
|
||||
// binary the deterministic eval binary checks, from binary_checks.mts
|
||||
import { existsSync, readdirSync, readFileSync, writeFileSync } from 'node:fs';
|
||||
import { join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
import {
|
||||
LOGIC_TYPES,
|
||||
SINGLE_ITEM_TYPES,
|
||||
TRIGGER_FIXTURES,
|
||||
downstream,
|
||||
graph,
|
||||
isManualTrigger,
|
||||
isTrigger,
|
||||
loadSdk,
|
||||
lookup,
|
||||
reachable,
|
||||
repo,
|
||||
sample,
|
||||
shortType,
|
||||
showPath,
|
||||
strings,
|
||||
typeMatches,
|
||||
workflowReferences,
|
||||
} from './sdk.mjs';
|
||||
|
||||
const VERSION = 1;
|
||||
const SKILLS = 'https://github.com/n8n-io/skills/blob/27b6b9f429ac0be90cae764512dbc579d7107e95/';
|
||||
const GUIDES = 'packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/';
|
||||
const LC = '@n8n/n8n-nodes-langchain.';
|
||||
|
||||
const src = {
|
||||
validator: { label: '@n8n/workflow-sdk validateWorkflow with the node definitions' },
|
||||
canvas: { label: 'Node issues in the n8n editor (canvas status icons)' },
|
||||
graph: { label: 'Workflow connections' },
|
||||
expressions: { label: 'Expression references in node parameters' },
|
||||
execution: { label: 'Real output of the pinned-data runs (exec.json)' },
|
||||
wire: { label: 'In-process run with recorded API fixtures (wire.json)' },
|
||||
claims: { label: 'Final message, trace tool calls and workflow (claims.json)' },
|
||||
errorHandling: { label: 'n8n-error-handling-official', url: `${SKILLS}skills/n8n-error-handling-official/SKILL.md` },
|
||||
expressionsSkill: { label: 'n8n-expressions-official', url: `${SKILLS}skills/n8n-expressions-official/SKILL.md` },
|
||||
structured: {
|
||||
label: 'n8n-ai-nodes-official STRUCTURED_OUTPUT.md',
|
||||
url: `${SKILLS}skills/n8n-ai-nodes-official/references/STRUCTURED_OUTPUT.md`,
|
||||
},
|
||||
memory: { label: 'n8n-ai-nodes-official MEMORY.md', url: `${SKILLS}skills/n8n-ai-nodes-official/references/MEMORY.md` },
|
||||
comms: {
|
||||
label: 'n8n-node-configuration-official COMMS_NODES.md',
|
||||
url: `${SKILLS}skills/n8n-node-configuration-official/references/COMMS_NODES.md`,
|
||||
},
|
||||
triage: { label: 'workflow-sdk best practices: triage', path: `${GUIDES}triage.ts` },
|
||||
chatbot: { label: 'workflow-sdk best practices: chatbot', path: `${GUIDES}chatbot.ts` },
|
||||
multiItem: {
|
||||
label: 'WhatsAppTrigger.node.ts emits one item for each webhook change',
|
||||
path: 'packages/nodes-base/nodes/WhatsApp/WhatsAppTrigger.node.ts',
|
||||
},
|
||||
};
|
||||
|
||||
const check = (id, ok, text, source, details = [], extra = {}) => ({ id, ok, text, source, details, ...extra });
|
||||
const plural = (n, word) => `${n} ${word}${n === 1 ? '' : 's'}`;
|
||||
const readJson = (file) => (existsSync(file) ? JSON.parse(readFileSync(file, 'utf8')) : null);
|
||||
|
||||
// --- Node kinds ----------------------------------------------------------------
|
||||
|
||||
const isAiRoot = (g, node) => node.type.startsWith(LC) && !g.subnodes.has(node.name) && g.aiInputs(node.name, 'ai_languageModel').length > 0;
|
||||
const isBranch = (node) => typeMatches(node.type, 'if') || typeMatches(node.type, 'switch');
|
||||
/** Nodes that call a service: not logic, not a trigger, not a sub-node. AI root nodes count. */
|
||||
const isNetwork = (g, node) =>
|
||||
!g.subnodes.has(node.name) && !isTrigger(node.type) && !LOGIC_TYPES.has(node.type) && !typeMatches(node.type, 'stickyNote');
|
||||
|
||||
// Triggers that start one execution with one item. Other triggers can batch several events.
|
||||
const SINGLE_ITEM_TRIGGERS = ['manualTrigger', 'scheduleTrigger', 'cron', 'interval', 'webhook', 'chatTrigger', 'formTrigger', 'telegramTrigger', 'slackTrigger'];
|
||||
const MESSAGING_TRIGGERS = ['whatsAppTrigger', 'telegramTrigger', 'slackTrigger', 'chatTrigger', 'discordTrigger', 'microsoftTeamsTrigger', 'gmailTrigger'];
|
||||
const ITEM_SOURCES = ['splitOut', 'code', 'itemLists'];
|
||||
|
||||
/** Names of the nodes that can output more than one item in one execution. */
|
||||
function multiItemNodes(g) {
|
||||
const multi = new Set();
|
||||
const order = [...reachable(g)];
|
||||
let grew = true;
|
||||
while (grew) {
|
||||
grew = false;
|
||||
for (const name of order) {
|
||||
if (multi.has(name)) continue;
|
||||
const node = g.byName.get(name);
|
||||
if (!node || g.subnodes.has(name) || SINGLE_ITEM_TYPES.has(node.type) || node.executeOnce) continue;
|
||||
const fromTrigger = isTrigger(node.type) && !SINGLE_ITEM_TRIGGERS.some((t) => typeMatches(node.type, t));
|
||||
const source = ITEM_SOURCES.some((t) => typeMatches(node.type, t)) || (isNetwork(g, node) && !isAiRoot(g, node) && /^(read|getAll|search|list|getMany)$/.test(node.parameters?.operation ?? ''));
|
||||
if (fromTrigger || source || g.mainParents(name).some((p) => multi.has(p))) {
|
||||
multi.add(name);
|
||||
grew = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
return multi;
|
||||
}
|
||||
|
||||
/** Names of the nodes upstream of name. A sub-node uses the nodes upstream of the root that it serves. */
|
||||
function upstream(g, name) {
|
||||
const roots = new Set([name]);
|
||||
for (let grew = true; grew; ) {
|
||||
grew = false;
|
||||
for (const e of g.edges) {
|
||||
if (e.type.startsWith('ai_') && roots.has(e.from) && !roots.has(e.to)) {
|
||||
roots.add(e.to);
|
||||
grew = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
const seen = new Set();
|
||||
const queue = [...roots];
|
||||
while (queue.length) {
|
||||
for (const parent of g.mainParents(queue.shift())) {
|
||||
if (!seen.has(parent)) {
|
||||
seen.add(parent);
|
||||
queue.push(parent);
|
||||
}
|
||||
}
|
||||
}
|
||||
return seen;
|
||||
}
|
||||
|
||||
// --- Correctness ---------------------------------------------------------------
|
||||
|
||||
function checkDefinitions(workflow) {
|
||||
const loaded = loadSdk();
|
||||
if (!loaded || loaded.missingDirs.length) {
|
||||
const missing = loaded ? loaded.missingDirs.map((d) => d.replace(repo + '/', '')) : ['packages/@n8n/workflow-sdk/dist'];
|
||||
return check('C1', null, 'Node parameters match the node definitions', src.validator, [
|
||||
`Not checked. Build these first: ${missing.join(', ')}.`,
|
||||
]);
|
||||
}
|
||||
const result = loaded.sdk.validateWorkflow(workflow, { allowDisconnectedNodes: true, allowNoTrigger: true });
|
||||
const problems = [...result.errors, ...result.warnings.filter((w) => w.severity !== 'informational')];
|
||||
const details = problems.map((p) =>
|
||||
p.nodeName && !p.message.includes(`"${p.nodeName}"`) ? `${p.nodeName}: ${p.message}` : p.message,
|
||||
);
|
||||
const info = result.warnings.filter((w) => w.severity === 'informational').map((w) => `Info: ${w.message}`);
|
||||
return check(
|
||||
'C1',
|
||||
problems.length === 0,
|
||||
problems.length ? `${plural(problems.length, 'node parameter problem')} against the node definitions` : 'Node parameters match the node definitions',
|
||||
src.validator,
|
||||
details.concat(info),
|
||||
);
|
||||
}
|
||||
|
||||
const isEmpty = (value) =>
|
||||
value === undefined || value === null || value === '' || (typeof value === 'object' && value.__rl && (value.value === '' || value.value == null));
|
||||
|
||||
/** Sort the editor issues into setup work for the user and real problems. */
|
||||
function classifyCanvas(workflow, canvas) {
|
||||
const byName = new Map((workflow.nodes ?? []).map((n) => [n.name, n]));
|
||||
const setup = [];
|
||||
const problems = [];
|
||||
for (const node of canvas.nodes) {
|
||||
if (node.marker === 'node-not-installed') problems.push(`${node.name}: the node type is not installed`);
|
||||
if (node.marker === 'node-restricted') problems.push(`${node.name}: the node type is restricted`);
|
||||
for (const [kind, issue] of Object.entries(node.issues ?? {})) {
|
||||
if (kind === 'credentials') {
|
||||
for (const type of Object.keys(issue)) setup.push(`${node.name}: connect ${type} credentials`);
|
||||
} else if (kind === 'parameters') {
|
||||
for (const [param, messages] of Object.entries(issue)) {
|
||||
const value = byName.get(node.name)?.parameters?.[param];
|
||||
for (const message of messages) {
|
||||
if (/is required/i.test(message) && isEmpty(value)) setup.push(`${node.name}: ${message.replace(/\.$/, '')} (left for the user)`);
|
||||
else problems.push(`${node.name}: ${message}`);
|
||||
}
|
||||
}
|
||||
} else if (issue === true || (issue && typeof issue === 'object' && Object.keys(issue).length)) {
|
||||
const text = typeof issue === 'object' ? Object.values(issue).flat().join(' ') : '';
|
||||
problems.push(`${node.name}: ${kind === 'typeUnknown' ? 'n8n does not know this node type' : `${kind} issue`}${text ? ` (${text})` : ''}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
return { setup, problems };
|
||||
}
|
||||
|
||||
function checkCanvas(workflow, canvas) {
|
||||
if (!canvas) return [check('C2', null, 'The n8n editor shows no node errors', src.canvas, ['Not checked. Run capture_canvas.mjs first.'])];
|
||||
if (!canvas.store && canvas.nodes.some((n) => n.marker === 'node-issues')) {
|
||||
return [check('C2', null, 'The n8n editor shows no node errors', src.canvas, [
|
||||
'The editor store was not readable, so the node issues are not known. Icons: ' +
|
||||
canvas.nodes.map((n) => `${n.name} (${n.marker})`).join(', '),
|
||||
])];
|
||||
}
|
||||
const { setup, problems } = classifyCanvas(workflow, canvas);
|
||||
return [
|
||||
check('C2', problems.length === 0, problems.length ? `The n8n editor shows ${plural(problems.length, 'node error')}` : 'The n8n editor shows no node errors', src.canvas, problems),
|
||||
check('C2s', true, setup.length ? `Setup left for the user: ${plural(setup.length, 'item')}` : 'No setup left for the user', src.canvas, setup, { level: 'info' }),
|
||||
];
|
||||
}
|
||||
|
||||
function checkReachable(g) {
|
||||
if (g.triggers.length === 0) return check('C3', false, 'The workflow has a trigger', src.graph, ['No trigger node.']);
|
||||
const seen = reachable(g);
|
||||
const orphans = g.nodes.filter((n) => !seen.has(n.name)).map((n) => n.name);
|
||||
return check(
|
||||
'C3',
|
||||
orphans.length === 0,
|
||||
orphans.length ? `${plural(orphans.length, 'node')} cannot be reached from a trigger` : 'Every node can be reached from a trigger',
|
||||
src.graph,
|
||||
orphans.map((n) => `${n} is not connected to the flow`),
|
||||
);
|
||||
}
|
||||
|
||||
function checkNamedRefs(workflow, g, refs) {
|
||||
const allNames = new Set((workflow.nodes ?? []).map((n) => n.name));
|
||||
const problems = [];
|
||||
for (const { node, param, ref } of refs) {
|
||||
if (ref.kind !== 'named') continue;
|
||||
if (!allNames.has(ref.node)) problems.push(`${node} › ${param}: no node is called "${ref.node}"`);
|
||||
else if (!g.byName.has(ref.node)) problems.push(`${node} › ${param}: "${ref.node}" is disabled`);
|
||||
else if (!/isExecuted/.test(ref.raw) && !upstream(g, node).has(ref.node))
|
||||
problems.push(`${node} › ${param}: "${ref.node}" does not run before this node, so the reference has no data`);
|
||||
}
|
||||
const named = refs.filter((r) => r.ref.kind === 'named').length;
|
||||
return check(
|
||||
'C4',
|
||||
problems.length === 0,
|
||||
problems.length ? `${plural(problems.length, 'node reference')} cannot resolve` : `All ${plural(named, 'node reference')} name a node that runs before them`,
|
||||
src.expressions,
|
||||
[...new Set(problems)],
|
||||
);
|
||||
}
|
||||
|
||||
const DEFINITIVE = new Set(['executed', 'fixture', 'case', 'output-parser']);
|
||||
const INDICATIVE = new Set(['node-schema', 'prompt-example', 'ai-root']);
|
||||
// Outputs with the real number of array items. The other provenances are schema samples.
|
||||
const REAL_OUTPUT = new Set(['executed', 'fixture', 'case']);
|
||||
|
||||
/** The output items of a node across all scenarios, with their provenance. */
|
||||
function outputsOf(name, g, exec) {
|
||||
const items = [];
|
||||
for (const scenario of exec?.scenarios ?? []) {
|
||||
const out = scenario.nodes?.[name];
|
||||
if (out?.status === 'success' && Array.isArray(out.items)) items.push(...out.items.filter((i) => i && typeof i === 'object'));
|
||||
}
|
||||
const node = g.byName.get(name);
|
||||
let provenance = exec?.executed?.includes(name) ? 'executed' : exec?.pinned?.[name];
|
||||
if (items.length && provenance) return { items, provenance };
|
||||
if (TRIGGER_FIXTURES[node.type]) return { items: TRIGGER_FIXTURES[node.type], provenance: 'fixture' };
|
||||
const schema = loadSdk()?.sdk.discoverOutputSchemaForNode(node.type, node.typeVersion, node.parameters ?? {});
|
||||
if (schema) return { items: [sample(schema, 'item')], provenance: 'node-schema' };
|
||||
return { items: [], provenance: 'unknown' };
|
||||
}
|
||||
|
||||
function checkFieldPaths(g, refs, exec) {
|
||||
const missing = [];
|
||||
const unconfirmed = [];
|
||||
let checked = 0;
|
||||
let unknown = 0;
|
||||
const seen = new Set();
|
||||
for (const { node, param, ref, code } of refs) {
|
||||
if (code || !ref.path?.length) continue;
|
||||
if (ref.kind !== 'named' && g.subnodes.has(node)) continue;
|
||||
const sources = ref.kind === 'named' ? [ref.node] : g.mainParents(node);
|
||||
if (!sources.length || sources.some((s) => !g.byName.has(s))) continue;
|
||||
const key = `${node}|${param}|${ref.raw}`;
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
let found = false;
|
||||
let anyUnknown = false;
|
||||
const misses = [];
|
||||
for (const source of sources) {
|
||||
const { items, provenance } = outputsOf(source, g, exec);
|
||||
if (!items.length || provenance === 'unknown' || provenance === 'empty' || provenance === 'placeholder') {
|
||||
anyUnknown = true;
|
||||
continue;
|
||||
}
|
||||
const results = items.map((item) => lookup(item, ref.path, { sample: !REAL_OUTPUT.has(provenance) }));
|
||||
if (results.includes('found')) found = true;
|
||||
else if (results.includes('unknown')) anyUnknown = true;
|
||||
else misses.push({ source, provenance });
|
||||
}
|
||||
if (found) checked++;
|
||||
else if (anyUnknown || !misses.length) unknown++;
|
||||
else {
|
||||
checked++;
|
||||
const line = `${node} › ${param}: ${ref.raw.trim()} reads "${showPath(ref.path)}", which is not in the output of ${misses.map((m) => `${m.source} (${m.provenance})`).join(' or ')}`;
|
||||
if (misses.some((m) => DEFINITIVE.has(m.provenance))) missing.push(line);
|
||||
else if (misses.some((m) => INDICATIVE.has(m.provenance))) unconfirmed.push(`Unconfirmed: ${line}`);
|
||||
}
|
||||
}
|
||||
const note = exec?.scenarios?.length ? [] : ['No pinned-data run, so node outputs come from fixtures and node schemas only.'];
|
||||
return check(
|
||||
'C5',
|
||||
missing.length === 0,
|
||||
missing.length
|
||||
? `${plural(missing.length, 'expression')} read a field that the source node does not output`
|
||||
: `${plural(checked, 'expression field')} found in the source node output${unknown ? `, ${unknown} not checkable` : ''}`,
|
||||
src.execution,
|
||||
[...missing, ...unconfirmed, ...note],
|
||||
);
|
||||
}
|
||||
|
||||
const SECRET_PATTERNS = [
|
||||
/sk-(?:proj-|ant-)?[A-Za-z0-9_-]{20,}/,
|
||||
/xox[abpr]-[A-Za-z0-9-]{10,}/,
|
||||
/AIza[0-9A-Za-z_-]{35}/,
|
||||
/gh[pousr]_[A-Za-z0-9]{36}/,
|
||||
/Bearer\s+(?!\{\{)[A-Za-z0-9._~+/-]{24,}/,
|
||||
/-----BEGIN [A-Z ]*PRIVATE KEY-----/,
|
||||
];
|
||||
|
||||
function checkSecrets(g) {
|
||||
const hits = [];
|
||||
for (const node of g.nodes) {
|
||||
for (const { path, value } of strings(node.parameters ?? {})) {
|
||||
if (SECRET_PATTERNS.some((re) => re.test(value))) hits.push(`${node.name} › ${path}: a value looks like a key or token`);
|
||||
}
|
||||
}
|
||||
return check('C6', hits.length === 0, hits.length ? 'Secrets are written into node parameters' : 'No secrets in node parameters', src.expressions, hits);
|
||||
}
|
||||
|
||||
// --- Execution -----------------------------------------------------------------
|
||||
|
||||
function executionChecks(exec) {
|
||||
if (!exec) return [check('E0', null, 'The workflow runs with pinned data', src.execution, ['Not run. Run run_scenarios.py first.'])];
|
||||
if (exec.skipped) return [check('E0', null, 'The workflow runs with pinned data', src.execution, [exec.skipped])];
|
||||
const pinned = Object.entries(exec.pinned ?? {}).map(([n, p]) => `${n} (${p})`);
|
||||
const intro = [`Pinned: ${pinned.join(', ') || 'nothing'}.`, `Ran for real: ${(exec.executed ?? []).join(', ') || 'nothing'}.`];
|
||||
const out = [];
|
||||
exec.scenarios.forEach((s, i) => {
|
||||
const errors = Object.entries(s.nodes ?? {}).filter(([, n]) => n.error).map(([name, n]) => `${name}: ${n.error}`);
|
||||
const ran = Object.keys(s.nodes ?? {});
|
||||
const ok = s.status === 'success' && errors.length === 0;
|
||||
out.push(check(`E${i + 1}`, ok, `"${s.name}" runs without an error`, src.execution, [
|
||||
...(i === 0 ? intro : []),
|
||||
`Status: ${s.status}${s.error ? ` (${s.error})` : ''}.`,
|
||||
...errors,
|
||||
`Nodes that ran: ${ran.join(', ')}.`,
|
||||
], { executionId: s.executionId }));
|
||||
(s.checks ?? []).forEach((c, j) => {
|
||||
const verb = c.kind === 'reach' ? 'reaches' : 'does not reach';
|
||||
out.push(check(`E${i + 1}.${j + 1}`, c.ok, `"${s.name}" ${verb} a ${c.type} node`, src.execution, []));
|
||||
});
|
||||
});
|
||||
return out;
|
||||
}
|
||||
|
||||
// --- In-process runs, eval checks and claims -------------------------------------
|
||||
|
||||
const WIRE_KINDS = {
|
||||
contract: { id: 'W', title: 'Service calls', gating: true, note: 'The workflow runs in process. Service nodes run for real against recorded API fixtures, so the check sees each HTTP request that a node sends. The AI node is pinned. A failure here changes the verdict.' },
|
||||
edge: { id: 'X', title: 'Edge cases', gating: false, note: 'Inputs that the prompt does not describe but the trigger can send: several messages in one call, status updates, empty data. This score does not change the verdict.' },
|
||||
ai: { id: 'A', title: 'Real AI runs', gating: false, note: 'The AI node calls the real model on labeled questions. Service nodes use the recorded API fixtures. Model output can change between runs, so this score does not change the verdict.' },
|
||||
};
|
||||
|
||||
function wireCheckText(name, c) {
|
||||
if (c.kind === 'reach') return `"${name}" reaches a ${c.type} node`;
|
||||
if (c.kind === 'skip') return `"${name}" does not reach a ${c.type} node`;
|
||||
if (c.kind === 'noWrites') return `"${name}" sends no write request`;
|
||||
if (c.kind === 'eachItem') return `"${name}" sends a request for each of the ${c.ids.length} items`;
|
||||
const what = c.contains?.length ? ` that carries ${c.contains.map((m) => m.split(' or ').map((x) => (x.startsWith('@') ? `\`${x}\`` : x)).join(' or ')).join(', ')}` : '';
|
||||
const count = c.count === undefined ? 'a request' : plural(c.count, 'request');
|
||||
return `"${name}" sends ${count} from a ${c.type} node${what}`;
|
||||
}
|
||||
|
||||
function requestLines(s) {
|
||||
return (s.requests ?? []).map((r) => `Request: ${r.node} › ${r.method} ${r.url}${r.answeredBy && r.answeredBy !== 'fixture' ? ` (${r.answeredBy})` : ''}`);
|
||||
}
|
||||
|
||||
function wireGroups(wire) {
|
||||
if (!wire) return [];
|
||||
const groups = [];
|
||||
for (const [kind, meta] of Object.entries(WIRE_KINDS)) {
|
||||
const scenarios = (wire.scenarios ?? []).filter((s) => s.kind === kind);
|
||||
if (!scenarios.length && !(kind === 'contract' && wire.skipped)) continue;
|
||||
const checks = [];
|
||||
if (wire.skipped) checks.push(check(`${meta.id}0`, null, 'The workflow runs in process', src.wire, [wire.skipped]));
|
||||
scenarios.forEach((s, i) => {
|
||||
const id = `${meta.id}${i + 1}`;
|
||||
if (s.status === 'skipped') {
|
||||
checks.push(check(id, null, `"${s.name}" runs`, src.wire, [s.error]));
|
||||
return;
|
||||
}
|
||||
const errors = Object.entries(s.nodes ?? {}).filter(([, n]) => n.error).map(([name, n]) => `${name}: ${n.error}`);
|
||||
const ai = Object.entries(s.aiOutputs ?? {}).map(([node, items]) => `AI output of ${node}: ${JSON.stringify(items?.[0]?.output ?? items?.[0] ?? null)}`);
|
||||
const ran = Object.entries(s.nodes ?? {}).map(([name, n]) => `${name} (${plural((n.outputCounts ?? []).reduce((a, b) => a + b, 0), 'item')})`);
|
||||
checks.push(check(id, s.status === 'success' && !errors.length, `"${s.name}" runs without an error`, src.wire, [
|
||||
...(s.why ? [s.why[0].toUpperCase() + s.why.slice(1)] : []),
|
||||
`Status: ${s.status}${s.error ? ` (${s.error})` : ''}.`,
|
||||
...errors,
|
||||
`Nodes that ran: ${ran.join(', ') || 'none'}.`,
|
||||
...ai,
|
||||
...requestLines(s),
|
||||
...(s.blocked?.length ? [`Blocked network calls: ${s.blocked.join(', ')}`] : []),
|
||||
]));
|
||||
(s.checks ?? []).forEach((c, j) => checks.push(check(`${id}.${j + 1}`, c.ok, wireCheckText(s.name, c), src.wire, c.details ?? [])));
|
||||
});
|
||||
const scored = checks.filter((c) => c.ok !== null);
|
||||
const filled = kind === 'contract' && wire.filled?.length ? [`Setup values filled for the run: ${wire.filled.join(', ')}.`] : [];
|
||||
groups.push({
|
||||
key: kind === 'contract' ? 'contract' : kind,
|
||||
title: meta.title,
|
||||
gating: meta.gating,
|
||||
note: [meta.note, ...filled].join(' '),
|
||||
...(meta.gating ? {} : { score: { passed: scored.filter((c) => c.ok).length, total: scored.length } }),
|
||||
checks,
|
||||
});
|
||||
}
|
||||
return groups;
|
||||
}
|
||||
|
||||
function binaryGroup(binary) {
|
||||
if (!binary?.outcomes) return null;
|
||||
const measured = binary.outcomes.filter((o) => o.status !== 'n_a');
|
||||
const checks = measured.map((o, i) =>
|
||||
check(`B${i + 1}`, o.status === 'error' ? null : o.status === 'pass', `${o.name}: ${o.description}`, { label: `Instance AI eval binary checks (${o.dimension})`, path: binary.source }, o.comment ? [o.comment] : []),
|
||||
);
|
||||
const scored = checks.filter((c) => c.ok !== null);
|
||||
const na = binary.outcomes.length - measured.length;
|
||||
return {
|
||||
key: 'binary',
|
||||
title: 'Eval binary checks',
|
||||
gating: false,
|
||||
note: `The deterministic binary checks of the Instance AI eval suite.${na ? ` ${na} do not apply to this workflow.` : ''} This score does not change the verdict.`,
|
||||
score: { passed: scored.filter((c) => c.ok).length, total: scored.length },
|
||||
checks,
|
||||
};
|
||||
}
|
||||
|
||||
function claimsGroup(claims) {
|
||||
if (!claims) return null;
|
||||
const checks = claims.checks.map((c) => check(c.id, c.ok, c.text, src.claims, c.details));
|
||||
if (claims.error) checks.unshift(check('M0', null, 'The final message and trace are readable', src.claims, [claims.error]));
|
||||
const scored = checks.filter((c) => c.ok !== null);
|
||||
return {
|
||||
key: 'claims',
|
||||
title: 'Final message claims',
|
||||
gating: false,
|
||||
note: 'What the final message says about the build, compared with the trace tool calls, the workflow and the service-call runs. This score does not change the verdict.',
|
||||
score: { passed: scored.filter((c) => c.ok).length, total: scored.length },
|
||||
checks,
|
||||
};
|
||||
}
|
||||
|
||||
// --- Standards -----------------------------------------------------------------
|
||||
|
||||
function unattended(g) {
|
||||
return g.triggers.some((t) => !isManualTrigger(t.type) && !typeMatches(t.type, 'chatTrigger') && !typeMatches(t.type, 'formTrigger'));
|
||||
}
|
||||
|
||||
function errorWorkflowRule(workflow, g) {
|
||||
if (!unattended(g)) return null;
|
||||
const set = Boolean(workflow.settings?.errorWorkflow);
|
||||
return check('S1', set, 'An unattended workflow has an error workflow', src.errorHandling, set ? [] : ['settings.errorWorkflow is not set, so a failure goes nowhere.']);
|
||||
}
|
||||
|
||||
function errorOutputRule(g) {
|
||||
const problems = [];
|
||||
for (const node of g.nodes) {
|
||||
if (node.onError === 'continueRegularOutput') problems.push(`${node.name}: onError continueRegularOutput passes the error on as normal data`);
|
||||
// The error output comes after the normal outputs, so it is never output 0.
|
||||
if (node.onError === 'continueErrorOutput' && !g.mainChildren(node.name).some((e) => e.output > 0))
|
||||
problems.push(`${node.name}: onError continueErrorOutput has no node on its error output`);
|
||||
}
|
||||
if (!g.nodes.some((n) => n.onError)) return null;
|
||||
return check('S12', problems.length === 0, 'Failed nodes go to a wired error output, not on as data', src.errorHandling, problems);
|
||||
}
|
||||
|
||||
function retryRule(g) {
|
||||
const nodes = g.nodes.filter((n) => isNetwork(g, n));
|
||||
if (!nodes.length) return null;
|
||||
const missing = nodes.filter((n) => !n.retryOnFail).map((n) => `${n.name}: no retryOnFail`);
|
||||
return check('S2', missing.length === 0, `Nodes that call a service retry on failure (${nodes.length - missing.length} of ${nodes.length})`, src.errorHandling, missing);
|
||||
}
|
||||
|
||||
function namedRefRule(refs) {
|
||||
const uses = refs.filter((r) => !r.code && (r.ref.kind === 'json' || r.ref.kind === 'input'));
|
||||
if (!refs.some((r) => !r.code)) return null;
|
||||
const where = [...new Set(uses.map((r) => `${r.node} › ${r.param}`))];
|
||||
return check('S3', uses.length === 0, uses.length ? `${plural(where.length, 'parameter')} read $json instead of a node name` : 'Expressions read data by node name', src.expressionsSkill, where);
|
||||
}
|
||||
|
||||
function parserParams(g, root) {
|
||||
const parser = g.aiInputs(root.name, 'ai_outputParser')[0];
|
||||
return parser ? { parser, p: parser.parameters ?? {} } : null;
|
||||
}
|
||||
|
||||
function structuredRule(g) {
|
||||
const roots = g.nodes.filter((n) => isAiRoot(g, n) && g.mainChildren(n.name).length);
|
||||
if (!roots.length) return null;
|
||||
const problems = [];
|
||||
for (const root of roots) {
|
||||
const found = parserParams(g, root);
|
||||
if (!found) {
|
||||
const declared = strings(root.parameters ?? {}).some(({ value }) => /json/i.test(value));
|
||||
if (declared) problems.push(`${root.name}: the JSON format is only in the prompt, with no structured output parser`);
|
||||
continue;
|
||||
}
|
||||
if ((found.p.schemaType ?? 'fromJson') !== 'manual') problems.push(`${found.parser.name}: schemaType is not 'manual'`);
|
||||
if (!found.p.autoFix) problems.push(`${found.parser.name}: autoFix is off`);
|
||||
}
|
||||
return check('S4', problems.length === 0, 'Structured AI output uses a manual schema with autoFix', src.structured, problems);
|
||||
}
|
||||
|
||||
const ATTRIBUTION_FIELDS = ['appendAttribution', 'includeLinkToWorkflow'];
|
||||
const definitionCache = new Map();
|
||||
|
||||
/** The text of the node definition file for a node, or ''. */
|
||||
function definitionText(node) {
|
||||
const [pkg, short] = node.type.startsWith(LC) ? ['@n8n/nodes-langchain', node.type.slice(LC.length)] : ['nodes-base', shortType(node.type)];
|
||||
const pkgDir = node.type.startsWith(LC) ? '@n8n/n8n-nodes-langchain' : 'n8n-nodes-base';
|
||||
const dir = join(repo, 'packages', pkg, 'dist/node-definitions/nodes', pkgDir, short, `v${String(node.typeVersion ?? 1).replace('.', '')}`);
|
||||
const snake = (s) => String(s ?? '').replace(/([a-z0-9])([A-Z])/g, '$1_$2').toLowerCase();
|
||||
const p = node.parameters ?? {};
|
||||
const files = [join(dir, `resource_${snake(p.resource)}`, `operation_${snake(p.operation)}.ts`), `${dir}.ts`];
|
||||
if (!p.resource && p.operation && existsSync(dir)) {
|
||||
for (const entry of readdirSync(dir)) if (entry.startsWith('resource_')) files.push(join(dir, entry, `operation_${snake(p.operation)}.ts`));
|
||||
}
|
||||
const key = files.join('|');
|
||||
if (!definitionCache.has(key)) definitionCache.set(key, files.filter(existsSync).map((f) => readFileSync(f, 'utf8')).join('\n'));
|
||||
return definitionCache.get(key);
|
||||
}
|
||||
|
||||
function findKey(value, key) {
|
||||
if (!value || typeof value !== 'object') return undefined;
|
||||
if (key in value) return value[key];
|
||||
for (const v of Object.values(value)) {
|
||||
const hit = findKey(v, key);
|
||||
if (hit !== undefined) return hit;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function attributionRule(g) {
|
||||
const nodes = [];
|
||||
for (const node of g.nodes) {
|
||||
const text = definitionText(node);
|
||||
const field = ATTRIBUTION_FIELDS.find((f) => text.includes(`${f}?:`));
|
||||
if (field) nodes.push({ node, field });
|
||||
}
|
||||
if (!nodes.length) return null;
|
||||
const on = nodes.filter(({ node, field }) => findKey(node.parameters, field) !== false).map(({ node, field }) => `${node.name}: ${field} is not false (the default adds an n8n attribution)`);
|
||||
return check('S5', on.length === 0, 'Messages do not carry the n8n attribution', src.comms, on);
|
||||
}
|
||||
|
||||
function multiItemRule(g, refs) {
|
||||
const multi = multiItemNodes(g);
|
||||
if (!multi.size) return null;
|
||||
const hits = refs.filter((r) => r.ref.kind === 'named' && ['first', 'last', 'all-index'].includes(r.ref.accessor) && multi.has(r.ref.node));
|
||||
const where = [...new Set(hits.map((r) => `${r.node} › ${r.param}: ${r.ref.raw.trim()}`))];
|
||||
const triggers = g.triggers.filter((t) => multi.has(t.name));
|
||||
const source = triggers.some((t) => typeMatches(t.type, 'whatsAppTrigger')) ? { ...src.multiItem, also: src.expressionsSkill } : src.expressionsSkill;
|
||||
return check(
|
||||
'S6',
|
||||
where.length === 0,
|
||||
where.length ? `${plural(where.length, 'reference')} read only the first item of a node that can output several` : 'References keep each item paired with its own data',
|
||||
source,
|
||||
where.length ? [...where, `Nodes that can output several items: ${[...multi].join(', ')}.`] : [],
|
||||
);
|
||||
}
|
||||
|
||||
/** AI root nodes whose output decides a branch, with the IF or Switch nodes that they feed. */
|
||||
function decidingRoots(g) {
|
||||
return g.nodes
|
||||
.filter((n) => isAiRoot(g, n))
|
||||
.map((root) => ({ root, branches: [...downstream(g, root.name)].map((n) => g.byName.get(n)).filter((n) => n && isBranch(n)) }))
|
||||
.filter((r) => r.branches.length);
|
||||
}
|
||||
|
||||
function temperatureRule(g) {
|
||||
const roots = decidingRoots(g);
|
||||
if (!roots.length) return null;
|
||||
const problems = [];
|
||||
for (const { root } of roots) {
|
||||
for (const model of g.aiInputs(root.name, 'ai_languageModel')) {
|
||||
const t = findKey(model.parameters?.options, 'temperature') ?? findKey(model.parameters, 'temperature');
|
||||
if (t === undefined) problems.push(`${model.name}: temperature is not set, so the model default is used`);
|
||||
else if (typeof t === 'number' && t > 0.2) problems.push(`${model.name}: temperature is ${t}`);
|
||||
}
|
||||
}
|
||||
return check('S7', problems.length === 0, 'A model that decides a branch has a temperature of 0.2 or lower', src.triage, problems);
|
||||
}
|
||||
|
||||
/** Field names of a JSON Schema, or of a JSON example when the value is not a schema. */
|
||||
function schemaKeys(value, out = new Set()) {
|
||||
if (!value || typeof value !== 'object') return out;
|
||||
if (Array.isArray(value)) {
|
||||
value.forEach((v) => schemaKeys(v, out));
|
||||
return out;
|
||||
}
|
||||
const isSchema = value.properties || value.items || typeof value.type === 'string';
|
||||
if (isSchema) {
|
||||
for (const [k, v] of Object.entries(value.properties ?? {})) {
|
||||
out.add(k);
|
||||
schemaKeys(v, out);
|
||||
}
|
||||
if (value.items) schemaKeys(value.items, out);
|
||||
return out;
|
||||
}
|
||||
for (const [k, v] of Object.entries(value)) {
|
||||
out.add(k);
|
||||
schemaKeys(v, out);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function reasoningRule(g) {
|
||||
const roots = decidingRoots(g);
|
||||
if (!roots.length) return null;
|
||||
const problems = [];
|
||||
let judged = 0;
|
||||
for (const { root } of roots) {
|
||||
const found = parserParams(g, root);
|
||||
let text = '';
|
||||
if (found) text = String(found.p.inputSchema ?? found.p.jsonSchemaExample ?? found.p.jsonSchema ?? '');
|
||||
else text = strings(root.parameters ?? {}).map((s) => s.value).join('\n');
|
||||
if (!text.includes('{')) continue;
|
||||
judged++;
|
||||
let keys;
|
||||
try {
|
||||
keys = [...schemaKeys(JSON.parse(text))];
|
||||
} catch {
|
||||
keys = [...text.matchAll(/"([A-Za-z_]\w*)"\s*:/g)].map((m) => m[1]);
|
||||
}
|
||||
if (!keys.some((k) => /reason|rationale|explanation|why/i.test(k))) problems.push(`${root.name}: the output has no reasoning field (fields: ${keys.slice(0, 8).join(', ')})`);
|
||||
}
|
||||
if (!judged) return null;
|
||||
return check('S8', problems.length === 0, 'AI output that decides a branch includes a reasoning field', src.triage, problems);
|
||||
}
|
||||
|
||||
function fallbackRule(g) {
|
||||
const roots = decidingRoots(g);
|
||||
if (!roots.length) return null;
|
||||
const problems = [];
|
||||
const branches = new Map(roots.flatMap((r) => r.branches.map((b) => [b.name, b])));
|
||||
for (const node of branches.values()) {
|
||||
const used = new Set(g.mainChildren(node.name).map((e) => e.output));
|
||||
if (typeMatches(node.type, 'if')) {
|
||||
for (const [i, label] of [[0, 'true'], [1, 'false']]) if (!used.has(i)) problems.push(`${node.name}: the ${label} output goes nowhere`);
|
||||
} else {
|
||||
const rules = node.parameters?.rules?.values ?? node.parameters?.rules?.rules ?? [];
|
||||
const fallback = node.parameters?.options?.fallbackOutput;
|
||||
if (fallback !== 'extra') problems.push(`${node.name}: options.fallbackOutput is not 'extra', so unmatched items are dropped`);
|
||||
else if (!used.has(rules.length)) problems.push(`${node.name}: the fallback output goes nowhere`);
|
||||
rules.forEach((_, i) => !used.has(i) && problems.push(`${node.name}: output ${i} goes nowhere`));
|
||||
}
|
||||
}
|
||||
return check('S9', problems.length === 0, 'Every branch after an AI decision, including the fallback, goes somewhere', src.triage, problems);
|
||||
}
|
||||
|
||||
function memoryRule(g) {
|
||||
const chat = g.triggers.some((t) => MESSAGING_TRIGGERS.some((m) => typeMatches(t.type, m)));
|
||||
const agents = g.nodes.filter((n) => typeMatches(n.type, `${LC}agent`) && !g.subnodes.has(n.name));
|
||||
if (!chat || !agents.length) return null;
|
||||
const missing = agents.filter((a) => !g.aiInputs(a.name, 'ai_memory').length).map((a) => `${a.name}: no memory sub-node`);
|
||||
return check('S10', missing.length === 0, 'A chatbot agent has memory', src.chatbot, missing);
|
||||
}
|
||||
|
||||
function sessionKeyRule(g) {
|
||||
const memories = g.nodes.filter((n) => g.subnodes.has(n.name) && /memory/i.test(shortType(n.type)) && n.parameters?.sessionIdType === 'customKey');
|
||||
if (!memories.length) return null;
|
||||
const problems = [];
|
||||
for (const m of memories) {
|
||||
const key = String(m.parameters?.sessionKey ?? '');
|
||||
if (!key.startsWith('=')) problems.push(`${m.name}: the session key is a fixed value, so every conversation shares one memory`);
|
||||
else if (/\$json\b/.test(key)) problems.push(`${m.name}: the session key reads $json, which a memory sub-node does not have`);
|
||||
}
|
||||
return check('S11', problems.length === 0, 'The memory session key names a node and identifies the conversation', { ...src.chatbot, also: src.memory }, problems);
|
||||
}
|
||||
|
||||
// --- Main ----------------------------------------------------------------------
|
||||
|
||||
export function deriveChecks(workflow, canvas, exec, { wire, binary, claims } = {}) {
|
||||
const g = graph(workflow);
|
||||
const refs = workflowReferences(g);
|
||||
const correctness = [
|
||||
checkDefinitions(workflow),
|
||||
...checkCanvas(workflow, canvas),
|
||||
checkReachable(g),
|
||||
checkNamedRefs(workflow, g, refs),
|
||||
checkFieldPaths(g, refs, exec),
|
||||
checkSecrets(g),
|
||||
];
|
||||
const standards = [
|
||||
errorWorkflowRule(workflow, g),
|
||||
errorOutputRule(g),
|
||||
retryRule(g),
|
||||
namedRefRule(refs),
|
||||
structuredRule(g),
|
||||
attributionRule(g),
|
||||
multiItemRule(g, refs),
|
||||
temperatureRule(g),
|
||||
reasoningRule(g),
|
||||
fallbackRule(g),
|
||||
memoryRule(g),
|
||||
sessionKeyRule(g),
|
||||
].filter(Boolean);
|
||||
const scored = standards.filter((c) => c.ok !== null);
|
||||
return {
|
||||
version: VERSION,
|
||||
groups: [
|
||||
{ key: 'correctness', title: 'Correctness', gating: true, note: 'From the node definitions, the n8n editor and the workflow graph. A failure here changes the verdict.', checks: correctness },
|
||||
{ key: 'execution', title: 'Execution with pinned data', gating: true, note: 'Triggers, service nodes and AI nodes get pinned data. Logic nodes run for real. A failure here changes the verdict.', checks: executionChecks(exec) },
|
||||
...wireGroups(wire),
|
||||
claimsGroup(claims),
|
||||
{
|
||||
key: 'standards',
|
||||
title: 'n8n standards',
|
||||
gating: false,
|
||||
note: 'Rules from n8n-io/skills and the workflow-sdk best practices. The builders did not get these rules. This score does not change the verdict.',
|
||||
score: { passed: scored.filter((c) => c.ok).length, total: scored.length },
|
||||
checks: standards,
|
||||
},
|
||||
binaryGroup(binary),
|
||||
].filter(Boolean),
|
||||
};
|
||||
}
|
||||
|
||||
if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
const runDir = resolve(process.argv[2] ?? '.');
|
||||
for (const entry of readdirSync(runDir, { withFileTypes: true })) {
|
||||
const dir = join(runDir, entry.name);
|
||||
if (!entry.isDirectory() || !existsSync(join(dir, 'workflow.json'))) continue;
|
||||
const result = deriveChecks(readJson(join(dir, 'workflow.json')), readJson(join(dir, 'canvas-issues.json')), readJson(join(dir, 'exec.json')), {
|
||||
wire: readJson(join(dir, 'wire.json')),
|
||||
binary: readJson(join(dir, 'binary-checks.json')),
|
||||
claims: readJson(join(dir, 'claims.json')),
|
||||
});
|
||||
writeFileSync(join(dir, 'auto-checks.json'), JSON.stringify(result, null, 1));
|
||||
const line = result.groups.map((group) => {
|
||||
const judged = group.checks.filter((c) => c.ok !== null && c.level !== 'info');
|
||||
return `${group.key} ${judged.filter((c) => c.ok).length}/${judged.length}`;
|
||||
});
|
||||
console.log(`${entry.name}: auto-checks.json written (${line.join(', ')})`);
|
||||
}
|
||||
}
|
||||
|
|
@ -1,160 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Fetch per-call token usage and cost of an Instance AI run from LangSmith.
|
||||
|
||||
n8n does not store token usage. LangSmith has it when the instance runs with
|
||||
LANGSMITH_TRACING=true. The script reads LANGSMITH_API_KEY, LANGSMITH_ENDPOINT and
|
||||
LANGSMITH_PROJECT from the config (see n8n_api.config).
|
||||
|
||||
langsmith_usage.py --run-dir <run-dir>/iai
|
||||
|
||||
It takes all LLM runs in the project between startedAt and finishedAt from run.json.
|
||||
Do not start other Instance AI work on the same instance during the run, or use --by-trace.
|
||||
|
||||
langsmith_usage.py --run-dir <dir> --by-trace
|
||||
|
||||
--by-trace keeps only the calls of the traces in run.json "langsmithRunIds" (the root run id of
|
||||
each run-start event is also its LangSmith trace id). Use it when other Instance AI work runs
|
||||
at the same time.
|
||||
|
||||
The script also writes context.json: for each agent, the system prompt, the messages before
|
||||
the first model response, and the tools, from the first LLM call of that agent. Tools that the
|
||||
agent loads later with load_tool come from its last call.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
from n8n_api import setting
|
||||
|
||||
|
||||
def langsmith_client():
|
||||
key = setting('LANGSMITH_API_KEY')
|
||||
if not key:
|
||||
raise SystemExit('LANGSMITH_API_KEY is not set. Instance AI cost and tokens are not available.')
|
||||
base = setting('LANGSMITH_ENDPOINT', default='https://api.smith.langchain.com').rstrip('/')
|
||||
|
||||
def call(method, path, body=None):
|
||||
req = urllib.request.Request(
|
||||
base + path, method=method,
|
||||
data=json.dumps(body).encode() if body is not None else None,
|
||||
headers={'x-api-key': key, 'content-type': 'application/json'},
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=60) as res:
|
||||
return json.load(res)
|
||||
|
||||
return call
|
||||
|
||||
|
||||
def fetch_calls(started_at, finished_at, traces=None):
|
||||
"""LLM calls in the LangSmith project between two ISO times. traces keeps only the calls of these traces."""
|
||||
call = langsmith_client()
|
||||
project = setting('LANGSMITH_PROJECT', 'LANGCHAIN_PROJECT', default='instance-ai')
|
||||
|
||||
sessions = call('GET', '/api/v1/sessions?' + urllib.parse.urlencode({'name': project}))
|
||||
if not sessions:
|
||||
raise SystemExit(f'LangSmith project "{project}" not found.')
|
||||
session_id = sessions[0]['id']
|
||||
|
||||
start = datetime.fromisoformat(started_at) - timedelta(seconds=5)
|
||||
end = datetime.fromisoformat(finished_at) + timedelta(seconds=5)
|
||||
calls, cursor = [], None
|
||||
while True:
|
||||
body = {
|
||||
'session': [session_id], 'run_type': 'llm', 'limit': 100,
|
||||
'start_time': start.isoformat(),
|
||||
'select': ['id', 'name', 'start_time', 'end_time', 'prompt_tokens', 'completion_tokens',
|
||||
'total_cost', 'prompt_token_details', 'extra', 'trace_id'],
|
||||
}
|
||||
if cursor:
|
||||
body['cursor'] = cursor
|
||||
res = call('POST', '/api/v1/runs/query', body)
|
||||
for run in res.get('runs', []):
|
||||
started = datetime.fromisoformat(run['start_time'].replace('Z', '+00:00'))
|
||||
if started.tzinfo is None:
|
||||
started = started.replace(tzinfo=start.tzinfo)
|
||||
if started > end or (traces is not None and run.get('trace_id') not in traces):
|
||||
continue
|
||||
details = run.get('prompt_token_details') or {}
|
||||
metadata = (run.get('extra') or {}).get('metadata') or {}
|
||||
calls.append({
|
||||
'id': run.get('id'), 'trace': run.get('trace_id'), 'name': run.get('name'),
|
||||
'start': started.isoformat(), 'end': run.get('end_time'),
|
||||
'in': run.get('prompt_tokens') or 0,
|
||||
'cache_read': details.get('cache_read') or 0,
|
||||
'cache_write': details.get('cache_creation') or 0,
|
||||
'out': run.get('completion_tokens') or 0,
|
||||
'cost': float(run.get('total_cost') or 0),
|
||||
'model': metadata.get('ls_model_name'),
|
||||
})
|
||||
cursor = (res.get('cursors') or {}).get('next')
|
||||
if not cursor:
|
||||
break
|
||||
return sorted(calls, key=lambda c: c['start'])
|
||||
|
||||
|
||||
def message_text(content):
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
return '\n'.join(p.get('text', '') for p in content or [] if isinstance(p, dict) and p.get('type') == 'text')
|
||||
|
||||
|
||||
def tool_spec(t):
|
||||
return {'name': t.get('name') or t.get('id'), 'description': t.get('description') or '',
|
||||
'schema': t.get('inputSchema') or t.get('parameters'), 'type': t.get('type')}
|
||||
|
||||
|
||||
def fetch_contexts(calls):
|
||||
"""What each agent got before its first model response, in the order that the agents started."""
|
||||
call = langsmith_client()
|
||||
by_agent = {}
|
||||
for c in calls:
|
||||
if c.get('id'):
|
||||
by_agent.setdefault(c['name'], []).append(c)
|
||||
contexts = []
|
||||
for name, agent_calls in by_agent.items():
|
||||
first = call('GET', f'/api/v1/runs/{agent_calls[0]["id"]}').get('inputs') or {}
|
||||
messages = []
|
||||
for m in first.get('input') or []:
|
||||
if m.get('role') not in ('system', 'user'):
|
||||
break
|
||||
messages.append({'role': m['role'], 'text': message_text(m.get('content'))})
|
||||
tools = [tool_spec(t) for t in first.get('tools') or []]
|
||||
loaded = []
|
||||
if len(agent_calls) > 1:
|
||||
last = call('GET', f'/api/v1/runs/{agent_calls[-1]["id"]}').get('inputs') or {}
|
||||
known = {t['name'] for t in tools}
|
||||
loaded = [tool_spec(t) for t in last.get('tools') or [] if (t.get('name') or t.get('id')) not in known]
|
||||
contexts.append({'agent': name, 'messages': messages, 'tools': tools, 'loadedTools': loaded})
|
||||
return contexts
|
||||
|
||||
|
||||
def totals(calls):
|
||||
return {
|
||||
'calls': len(calls), 'models': sorted({c['model'] for c in calls if c['model']}),
|
||||
'input': sum(c['in'] for c in calls), 'output': sum(c['out'] for c in calls),
|
||||
'cost': round(sum(c['cost'] for c in calls), 4),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--run-dir', required=True)
|
||||
ap.add_argument('--by-trace', action='store_true')
|
||||
args = ap.parse_args()
|
||||
run_dir = Path(args.run_dir)
|
||||
meta = json.loads((run_dir / 'run.json').read_text())
|
||||
traces = set(meta.get('langsmithRunIds') or [])
|
||||
if args.by_trace and not traces:
|
||||
raise SystemExit('run.json has no langsmithRunIds. Is LANGSMITH_TRACING=true on the instance?')
|
||||
calls = fetch_calls(meta['startedAt'], meta['finishedAt'], traces if args.by_trace else None)
|
||||
(run_dir / 'usage.json').write_text(json.dumps(calls, indent=1))
|
||||
(run_dir / 'context.json').write_text(json.dumps(fetch_contexts(calls), indent=1))
|
||||
print(json.dumps(totals(calls), indent=2))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -1,84 +0,0 @@
|
|||
"""Live views of a run.
|
||||
|
||||
COMPARE_LIVE=1 (default): compare.py serves a live page (live_server.py). COMPARE_LIVE=0 turns it off.
|
||||
COMPARE_LIVE_WINDOWS=1 also opens the Instance AI thread in the browser and a terminal window for
|
||||
each CLI builder. If a view cannot open, the script prints how to open it.
|
||||
COMPARE_RECORD=0 skips the screen recordings that the report replays.
|
||||
"""
|
||||
|
||||
import shlex
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import webbrowser
|
||||
|
||||
from n8n_api import setting
|
||||
|
||||
|
||||
def is_on(name, default='1'):
|
||||
return setting(name, default=default).strip().lower() not in ('0', 'false', 'no', 'off')
|
||||
|
||||
|
||||
def enabled():
|
||||
return is_on('COMPARE_LIVE')
|
||||
|
||||
|
||||
def windows():
|
||||
return enabled() and is_on('COMPARE_LIVE_WINDOWS', default='0')
|
||||
|
||||
|
||||
def recording():
|
||||
return is_on('COMPARE_RECORD')
|
||||
|
||||
|
||||
def open_url(url):
|
||||
if not windows():
|
||||
return
|
||||
try:
|
||||
opened = webbrowser.open(url)
|
||||
except webbrowser.Error:
|
||||
opened = False
|
||||
if not opened:
|
||||
print(f'open {url} to watch the run', flush=True)
|
||||
|
||||
|
||||
def log_window_launchers(command):
|
||||
if sys.platform == 'darwin' and shutil.which('osascript'):
|
||||
# The command goes in as an argument, so it needs no AppleScript quoting.
|
||||
yield ['osascript', '-e', 'on run argv', '-e',
|
||||
'tell application "Terminal" to do script (item 1 of argv)', '-e', 'end run', command]
|
||||
for terminal, flag in (('x-terminal-emulator', '-e'), ('gnome-terminal', '--'), ('konsole', '-e'), ('xterm', '-e')):
|
||||
if shutil.which(terminal):
|
||||
yield [terminal, flag, 'sh', '-c', command]
|
||||
return
|
||||
|
||||
|
||||
def open_terminal(title, command, cols=None, rows=None):
|
||||
"""Open a terminal window with a title that runs a shell command. Returns False if none opened."""
|
||||
if not windows():
|
||||
return True
|
||||
prefix = f"printf '\\033]0;%s\\007' {shlex.quote(title)}; "
|
||||
if cols and rows:
|
||||
# Resize the window so that the full screen of the CLI fits.
|
||||
prefix += f"printf '\\033[8;{rows};{cols}t'; "
|
||||
for launcher in log_window_launchers(prefix + command):
|
||||
# osascript returns at once, but xterm and other emulators run as long as the window is open.
|
||||
# A launcher that still runs after a short wait has opened its window.
|
||||
try:
|
||||
proc = subprocess.Popen(launcher, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL, start_new_session=True)
|
||||
except OSError:
|
||||
continue
|
||||
try:
|
||||
if proc.wait(timeout=3) == 0:
|
||||
return True
|
||||
except subprocess.TimeoutExpired:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def open_log_window(title, path):
|
||||
"""Open a terminal window that follows the log file at path."""
|
||||
quoted = shlex.quote(str(path))
|
||||
if not open_terminal(title, f'tail -n +1 -f {quoted}'):
|
||||
print(f'watch {title}: tail -f {quoted}', flush=True)
|
||||
|
|
@ -1,635 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Serve a page that shows the builder runs while they run.
|
||||
|
||||
live_server.py --run-dir <run-dir> [--parent-pid <pid>]
|
||||
|
||||
compare.py starts this server in the background. The server listens on 127.0.0.1 only and
|
||||
writes its address to <run-dir>/live-url.txt. Routes:
|
||||
|
||||
/report/live the live page
|
||||
/report/<file> the final report and its files, when they exist
|
||||
/api/state the status, table row and LLM steps of each builder, as JSON
|
||||
/api/cast/<name> the terminal screen of a CLI builder, as Server-Sent Events in the
|
||||
live format of the asciinema player
|
||||
/api/shot/<name> the last screenshot of the Instance AI thread
|
||||
|
||||
compare.py writes the status of each builder to <run-dir>/status.json. The server stops
|
||||
30 minutes after the last request, when the run is done or compare.py has stopped.
|
||||
|
||||
Other skills reuse the server: they call serve() with their own app (see BuilderApp).
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import mimetypes
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
from urllib.parse import unquote, urlparse
|
||||
|
||||
import build_report as br
|
||||
from run_cli_builder import find_session_file
|
||||
|
||||
IDLE_STOP_S = 1800
|
||||
KINDS = {'iai': 'instance-ai', 'claude': 'claude', 'codex': 'codex'}
|
||||
STATUS_LABELS = {'waiting': 'Waiting', 'running': 'Running', 'done': 'Done', 'failed': 'Failed', 'skipped': 'Skipped'}
|
||||
mimetypes.add_type('font/woff2', '.woff2')
|
||||
mimetypes.add_type('text/javascript', '.js')
|
||||
mimetypes.add_type('video/webm', '.webm')
|
||||
|
||||
|
||||
def read_status(run_dir):
|
||||
try:
|
||||
return json.loads((run_dir / 'status.json').read_text())
|
||||
except (OSError, ValueError):
|
||||
return {'phase': 'running', 'builders': {}}
|
||||
|
||||
|
||||
def empty_data(model=None):
|
||||
return {'events': [], 'calls': [], 'model': model, 'duration': '-', 'cost': '-', 'costSource': None}
|
||||
|
||||
|
||||
def load_instance_ai_live(sub):
|
||||
"""Trace events from the event stream that run_instance_ai.py writes while the thread runs."""
|
||||
path = sub / 'events.jsonl'
|
||||
rows = br.read_jsonl(path) if path.exists() else []
|
||||
if not rows:
|
||||
return empty_data()
|
||||
first = next((r for r in rows if r.get('type') == 'run-start'), rows[0])
|
||||
start, root = first['ts'] / 1000, first.get('agentId')
|
||||
events, texts, tools = [], {}, {}
|
||||
for r in rows:
|
||||
kind, p, agent = r.get('type'), r.get('payload') or {}, r.get('agentId')
|
||||
t = (r.get('ts') or first['ts']) / 1000 - start
|
||||
label = '' if agent in (root, None) else f'{agent.split("-run_")[0]} › '
|
||||
if kind in ('text-delta', 'reasoning-delta'):
|
||||
event_kind = 'text' if kind == 'text-delta' else 'reasoning'
|
||||
key = (agent, r.get('responseId'), event_kind)
|
||||
if key not in texts:
|
||||
texts[key] = {'kind': event_kind, 't': t, 'text': ''}
|
||||
events.append(texts[key])
|
||||
texts[key]['text'] += p.get('text') or ''
|
||||
elif kind == 'tool-call':
|
||||
tools[p.get('toolCallId')] = {'kind': 'tool', 't': t, 'name': label + (p.get('toolName') or ''),
|
||||
'args': p.get('args'), 'result': 'Waiting for the result…',
|
||||
'error': False, 'duration': None}
|
||||
events.append(tools[p.get('toolCallId')])
|
||||
elif kind in ('tool-result', 'tool-error') and p.get('toolCallId') in tools:
|
||||
result = p.get('result') if kind == 'tool-result' else p.get('error')
|
||||
tools[p['toolCallId']].update(
|
||||
result=result, duration=t - tools[p['toolCallId']]['t'],
|
||||
error=kind == 'tool-error' or (isinstance(result, dict) and (result.get('success') is False or 'error' in result)))
|
||||
elif kind == 'confirmation-request' and p.get('toolCallId') in tools:
|
||||
tools[p['toolCallId']]['note'] = 'Confirmation card answered by the auto-approve policy.'
|
||||
data = empty_data()
|
||||
data['events'] = [e for e in events if e['kind'] == 'tool' or e['text'].strip()]
|
||||
return data
|
||||
|
||||
|
||||
def live_run(run_dir, name, case):
|
||||
"""Meta and trace data of one builder. A finished builder has run.json. A running one does not."""
|
||||
sub = run_dir / name
|
||||
meta_file = sub / 'run.json'
|
||||
if meta_file.exists():
|
||||
meta = json.loads(meta_file.read_text())
|
||||
if meta.get('kind') in br.LOADERS:
|
||||
return meta, br.load_run(sub, meta, case.get('prices'))
|
||||
meta = {'kind': KINDS.get(name, name)}
|
||||
if meta['kind'] == 'instance-ai':
|
||||
return meta, load_instance_ai_live(sub)
|
||||
session = find_session_file(meta['kind'], sub / 'home') if (sub / 'home').exists() else None
|
||||
if not session:
|
||||
return meta, empty_data()
|
||||
meta['sessionFile'] = str(session.relative_to(sub))
|
||||
return meta, br.load_run(sub, meta, case.get('prices'))
|
||||
|
||||
|
||||
def seconds(value):
|
||||
return f'{value:.0f}s'
|
||||
|
||||
|
||||
def gather(run_dir):
|
||||
case, status, verdicts = br.load_case(run_dir), read_status(run_dir), br.load_verdicts(run_dir)
|
||||
builders = status.get('builders', {})
|
||||
names = sorted((n for n, b in builders.items() if b.get('status') != 'skipped'),
|
||||
key=lambda n: (br.RUN_ORDER.get(KINDS.get(n, n), 9), n))
|
||||
now, runs = time.time(), []
|
||||
for name in names:
|
||||
entry = builders[name]
|
||||
state = entry.get('status', 'waiting')
|
||||
meta, data = live_run(run_dir, name, case)
|
||||
running = state in ('waiting', 'running')
|
||||
if running and data.get('loadError'):
|
||||
# A running builder can leave a file in a state that the loader cannot read yet.
|
||||
data = empty_data(data.get('model'))
|
||||
started, finished = entry.get('startedAt'), entry.get('finishedAt')
|
||||
if started and (running or data['duration'] == '-'):
|
||||
data['duration'] = seconds((finished or now) - started)
|
||||
if running:
|
||||
data['cost'] = '-'
|
||||
runs.append((name, meta, data, entry))
|
||||
return case, status, verdicts, runs
|
||||
|
||||
|
||||
def status_badge(state):
|
||||
return f'<span class="badge st-{br.esc(state)}">{STATUS_LABELS.get(state, br.esc(state))}</span>'
|
||||
|
||||
|
||||
def clock(status):
|
||||
started = status.get('startedAt')
|
||||
elapsed = seconds((status.get('finishedAt') or time.time()) - started) if started else ''
|
||||
mode = 'Parallel' if status.get('mode') == 'parallel' else 'One at a time'
|
||||
return ' · '.join(x for x in (mode, elapsed, status.get('step')) if x)
|
||||
|
||||
|
||||
def render_run(run_dir, case, name, meta, data, entry, verdict, scale):
|
||||
row = br.render_row_cells(name, meta, data, status_badge(entry.get('status', 'waiting')))
|
||||
canvas = br.render_canvas_box(name, meta, run_dir / 'report', case.get('baseUrl', 'http://localhost:5678'),
|
||||
empty='The canvas appears when all builders finish.')
|
||||
view, auto = br.verdict_view(verdict, run_dir / name, case.get('expected') or '')
|
||||
return row, br.render_panel_body(data, view, canvas, *scale, case['prompt'], case.get('expected') or '', auto=auto,
|
||||
pending=True)
|
||||
|
||||
|
||||
def render_live_box(status, runs):
|
||||
cards = []
|
||||
for name, meta, _, entry in runs:
|
||||
title = br.esc(br.run_title(name, meta))
|
||||
screen = (status.get('screens') or {}).get(name)
|
||||
if screen == 'cast':
|
||||
inner = (f'<div data-cast="{name}"></div><button type="button" class="live-zoom overlay" '
|
||||
f'data-zoom="{name}" aria-label="Show {title} full screen"></button>')
|
||||
elif screen == 'shot':
|
||||
inner = (f'<button type="button" class="live-zoom" data-zoom="{name}" aria-label="Show {title} full screen">'
|
||||
f'<img data-shot="{name}" alt="{title} in the n8n UI" hidden></button>'
|
||||
f'<div class="live-empty" data-shot-empty="{name}">Waiting for the first screenshot</div>')
|
||||
else:
|
||||
inner = '<div class="live-empty">No live screen for this builder. The LLM steps below update live.</div>'
|
||||
cards.append(
|
||||
f'<div class="live-card"><div class="live-screen">{inner}</div>'
|
||||
f'<div class="reel-meta"><span class="run-name">{title}</span>'
|
||||
f'<span data-live-badge="{name}">{status_badge(entry.get("status", "waiting"))}</span></div></div>'
|
||||
)
|
||||
return (f'<div class="box"><div class="head"><h2>Live</h2><span class="mono-label" id="live-clock">'
|
||||
f'{br.esc(clock(status))}</span></div><div class="live-grid">{"".join(cards)}</div></div>')
|
||||
|
||||
|
||||
def render_page(run_dir):
|
||||
case, status, verdicts, runs = gather(run_dir)
|
||||
scale = br.token_scale([r[:3] for r in runs])
|
||||
rows, panels = [], []
|
||||
for i, (name, meta, data, entry) in enumerate(runs):
|
||||
active = ' active' if i == 0 else ''
|
||||
row, panel = render_run(run_dir, case, name, meta, data, entry, verdicts.get(name), scale)
|
||||
rows.append(f'<tr class="row{active}" data-run="{name}" onclick="show(\'{name}\')">{row}</tr>')
|
||||
panels.append(f'<section class="run{active}" id="{name}">{panel}</section>')
|
||||
try:
|
||||
br.ensure_player_cache()
|
||||
player_js, player_css = (f'vendor/{n}' for n in br.PLAYER_FILES)
|
||||
except OSError:
|
||||
player_js, player_css = (br.PLAYER_CDN + n for n in br.PLAYER_FILES)
|
||||
notes = ('This page updates while the builders run, and changes to the full report when the run is done. '
|
||||
'Costs, canvases, checks and verdicts come after the run. Times are seconds from the start of each builder.')
|
||||
return br.fill_template(
|
||||
br.esc(case.get('title') or 'Builder comparison'), ''.join(rows), br.esc(notes), ''.join(panels),
|
||||
replay=render_live_box(status, runs), head=f'<link rel="stylesheet" href="{player_css}">',
|
||||
scripts=f'<script src="{player_js}"></script><script>const SCREENS = [];</script><script>{LIVE_JS}</script>',
|
||||
prompt=br.esc(case['prompt']), expected=br.esc(case.get('expected') or '-'))
|
||||
|
||||
|
||||
def render_state(run_dir):
|
||||
case, status, verdicts, runs = gather(run_dir)
|
||||
scale = br.token_scale([r[:3] for r in runs])
|
||||
state = {'phase': status.get('phase', 'running'), 'clock': clock(status), 'runs': {}}
|
||||
if state['phase'] == 'done' and (run_dir / 'report' / 'index.html').exists():
|
||||
state['report'] = '/report/index.html'
|
||||
for name, meta, data, entry in runs:
|
||||
row, panel = render_run(run_dir, case, name, meta, data, entry, verdicts.get(name), scale)
|
||||
shot = run_dir / name / 'live.png'
|
||||
state['runs'][name] = {'row': row, 'panel': panel, 'badge': status_badge(entry.get('status', 'waiting')),
|
||||
'shot': shot.stat().st_mtime if shot.exists() else None}
|
||||
return state
|
||||
|
||||
|
||||
LIVE_JS = """
|
||||
window.addEventListener('DOMContentLoaded', ()=>{
|
||||
const seen = {}, shots = {}, userFolds = {};
|
||||
// A box that the user opened or closed stays that way when its panel updates. Other boxes take the new default.
|
||||
document.addEventListener('click', e=>{
|
||||
const summary = e.target.closest && e.target.closest('details.fold > summary');
|
||||
if (!summary || e.target.closest('.run-picker')) return;
|
||||
const section = summary.closest('section.run');
|
||||
if (section) (userFolds[section.id] ??= {})[summary.querySelector('h2').textContent] = !summary.parentElement.open;
|
||||
});
|
||||
document.addEventListener('change', e=>{
|
||||
if (!e.target.matches || !e.target.matches('.run-picker')) return;
|
||||
const steps = document.getElementById(e.target.value)?.querySelector('.steps');
|
||||
if (steps) (userFolds[e.target.value] ??= {})['LLM steps'] = steps.open;
|
||||
});
|
||||
document.querySelectorAll('[data-cast]').forEach(el=>{
|
||||
AsciinemaPlayer.create({driver: 'eventsource', url: `/api/cast/${el.dataset.cast}`, bufferTime: 0}, el,
|
||||
{fit: 'both', autoPlay: true, controls: false});
|
||||
});
|
||||
// A live screen opens larger in a full-screen viewer. The viewer follows the newest screenshot, and a
|
||||
// terminal gets its own player, because the stream starts with the whole recording so far.
|
||||
const zoomNames = [...document.querySelectorAll('[data-zoom]')].map(b=>b.dataset.zoom);
|
||||
const viewer = document.createElement('div');
|
||||
let zoomed = null, opener = null, castPlayer = null, castName = null;
|
||||
viewer.className = 'theater live-theater';
|
||||
viewer.hidden = true;
|
||||
viewer.setAttribute('role', 'dialog');
|
||||
viewer.setAttribute('aria-modal', 'true');
|
||||
viewer.setAttribute('aria-label', 'Live screen');
|
||||
viewer.innerHTML = '<div class="theater-bar"><h2>Live</h2><span data-zoom-badge></span>'
|
||||
+ '<button type="button" class="btn" data-zoom-close>Close</button></div><div class="tabs"></div>'
|
||||
+ '<div class="live-zoomed"><img alt=""><div class="live-empty">Waiting for the first screenshot</div>'
|
||||
+ '<div class="live-zoomed-cast" hidden></div></div>'
|
||||
+ '<p class="fineprint">The screen updates every few seconds. Press the left or right arrow key to change the '
|
||||
+ 'scenario. Press Esc to close.</p>';
|
||||
for (const name of zoomNames) {
|
||||
const tab = Object.assign(document.createElement('button'), {type: 'button', className: 'tab'});
|
||||
tab.dataset.zoomTab = name;
|
||||
tab.textContent = document.querySelector(`[data-zoom="${name}"]`).closest('.live-card').querySelector('.run-name').textContent;
|
||||
tab.addEventListener('click', ()=>showZoom(name));
|
||||
viewer.querySelector('.tabs').append(tab);
|
||||
}
|
||||
document.body.append(viewer);
|
||||
function stopCast(){
|
||||
castPlayer?.dispose();
|
||||
castPlayer = castName = null;
|
||||
viewer.querySelector('.live-zoomed-cast').replaceChildren();
|
||||
}
|
||||
function showZoom(name){
|
||||
zoomed = name;
|
||||
viewer.querySelectorAll('[data-zoom-tab]').forEach(t=>t.classList.toggle('active', t.dataset.zoomTab === name));
|
||||
const shot = document.querySelector(`img[data-shot="${name}"]`);
|
||||
const isCast = !!document.querySelector(`[data-cast="${name}"]`);
|
||||
const img = viewer.querySelector('.live-zoomed img'), castBox = viewer.querySelector('.live-zoomed-cast');
|
||||
if (castName !== name) stopCast();
|
||||
castBox.hidden = !isCast;
|
||||
if (isCast && !castPlayer) {
|
||||
castName = name;
|
||||
castPlayer = AsciinemaPlayer.create({driver: 'eventsource', url: `/api/cast/${name}`, bufferTime: 0}, castBox,
|
||||
{fit: 'both', autoPlay: true, controls: false});
|
||||
}
|
||||
const shotHidden = !shot || shot.hidden;
|
||||
img.hidden = isCast || shotHidden;
|
||||
if (shot && !shot.hidden && img.getAttribute('src') !== shot.getAttribute('src')) { img.src = shot.src; img.alt = shot.alt; }
|
||||
viewer.querySelector('.live-zoomed .live-empty').hidden = isCast || !shotHidden;
|
||||
viewer.querySelector('[data-zoom-badge]').innerHTML = document.querySelector(`[data-live-badge="${name}"]`)?.innerHTML || '';
|
||||
}
|
||||
function closeZoom(){
|
||||
stopCast();
|
||||
viewer.hidden = true;
|
||||
zoomed = null;
|
||||
document.body.classList.remove('theater-open');
|
||||
opener?.focus();
|
||||
}
|
||||
document.querySelectorAll('[data-zoom]').forEach(b=>b.addEventListener('click', ()=>{
|
||||
opener = b;
|
||||
viewer.hidden = false;
|
||||
document.body.classList.add('theater-open');
|
||||
showZoom(b.dataset.zoom);
|
||||
viewer.querySelector('[data-zoom-close]').focus();
|
||||
}));
|
||||
viewer.querySelector('[data-zoom-close]').addEventListener('click', closeZoom);
|
||||
document.addEventListener('keydown', e=>{
|
||||
if (zoomed === null) return;
|
||||
if (e.key === 'Escape') closeZoom();
|
||||
else if (e.key === 'ArrowLeft' || e.key === 'ArrowRight') {
|
||||
const step = e.key === 'ArrowRight' ? 1 : -1;
|
||||
showZoom(zoomNames[(zoomNames.indexOf(zoomed) + step + zoomNames.length) % zoomNames.length]);
|
||||
}
|
||||
});
|
||||
function selectQuietly(root, step){
|
||||
root.querySelectorAll('.step').forEach(b=>b.classList.toggle('active', b.dataset.step === step));
|
||||
root.querySelectorAll('.step-detail').forEach(d=>{ d.hidden = d.dataset.step !== step; });
|
||||
}
|
||||
// Keeps the selected step, the scroll positions and the open sections. When the last step is
|
||||
// selected, the newest step is selected after the update.
|
||||
function swapPanel(section, html, first){
|
||||
const old = section.querySelector('.steps');
|
||||
const items = old ? [...old.querySelectorAll('.step')] : [];
|
||||
const active = items.find(b=>b.classList.contains('active'));
|
||||
const follow = first || !active || active === items[items.length - 1];
|
||||
// A run with no trace yet has a .steps section without the list and the pane.
|
||||
const listTop = old?.querySelector('.step-list')?.scrollTop ?? 0;
|
||||
const paneTop = old?.querySelector('.step-pane')?.scrollTop ?? 0;
|
||||
const openState = {};
|
||||
section.querySelectorAll('.step-detail').forEach(d=>d.querySelectorAll('details').forEach((x, i)=>{ openState[`${d.dataset.step}:${i}`] = x.open; }));
|
||||
section.innerHTML = html;
|
||||
section.querySelectorAll('details.fold').forEach(d=>{
|
||||
const open = (userFolds[section.id] || {})[d.querySelector('summary h2')?.textContent];
|
||||
if (open !== undefined) d.open = open;
|
||||
});
|
||||
addRunPickers();
|
||||
section.querySelectorAll('.step-detail').forEach(d=>d.querySelectorAll('details').forEach((x, i)=>{
|
||||
const key = `${d.dataset.step}:${i}`;
|
||||
if (key in openState) x.open = openState[key];
|
||||
}));
|
||||
const root = section.querySelector('.steps');
|
||||
const next = root ? [...root.querySelectorAll('.step')] : [];
|
||||
if (!next.length) return;
|
||||
const list = root.querySelector('.step-list'), pane = root.querySelector('.step-pane');
|
||||
if (!list || !pane) return;
|
||||
if (follow) {
|
||||
selectQuietly(root, next[next.length - 1].dataset.step);
|
||||
list.scrollTop = list.scrollHeight;
|
||||
} else {
|
||||
selectQuietly(root, active.dataset.step);
|
||||
list.scrollTop = listTop;
|
||||
pane.scrollTop = paneTop;
|
||||
}
|
||||
}
|
||||
async function poll(){
|
||||
let state;
|
||||
try {
|
||||
state = await (await fetch('/api/state', {cache: 'no-store'})).json();
|
||||
} catch {
|
||||
$('live-clock').textContent = 'Lost the connection to the live server. Trying again…';
|
||||
setTimeout(poll, 3000);
|
||||
return;
|
||||
}
|
||||
if (state.report) { location.replace(state.report); return; }
|
||||
$('live-clock').textContent = state.clock;
|
||||
for (const [name, run] of Object.entries(state.runs)) {
|
||||
const row = document.querySelector(`.row[data-run="${name}"]`);
|
||||
if (row && seen[`row:${name}`] !== run.row) { row.innerHTML = run.row; seen[`row:${name}`] = run.row; }
|
||||
const section = document.getElementById(name);
|
||||
if (section && seen[`panel:${name}`] !== run.panel) {
|
||||
swapPanel(section, run.panel, seen[`panel:${name}`] === undefined);
|
||||
seen[`panel:${name}`] = run.panel;
|
||||
}
|
||||
const badge = document.querySelector(`[data-live-badge="${name}"]`);
|
||||
if (badge) badge.innerHTML = run.badge;
|
||||
const img = document.querySelector(`img[data-shot="${name}"]`);
|
||||
if (img && run.shot && shots[name] !== run.shot) {
|
||||
shots[name] = run.shot;
|
||||
img.src = `/api/shot/${name}?v=${run.shot}`;
|
||||
img.hidden = false;
|
||||
document.querySelector(`[data-shot-empty="${name}"]`)?.remove();
|
||||
}
|
||||
}
|
||||
if (zoomed !== null) showZoom(zoomed);
|
||||
setTimeout(poll, 2000);
|
||||
}
|
||||
poll();
|
||||
});
|
||||
"""
|
||||
|
||||
|
||||
class BuilderApp:
|
||||
"""What the server shows for compare.py: one live card, table row and panel for each builder.
|
||||
|
||||
Another skill can serve its own page with the same server: give LiveServer an object with
|
||||
these methods."""
|
||||
|
||||
def __init__(self, run_dir):
|
||||
self.run_dir = run_dir
|
||||
|
||||
def entry(self, name):
|
||||
"""The status.json entry of a live card, or None for an unknown name."""
|
||||
return read_status(self.run_dir).get('builders', {}).get(name)
|
||||
|
||||
def finished(self, name):
|
||||
state = (self.entry(name) or {}).get('status')
|
||||
return state in ('done', 'failed', 'skipped') or (self.run_dir / name / 'run.json').exists()
|
||||
|
||||
def done(self):
|
||||
return read_status(self.run_dir).get('phase') == 'done' and (self.run_dir / 'report' / 'index.html').exists()
|
||||
|
||||
def page(self):
|
||||
return render_page(self.run_dir)
|
||||
|
||||
def state(self):
|
||||
return render_state(self.run_dir)
|
||||
|
||||
def shot(self, name):
|
||||
return self.run_dir / name / 'live.png'
|
||||
|
||||
def casts(self, name):
|
||||
"""The live recording, then the place where the runner moves it when the builder stops."""
|
||||
sub = self.run_dir / name
|
||||
return sub / '.screen' / 'screen.cast', sub / 'screen.cast'
|
||||
|
||||
def file(self, path):
|
||||
"""A file for a route that the server does not know, or None."""
|
||||
return None
|
||||
|
||||
|
||||
class LiveServer(ThreadingHTTPServer):
|
||||
daemon_threads = True
|
||||
|
||||
def __init__(self, address, app):
|
||||
super().__init__(address, Handler)
|
||||
port = self.server_address[1]
|
||||
self.app, self.run_dir = app, app.run_dir
|
||||
self.allowed_hosts = {f'127.0.0.1:{port}', f'localhost:{port}'}
|
||||
self.last_request = time.time()
|
||||
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
server_version = 'cwb-live'
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
def do_GET(self):
|
||||
srv = self.server
|
||||
srv.last_request = time.time()
|
||||
# Only same-origin requests. A page that resolves its own host name to 127.0.0.1 gets a 403.
|
||||
if self.headers.get('Host', '') not in srv.allowed_hosts:
|
||||
return self.send_error(403)
|
||||
path = unquote(urlparse(self.path).path)
|
||||
app = srv.app
|
||||
try:
|
||||
if path in ('/', '/report', '/report/'):
|
||||
return self.redirect('/report/index.html' if app.done() else '/report/live')
|
||||
if path == '/report/live':
|
||||
return self.send_body(app.page().encode(), 'text/html; charset=utf-8')
|
||||
if path == '/api/state':
|
||||
return self.send_body(json.dumps(app.state()).encode(), 'application/json')
|
||||
name = path.rsplit('/', 1)[-1]
|
||||
if path.startswith(('/api/cast/', '/api/shot/')) and not app.entry(name):
|
||||
return self.send_error(404)
|
||||
if path.startswith('/api/cast/'):
|
||||
return self.stream_cast(name)
|
||||
if path.startswith('/api/shot/'):
|
||||
return self.send_file(app.shot(name))
|
||||
if path.startswith('/report/'):
|
||||
return self.send_file(self.static_file(path.removeprefix('/report/')))
|
||||
self.send_file(app.file(path))
|
||||
except (BrokenPipeError, ConnectionResetError):
|
||||
pass
|
||||
|
||||
def redirect(self, location):
|
||||
self.send_response(302)
|
||||
self.send_header('Location', location)
|
||||
self.send_header('Content-Length', '0')
|
||||
self.end_headers()
|
||||
|
||||
def send_body(self, body, content_type):
|
||||
self.send_response(200)
|
||||
self.send_header('Content-Type', content_type)
|
||||
self.send_header('Content-Length', str(len(body)))
|
||||
self.send_header('Cache-Control', 'no-store')
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def static_file(self, rel):
|
||||
"""A file of the report. Fonts, logos and the player come from the asset cache until the report exists."""
|
||||
base = (self.server.run_dir / 'report').resolve()
|
||||
candidate = (base / rel).resolve()
|
||||
if candidate.is_relative_to(base) and candidate.is_file():
|
||||
return candidate
|
||||
parts = Path(rel).parts
|
||||
if parts[:1] in (('fonts',), ('brand',)):
|
||||
cache = br.ASSET_CACHE.resolve()
|
||||
candidate = (cache / rel).resolve()
|
||||
return candidate if candidate.is_relative_to(cache) and candidate.is_file() else None
|
||||
if len(parts) == 2 and parts[0] == 'vendor' and parts[1] in br.PLAYER_FILES:
|
||||
return br.PLAYER_CACHE / parts[1]
|
||||
return None
|
||||
|
||||
def send_file(self, file):
|
||||
if not file or not file.is_file():
|
||||
return self.send_error(404)
|
||||
size = file.stat().st_size
|
||||
start, end = 0, size - 1
|
||||
match = re.fullmatch(r'bytes=(\d*)-(\d*)', self.headers.get('Range') or '')
|
||||
if match and size and (match[1] or match[2]):
|
||||
# Videos need byte ranges to seek.
|
||||
if match[1]:
|
||||
start, end = int(match[1]), int(match[2]) if match[2] else size - 1
|
||||
else:
|
||||
start = max(size - int(match[2]), 0)
|
||||
end = min(end, size - 1)
|
||||
if start > end:
|
||||
self.send_response(416)
|
||||
self.send_header('Content-Range', f'bytes */{size}')
|
||||
self.end_headers()
|
||||
return
|
||||
self.send_response(206)
|
||||
self.send_header('Content-Range', f'bytes {start}-{end}/{size}')
|
||||
else:
|
||||
self.send_response(200)
|
||||
self.send_header('Content-Type', mimetypes.guess_type(file.name)[0] or 'application/octet-stream')
|
||||
self.send_header('Content-Length', str(end - start + 1))
|
||||
self.send_header('Accept-Ranges', 'bytes')
|
||||
self.send_header('Cache-Control', 'no-cache')
|
||||
self.end_headers()
|
||||
with file.open('rb') as f:
|
||||
f.seek(start)
|
||||
left = end - start + 1
|
||||
while left > 0:
|
||||
chunk = f.read(min(left, 1 << 16))
|
||||
if not chunk:
|
||||
break
|
||||
self.wfile.write(chunk)
|
||||
left -= len(chunk)
|
||||
|
||||
def sse(self, data, event=None):
|
||||
self.wfile.write(((f'event: {event}\n' if event else '') + f'data: {data}\n\n').encode())
|
||||
self.wfile.flush()
|
||||
|
||||
def stream_cast(self, name):
|
||||
"""Send the screen recording so far as one reset frame, then each new frame as it arrives."""
|
||||
srv = self.server
|
||||
self.send_response(200)
|
||||
self.send_header('Content-Type', 'text/event-stream')
|
||||
self.send_header('Cache-Control', 'no-store')
|
||||
self.end_headers()
|
||||
path = None
|
||||
while path is None:
|
||||
path = next((p for p in srv.app.casts(name) if p.exists()), None)
|
||||
if path is None:
|
||||
if srv.app.finished(name):
|
||||
return self.sse('{}', 'done')
|
||||
self.wfile.write(b': waiting\n\n')
|
||||
self.wfile.flush()
|
||||
srv.last_request = time.time()
|
||||
time.sleep(1)
|
||||
# The runner moves the recording when the builder stops. The open file stays readable.
|
||||
with path.open('rb') as f:
|
||||
buffer = b''
|
||||
|
||||
def complete_lines():
|
||||
nonlocal buffer
|
||||
buffer += f.read()
|
||||
*lines, buffer = buffer.split(b'\n')
|
||||
return [line for line in lines if line.strip()]
|
||||
|
||||
lines = complete_lines()
|
||||
while not lines:
|
||||
if srv.app.finished(name):
|
||||
return self.sse('{}', 'done')
|
||||
time.sleep(0.2)
|
||||
lines = complete_lines()
|
||||
header, frames = json.loads(lines[0]), [json.loads(line) for line in lines[1:]]
|
||||
output = [f for f in frames if f[1] == 'o']
|
||||
self.sse(json.dumps({'cols': header.get('width'), 'rows': header.get('height'),
|
||||
'time': output[-1][0] if output else 0, 'init': ''.join(f[2] for f in output)}))
|
||||
last_data = last_ping = time.time()
|
||||
while True:
|
||||
lines = complete_lines()
|
||||
for line in lines:
|
||||
frame = json.loads(line)
|
||||
if frame[1] == 'o':
|
||||
self.sse(json.dumps([frame[0], 'o', frame[2]]))
|
||||
now = time.time()
|
||||
srv.last_request = now
|
||||
if lines:
|
||||
last_data = now
|
||||
continue
|
||||
if srv.app.finished(name) and now - last_data > 3:
|
||||
return self.sse('{}', 'done')
|
||||
if now - last_ping > 15:
|
||||
self.wfile.write(b': ping\n\n')
|
||||
self.wfile.flush()
|
||||
last_ping = now
|
||||
time.sleep(0.2)
|
||||
|
||||
|
||||
def pid_alive(pid):
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
except ProcessLookupError:
|
||||
return False
|
||||
except PermissionError:
|
||||
return True
|
||||
return True
|
||||
|
||||
|
||||
def stop_when_idle(server, parent_pid):
|
||||
while True:
|
||||
time.sleep(30)
|
||||
if time.time() - server.last_request < IDLE_STOP_S:
|
||||
continue
|
||||
done = read_status(server.run_dir).get('phase') == 'done'
|
||||
if done or (parent_pid and not pid_alive(parent_pid)):
|
||||
server.shutdown()
|
||||
return
|
||||
|
||||
|
||||
def serve(app, parent_pid=None, port=0):
|
||||
"""Serve app until it is idle. Writes the address to <run-dir>/live-url.txt."""
|
||||
server = LiveServer(('127.0.0.1', port), app)
|
||||
(app.run_dir / 'live-url.txt').write_text(f'http://127.0.0.1:{server.server_address[1]}\n')
|
||||
threading.Thread(target=stop_when_idle, args=(server, parent_pid), daemon=True).start()
|
||||
server.serve_forever()
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--run-dir', required=True)
|
||||
ap.add_argument('--parent-pid', type=int)
|
||||
ap.add_argument('--port', type=int, default=0)
|
||||
args = ap.parse_args()
|
||||
serve(BuilderApp(Path(args.run_dir).resolve()), args.parent_pid, args.port)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -1,122 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Merge the blind judge panel into verdicts.json.
|
||||
|
||||
merge_verdicts.py <run-dir>
|
||||
|
||||
Reads judge/blind-map.json and judge/verdicts-*.json. Each judge file has one entry for each
|
||||
builder letter:
|
||||
|
||||
{"A": {"summary": "...", "goalMet": true, "goalReason": "...", "verdict": "pass",
|
||||
"checks": [{"id": "4", "ok": true, "text": "...", "evidence": [{"node": "...", "path": "...", "value": "..."}]}]}}
|
||||
|
||||
For each criterion id, the merged check takes the value that all judges or a majority of the judges
|
||||
gave. A split is listed under disagreements. Without a majority, the check is not confirmed (ok null).
|
||||
The summary and the findings outside the criteria come from the first judge.
|
||||
"""
|
||||
|
||||
import json
|
||||
import sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from build_report import load_case, parse_criteria # noqa: E402
|
||||
|
||||
VERSION = 2
|
||||
VOTE_TEXT = {True: 'met', False: 'not met', None: 'not confirmed'}
|
||||
|
||||
|
||||
def majority(values):
|
||||
"""The value that more than half of the votes have, or None when no value has a majority."""
|
||||
value, count = Counter(values).most_common(1)[0]
|
||||
return value if count * 2 > len(values) else None
|
||||
|
||||
|
||||
def first_check(judge, cid):
|
||||
return next((c for c in judge.get('checks') or [] if str(c.get('id')) == cid), None)
|
||||
|
||||
|
||||
def merge_quality(judges):
|
||||
"""The quality checks by majority, in the order of the first judge that answered each check."""
|
||||
ids = list(dict.fromkeys(str(q.get('id')) for j in judges for q in j.get('quality') or [] if q.get('id')))
|
||||
merged = []
|
||||
for qid in ids:
|
||||
found = [next((q for q in j.get('quality') or [] if str(q.get('id')) == qid), None) for j in judges]
|
||||
votes = [q.get('ok') if q else None for q in found]
|
||||
ok = majority(votes)
|
||||
pick = next((q for q, v in zip(found, votes) if q and v == ok), None) or next(q for q in found if q)
|
||||
item = {'id': qid, 'ok': ok, 'text': pick.get('text', ''), 'votes': votes}
|
||||
if len(set(votes)) > 1:
|
||||
item['note'] = f'{Counter(votes)[ok]} of {len(judges)} judges.' if ok is not None else 'Judges disagree.'
|
||||
merged.append(item)
|
||||
return merged
|
||||
|
||||
|
||||
def merge_builder(judges, criteria_ids):
|
||||
n = len(judges)
|
||||
lead = judges[0]
|
||||
checks, disagreements = [], []
|
||||
for cid in criteria_ids:
|
||||
found = [first_check(j, cid) for j in judges]
|
||||
votes = [c.get('ok') if c else None for c in found]
|
||||
if all(c is None for c in found):
|
||||
continue
|
||||
ok = majority(votes)
|
||||
pick = next((c for c, v in zip(found, votes) if c and v == ok), None) or next(c for c in found if c)
|
||||
check = {k: v for k, v in pick.items() if k != 'ok'}
|
||||
check.update(id=cid, ok=ok, votes=votes)
|
||||
if len(set(votes)) > 1:
|
||||
counts = ', '.join(f'{count} {VOTE_TEXT[v]}' for v, count in Counter(votes).items())
|
||||
disagreements.append({'id': cid, 'votes': votes, 'texts': [c.get('text') if c else None for c in found]})
|
||||
note = f'Judges disagree ({counts}).' if ok is None else f'{Counter(votes)[ok]} of {n} judges.'
|
||||
check['note'] = note
|
||||
checks.append(check)
|
||||
checks += [dict(c) for c in lead.get('checks') or [] if str(c.get('id')) not in criteria_ids]
|
||||
|
||||
goal_votes = [j.get('goalMet') for j in judges]
|
||||
goal = majority(goal_votes)
|
||||
if goal is None:
|
||||
goal = False if goal_votes.count(False) * 2 >= n else None
|
||||
if len(set(goal_votes)) > 1:
|
||||
reasons = [j.get('goalReason') for j in judges if j.get('goalMet') is False and j.get('goalReason')]
|
||||
disagreements.append({'id': 'goal', 'votes': goal_votes, 'texts': reasons})
|
||||
verdicts = [j.get('verdict') for j in judges if j.get('verdict')]
|
||||
return {
|
||||
'quality': merge_quality(judges),
|
||||
'version': VERSION,
|
||||
'judges': n,
|
||||
'verdict': majority(verdicts) if verdicts else None,
|
||||
'summary': lead.get('summary', ''),
|
||||
'goalMet': goal,
|
||||
'goalReason': next((j.get('goalReason') for j in judges if j.get('goalMet') is goal and j.get('goalReason')), ''),
|
||||
'checks': checks,
|
||||
'disagreements': disagreements,
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
run_dir = Path(sys.argv[1] if len(sys.argv) > 1 else '.')
|
||||
judge_dir = run_dir / 'judge'
|
||||
blind_map = json.loads((judge_dir / 'blind-map.json').read_text())
|
||||
files = sorted(judge_dir.glob('verdicts-*.json'))
|
||||
if not files:
|
||||
sys.exit(f'No judge/verdicts-*.json in {run_dir}.')
|
||||
panel = [json.loads(f.read_text()) for f in files]
|
||||
groups = parse_criteria(load_case(run_dir).get('expected') or '') or []
|
||||
criteria_ids = [num for _, items in groups for num, _ in items]
|
||||
|
||||
merged = {}
|
||||
for letter, builder in blind_map.items():
|
||||
judges = [p[letter] for p in panel if letter in p]
|
||||
if not judges:
|
||||
print(f'{builder}: no judge gave a verdict for builder {letter}')
|
||||
continue
|
||||
merged[builder] = merge_builder(judges, criteria_ids)
|
||||
split = len(merged[builder]['disagreements'])
|
||||
print(f'{builder} (builder {letter}): {len(judges)} judges, {split} disagreement{"" if split == 1 else "s"}')
|
||||
(run_dir / 'verdicts.json').write_text(json.dumps(merged, indent=2, ensure_ascii=False))
|
||||
print(run_dir / 'verdicts.json')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -1,123 +0,0 @@
|
|||
"""Config loader and small REST client for a local n8n instance. Standard library only."""
|
||||
|
||||
import http.cookiejar
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid
|
||||
from functools import cache
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[4]
|
||||
DEFAULT_BASE_URL = 'http://localhost:5678'
|
||||
TOKEN_FILE = Path.home() / '.config' / 'n8n-builder-compare' / 'mcp-token'
|
||||
|
||||
|
||||
def read_env_file(path):
|
||||
"""Return KEY=VALUE pairs from a dotenv file. Values are never printed."""
|
||||
values = {}
|
||||
p = Path(path)
|
||||
if not p.exists():
|
||||
return values
|
||||
for line in p.read_text().splitlines():
|
||||
line = line.strip()
|
||||
if line.startswith('export '):
|
||||
line = line[7:]
|
||||
if not line or line.startswith('#') or '=' not in line:
|
||||
continue
|
||||
key, value = line.split('=', 1)
|
||||
value = value.strip()
|
||||
if value[:1] in ('"', "'"):
|
||||
value = value[1:].split(value[0], 1)[0]
|
||||
else:
|
||||
value = value.split(' #', 1)[0].strip()
|
||||
values[key.strip()] = value
|
||||
return values
|
||||
|
||||
|
||||
def env_files():
|
||||
"""Config files from lowest to highest priority."""
|
||||
custom = os.environ.get('AI_SURFACES_ENV')
|
||||
return [REPO_ROOT / '.env.local', Path(custom) if custom else REPO_ROOT / '.env.ai-surfaces']
|
||||
|
||||
|
||||
@cache
|
||||
def config():
|
||||
"""Merged settings: .env.local, then .env.ai-surfaces, then the process environment."""
|
||||
merged = {}
|
||||
for path in env_files():
|
||||
merged.update({k: v for k, v in read_env_file(path).items() if v != ''})
|
||||
merged.update({k: v for k, v in os.environ.items() if v != ''})
|
||||
return merged
|
||||
|
||||
|
||||
def setting(*names, default=None):
|
||||
"""Return the first name that has a value."""
|
||||
values = config()
|
||||
for name in names:
|
||||
if values.get(name):
|
||||
return values[name]
|
||||
return default
|
||||
|
||||
|
||||
class N8nClient:
|
||||
def __init__(self):
|
||||
self.base_url = setting('N8N_BASE_URL', default=DEFAULT_BASE_URL).rstrip('/')
|
||||
self.email = setting('N8N_EVAL_EMAIL', default='nathan@n8n.io')
|
||||
self.password = setting('N8N_EVAL_PASSWORD', default='PlaywrightTest123')
|
||||
# The auth cookie is bound to the browser-id header that was sent at login.
|
||||
self.browser_id = str(uuid.uuid4())
|
||||
self.cookies = http.cookiejar.CookieJar()
|
||||
self.opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(self.cookies))
|
||||
|
||||
def headers(self, extra=None):
|
||||
h = {'content-type': 'application/json', 'browser-id': self.browser_id}
|
||||
h.update(extra or {})
|
||||
return h
|
||||
|
||||
def request(self, method, path, body=None, timeout=60):
|
||||
data = json.dumps(body).encode() if body is not None else None
|
||||
req = urllib.request.Request(self.base_url + path, method=method, data=data, headers=self.headers())
|
||||
try:
|
||||
with self.opener.open(req, timeout=timeout) as res:
|
||||
raw = res.read()
|
||||
except urllib.error.HTTPError as e:
|
||||
detail = e.read()[:500].decode(errors='replace')
|
||||
raise RuntimeError(f'{method} {path} -> {e.code}: {detail}') from None
|
||||
parsed = json.loads(raw) if raw else None
|
||||
if isinstance(parsed, dict) and set(parsed) == {'data'}:
|
||||
return parsed['data']
|
||||
return parsed
|
||||
|
||||
def open_stream(self, path):
|
||||
req = urllib.request.Request(
|
||||
self.base_url + path, headers=self.headers({'accept': 'text/event-stream'})
|
||||
)
|
||||
return self.opener.open(req, timeout=600)
|
||||
|
||||
def login(self):
|
||||
self.request('POST', '/rest/login', {'emailOrLdapLoginId': self.email, 'password': self.password})
|
||||
return self
|
||||
|
||||
def workflow(self, workflow_id):
|
||||
return self.request('GET', f'/rest/workflows/{workflow_id}')
|
||||
|
||||
|
||||
def save_token(token):
|
||||
TOKEN_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
TOKEN_FILE.write_text(token)
|
||||
os.chmod(TOKEN_FILE, stat.S_IRUSR | stat.S_IWUSR)
|
||||
|
||||
|
||||
def load_token():
|
||||
"""N8N_MCP_TOKEN from the config wins over the cached token."""
|
||||
token = setting('N8N_MCP_TOKEN')
|
||||
if token:
|
||||
return token
|
||||
return TOKEN_FILE.read_text().strip() if TOKEN_FILE.exists() else None
|
||||
|
||||
|
||||
def is_full_token(value):
|
||||
return isinstance(value, str) and value.count('.') == 2 and '*' not in value
|
||||
|
|
@ -1,50 +0,0 @@
|
|||
// Shared browser helpers: settings, Playwright launch, and n8n sign-in.
|
||||
// Uses the Playwright copy in the repo test package. Falls back to the local Chrome install.
|
||||
import { createRequire } from 'node:module';
|
||||
import { existsSync, readFileSync } from 'node:fs';
|
||||
import { dirname, join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const here = dirname(fileURLToPath(import.meta.url));
|
||||
const repo = resolve(here, '../../../..');
|
||||
const require = createRequire(join(repo, 'packages/quality/testing/playwright/package.json'));
|
||||
const { chromium, expect, selectors } = require('@playwright/test');
|
||||
export { expect, selectors };
|
||||
// pngjs, as Playwright bundles it. Image diffs use it, so the skills need no extra package.
|
||||
export const loadPng = () =>
|
||||
createRequire(require.resolve('@playwright/test'))('playwright-core/lib/utilsBundle').PNG;
|
||||
|
||||
// Same order as n8n_api.config(): .env.local, then .env.ai-surfaces, then the environment.
|
||||
const settings = {};
|
||||
for (const file of [
|
||||
join(repo, '.env.local'),
|
||||
process.env.AI_SURFACES_ENV ?? join(repo, '.env.ai-surfaces'),
|
||||
]) {
|
||||
if (!existsSync(file)) continue;
|
||||
for (const line of readFileSync(file, 'utf8').split('\n')) {
|
||||
const match = line.match(/^\s*(?:export\s+)?([A-Z0-9_]+)\s*=\s*(.*)$/);
|
||||
if (!match) continue;
|
||||
const value = match[2].trim().replace(/^(['"])(.*)\1$/, '$2');
|
||||
if (value) settings[match[1]] = value;
|
||||
}
|
||||
}
|
||||
export const setting = (name, fallback) => process.env[name] || settings[name] || fallback;
|
||||
|
||||
export const baseUrl = setting('N8N_BASE_URL', 'http://localhost:5678').replace(/\/$/, '');
|
||||
|
||||
export async function launchBrowser() {
|
||||
const launch = existsSync(chromium.executablePath()) ? {} : { channel: 'chrome' };
|
||||
return await chromium.launch({ headless: true, ...launch });
|
||||
}
|
||||
|
||||
export async function signIn(page) {
|
||||
await page.goto(`${baseUrl}/signin`);
|
||||
await page
|
||||
.getByRole('textbox', { name: 'Email' })
|
||||
.fill(setting('N8N_EVAL_EMAIL', 'nathan@n8n.io'));
|
||||
await page
|
||||
.getByRole('textbox', { name: 'Password' })
|
||||
.fill(setting('N8N_EVAL_PASSWORD', 'PlaywrightTest123'));
|
||||
await page.getByRole('button', { name: 'Sign in', exact: true }).click();
|
||||
await page.waitForURL((url) => !url.pathname.startsWith('/signin'), { timeout: 30_000 });
|
||||
}
|
||||
|
|
@ -1,109 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Check the local setup and make the n8n MCP token available.
|
||||
|
||||
Prints a JSON status object. It never prints a token or a key.
|
||||
|
||||
prepare.py check everything, reuse the cached MCP token
|
||||
prepare.py --rotate create a new MCP key (this breaks other clients that use the old key)
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
from n8n_api import TOKEN_FILE, N8nClient, env_files, is_full_token, load_token, save_token, setting
|
||||
|
||||
|
||||
def mcp_handshake(base_url, token):
|
||||
body = json.dumps({
|
||||
'jsonrpc': '2.0', 'id': 1, 'method': 'initialize',
|
||||
'params': {'protocolVersion': '2025-03-26', 'capabilities': {},
|
||||
'clientInfo': {'name': 'testmax-n8n-ai', 'version': '1'}},
|
||||
}).encode()
|
||||
req = urllib.request.Request(base_url + '/mcp-server/http', data=body, method='POST', headers={
|
||||
'content-type': 'application/json',
|
||||
'accept': 'application/json, text/event-stream',
|
||||
'authorization': f'Bearer {token}',
|
||||
})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=20) as res:
|
||||
return res.status == 200
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def docker_ready():
|
||||
if not shutil.which('docker'):
|
||||
return False
|
||||
return subprocess.run(['docker', 'info'], capture_output=True).returncode == 0
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--rotate', action='store_true')
|
||||
args = ap.parse_args()
|
||||
|
||||
status = {
|
||||
'configFiles': [str(p) for p in env_files() if p.exists()],
|
||||
'tokenFile': str(TOKEN_FILE),
|
||||
'builders': setting('COMPARE_BUILDERS', default='iai,claude,codex'),
|
||||
'keys': {
|
||||
'claude': bool(setting('CLAUDE_API_KEY', 'ANTHROPIC_API_KEY')),
|
||||
'codex': bool(setting('CODEX_API_KEY', 'OPENAI_API_KEY')),
|
||||
'langsmith': bool(setting('LANGSMITH_API_KEY')),
|
||||
},
|
||||
}
|
||||
client = N8nClient()
|
||||
status['baseUrl'] = client.base_url
|
||||
try:
|
||||
client.login()
|
||||
status['login'] = True
|
||||
except Exception as e:
|
||||
status['login'] = False
|
||||
status['error'] = str(e)
|
||||
print(json.dumps(status, indent=2))
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
settings = client.request('GET', '/rest/instance-ai/settings')
|
||||
status['instanceAiEnabled'] = bool(settings.get('enabled'))
|
||||
except Exception:
|
||||
status['instanceAiEnabled'] = False
|
||||
|
||||
client.request('PATCH', '/rest/mcp/settings', {'mcpAccessEnabled': True})
|
||||
status['mcpAccessEnabled'] = True
|
||||
|
||||
token = None
|
||||
if args.rotate:
|
||||
if setting('N8N_MCP_TOKEN'):
|
||||
# The clients would keep the configured token, which the rotation makes invalid.
|
||||
sys.exit('Remove N8N_MCP_TOKEN from the config before you rotate the MCP key, then run prepare.py --rotate again.')
|
||||
token = client.request('POST', '/rest/mcp/api-key/rotate')['apiKey']
|
||||
save_token(token)
|
||||
else:
|
||||
current = client.request('GET', '/rest/mcp/api-key')['apiKey']
|
||||
if is_full_token(current):
|
||||
# A new key was created for this user, so the full value is visible once.
|
||||
save_token(current)
|
||||
token = load_token()
|
||||
status['mcpTokenSource'] = 'N8N_MCP_TOKEN' if setting('N8N_MCP_TOKEN') else 'cache'
|
||||
|
||||
status['mcpTokenAvailable'] = bool(token)
|
||||
status['mcpTokenWorks'] = bool(token) and mcp_handshake(client.base_url, token)
|
||||
if not status['mcpTokenWorks']:
|
||||
status['mcpTokenHelp'] = (
|
||||
'The MCP token is missing or invalid. Set N8N_MCP_TOKEN in .env.ai-surfaces, or run '
|
||||
'prepare.py --rotate after the user agrees. Rotation invalidates the old key in every other MCP client.'
|
||||
)
|
||||
|
||||
status['docker'] = docker_ready()
|
||||
status['claudeOnHost'] = bool(shutil.which('claude'))
|
||||
status['codexOnHost'] = bool(shutil.which('codex'))
|
||||
print(json.dumps(status, indent=2))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -1,82 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
// Record a video of an Instance AI thread in the n8n UI while the run continues.
|
||||
//
|
||||
// node record_iai.mjs <out-dir> [--live]
|
||||
//
|
||||
// --live also saves the screen to <out-dir>/live.png every few seconds, for the live page.
|
||||
//
|
||||
// Protocol on stdin and stdout, one line each:
|
||||
// prints "ready" after sign-in
|
||||
// reads "open <thread-url>" and opens the thread
|
||||
// reads "stop", or stdin closes: keeps the last screen for a moment, then saves <out-dir>/screen.webm
|
||||
// and the last screen as <out-dir>/screen-poster.png
|
||||
// The sign-in happens in a separate context, so the video starts at the thread.
|
||||
import { copyFileSync, mkdtempSync, renameSync, rmSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join, resolve } from 'node:path';
|
||||
import { createInterface } from 'node:readline';
|
||||
|
||||
import { launchBrowser, signIn } from './n8n_browser.mjs';
|
||||
|
||||
const outDir = resolve(process.argv[2] ?? '.');
|
||||
const live = process.argv.includes('--live');
|
||||
const SIZE = { width: 1600, height: 1000 };
|
||||
const LIVE_SHOT_MS = 2500;
|
||||
|
||||
const browser = await launchBrowser();
|
||||
const login = await browser.newContext();
|
||||
const loginPage = await login.newPage();
|
||||
await signIn(loginPage);
|
||||
const storageState = await login.storageState();
|
||||
await login.close();
|
||||
|
||||
const videoDir = mkdtempSync(join(tmpdir(), 'cwb-video-'));
|
||||
const context = await browser.newContext({
|
||||
viewport: SIZE,
|
||||
storageState,
|
||||
recordVideo: { dir: videoDir, size: SIZE },
|
||||
});
|
||||
const page = await context.newPage();
|
||||
console.log('ready');
|
||||
|
||||
let liveTimer;
|
||||
let shooting = false;
|
||||
async function saveLiveShot() {
|
||||
if (shooting) return;
|
||||
shooting = true;
|
||||
// The live page can read the file at any time, so it gets only complete images.
|
||||
const partial = join(outDir, 'live.tmp.png');
|
||||
try {
|
||||
await page.screenshot({ path: partial });
|
||||
renameSync(partial, join(outDir, 'live.png'));
|
||||
} catch {
|
||||
// The page can be in the middle of a navigation. The next interval tries again.
|
||||
} finally {
|
||||
shooting = false;
|
||||
}
|
||||
}
|
||||
|
||||
const lines = createInterface({ input: process.stdin });
|
||||
for await (const line of lines) {
|
||||
if (line.startsWith('open ')) {
|
||||
await page.goto(line.slice(5).trim(), { waitUntil: 'domcontentloaded' });
|
||||
console.log('opened');
|
||||
if (live && !liveTimer) liveTimer = setInterval(saveLiveShot, LIVE_SHOT_MS);
|
||||
} else if (line.trim() === 'stop') {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
clearInterval(liveTimer);
|
||||
await page.waitForTimeout(3000);
|
||||
if (live) await saveLiveShot();
|
||||
// The video starts on a blank page, so the report shows the last screen until playback starts.
|
||||
await page.screenshot({ path: join(outDir, 'screen-poster.png') }).catch(() => {});
|
||||
const video = page.video();
|
||||
await context.close();
|
||||
if (video) {
|
||||
copyFileSync(await video.path(), join(outDir, 'screen.webm'));
|
||||
console.log('saved screen.webm');
|
||||
}
|
||||
await browser.close();
|
||||
rmSync(videoDir, { recursive: true, force: true });
|
||||
|
|
@ -1,192 +0,0 @@
|
|||
"""Record the request bodies that a CLI sends to its model API. Standard library only.
|
||||
|
||||
The recorder is a local HTTP proxy. The CLI sends its model requests to the recorder, and the
|
||||
recorder sends them to the real API without changes. It writes the request bodies to a JSONL file.
|
||||
It never writes headers, so the API key stays out of the file.
|
||||
|
||||
Each row has the request settings without the conversation input. The tools and the instructions
|
||||
are in a row only when they differ from the previous request of the same thread, because the CLI
|
||||
sends them every turn.
|
||||
The tools field also holds the tools of additional_tools input items.
|
||||
"""
|
||||
|
||||
import gzip
|
||||
import http.client
|
||||
import json
|
||||
import threading
|
||||
import time
|
||||
import zlib
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from urllib.parse import urlparse
|
||||
|
||||
# Headers that apply to one connection only. The recorder sets its own values.
|
||||
HOP_HEADERS = {'connection', 'keep-alive', 'proxy-connection', 'proxy-authorization', 'te', 'trailer',
|
||||
'transfer-encoding', 'upgrade', 'host', 'content-length', 'accept-encoding'}
|
||||
CHANGING_KEYS = ('tools', 'instructions')
|
||||
|
||||
|
||||
def decode_body(data, encoding):
|
||||
encoding = (encoding or '').lower()
|
||||
if encoding == 'gzip':
|
||||
return gzip.decompress(data)
|
||||
if encoding == 'deflate':
|
||||
return zlib.decompress(data)
|
||||
if encoding == 'zstd':
|
||||
from compression import zstd
|
||||
return zstd.decompress(data)
|
||||
return data
|
||||
|
||||
|
||||
class RequestRecorder:
|
||||
"""Start with start(). Point the CLI at base_url(host). Call stop() when the CLI is done."""
|
||||
|
||||
def __init__(self, upstream, out_file, bind='127.0.0.1'):
|
||||
self.upstream = urlparse(upstream)
|
||||
self.out_file = out_file
|
||||
self.lock = threading.Lock()
|
||||
self.idle = threading.Condition(self.lock)
|
||||
self.active = 0
|
||||
self.last = {}
|
||||
self.count = 0
|
||||
self.started = time.time()
|
||||
self.server = ThreadingHTTPServer((bind, 0), self.handler())
|
||||
self.server.daemon_threads = True
|
||||
self.thread = threading.Thread(target=self.server.serve_forever, daemon=True)
|
||||
|
||||
@property
|
||||
def port(self):
|
||||
return self.server.server_address[1]
|
||||
|
||||
def base_url(self, host):
|
||||
return f'http://{host}:{self.port}{self.upstream.path.rstrip("/")}'
|
||||
|
||||
def start(self):
|
||||
self.out_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
self.out_file.write_text('')
|
||||
self.thread.start()
|
||||
return self
|
||||
|
||||
def stop(self, timeout=30):
|
||||
"""Stop the server. Wait until the open requests are written, so count is final."""
|
||||
self.server.shutdown()
|
||||
with self.idle:
|
||||
self.idle.wait_for(lambda: self.active == 0, timeout=timeout)
|
||||
self.server.server_close()
|
||||
|
||||
def begin(self):
|
||||
with self.lock:
|
||||
self.active += 1
|
||||
|
||||
def end(self):
|
||||
with self.idle:
|
||||
self.active -= 1
|
||||
self.idle.notify_all()
|
||||
|
||||
def record(self, method, path, body, encoding, status, error_text):
|
||||
row = {'t': round(time.time() - self.started, 3), 'method': method, 'path': path, 'status': status}
|
||||
try:
|
||||
payload = json.loads(decode_body(body, encoding)) if body else None
|
||||
except (ValueError, OSError, ImportError):
|
||||
payload, row['bodyBytes'] = None, len(body)
|
||||
with self.lock:
|
||||
if isinstance(payload, dict):
|
||||
items = payload.get('input') if isinstance(payload.get('input'), list) else []
|
||||
if items:
|
||||
row['inputItems'] = len(items)
|
||||
# Codex sends its tools as an additional_tools input item, not in the top-level tools field.
|
||||
extra = [t for i in items if isinstance(i, dict) and i.get('type') == 'additional_tools'
|
||||
for t in i.get('tools') or []]
|
||||
if extra:
|
||||
payload = {**payload, 'tools': (payload.get('tools') or []) + extra}
|
||||
# Side tasks, such as a title request, run on their own thread with other tools.
|
||||
metadata = payload.get('client_metadata') if isinstance(payload.get('client_metadata'), dict) else {}
|
||||
thread = metadata.get('thread_id') or payload.get('prompt_cache_key') or ''
|
||||
for key, value in payload.items():
|
||||
if key == 'input':
|
||||
continue
|
||||
if key in CHANGING_KEYS:
|
||||
if self.last.get((thread, key)) == value:
|
||||
continue
|
||||
self.last[(thread, key)] = value
|
||||
row[key] = value
|
||||
if error_text:
|
||||
row['error'] = error_text
|
||||
self.count += 1
|
||||
with self.out_file.open('a') as f:
|
||||
f.write(json.dumps(row, ensure_ascii=False) + '\n')
|
||||
|
||||
def handler(self):
|
||||
recorder = self
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
protocol_version = 'HTTP/1.1'
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
def read_body(self):
|
||||
if self.headers.get('Content-Length'):
|
||||
return self.rfile.read(int(self.headers['Content-Length']))
|
||||
if 'chunked' not in (self.headers.get('Transfer-Encoding') or '').lower():
|
||||
return b''
|
||||
parts = []
|
||||
while True:
|
||||
size = int(self.rfile.readline().split(b';')[0].strip() or b'0', 16)
|
||||
if not size:
|
||||
self.rfile.readline()
|
||||
return b''.join(parts)
|
||||
parts.append(self.rfile.read(size))
|
||||
self.rfile.readline()
|
||||
|
||||
def forward(self):
|
||||
recorder.begin()
|
||||
try:
|
||||
self.forward_and_record()
|
||||
finally:
|
||||
recorder.end()
|
||||
|
||||
def forward_and_record(self):
|
||||
body = self.read_body()
|
||||
up = recorder.upstream
|
||||
conn_class = http.client.HTTPSConnection if up.scheme == 'https' else http.client.HTTPConnection
|
||||
conn = conn_class(up.hostname, up.port, timeout=900)
|
||||
headers = {k: v for k, v in self.headers.items() if k.lower() not in HOP_HEADERS}
|
||||
headers['Host'] = up.netloc
|
||||
headers['Content-Length'] = str(len(body))
|
||||
status, error_text, replied = 502, None, False
|
||||
try:
|
||||
conn.request(self.command, self.path, body=body, headers=headers)
|
||||
resp = conn.getresponse()
|
||||
status, replied = resp.status, True
|
||||
self.send_response(resp.status, resp.reason)
|
||||
for key, value in resp.getheaders():
|
||||
if key.lower() not in HOP_HEADERS:
|
||||
self.send_header(key, value)
|
||||
self.send_header('Transfer-Encoding', 'chunked')
|
||||
self.end_headers()
|
||||
kept = b''
|
||||
while chunk := resp.read1(65536):
|
||||
if status >= 400 and len(kept) < 4000:
|
||||
kept += chunk
|
||||
self.wfile.write(b'%x\r\n%s\r\n' % (len(chunk), chunk))
|
||||
self.wfile.flush()
|
||||
self.wfile.write(b'0\r\n\r\n')
|
||||
self.wfile.flush()
|
||||
if kept:
|
||||
error_text = kept[:4000].decode('utf-8', 'replace')
|
||||
except (OSError, http.client.HTTPException) as e:
|
||||
error_text = f'recorder: {type(e).__name__}'
|
||||
if not replied:
|
||||
try:
|
||||
self.send_error(502, 'Recorder could not reach the API')
|
||||
except OSError:
|
||||
pass
|
||||
self.close_connection = True
|
||||
finally:
|
||||
conn.close()
|
||||
recorder.record(self.command, self.path, body, self.headers.get('Content-Encoding'),
|
||||
status, error_text)
|
||||
|
||||
do_GET = do_POST = do_PUT = do_DELETE = do_PATCH = forward
|
||||
|
||||
return Handler
|
||||
|
|
@ -1,613 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Run one prompt through Claude Code or Codex with only the n8n MCP server.
|
||||
|
||||
run_cli_builder.py --builder claude --out <run-dir>/claude --prompt-file <run-dir>/prompt.txt
|
||||
run_cli_builder.py --builder codex --out <run-dir>/codex --prompt-file <run-dir>/prompt.txt
|
||||
|
||||
Isolation modes:
|
||||
container (default) Docker container. The only mount is the output directory.
|
||||
The builder cannot see host files, skills, memory or settings.
|
||||
clean Claude only. A clean process with an empty config directory, no built-in
|
||||
tools and only the n8n MCP server. It needs an API key, not a subscription login.
|
||||
|
||||
Screen modes:
|
||||
interactive (default in a container) The normal terminal screen of the CLI runs in tmux.
|
||||
screen.cast records the exact screen, and a terminal window can watch it live.
|
||||
The trace comes from the session file of the CLI.
|
||||
headless `claude -p` or `codex exec` with a JSON event stream. Clean isolation uses this mode.
|
||||
|
||||
Secrets: the model key and the MCP token go through environment variables only.
|
||||
They are never written to a file or put on a command line.
|
||||
|
||||
Codex sends its model requests through a local recorder (request_recorder.py), which writes the
|
||||
request bodies, but not the headers, to requests.jsonl. Set COMPARE_RECORD_REQUESTS=0 to turn it off.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from live import enabled as live_enabled
|
||||
from live import open_log_window, open_terminal
|
||||
from n8n_api import DEFAULT_BASE_URL, load_token, setting
|
||||
from request_recorder import RequestRecorder
|
||||
|
||||
SKILL_DIR = Path(__file__).resolve().parents[1]
|
||||
IMAGE = 'n8n-builder-sandbox'
|
||||
# The builder-specific name comes first, so the CLI can use a different key than
|
||||
# the ANTHROPIC_API_KEY or OPENAI_API_KEY that the instance reads from .env.local.
|
||||
KEY_SETTINGS = {'claude': ['CLAUDE_API_KEY', 'ANTHROPIC_API_KEY'], 'codex': ['CODEX_API_KEY', 'OPENAI_API_KEY']}
|
||||
# Codex features that add tools or downloads beyond the n8n MCP server. shell_snapshot is off
|
||||
# because it writes the environment, with the API key, to a file.
|
||||
CODEX_FEATURES_OFF = ['apps', 'browser_use', 'browser_use_external', 'computer_use', 'image_generation',
|
||||
'in_app_browser', 'plugins', 'remote_plugin', 'shell_snapshot']
|
||||
SCREEN = {'cols': 120, 'rows': 36}
|
||||
OPENAI_BASE_URL = 'https://api.openai.com/v1'
|
||||
# Interactive mode: the variable that the CLI reads the model key from.
|
||||
SANDBOX_KEY = {'claude': 'CLAUDE_SANDBOX_KEY', 'codex': 'CODEX_SANDBOX_KEY'}
|
||||
IDLE_LIMIT_S = 600
|
||||
|
||||
|
||||
def now_iso():
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def ensure_image(rebuild):
|
||||
# The label holds a hash of the Dockerfile, so an edit to the Dockerfile rebuilds the image.
|
||||
digest = hashlib.sha256((SKILL_DIR / 'sandbox' / 'Dockerfile').read_bytes()).hexdigest()[:16]
|
||||
found = subprocess.run(['docker', 'image', 'inspect', '--format',
|
||||
'{{index .Config.Labels "dockerfile-sha"}}', IMAGE], capture_output=True, text=True)
|
||||
if found.returncode == 0 and found.stdout.strip() == digest and not rebuild:
|
||||
return
|
||||
print(f'building docker image {IMAGE}', flush=True)
|
||||
subprocess.run(['docker', 'build', '--label', f'dockerfile-sha={digest}', '-t', IMAGE,
|
||||
str(SKILL_DIR / 'sandbox')], check=True)
|
||||
|
||||
|
||||
def mcp_url(base_url, isolation):
|
||||
if isolation != 'container':
|
||||
return f'{base_url}/mcp-server/http'
|
||||
parsed = urlparse(base_url)
|
||||
port = parsed.port or (443 if parsed.scheme == 'https' else 80)
|
||||
host = 'host.docker.internal' if parsed.hostname in ('localhost', '127.0.0.1') else parsed.hostname
|
||||
return f'{parsed.scheme}://{host}:{port}/mcp-server/http'
|
||||
|
||||
|
||||
def claude_tool_args(tools):
|
||||
# 'default' gives the normal Claude Code tools, all allowed without a question, like Codex in the
|
||||
# container. Only a container may use it, because Bash and Write can change the files they see.
|
||||
if tools == 'default':
|
||||
return ['--permission-mode', 'bypassPermissions']
|
||||
# dontAsk allows the pre-approved n8n tools and denies the rest without a prompt.
|
||||
return ['--tools', '', '--allowedTools', 'mcp__n8n', '--permission-mode', 'dontAsk']
|
||||
|
||||
|
||||
def claude_args(prompt, model, config_path, bare, tools='none'):
|
||||
# --bare also replaces the system prompt with a minimal one. A container already isolates the CLI,
|
||||
# so only the clean mode, which runs on the host, uses it.
|
||||
args = [
|
||||
'claude', '-p', prompt, *(['--bare'] if bare else []),
|
||||
'--strict-mcp-config', '--mcp-config', config_path, *claude_tool_args(tools),
|
||||
'--disable-slash-commands', '--output-format', 'stream-json', '--verbose',
|
||||
]
|
||||
if model:
|
||||
args += ['--model', model]
|
||||
return args
|
||||
|
||||
|
||||
def claude_interactive_args(model, tools='none'):
|
||||
args = [
|
||||
'claude', '--strict-mcp-config', '--mcp-config', '/out/mcp.json', *claude_tool_args(tools),
|
||||
'--disable-slash-commands', '--settings', '/out/.screen/claude-settings.json',
|
||||
]
|
||||
if model:
|
||||
args += ['--model', model]
|
||||
return args
|
||||
|
||||
|
||||
def codex_args(prompt, model):
|
||||
args = ['codex', 'exec', '--json', '--skip-git-repo-check', '--sandbox', 'read-only']
|
||||
if model:
|
||||
args += ['--model', model]
|
||||
return args + [prompt]
|
||||
|
||||
|
||||
def codex_interactive_args(model):
|
||||
# The background app server needs `ps`, which the slim image does not have.
|
||||
args = ['codex', '--no-daemon']
|
||||
if model:
|
||||
args += ['--model', model]
|
||||
return args
|
||||
|
||||
|
||||
def write_codex_config(home, url, key_env=None, api_base=OPENAI_BASE_URL):
|
||||
"""Write config.toml. With key_env, a provider reads the key from that variable, so the
|
||||
interactive screen does not ask for a sign-in and no key is stored in auth.json."""
|
||||
codex_home = home / '.codex'
|
||||
codex_home.mkdir(parents=True, exist_ok=True)
|
||||
lines = ['approval_policy = "never"', 'sandbox_mode = "read-only"']
|
||||
if key_env:
|
||||
lines.append('model_provider = "sandbox-openai"')
|
||||
lines += ['', '[features]'] + [f'{name} = false' for name in CODEX_FEATURES_OFF]
|
||||
if key_env:
|
||||
lines += ['', '[model_providers.sandbox-openai]', 'name = "OpenAI"',
|
||||
f'base_url = "{api_base}"', f'env_key = "{key_env}"', 'wire_api = "responses"']
|
||||
lines += ['', '[projects."/out/work"]', 'trust_level = "trusted"',
|
||||
'', '[mcp_servers.n8n]', f'url = "{url}"', 'bearer_token_env_var = "N8N_MCP_TOKEN"', '']
|
||||
(codex_home / 'config.toml').write_text('\n'.join(lines))
|
||||
|
||||
|
||||
def write_claude_interactive_config(out, home, tools):
|
||||
# Skip the first-run screens: theme, onboarding and the folder trust question.
|
||||
(home / '.claude' / '.claude.json').write_text(json.dumps({
|
||||
'hasCompletedOnboarding': True, 'theme': 'dark',
|
||||
'projects': {'/out/work': {'hasTrustDialogAccepted': True, 'hasCompletedProjectOnboarding': True}},
|
||||
}))
|
||||
# With ANTHROPIC_API_KEY the screen asks to confirm the key and shows part of it.
|
||||
# apiKeyHelper reads the key from the environment without that question.
|
||||
settings = {'apiKeyHelper': f'printenv {SANDBOX_KEY["claude"]}'}
|
||||
if tools == 'default':
|
||||
# The bypass-mode warning screen also has a ❯ marker, so the prompt would answer "No, exit".
|
||||
settings['skipDangerousModePermissionPrompt'] = True
|
||||
(out / '.screen' / 'claude-settings.json').write_text(json.dumps(settings))
|
||||
|
||||
|
||||
def live_line(builder, event):
|
||||
"""One short terminal line for an interesting stream event, or None."""
|
||||
if builder == 'claude':
|
||||
if event.get('subtype') == 'permission_denied':
|
||||
return f'denied {event.get("tool_name")}'
|
||||
if event.get('type') == 'assistant':
|
||||
for block in (event.get('message') or {}).get('content') or []:
|
||||
if block.get('type') == 'tool_use':
|
||||
return f'tool {block.get("name")}'
|
||||
if block.get('type') == 'text' and block.get('text', '').strip():
|
||||
return 'text ' + block['text'].strip().splitlines()[0][:120]
|
||||
if event.get('type') == 'result':
|
||||
return f'result {event.get("subtype")} turns={event.get("num_turns")}'
|
||||
else:
|
||||
item = event.get('item') or {}
|
||||
if event.get('type') == 'item.started' and item.get('type') == 'mcp_tool_call':
|
||||
return f'tool {item.get("server")}.{item.get("tool")}'
|
||||
if event.get('type') == 'item.completed' and item.get('type') == 'agent_message':
|
||||
return 'text ' + (item.get('text') or '').strip()[:120]
|
||||
if event.get('type') in ('turn.completed', 'turn.failed', 'error'):
|
||||
return event['type']
|
||||
return None
|
||||
|
||||
|
||||
def session_lines(builder, row):
|
||||
"""Short terminal lines for one session-file row, and the end state: None, 'done' or 'error'."""
|
||||
lines = []
|
||||
if builder == 'claude':
|
||||
if row.get('type') != 'assistant':
|
||||
return lines, None
|
||||
msg = row.get('message') or {}
|
||||
for block in msg.get('content') or []:
|
||||
if block.get('type') == 'tool_use':
|
||||
lines.append(f'tool {block.get("name")}')
|
||||
elif block.get('type') == 'text' and block.get('text', '').strip():
|
||||
lines.append('text ' + block['text'].strip().splitlines()[0][:120])
|
||||
if row.get('isApiErrorMessage'):
|
||||
return lines, 'error'
|
||||
return lines, 'done' if msg.get('stop_reason') == 'end_turn' else None
|
||||
payload = row.get('payload') or {}
|
||||
if row.get('type') != 'event_msg':
|
||||
return lines, None
|
||||
kind, item = payload.get('type'), payload.get('item') or {}
|
||||
if kind == 'item_completed' and item.get('type') == 'McpToolCall':
|
||||
lines.append(f'tool {item.get("server")}.{item.get("tool")}')
|
||||
elif kind == 'item_completed' and item.get('type') == 'AgentMessage':
|
||||
text = ' '.join(c.get('text', '') for c in item.get('content') or [] if isinstance(c, dict))
|
||||
lines.append('text ' + text.strip()[:120])
|
||||
elif kind == 'error':
|
||||
lines.append(f'error {str(payload.get("message"))[:200]}')
|
||||
return lines, {'task_complete': 'done', 'turn_aborted': 'error'}.get(kind)
|
||||
|
||||
|
||||
BOLD, DIM, CYAN, GREEN, RED, RESET = '\033[1m', '\033[2m', '\033[36m', '\033[32m', '\033[31m', '\033[0m'
|
||||
LABELS = {'claude': 'Claude Code + n8n MCP', 'codex': 'Codex + n8n MCP'}
|
||||
|
||||
|
||||
def short(value, limit=400):
|
||||
"""Readable one-line text for a tool input or result."""
|
||||
if isinstance(value, dict) and isinstance(value.get('content'), list):
|
||||
value = value['content']
|
||||
if isinstance(value, list) and all(isinstance(v, dict) and 'text' in v for v in value):
|
||||
value = ' '.join(v['text'] for v in value)
|
||||
text = value if isinstance(value, str) else json.dumps(value, ensure_ascii=False)
|
||||
text = ' '.join(text.split())
|
||||
return text if len(text) <= limit else text[:limit] + ' …'
|
||||
|
||||
|
||||
def log_entries(builder, event):
|
||||
"""Detailed, colored lines for live.log, the file that the headless live window follows."""
|
||||
lines = []
|
||||
if builder == 'claude':
|
||||
# A permission_denied event has a string message, not an API message.
|
||||
if event.get('subtype') == 'permission_denied':
|
||||
lines.append(f'{RED}denied {event.get("tool_name")}{RESET}')
|
||||
message = event.get('message')
|
||||
content = message.get('content') if isinstance(message, dict) else None
|
||||
for block in content if isinstance(content, list) else []:
|
||||
kind = block.get('type')
|
||||
if kind == 'thinking' and block.get('thinking', '').strip():
|
||||
lines.append(f'{DIM}thinking: {short(block["thinking"], 600)}{RESET}')
|
||||
elif kind == 'text' and block.get('text', '').strip():
|
||||
lines.append(block['text'].strip())
|
||||
elif kind == 'tool_use':
|
||||
name = block.get('name', '').removeprefix('mcp__n8n__')
|
||||
lines.append(f'{BOLD}{CYAN}→ {name}{RESET} {short(block.get("input"))}')
|
||||
elif kind == 'tool_result':
|
||||
color = RED if block.get('is_error') else DIM
|
||||
lines.append(f'{color}← {short(block.get("content"))}{RESET}')
|
||||
if event.get('type') == 'result':
|
||||
cost = event.get('total_cost_usd')
|
||||
lines.append(f'{GREEN}done: {event.get("subtype")}, {event.get("num_turns")} turns'
|
||||
+ (f', ${cost:.2f} (Claude Code estimate)' if cost is not None else '') + RESET)
|
||||
else:
|
||||
item, etype = event.get('item') or {}, event.get('type')
|
||||
kind = item.get('type')
|
||||
if etype == 'item.started' and kind == 'mcp_tool_call':
|
||||
lines.append(f'{BOLD}{CYAN}→ {item.get("tool")}{RESET} {short(item.get("arguments"))}')
|
||||
elif etype == 'item.completed' and kind == 'mcp_tool_call':
|
||||
if item.get('error'):
|
||||
lines.append(f'{RED}← {short(item["error"])}{RESET}')
|
||||
else:
|
||||
lines.append(f'{DIM}← {short(item.get("result"))}{RESET}')
|
||||
elif etype == 'item.completed' and kind == 'reasoning' and item.get('text'):
|
||||
lines.append(f'{DIM}thinking: {short(item["text"], 600)}{RESET}')
|
||||
elif etype == 'item.completed' and kind == 'agent_message' and item.get('text'):
|
||||
lines.append(item['text'].strip())
|
||||
elif etype == 'turn.completed':
|
||||
lines.append(f'{GREEN}done: turn completed{RESET}')
|
||||
elif etype in ('turn.failed', 'error'):
|
||||
lines.append(f'{RED}{etype}: {short(event.get("error") or event.get("message"))}{RESET}')
|
||||
return lines
|
||||
|
||||
|
||||
WORKFLOW_ID = re.compile(r'(?:workflowId|WORKFLOW_ID)\\*"?\s*[:=]\s*\\*"?([A-Za-z0-9]{10,})')
|
||||
|
||||
|
||||
def session_candidates(builder, home):
|
||||
if builder == 'claude':
|
||||
# Subagent transcripts are in deeper folders. The main session is at projects/<cwd>/<id>.jsonl.
|
||||
root = home / '.claude' / 'projects'
|
||||
return list(root.glob('*/*.jsonl')) if root.exists() else []
|
||||
root = home / '.codex' / 'sessions'
|
||||
return list(root.rglob('rollout-*.jsonl')) if root.exists() else []
|
||||
|
||||
|
||||
def find_session_file(builder, home, session_id=None):
|
||||
matches = session_candidates(builder, home)
|
||||
if session_id:
|
||||
matches = [p for p in matches if session_id in p.name] or matches
|
||||
matches.sort(key=lambda p: p.stat().st_mtime)
|
||||
return matches[-1] if matches else None
|
||||
|
||||
|
||||
class SessionReader:
|
||||
"""Reads the rows that the CLI appends to its session file."""
|
||||
|
||||
def __init__(self, builder, home):
|
||||
self.builder, self.home, self.path, self.offset = builder, home, None, 0
|
||||
|
||||
def rows(self):
|
||||
self.path = self.path or find_session_file(self.builder, self.home)
|
||||
if not self.path:
|
||||
return []
|
||||
with self.path.open('rb') as f:
|
||||
f.seek(self.offset)
|
||||
data = f.read()
|
||||
end = data.rfind(b'\n') + 1
|
||||
self.offset += end
|
||||
rows = []
|
||||
for line in data[:end].splitlines():
|
||||
try:
|
||||
rows.append(json.loads(line))
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
return rows
|
||||
|
||||
|
||||
def docker_command(container, out, env_names):
|
||||
cmd = [
|
||||
'docker', 'run', '--rm', '-i', '--name', container,
|
||||
'--user', f'{os.getuid()}:{os.getgid()}',
|
||||
'--cap-drop', 'ALL', '--security-opt', 'no-new-privileges',
|
||||
'--read-only', '--tmpfs', '/tmp',
|
||||
'--add-host', 'host.docker.internal:host-gateway',
|
||||
'-v', f'{out}:/out', '-w', '/out/work',
|
||||
'-e', 'HOME=/out/home',
|
||||
'-e', 'CLAUDE_CONFIG_DIR=/out/home/.claude',
|
||||
'-e', 'CODEX_HOME=/out/home/.codex',
|
||||
'-e', 'CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1',
|
||||
'-e', 'COLORTERM=truecolor',
|
||||
]
|
||||
# `-e NAME` without a value copies the value from this process, so secrets stay off argv.
|
||||
for name in env_names:
|
||||
cmd += ['-e', name]
|
||||
return cmd + [IMAGE]
|
||||
|
||||
|
||||
def check_no_secrets(out, secrets):
|
||||
"""Warn if a secret value ended up in a file of the run directory. Never prints the value."""
|
||||
needles = [s.encode() for s in secrets if s and len(s) >= 12]
|
||||
for path in out.rglob('*'):
|
||||
if not path.is_file() or path.stat().st_size > 50_000_000:
|
||||
continue
|
||||
data = path.read_bytes()
|
||||
if any(n in data for n in needles):
|
||||
print(f'WARNING: a secret value is in {path}. Delete the file before you share the run.', flush=True)
|
||||
|
||||
|
||||
def run_headless(args, out, prompt, cmd, run_env, cwd, container):
|
||||
stream_path = out / 'stream.jsonl'
|
||||
session_id, workflow_ids, result, errors_in_row = None, [], None, 0
|
||||
live_log = (out / 'live.log').open('w', buffering=1)
|
||||
live_log.write(f'{BOLD}{LABELS[args.builder]}{RESET} model: {args.model or "CLI default"}, '
|
||||
f'isolation: {args.isolation}\n{DIM}prompt: {short(prompt, 600)}{RESET}\n\n')
|
||||
open_log_window(LABELS[args.builder], out / 'live.log')
|
||||
# The Codex stream has no timestamps, so stream.times keeps the arrival time of each line.
|
||||
with stream_path.open('w') as stream, (out / 'stream.times').open('w') as times, \
|
||||
(out / 'stderr.log').open('w') as err, live_log:
|
||||
proc = subprocess.Popen(cmd, env=run_env, cwd=cwd, stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.PIPE, stderr=err, text=True)
|
||||
watchdog = threading.Timer(args.timeout_min * 60, proc.kill)
|
||||
watchdog.start()
|
||||
for line in proc.stdout:
|
||||
try:
|
||||
event = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
stream.write(line)
|
||||
stream.flush()
|
||||
times.write(f'{time.time():.3f}\n')
|
||||
if event.get('type') == 'system' and event.get('subtype') == 'init':
|
||||
session_id = event.get('session_id')
|
||||
if event.get('type') == 'thread.started':
|
||||
session_id = event.get('thread_id')
|
||||
if event.get('type') == 'result':
|
||||
result = event
|
||||
for match in WORKFLOW_ID.findall(line):
|
||||
if match not in workflow_ids:
|
||||
workflow_ids.append(match)
|
||||
text = live_line(args.builder, event)
|
||||
if text:
|
||||
print(f'[{args.builder}] {text}', flush=True)
|
||||
for entry in log_entries(args.builder, event):
|
||||
live_log.write(entry + '\n\n')
|
||||
errors_in_row = errors_in_row + 1 if event.get('type') == 'error' else 0
|
||||
if errors_in_row >= 10:
|
||||
print(f'[{args.builder}] stopped after 10 errors in a row: {event.get("message", "")[:200]}', flush=True)
|
||||
proc.kill()
|
||||
break
|
||||
proc.wait()
|
||||
if not watchdog.is_alive():
|
||||
print('timeout: killed the builder', flush=True)
|
||||
watchdog.cancel()
|
||||
if args.isolation == 'container':
|
||||
subprocess.run(['docker', 'kill', container], capture_output=True)
|
||||
live_log.write(f'{BOLD}=== finished, exit code {proc.returncode}. You can close this window. ==={RESET}\n')
|
||||
return {'exitCode': proc.returncode, 'sessionId': session_id, 'workflowIds': workflow_ids,
|
||||
'reportedCostUsd': (result or {}).get('total_cost_usd')}
|
||||
|
||||
|
||||
def run_interactive(args, out, home, prompt, run_env, container):
|
||||
screen_dir = out / '.screen'
|
||||
argv = (claude_interactive_args(args.model, args.claude_tools) if args.builder == 'claude'
|
||||
else codex_interactive_args(args.model))
|
||||
# Claude Code shows its input box before the MCP servers connect, so it waits a few seconds more.
|
||||
ready = {'claude': ('❯', 5000), 'codex': ('›', 2000)}[args.builder]
|
||||
(screen_dir / 'command.json').write_text(json.dumps(
|
||||
{'argv': argv, **SCREEN, 'input': prompt, 'readyText': ready[0], 'settleMs': ready[1]}))
|
||||
cmd = docker_command(container, out, [n for n in run_env if n in (SANDBOX_KEY[args.builder], 'N8N_MCP_TOKEN')])
|
||||
cmd += ['node', '/out/.screen/screen.mjs', 'run', '/out/.screen']
|
||||
|
||||
reader, workflow_ids, state, timed_out = SessionReader(args.builder, home), [], None, False
|
||||
with (out / 'stderr.log').open('w') as err:
|
||||
proc = subprocess.Popen(cmd, env=run_env, stdin=subprocess.DEVNULL, stdout=err, stderr=err)
|
||||
for _ in range(120):
|
||||
if (screen_dir / 'screen.cast').exists() or proc.poll() is not None:
|
||||
break
|
||||
time.sleep(0.25)
|
||||
watch = f'docker exec -it {container} tmux attach -r -t run'
|
||||
if live_enabled() and not open_terminal(LABELS[args.builder], watch, SCREEN['cols'], SCREEN['rows']):
|
||||
print(f'watch {LABELS[args.builder]}: {watch}', flush=True)
|
||||
deadline, last_growth = time.time() + args.timeout_min * 60, time.time()
|
||||
while proc.poll() is None and not state:
|
||||
rows = reader.rows()
|
||||
if rows:
|
||||
last_growth = time.time()
|
||||
for row in rows:
|
||||
lines, row_state = session_lines(args.builder, row)
|
||||
for line in lines:
|
||||
print(f'[{args.builder}] {line}', flush=True)
|
||||
for match in WORKFLOW_ID.findall(json.dumps(row)):
|
||||
if match not in workflow_ids:
|
||||
workflow_ids.append(match)
|
||||
state = state or row_state
|
||||
if time.time() > deadline:
|
||||
print('timeout: stopped the builder', flush=True)
|
||||
timed_out = True
|
||||
break
|
||||
if reader.path and time.time() - last_growth > IDLE_LIMIT_S:
|
||||
print(f'no progress for {IDLE_LIMIT_S // 60} minutes: stopped the builder', flush=True)
|
||||
timed_out = True
|
||||
break
|
||||
time.sleep(1)
|
||||
(screen_dir / 'stop').write_text('')
|
||||
try:
|
||||
proc.wait(timeout=30)
|
||||
except subprocess.TimeoutExpired:
|
||||
subprocess.run(['docker', 'kill', container], capture_output=True)
|
||||
proc.wait()
|
||||
for row in reader.rows():
|
||||
for match in WORKFLOW_ID.findall(json.dumps(row)):
|
||||
if match not in workflow_ids:
|
||||
workflow_ids.append(match)
|
||||
if (screen_dir / 'screen.cast').exists():
|
||||
shutil.move(screen_dir / 'screen.cast', out / 'screen.cast')
|
||||
# After a finished turn the driver stops tmux, so the exit code of the CLI means nothing.
|
||||
exited = screen_dir / 'exited'
|
||||
if state == 'done' and not timed_out:
|
||||
code = 0
|
||||
else:
|
||||
code = int(exited.read_text() or 1) if exited.exists() else 1
|
||||
code = code or 1
|
||||
session = reader.path
|
||||
return {'exitCode': code, 'sessionId': session.stem.split('rollout-')[-1] if session else None,
|
||||
'workflowIds': workflow_ids, 'reportedCostUsd': None}
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--builder', choices=['claude', 'codex'], required=True)
|
||||
ap.add_argument('--out', required=True)
|
||||
ap.add_argument('--prompt-file', required=True)
|
||||
ap.add_argument('--isolation', choices=['container', 'clean'],
|
||||
default=setting('COMPARE_ISOLATION', default='container'))
|
||||
ap.add_argument('--mode', choices=['interactive', 'headless'], default=setting('COMPARE_CLI_MODE'),
|
||||
help='Default: interactive in a container, headless in clean isolation.')
|
||||
ap.add_argument('--model', help='Model id. Default: CLAUDE_MODEL or CODEX_MODEL, else the CLI default.')
|
||||
ap.add_argument('--key-env', help='Name of the setting with the model API key.')
|
||||
ap.add_argument('--claude-tools', choices=['none', 'default'],
|
||||
default=setting('COMPARE_CLAUDE_TOOLS', default='none'),
|
||||
help='none: only the n8n tools. default: also the built-in Claude Code tools, all allowed.')
|
||||
ap.add_argument('--rebuild-image', action='store_true')
|
||||
ap.add_argument('--timeout-min', type=float, default=30)
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.claude_tools not in ('none', 'default'):
|
||||
sys.exit(f'COMPARE_CLAUDE_TOOLS must be none or default, not {args.claude_tools!r}.')
|
||||
if args.builder == 'codex' and args.isolation != 'container':
|
||||
sys.exit('Codex runs only in a container. Its shell tool can read the host disk.')
|
||||
if args.isolation != 'container':
|
||||
args.mode = 'headless'
|
||||
if args.builder == 'claude' and args.claude_tools != 'none':
|
||||
print('[claude] clean isolation runs on the host, so COMPARE_CLAUDE_TOOLS=default is ignored.', flush=True)
|
||||
args.claude_tools = 'none'
|
||||
args.mode = args.mode or 'interactive'
|
||||
|
||||
prefix = args.builder.upper()
|
||||
args.model = args.model or setting(f'{prefix}_MODEL')
|
||||
# Gateway-style ids such as anthropic/claude-opus-5-5 are not valid for the CLIs.
|
||||
provider = {'claude': 'anthropic/', 'codex': 'openai/'}[args.builder]
|
||||
if args.model and args.model.startswith(provider):
|
||||
args.model = args.model.removeprefix(provider)
|
||||
key_names = [args.key_env] if args.key_env else KEY_SETTINGS[args.builder]
|
||||
api_key = setting(*key_names)
|
||||
if not api_key:
|
||||
sys.exit(f'Set {" or ".join(key_names)} in .env.ai-surfaces, .env.local or the environment.')
|
||||
token = load_token()
|
||||
if not token:
|
||||
sys.exit('No MCP token. Run prepare.py first.')
|
||||
|
||||
out = Path(args.out).resolve()
|
||||
if out.exists():
|
||||
shutil.rmtree(out)
|
||||
work, home = out / 'work', out / 'home'
|
||||
work.mkdir(parents=True)
|
||||
(home / '.claude').mkdir(parents=True)
|
||||
prompt = Path(args.prompt_file).read_text().strip()
|
||||
base_url = setting('N8N_BASE_URL', default=DEFAULT_BASE_URL).rstrip('/')
|
||||
url = mcp_url(base_url, args.isolation)
|
||||
container = f'cwb-{args.builder}-{int(time.time())}'
|
||||
interactive = args.mode == 'interactive'
|
||||
|
||||
# Codex does not save the tools it sends to the model, so a local proxy records the request bodies.
|
||||
recorder = None
|
||||
if args.builder == 'codex' and setting('COMPARE_RECORD_REQUESTS', default='1') != '0':
|
||||
recorder = RequestRecorder(OPENAI_BASE_URL, out / 'requests.jsonl',
|
||||
setting('COMPARE_RECORDER_BIND', default='127.0.0.1')).start()
|
||||
api_base = recorder.base_url('host.docker.internal') if recorder else OPENAI_BASE_URL
|
||||
|
||||
child_env = {'N8N_MCP_TOKEN': token}
|
||||
if args.builder == 'claude':
|
||||
(out / 'mcp.json').write_text(json.dumps({'mcpServers': {'n8n': {
|
||||
'type': 'http', 'url': url, 'headers': {'Authorization': 'Bearer ${N8N_MCP_TOKEN}'},
|
||||
}}}))
|
||||
if interactive:
|
||||
(out / '.screen').mkdir()
|
||||
shutil.copy(SKILL_DIR / 'sandbox' / 'screen.mjs', out / '.screen' / 'screen.mjs')
|
||||
child_env[SANDBOX_KEY[args.builder]] = api_key
|
||||
if args.builder == 'claude':
|
||||
write_claude_interactive_config(out, home, args.claude_tools)
|
||||
else:
|
||||
write_codex_config(home, url, SANDBOX_KEY['codex'], api_base)
|
||||
elif args.builder == 'claude':
|
||||
child_env['ANTHROPIC_API_KEY'] = api_key
|
||||
else:
|
||||
child_env['OPENAI_API_KEY'] = api_key
|
||||
child_env['CODEX_API_KEY'] = api_key
|
||||
write_codex_config(home, url, 'CODEX_API_KEY' if recorder else None, api_base)
|
||||
|
||||
started_at = now_iso()
|
||||
if args.isolation == 'container':
|
||||
ensure_image(args.rebuild_image)
|
||||
run_env = {**os.environ, **child_env}
|
||||
try:
|
||||
if interactive:
|
||||
result = run_interactive(args, out, home, prompt, run_env, container)
|
||||
else:
|
||||
cmd = docker_command(container, out, child_env)
|
||||
cmd += (claude_args(prompt, args.model, '/out/mcp.json', False, args.claude_tools)
|
||||
if args.builder == 'claude' else codex_args(prompt, args.model))
|
||||
result = run_headless(args, out, prompt, cmd, run_env, None, container)
|
||||
finally:
|
||||
if recorder:
|
||||
recorder.stop()
|
||||
if recorder and not recorder.count:
|
||||
print(f'[{args.builder}] the request recorder saw no requests, so the report has no Codex tools.', flush=True)
|
||||
else:
|
||||
if not shutil.which('claude'):
|
||||
sys.exit('claude is not on PATH.')
|
||||
run_env = {
|
||||
'PATH': os.environ.get('PATH', '/usr/bin:/bin'),
|
||||
'HOME': str(home),
|
||||
'CLAUDE_CONFIG_DIR': str(home / '.claude'),
|
||||
'DISABLE_AUTOUPDATER': '1',
|
||||
'DISABLE_TELEMETRY': '1',
|
||||
**child_env,
|
||||
}
|
||||
cmd = claude_args(prompt, args.model, str(out / 'mcp.json'), True)
|
||||
result = run_headless(args, out, prompt, cmd, run_env, str(work), container)
|
||||
|
||||
session_file = find_session_file(args.builder, home, result['sessionId'])
|
||||
meta = {
|
||||
'kind': args.builder,
|
||||
'isolation': args.isolation,
|
||||
'mode': args.mode,
|
||||
'model': args.model,
|
||||
**({'claudeTools': args.claude_tools} if args.builder == 'claude' else {}),
|
||||
'startedAt': started_at,
|
||||
'finishedAt': now_iso(),
|
||||
'exitCode': result['exitCode'],
|
||||
'sessionId': result['sessionId'],
|
||||
'sessionFile': str(session_file.relative_to(out)) if session_file else None,
|
||||
'workflowIds': result['workflowIds'],
|
||||
'workflowId': result['workflowIds'][-1] if result['workflowIds'] else None,
|
||||
'reportedCostUsd': result['reportedCostUsd'],
|
||||
'screen': 'screen.cast' if (out / 'screen.cast').exists() else None,
|
||||
'requests': 'requests.jsonl' if recorder and recorder.count else None,
|
||||
}
|
||||
(out / 'run.json').write_text(json.dumps(meta, indent=2))
|
||||
shutil.rmtree(home / '.codex' / 'shell_snapshots', ignore_errors=True)
|
||||
check_no_secrets(out, [api_key, token])
|
||||
print(json.dumps({k: meta[k] for k in ('exitCode', 'sessionId', 'workflowId', 'sessionFile', 'screen')}, indent=2))
|
||||
if meta['exitCode']:
|
||||
print(f'see {out / "stderr.log"}', flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -1,294 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Run one prompt through Instance AI on the local n8n instance.
|
||||
|
||||
The script sends the prompt to a new thread, answers every confirmation card with
|
||||
a fixed policy, waits for the thread to go idle, and exports the trace.
|
||||
|
||||
run_instance_ai.py --out <run-dir>/iai --prompt-file <run-dir>/prompt.txt
|
||||
|
||||
Approval policy (the same choices for every run, so runs are comparable):
|
||||
- approval, plan review, web search, continue: approve
|
||||
- text input: approve with "Use sensible defaults and continue."
|
||||
- questions: pick the first option, or skip a free-text question
|
||||
- domain access: allow once
|
||||
- workflow or credential setup cards: defer (the builder must not wait for real credentials)
|
||||
- local gateway resources and MCP connect cards: deny
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import queue
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from live import enabled as live_enabled
|
||||
from live import open_url, recording
|
||||
from n8n_api import N8nClient
|
||||
|
||||
DEFAULT_TEXT_ANSWER = 'Use sensible defaults and continue.'
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
class Recorder:
|
||||
"""Video of the thread in the n8n UI, made by record_iai.mjs."""
|
||||
|
||||
def __init__(self, out):
|
||||
self.proc, self.lines = None, queue.Queue()
|
||||
node = shutil.which('node')
|
||||
if not recording() or not node:
|
||||
return
|
||||
live_flag = ['--live'] if live_enabled() else []
|
||||
self.proc = subprocess.Popen([node, str(HERE / 'record_iai.mjs'), str(out.resolve()), *live_flag], cwd=HERE,
|
||||
stdin=subprocess.PIPE, stdout=subprocess.PIPE,
|
||||
stderr=(out / 'record.log').open('w'), text=True)
|
||||
threading.Thread(target=self._read, daemon=True).start()
|
||||
|
||||
def _read(self):
|
||||
for line in self.proc.stdout:
|
||||
self.lines.put(line.strip())
|
||||
|
||||
def wait_for(self, word, timeout):
|
||||
deadline = time.time() + timeout
|
||||
while self.proc and time.time() < deadline:
|
||||
try:
|
||||
if self.lines.get(timeout=min(max(deadline - time.time(), 0.1), 1)) == word:
|
||||
return True
|
||||
except queue.Empty:
|
||||
if self.proc.poll() is not None:
|
||||
return False
|
||||
return False
|
||||
|
||||
def send(self, line):
|
||||
if self.proc and self.proc.poll() is None:
|
||||
self.proc.stdin.write(line + '\n')
|
||||
self.proc.stdin.flush()
|
||||
|
||||
def stop(self):
|
||||
if not self.proc:
|
||||
return
|
||||
self.send('stop')
|
||||
self.proc.stdin.close()
|
||||
try:
|
||||
self.proc.wait(timeout=120)
|
||||
except subprocess.TimeoutExpired:
|
||||
self.proc.kill()
|
||||
|
||||
|
||||
def now_iso():
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def local_time_zone():
|
||||
zone = os.environ.get('TZ')
|
||||
if zone:
|
||||
return zone
|
||||
link = os.path.realpath('/etc/localtime')
|
||||
return link.split('zoneinfo/', 1)[1] if 'zoneinfo/' in link else 'UTC'
|
||||
|
||||
|
||||
def answer_questions(questions):
|
||||
answers = []
|
||||
for q in questions or []:
|
||||
options = q.get('options') or []
|
||||
if q.get('type') in ('single', 'multi') and options:
|
||||
answers.append({'questionId': q['id'], 'selectedOptions': [options[0]]})
|
||||
elif q.get('required'):
|
||||
answers.append({'questionId': q['id'], 'selectedOptions': [], 'customText': DEFAULT_TEXT_ANSWER})
|
||||
else:
|
||||
answers.append({'questionId': q['id'], 'selectedOptions': [], 'skipped': True})
|
||||
return {'kind': 'questions', 'answers': answers}
|
||||
|
||||
|
||||
def decide(payload):
|
||||
"""Return (confirm body, short label) for one confirmation-request payload."""
|
||||
input_type = payload.get('inputType') or 'approval'
|
||||
if payload.get('credentialDestination'):
|
||||
origin = payload['credentialDestination'].get('origin')
|
||||
return {'kind': 'credentialDestination', 'origin': origin, 'approved': False}, 'defer credential destination'
|
||||
if payload.get('setupRequests') or payload.get('credentialRequests') or payload.get('credentialFlow'):
|
||||
return {'kind': 'approval', 'approved': False}, 'defer setup'
|
||||
if payload.get('domainAccess'):
|
||||
return {'kind': 'domainAccessApprove', 'domainAccessAction': 'allow_once'}, 'allow domain once'
|
||||
if payload.get('mcpConnectRequest'):
|
||||
return {'kind': 'mcpConnect', 'approved': False}, 'deny MCP connect'
|
||||
if input_type == 'resource-decision':
|
||||
return {'kind': 'resourceDecision', 'resourceDecision': 'denyOnce'}, 'deny local resource'
|
||||
if input_type == 'questions':
|
||||
return answer_questions(payload.get('questions')), 'answer questions'
|
||||
if input_type == 'text':
|
||||
return {'kind': 'approval', 'approved': True, 'userInput': DEFAULT_TEXT_ANSWER}, 'answer text'
|
||||
return {'kind': 'approval', 'approved': True}, f'approve {input_type}'
|
||||
|
||||
|
||||
def read_stream(client, thread_id, out_queue, stop):
|
||||
try:
|
||||
res = client.open_stream(f'/rest/instance-ai/events/{thread_id}')
|
||||
except Exception as e:
|
||||
out_queue.put(('error', str(e)))
|
||||
return
|
||||
out_queue.put(('open', None))
|
||||
data_lines, event_name = [], None
|
||||
while not stop.is_set():
|
||||
raw = res.readline()
|
||||
if not raw:
|
||||
out_queue.put(('closed', None))
|
||||
return
|
||||
line = raw.decode(errors='replace').rstrip('\r\n')
|
||||
if line == '':
|
||||
if data_lines:
|
||||
try:
|
||||
out_queue.put(('event', (event_name, json.loads('\n'.join(data_lines)))))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
data_lines, event_name = [], None
|
||||
elif line.startswith('data:'):
|
||||
data_lines.append(line[5:].lstrip())
|
||||
elif line.startswith('event:'):
|
||||
event_name = line[6:].strip()
|
||||
|
||||
|
||||
def collect_workflow_ids(value, found):
|
||||
text = json.dumps(value)
|
||||
for match in re.finditer(r'"workflowId"\s*:\s*"([A-Za-z0-9]{10,})"', text):
|
||||
if match.group(1) not in found:
|
||||
found.append(match.group(1))
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--out', required=True)
|
||||
ap.add_argument('--prompt-file', required=True)
|
||||
ap.add_argument('--timeout-min', type=float, default=30)
|
||||
args = ap.parse_args()
|
||||
|
||||
out = Path(args.out)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
prompt = Path(args.prompt_file).read_text().strip()
|
||||
|
||||
# The recorder signs in while the event stream opens, so the video starts soon after the prompt.
|
||||
recorder = Recorder(out)
|
||||
client = N8nClient().login()
|
||||
thread_id = str(uuid.uuid4())
|
||||
events_file = (out / 'events.jsonl').open('w')
|
||||
q, stop = queue.Queue(), threading.Event()
|
||||
threading.Thread(target=read_stream, args=(client, thread_id, q, stop), daemon=True).start()
|
||||
|
||||
kind, detail = q.get(timeout=30)
|
||||
if kind != 'open':
|
||||
sys.exit(f'Event stream did not open: {detail}')
|
||||
|
||||
project = client.request('GET', '/rest/projects/personal')
|
||||
client.request('POST', '/rest/instance-ai/threads',
|
||||
{'threadId': thread_id, 'projectId': project['id'], 'source': 'assistant_page'})
|
||||
|
||||
if recorder.proc and not recorder.wait_for('ready', 60):
|
||||
print(f'recording failed to start, see {out / "record.log"}', flush=True)
|
||||
recorder.proc.kill()
|
||||
recorder.proc = None
|
||||
|
||||
started_at = now_iso()
|
||||
chat = client.request('POST', f'/rest/instance-ai/chat/{thread_id}',
|
||||
{'message': prompt, 'timeZone': local_time_zone()})
|
||||
print(f'thread {thread_id} run {chat.get("runId") if isinstance(chat, dict) else chat}', flush=True)
|
||||
thread_url = f'{client.base_url}/instance-ai/{thread_id}'
|
||||
# The UI shows the user message only after the chat request, so the recorder opens the thread now.
|
||||
recorder.send(f'open {thread_url}')
|
||||
print(f'watch: {thread_url}', flush=True)
|
||||
open_url(thread_url)
|
||||
|
||||
workflow_ids, decisions, langsmith_run_ids = [], [], []
|
||||
saw_finish, idle_since = False, None
|
||||
deadline = time.time() + args.timeout_min * 60
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
kind, detail = q.get(timeout=5)
|
||||
except queue.Empty:
|
||||
kind, detail = 'tick', None
|
||||
|
||||
if kind == 'event':
|
||||
_, event = detail
|
||||
events_file.write(json.dumps(event) + '\n')
|
||||
events_file.flush()
|
||||
etype, payload = event.get('type'), event.get('payload') or {}
|
||||
if etype == 'run-start':
|
||||
if payload.get('langsmithRunId'):
|
||||
langsmith_run_ids.append(payload['langsmithRunId'])
|
||||
idle_since = None
|
||||
elif etype == 'run-finish':
|
||||
saw_finish = True
|
||||
print(f'run-finish {payload.get("status", "")}', flush=True)
|
||||
elif etype == 'tool-call':
|
||||
print(f'tool {payload.get("toolName")}', flush=True)
|
||||
elif etype in ('tool-result', 'confirmation-request'):
|
||||
collect_workflow_ids(payload, workflow_ids)
|
||||
if etype == 'confirmation-request':
|
||||
body, label = decide(payload)
|
||||
try:
|
||||
client.request('POST', f'/rest/instance-ai/confirm/{payload["requestId"]}', body)
|
||||
ok = True
|
||||
except RuntimeError as e:
|
||||
ok = False
|
||||
label += f' (failed: {e})'
|
||||
decisions.append({'at': now_iso(), 'toolName': payload.get('toolName'),
|
||||
'message': payload.get('message'), 'decision': label, 'ok': ok})
|
||||
print(f'confirm: {label} [{payload.get("toolName")}]', flush=True)
|
||||
elif kind in ('closed', 'error'):
|
||||
print(f'event stream {kind}: {detail or ""}', flush=True)
|
||||
|
||||
if saw_finish:
|
||||
status = client.request('GET', f'/rest/instance-ai/threads/{thread_id}/status')
|
||||
running = [t for t in status.get('backgroundTasks', []) if t.get('status') == 'running']
|
||||
if status.get('hasActiveRun') or status.get('isSuspended') or running:
|
||||
idle_since = None
|
||||
elif idle_since is None:
|
||||
idle_since = time.time()
|
||||
elif time.time() - idle_since > 10:
|
||||
break
|
||||
else:
|
||||
print('timeout: the thread did not go idle', flush=True)
|
||||
|
||||
finished_at = now_iso()
|
||||
stop.set()
|
||||
events_file.close()
|
||||
recorder.stop()
|
||||
|
||||
trace = {}
|
||||
for key, path in (
|
||||
('rich', f'/threads/{thread_id}/messages?limit=200'),
|
||||
('raw', f'/threads/{thread_id}/messages?limit=200&raw=true'),
|
||||
('thread', f'/threads/{thread_id}'),
|
||||
('debug', f'/debug/threads/{thread_id}/runs'),
|
||||
):
|
||||
try:
|
||||
trace[key] = client.request('GET', '/rest/instance-ai' + path)
|
||||
except RuntimeError as e:
|
||||
trace[key] = None
|
||||
print(f'export {key} failed: {str(e)[:200]}', flush=True)
|
||||
(out / 'trace.json').write_text(json.dumps(trace, indent=1))
|
||||
|
||||
meta = {
|
||||
'kind': 'instance-ai',
|
||||
'threadId': thread_id,
|
||||
'startedAt': started_at,
|
||||
'finishedAt': finished_at,
|
||||
'workflowIds': workflow_ids,
|
||||
'workflowId': workflow_ids[-1] if workflow_ids else None,
|
||||
'langsmithRunIds': langsmith_run_ids,
|
||||
'decisions': decisions,
|
||||
'threadUrl': thread_url,
|
||||
'screen': 'screen.webm' if (out / 'screen.webm').exists() else None,
|
||||
}
|
||||
(out / 'run.json').write_text(json.dumps(meta, indent=2))
|
||||
print(json.dumps({k: meta[k] for k in ('threadId', 'workflowId', 'startedAt', 'finishedAt')}, indent=2))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -1,172 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Run each workflow of a run directory once for each scenario, with pinned data.
|
||||
|
||||
run_scenarios.py <run-dir>
|
||||
|
||||
Reads <run-dir>/scenarios.json (optional, see SKILL.md) and <builder>/workflow.json.
|
||||
sdk.mjs makes the pin data: triggers, service nodes and AI root nodes are pinned, and logic
|
||||
nodes (Set, IF, Code, Aggregate, ...) run for real. The n8n MCP tool test_workflow runs the
|
||||
workflow, and get_workflow_execution reads the result of each node.
|
||||
Writes <builder>/exec.json. MCP runs a workflow only when its MCP access is on, so the script
|
||||
turns it on through the REST API when necessary, records that in exec.json, and turns it off
|
||||
again after the scenarios. With several triggers, each scenario starts from the trigger that
|
||||
sdk.mjs picks (scenarios.json "triggerNode", or the trigger that reaches the "reach" types).
|
||||
"""
|
||||
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
from n8n_api import N8nClient, load_token
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
TIMEOUT_SECONDS = 90
|
||||
ITEMS_PER_NODE = 3
|
||||
|
||||
|
||||
class Mcp:
|
||||
def __init__(self, base_url, token):
|
||||
self.url, self.token = base_url + '/mcp-server/http', token
|
||||
|
||||
def call(self, tool, arguments):
|
||||
body = json.dumps({'jsonrpc': '2.0', 'id': 1, 'method': 'tools/call',
|
||||
'params': {'name': tool, 'arguments': arguments}}).encode()
|
||||
req = urllib.request.Request(self.url, data=body, method='POST', headers={
|
||||
'content-type': 'application/json',
|
||||
'accept': 'application/json, text/event-stream',
|
||||
'authorization': f'Bearer {self.token}',
|
||||
})
|
||||
with urllib.request.urlopen(req, timeout=TIMEOUT_SECONDS + 60) as res:
|
||||
raw = res.read().decode()
|
||||
lines = [line[5:].strip() for line in raw.splitlines() if line.startswith('data:')]
|
||||
reply = json.loads(lines[-1] if lines else raw)
|
||||
if 'error' in reply:
|
||||
raise RuntimeError(f'{tool}: {reply["error"].get("message")}')
|
||||
result = reply['result']
|
||||
return result.get('structuredContent') or json.loads(result['content'][0]['text'])
|
||||
|
||||
|
||||
def matches(node_type, pattern):
|
||||
short = node_type.rsplit('.', 1)[-1]
|
||||
return node_type == pattern or short.lower() == pattern.lower()
|
||||
|
||||
|
||||
def read_run(run_data, types):
|
||||
"""Per node: did it run, its error, and its first output items (all outputs)."""
|
||||
nodes = {}
|
||||
for name, runs in (run_data or {}).items():
|
||||
run = runs[-1] if runs else {}
|
||||
items = []
|
||||
for output in ((run.get('data') or {}).get('main') or []):
|
||||
items.extend(item.get('json') for item in (output or [])[:ITEMS_PER_NODE])
|
||||
error = run.get('error')
|
||||
nodes[name] = {
|
||||
'type': types.get(name),
|
||||
'status': run.get('executionStatus'),
|
||||
'error': (error.get('message') or error.get('description')) if isinstance(error, dict) else error,
|
||||
'items': items[:ITEMS_PER_NODE * 2],
|
||||
}
|
||||
return nodes
|
||||
|
||||
|
||||
def expectations(expect, nodes):
|
||||
"""Check reach and skip: node types that must run, and node types that must not run."""
|
||||
results = []
|
||||
ran = [n for n in nodes.values() if n['status'] == 'success' and not n['error']]
|
||||
for pattern in expect.get('reach') or []:
|
||||
results.append({'kind': 'reach', 'type': pattern, 'ok': any(matches(n['type'] or '', pattern) for n in ran)})
|
||||
for pattern in expect.get('skip') or []:
|
||||
results.append({'kind': 'skip', 'type': pattern,
|
||||
'ok': not any(matches(n['type'] or '', pattern) for n in nodes.values())})
|
||||
return results
|
||||
|
||||
|
||||
def set_mcp_access(client, workflow_id, on):
|
||||
"""Set MCP access for the workflow. Returns True when the script changed the setting."""
|
||||
wf = client.workflow(workflow_id)
|
||||
settings = wf.get('settings') or {}
|
||||
if bool(settings.get('availableInMCP')) == on:
|
||||
return False
|
||||
client.request('PATCH', f'/rest/workflows/{workflow_id}',
|
||||
{'versionId': wf['versionId'], 'settings': {**settings, 'availableInMCP': on}})
|
||||
return True
|
||||
|
||||
|
||||
def run_builder(sub, scenarios_file, client, mcp, node):
|
||||
meta = json.loads((sub / 'run.json').read_text())
|
||||
workflow_id = meta.get('workflowId')
|
||||
wf_file = sub / 'workflow.json'
|
||||
out = {'workflowId': workflow_id, 'scenarios': []}
|
||||
if not workflow_id or not wf_file.exists():
|
||||
out['skipped'] = 'No workflow.'
|
||||
return out
|
||||
workflow = json.loads(wf_file.read_text())
|
||||
types = {n['name']: n['type'] for n in workflow.get('nodes', [])}
|
||||
cmd = [node, str(HERE / 'sdk.mjs'), 'pin-plan', str(wf_file)] + ([str(scenarios_file)] if scenarios_file else [])
|
||||
plan = json.loads(subprocess.run(cmd, cwd=HERE, capture_output=True, text=True, check=True).stdout)
|
||||
out.update(pinned=plan['pinned'], executed=plan['executed'])
|
||||
try:
|
||||
out['mcpAccessTurnedOn'] = set_mcp_access(client, workflow_id, True)
|
||||
except Exception as e:
|
||||
out['skipped'] = f'Could not turn on MCP access: {e}'
|
||||
return out
|
||||
try:
|
||||
run_plan(plan, workflow_id, types, mcp, sub, out)
|
||||
finally:
|
||||
if out['mcpAccessTurnedOn']:
|
||||
try:
|
||||
set_mcp_access(client, workflow_id, False)
|
||||
except Exception as e:
|
||||
out['mcpAccessRestoreError'] = str(e)
|
||||
print(f'[scenarios] {sub.name}: could not turn MCP access off again: {e}', flush=True)
|
||||
return out
|
||||
|
||||
|
||||
def run_plan(plan, workflow_id, types, mcp, sub, out):
|
||||
for scenario in plan['scenarios']:
|
||||
entry = {'name': scenario['name'], 'expect': scenario['expect'], 'provenance': scenario['provenance']}
|
||||
args = {'workflowId': workflow_id, 'pinData': scenario['pinData'], 'timeout': TIMEOUT_SECONDS}
|
||||
if scenario.get('triggerNode'):
|
||||
args['triggerNodeName'] = entry['triggerNode'] = scenario['triggerNode']
|
||||
try:
|
||||
result = mcp.call('test_workflow', args)
|
||||
entry.update(status=result.get('status'), error=result.get('error'), executionId=result.get('executionId'))
|
||||
if result.get('executionId'):
|
||||
execution = mcp.call('get_workflow_execution', {'workflowId': workflow_id,
|
||||
'executionId': result['executionId'], 'includeData': True})
|
||||
run_data = ((execution.get('data') or {}).get('resultData') or {}).get('runData')
|
||||
entry['nodes'] = read_run(run_data, types)
|
||||
entry['checks'] = expectations(scenario['expect'], entry['nodes'])
|
||||
except Exception as e:
|
||||
entry.update(status='error', error=f'{type(e).__name__}: {e}')
|
||||
print(f'[scenarios] {sub.name}: {scenario["name"]} -> {entry.get("status")}'
|
||||
+ (f' ({entry["error"]})' if entry.get('error') else ''), flush=True)
|
||||
out['scenarios'].append(entry)
|
||||
|
||||
|
||||
def main():
|
||||
run_dir = Path(sys.argv[1] if len(sys.argv) > 1 else '.').resolve()
|
||||
node = shutil.which('node')
|
||||
token = load_token()
|
||||
if not node or not token:
|
||||
sys.exit('Scenarios need node on PATH and a working MCP token.')
|
||||
client = N8nClient().login()
|
||||
mcp = Mcp(client.base_url, token)
|
||||
scenarios_file = run_dir / 'scenarios.json'
|
||||
scenarios_file = scenarios_file if scenarios_file.exists() else None
|
||||
for meta_file in sorted(run_dir.glob('*/run.json')):
|
||||
sub = meta_file.parent
|
||||
try:
|
||||
out = run_builder(sub, scenarios_file, client, mcp, node)
|
||||
except Exception as e:
|
||||
out = {'skipped': f'{type(e).__name__}: {e}', 'scenarios': []}
|
||||
(sub / 'exec.json').write_text(json.dumps(out, indent=1))
|
||||
if out.get('skipped'):
|
||||
print(f'[scenarios] {sub.name}: skipped, {out["skipped"]}', flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -1,313 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
// Run each workflow of a run directory with the real node code, and record its service requests.
|
||||
//
|
||||
// node run_wire.mjs <run-dir>
|
||||
//
|
||||
// wire.mjs runs the workflow in this process. Only the trigger is pinned, and the AI root
|
||||
// nodes too, except in the real-AI scenarios. Every other node runs, and the HTTP requests of
|
||||
// the service nodes get answers in the documented shape of each API. A node that runs must be on
|
||||
// the allow-list (runsInProcess), else the scenarios that run it are skipped. Three kinds of scenario:
|
||||
// contract the scenarios of scenarios.json, with the AI output pinned as in run_scenarios.py
|
||||
// edge the edge cases of the trigger (TRIGGER_KITS in sdk.mjs) and scenarios.json "edge"
|
||||
// ai scenarios.json "questions": the AI nodes call their real model with each question
|
||||
// Writes <builder>/wire.json.
|
||||
import { existsSync, readdirSync, readFileSync, writeFileSync } from 'node:fs';
|
||||
import { join, resolve } from 'node:path';
|
||||
|
||||
import { LOGIC_TYPES, TRIGGER_KITS, graph, isAiNode, isTrigger, pinPlan, typeMatches } from './sdk.mjs';
|
||||
import { MODEL_CREDENTIALS, fillSetup, isWrite, loadEngine, modelKey, nodeCredentialTypes, runWorkflow, startNode } from './wire.mjs';
|
||||
|
||||
const ITEMS_PER_NODE = 3;
|
||||
const BODY_CHARS = 1500;
|
||||
// This process holds the settings and has no sandbox. The eval hook answers only the requests that
|
||||
// go through the n8n request helpers. Database, SSH, FTP and SMTP clients, files and a shell bypass
|
||||
// it, so only the node types below run here.
|
||||
// These nodes run code from the builder. The pinned-data runs run them in the n8n task runner.
|
||||
const CODE_NODE_TYPES = new Set([
|
||||
'n8n-nodes-base.code',
|
||||
'n8n-nodes-base.function',
|
||||
'n8n-nodes-base.functionItem',
|
||||
'n8n-nodes-base.aiTransform',
|
||||
'@n8n/n8n-nodes-langchain.code',
|
||||
'@n8n/n8n-nodes-langchain.toolCode',
|
||||
]);
|
||||
// Telegram is here because its credential has no authenticate block.
|
||||
const IN_PROCESS_TYPES = new Set([
|
||||
...[...LOGIC_TYPES].filter((type) => !CODE_NODE_TYPES.has(type)),
|
||||
'n8n-nodes-base.httpRequest',
|
||||
'n8n-nodes-base.wait',
|
||||
'n8n-nodes-base.telegram',
|
||||
]);
|
||||
// Each model node has a key in MODEL_CREDENTIALS. The other nodes call only the model or the helpers.
|
||||
const IN_PROCESS_AI_TYPES = new Set(
|
||||
[
|
||||
'agent', 'chainLlm', 'chainSummarization', 'informationExtractor', 'textClassifier', 'sentimentAnalysis',
|
||||
'openAi', 'anthropic', 'googleGemini',
|
||||
'lmChatOpenAi', 'lmChatAnthropic', 'lmChatGoogleGemini', 'lmChatMistralCloud', 'lmChatGroq',
|
||||
'outputParserStructured', 'outputParserAutofixing', 'outputParserItemList',
|
||||
'memoryBufferWindow', 'toolCalculator', 'toolThink', 'toolHttpRequest',
|
||||
].map((type) => `@n8n/n8n-nodes-langchain.${type}`),
|
||||
);
|
||||
const readJson = (file) => (existsSync(file) ? JSON.parse(readFileSync(file, 'utf8')) : null);
|
||||
|
||||
/** Read a path such as `messages[0].from` from a value. */
|
||||
export function readPath(value, path) {
|
||||
const keys = Array.isArray(path) ? path : String(path).match(/[^.[\]]+/g) ?? [];
|
||||
return keys.reduce((v, k) => (v == null ? undefined : v[/^\d+$/.test(k) ? Number(k) : k]), value);
|
||||
}
|
||||
|
||||
const requestText = (r) => JSON.stringify({ url: r.url, qs: r.qs ?? undefined, body: r.body ?? undefined });
|
||||
|
||||
function nodeSummary(runData, types) {
|
||||
const nodes = {};
|
||||
for (const [name, runs] of Object.entries(runData ?? {})) {
|
||||
const run = runs.at(-1) ?? {};
|
||||
const outputs = (run.data?.main ?? []).map((o) => o ?? []);
|
||||
const error = run.error;
|
||||
nodes[name] = {
|
||||
type: types[name],
|
||||
status: run.executionStatus,
|
||||
error: error ? error.message ?? error.description ?? String(error) : null,
|
||||
runs: runs.length,
|
||||
outputCounts: outputs.map((o) => o.length),
|
||||
items: outputs.flat().slice(0, ITEMS_PER_NODE).map((i) => i.json),
|
||||
};
|
||||
}
|
||||
return nodes;
|
||||
}
|
||||
|
||||
const trim = (value) => {
|
||||
const text = value === undefined ? undefined : JSON.stringify(value);
|
||||
return text && text.length > BODY_CHARS ? `${text.slice(0, BODY_CHARS)}…` : value;
|
||||
};
|
||||
const trimRequest = (r) => ({ ...r, body: trim(r.body), response: trim(r.response) });
|
||||
|
||||
/**
|
||||
* Resolve a marker of an expectation:
|
||||
* @trigger.<path> a value of the first trigger item, such as @trigger.messages[0].from
|
||||
* @response.<type>.<path> a value of the response to the last write request of a node type,
|
||||
* such as @response.notion.id (the id of the page that Notion created)
|
||||
* Other strings stay as they are.
|
||||
*/
|
||||
function resolveMarker(value, triggerItem, requests) {
|
||||
if (typeof value !== 'string') return value;
|
||||
if (value.startsWith('@trigger.')) return readPath(triggerItem, value.slice('@trigger.'.length));
|
||||
const response = value.match(/^@response\.([^.]+)\.(.+)$/);
|
||||
if (response) {
|
||||
const last = requests.filter((r) => isWrite(r) && typeMatches(r.nodeType, response[1])).at(-1);
|
||||
return last ? readPath(last.response, response[2]) : undefined;
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
/** Check the expectations of a scenario against the nodes that ran and the requests that they sent. */
|
||||
export function evaluate(scenario, nodes, requests, trigger, kit) {
|
||||
const checks = [];
|
||||
const expect = scenario.expect ?? {};
|
||||
const ran = Object.values(nodes).filter((n) => n.status === 'success' && !n.error);
|
||||
for (const pattern of expect.reach ?? []) {
|
||||
checks.push({ kind: 'reach', type: pattern, ok: ran.some((n) => typeMatches(n.type ?? '', pattern)) });
|
||||
}
|
||||
for (const pattern of expect.skip ?? []) {
|
||||
checks.push({ kind: 'skip', type: pattern, ok: !Object.values(nodes).some((n) => typeMatches(n.type ?? '', pattern)) });
|
||||
}
|
||||
for (const want of expect.requests ?? []) {
|
||||
// An array in contains means any one of its markers.
|
||||
const contains = (want.contains ?? []).map((c) => (Array.isArray(c) ? c : [c]).map((m) => ({ marker: m, value: resolveMarker(m, trigger[0], requests) })));
|
||||
const sent = requests.filter((r) => typeMatches(r.nodeType, want.type) && (!want.method || r.method === want.method.toUpperCase()));
|
||||
const has = (r, any) => any.some((c) => c.value !== undefined && c.value !== null && requestText(r).includes(String(c.value)));
|
||||
const matching = sent.filter((r) => contains.every((any) => has(r, any)));
|
||||
const ok = want.count === undefined ? matching.length > 0 : matching.length === want.count;
|
||||
const show = (c) => (c.marker === c.value ? `"${c.value}"` : `${c.marker} ("${c.value ?? 'not found'}")`);
|
||||
const details = sent.length
|
||||
? sent.map((r) => {
|
||||
const missing = contains.filter((any) => !has(r, any)).map((any) => any.map(show).join(' or '));
|
||||
if (missing.length) return `${r.node}: ${r.method} ${r.url} does not carry ${missing.join(', ')}`;
|
||||
const body = r.body === undefined ? '' : JSON.stringify(r.body);
|
||||
return `${r.node}: ${r.method} ${r.url}${body ? ` with \`${body.length > 300 ? `${body.slice(0, 300)}…` : body}\`` : ''}`;
|
||||
})
|
||||
: [`No ${want.method ?? ''} request from a ${want.type} node.`.replace(' ', ' ')];
|
||||
checks.push({ kind: 'request', type: want.type, contains: contains.map((any) => any.map((c) => c.marker).join(' or ')), count: want.count, ok, details });
|
||||
}
|
||||
if (expect.noWrites) {
|
||||
const writes = requests.filter(isWrite);
|
||||
checks.push({ kind: 'noWrites', ok: writes.length === 0, details: writes.map((r) => `${r.node}: ${r.method} ${r.url}`) });
|
||||
}
|
||||
if (scenario.derive === 'each-item' && kit) {
|
||||
const ids = trigger.map((item) => String(readPath(item, kit.identity)));
|
||||
const carriers = requests.filter((r) => isWrite(r) && ids.some((id) => requestText(r).includes(id)));
|
||||
const byNode = new Map();
|
||||
for (const r of carriers) byNode.set(r.node, [...(byNode.get(r.node) ?? []), r]);
|
||||
const details = [];
|
||||
let ok = byNode.size > 0;
|
||||
for (const [node, list] of byNode) {
|
||||
const reached = ids.filter((id) => list.some((r) => requestText(r).includes(id)));
|
||||
const missed = ids.filter((id) => !reached.includes(id));
|
||||
if (missed.length) ok = false;
|
||||
details.push(`${node}: ${list.length} request${list.length === 1 ? '' : 's'}, for ${reached.join(', ') || 'nobody'}${missed.length ? `, nothing for ${missed.join(', ')}` : ''}`);
|
||||
}
|
||||
if (!byNode.size) details.push(`No outgoing request names a sender (${ids.join(', ')}).`);
|
||||
checks.push({ kind: 'eachItem', ids, ok, details });
|
||||
}
|
||||
return checks;
|
||||
}
|
||||
|
||||
/** The scenarios of one workflow: contract, edge and AI, with their pin data and case rows. */
|
||||
function buildScenarios(workflow, cfg) {
|
||||
const g = graph(workflow);
|
||||
const triggerNode = g.triggers.find((t) => isTrigger(t.type));
|
||||
const kit = triggerNode ? TRIGGER_KITS[triggerNode.type] : undefined;
|
||||
const contract = cfg.scenarios?.length ? cfg.scenarios : null;
|
||||
const edges = [...(kit?.edges ?? []), ...(cfg.edge ?? [])];
|
||||
// The triggerNode of pinPlan stays: a name from scenarios.json can be missing in this workflow.
|
||||
const merge = (s, entry, kind) => ({ ...s, ...entry, triggerNode: s.triggerNode, kind });
|
||||
const list = [
|
||||
...(contract ? pinPlan(workflow, cfg, contract).scenarios.map((s, i) => merge(s, contract[i], 'contract')) : pinPlan(workflow, cfg).scenarios.map((s) => merge(s, {}, 'contract'))),
|
||||
...pinPlan(workflow, cfg, edges).scenarios.map((s, i) => merge(s, edges[i], 'edge')),
|
||||
...pinPlan(workflow, cfg, cfg.questions ?? []).scenarios.map((s, i) => merge(s, cfg.questions[i], 'ai')),
|
||||
];
|
||||
const keep = (name) => {
|
||||
const node = g.byName.get(name);
|
||||
return node && (isTrigger(node.type) || isAiNode(g, node));
|
||||
};
|
||||
return {
|
||||
g,
|
||||
scenarios: list.map((s) => {
|
||||
const pinData = Object.fromEntries(
|
||||
Object.entries(s.pinData).filter(([name]) => keep(name) && !(s.kind === 'ai' && isAiNode(g, g.byName.get(name)))),
|
||||
);
|
||||
return { ...s, pinData };
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
/** Model sub-nodes and vendor model nodes, and whether a key exists for each, for the real-AI scenarios. */
|
||||
function modelReadiness(g) {
|
||||
const subnodes = g.edges.filter((e) => e.type === 'ai_languageModel').map((e) => g.byName.get(e.from));
|
||||
const vendors = g.nodes.filter((n) => isAiNode(g, n) && !g.aiInputs(n.name, 'ai_languageModel').length);
|
||||
return [...new Set([...subnodes, ...vendors])].map((node) => {
|
||||
const types = nodeCredentialTypes(node).filter((t) => MODEL_CREDENTIALS[t]);
|
||||
const type = types.find((t) => modelKey(t)) ?? types[0];
|
||||
const p = node.parameters ?? {};
|
||||
return { node: node.name, credentialType: type ?? null, model: p.modelName ?? p.model?.value ?? p.model ?? p.modelId?.value ?? null, ready: Boolean(type && modelKey(type)) };
|
||||
});
|
||||
}
|
||||
|
||||
const isHttpCredential = (name, { credentialType, parentTypes }) =>
|
||||
credentialType(name).authenticate !== undefined || [name, ...parentTypes(name)].some((p) => p === 'oAuth1Api' || p === 'oAuth2Api');
|
||||
|
||||
/**
|
||||
* True for a node on the allow-list: the types above, declarative nodes, and nodes whose credentials
|
||||
* all authenticate HTTP requests. A tool variant (slackTool) counts as its base node.
|
||||
*/
|
||||
export function runsInProcess(node) {
|
||||
const base = node.type.startsWith('n8n-nodes-base.') && node.type.endsWith('Tool') ? node.type.slice(0, -'Tool'.length) : node.type;
|
||||
if (IN_PROCESS_TYPES.has(base) || IN_PROCESS_AI_TYPES.has(base)) return true;
|
||||
if (!base.startsWith('n8n-nodes-base.')) return false;
|
||||
const engine = loadEngine();
|
||||
try {
|
||||
const { description } = engine.nodeTypes.getByNameAndVersion(base, node.typeVersion);
|
||||
if (description.requestDefaults !== undefined) return true;
|
||||
const credentials = (description.credentials ?? []).map((c) => c.name);
|
||||
return credentials.length > 0 && credentials.every((name) => isHttpCredential(name, engine));
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The names and types of the nodes that run from the start trigger and are not on the allow-list.
|
||||
* Pinned nodes do not run. A disabled node passes its input on, so the walk goes through it.
|
||||
*/
|
||||
function outsideAllowList(workflow, g, start, pinData) {
|
||||
const seen = new Set([start]);
|
||||
const running = new Set();
|
||||
// A Set visits the entries that the loop adds.
|
||||
for (const name of seen) {
|
||||
for (const targets of workflow.connections?.[name]?.main ?? []) {
|
||||
for (const { node } of targets ?? []) {
|
||||
if (seen.has(node)) continue;
|
||||
seen.add(node);
|
||||
if (g.byName.has(node) && !Object.hasOwn(pinData, node)) running.add(node);
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const name of running) {
|
||||
for (const e of g.edges) if (e.to === name && e.type.startsWith('ai_')) running.add(e.from);
|
||||
}
|
||||
return [...running].map((name) => g.byName.get(name)).filter((n) => !runsInProcess(n)).map((n) => `${n.name} (${n.type})`);
|
||||
}
|
||||
|
||||
async function runBuilder(dir, cfg) {
|
||||
const workflow = readJson(join(dir, 'workflow.json'));
|
||||
const { g, scenarios } = buildScenarios(workflow, cfg);
|
||||
const canvas = readJson(join(dir, 'canvas-issues.json'));
|
||||
const { workflow: filledWorkflow, filled } = fillSetup(workflow, canvas);
|
||||
const types = Object.fromEntries((workflow.nodes ?? []).map((n) => [n.name, n.type]));
|
||||
const models = modelReadiness(g);
|
||||
const out = { filled, models, scenarios: [] };
|
||||
for (const scenario of scenarios) {
|
||||
const entry = { name: scenario.name, kind: scenario.kind, why: scenario.why, expect: scenario.expect ?? {}, derive: scenario.derive, pinned: Object.keys(scenario.pinData) };
|
||||
if (scenario.kind === 'ai' && !models.length) {
|
||||
out.scenarios.push({ ...entry, status: 'skipped', error: 'The workflow has no model sub-node.' });
|
||||
continue;
|
||||
}
|
||||
const start = startNode(workflow, scenario.pinData, scenario.triggerNode);
|
||||
const outside = start ? outsideAllowList(workflow, g, start.name, scenario.pinData) : [];
|
||||
if (outside.length) {
|
||||
out.scenarios.push({ ...entry, status: 'skipped', error: `Nodes outside the in-process allow-list: ${outside.join(', ')}.` });
|
||||
continue;
|
||||
}
|
||||
if (scenario.kind === 'ai' && models.some((m) => !m.ready)) {
|
||||
const missing = models.filter((m) => !m.ready).map((m) => `${m.node} (${m.credentialType ?? 'unknown credential'}: set ${MODEL_CREDENTIALS[m.credentialType]?.keys[0] ?? 'a supported model'})`);
|
||||
out.scenarios.push({ ...entry, status: 'skipped', error: `No model key for ${missing.join(', ')}.` });
|
||||
continue;
|
||||
}
|
||||
const rows = (node) => Object.entries(scenario.byType ?? {}).find(([pattern]) => typeMatches(node.type, pattern))?.[1];
|
||||
const trigger = start ? scenario.pinData[start.name].map((i) => i.json) : [];
|
||||
const kit = start ? TRIGGER_KITS[start.type] : undefined;
|
||||
const result = await runWorkflow(filledWorkflow, { pinData: scenario.pinData, triggerNode: scenario.triggerNode, rows, realModels: scenario.kind === 'ai' });
|
||||
const nodes = nodeSummary(result.runData, types);
|
||||
const errors = Object.entries(nodes).filter(([, n]) => n.error).map(([name, n]) => `${name}: ${n.error}`);
|
||||
Object.assign(entry, {
|
||||
status: result.status,
|
||||
error: result.error,
|
||||
ms: result.ms,
|
||||
nodes,
|
||||
requests: result.requests.map(trimRequest),
|
||||
blocked: result.blocked,
|
||||
checks: evaluate(scenario, nodes, result.requests, trigger, kit),
|
||||
});
|
||||
if (scenario.kind === 'ai') {
|
||||
entry.aiOutputs = Object.fromEntries(Object.entries(nodes).filter(([name]) => isAiNode(g, g.byName.get(name))).map(([name, n]) => [name, n.items]));
|
||||
}
|
||||
console.log(`[wire] ${dir.split('/').pop()}: ${scenario.kind} "${scenario.name}" -> ${result.status}${errors.length ? ` (${errors[0]})` : result.error ? ` (${result.error})` : ''}`);
|
||||
out.scenarios.push(entry);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
if (process.argv[1] && resolve(process.argv[1]) === new URL(import.meta.url).pathname) {
|
||||
const runDir = resolve(process.argv[2] ?? '.');
|
||||
const loaded = loadEngine();
|
||||
const cfg = readJson(join(runDir, 'scenarios.json')) ?? {};
|
||||
for (const entry of readdirSync(runDir, { withFileTypes: true })) {
|
||||
const dir = join(runDir, entry.name);
|
||||
if (!entry.isDirectory() || !existsSync(join(dir, 'workflow.json'))) continue;
|
||||
let out;
|
||||
if (loaded.error) out = { skipped: loaded.error, scenarios: [] };
|
||||
else {
|
||||
try {
|
||||
out = await runBuilder(dir, cfg);
|
||||
} catch (error) {
|
||||
out = { skipped: `${error?.name ?? 'Error'}: ${error?.message ?? error}`, scenarios: [] };
|
||||
}
|
||||
}
|
||||
// A failed real model request can echo its URL, with the key, in the error message.
|
||||
const keys = Object.keys(MODEL_CREDENTIALS).map(modelKey).filter(Boolean);
|
||||
writeFileSync(join(dir, 'wire.json'), keys.reduce((text, key) => text.replaceAll(key, '[redacted]'), JSON.stringify(out, null, 1)));
|
||||
if (out.skipped) console.log(`[wire] ${entry.name}: skipped, ${out.skipped}`);
|
||||
}
|
||||
// Model SDKs can keep sockets open.
|
||||
process.exit(0);
|
||||
}
|
||||
|
|
@ -1,600 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
// Shared workflow knowledge for derive_checks.mjs and run_scenarios.py: the workflow SDK of
|
||||
// the repository, graph helpers, expression references, JSON Schema samples and pin data plans.
|
||||
//
|
||||
// node sdk.mjs pin-plan <workflow.json> [scenarios.json] print the pin data of each scenario
|
||||
import { createRequire } from 'node:module';
|
||||
import { existsSync, readFileSync } from 'node:fs';
|
||||
import { dirname, join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const here = dirname(fileURLToPath(import.meta.url));
|
||||
export const repo = resolve(here, '../../../..');
|
||||
const require = createRequire(import.meta.url);
|
||||
const SDK_DIST = join(repo, 'packages/@n8n/workflow-sdk/dist/index.js');
|
||||
// The same directories that the n8n node catalog gives to setSchemaBaseDirs().
|
||||
const NODE_DEFINITION_DIRS = [
|
||||
join(repo, 'packages/nodes-base/dist/node-definitions/'),
|
||||
join(repo, 'packages/@n8n/nodes-langchain/dist/node-definitions/'),
|
||||
];
|
||||
|
||||
let loaded;
|
||||
/** The workflow SDK with the node parameter schemas set, or null when the package is not built. */
|
||||
export function loadSdk() {
|
||||
if (loaded !== undefined) return loaded;
|
||||
if (!existsSync(SDK_DIST)) return (loaded = null);
|
||||
const sdk = require(SDK_DIST);
|
||||
const schemaDirs = NODE_DEFINITION_DIRS.filter((dir) => existsSync(dir));
|
||||
// Without schema directories validateNodeConfig reports every node as valid.
|
||||
sdk.setSchemaBaseDirs(schemaDirs);
|
||||
return (loaded = { sdk, schemaDirs, missingDirs: NODE_DEFINITION_DIRS.filter((d) => !existsSync(d)) });
|
||||
}
|
||||
|
||||
// --- Node kinds ----------------------------------------------------------------
|
||||
|
||||
const BASE = 'n8n-nodes-base.';
|
||||
const LC = '@n8n/n8n-nodes-langchain.';
|
||||
|
||||
/** Nodes that run for real in a scenario. They need no credentials and call no service. */
|
||||
export const LOGIC_TYPES = new Set(
|
||||
[
|
||||
'set', 'if', 'switch', 'filter', 'merge', 'aggregate', 'splitOut', 'code', 'function', 'functionItem',
|
||||
'noOp', 'splitInBatches', 'itemLists', 'sort', 'limit', 'removeDuplicates', 'summarize', 'dateTime',
|
||||
'renameKeys', 'compareDatasets', 'markdown', 'html', 'xml', 'crypto', 'executionData', 'stopAndError',
|
||||
'respondToWebhook', 'convertToFile', 'extractFromFile', 'debugHelper',
|
||||
].map((t) => BASE + t),
|
||||
);
|
||||
|
||||
/** Nodes that end up with one item, so `.first()` on them reads all of their output. */
|
||||
export const SINGLE_ITEM_TYPES = new Set(['aggregate', 'summarize'].map((t) => BASE + t));
|
||||
|
||||
const TRIGGER_TYPES = new Set(['webhook', 'cron', 'interval', 'emailReadImap', 'start'].map((t) => BASE + t));
|
||||
|
||||
export const isSticky = (type) => type === `${BASE}stickyNote`;
|
||||
export const isTrigger = (type) => TRIGGER_TYPES.has(type) || type.toLowerCase().includes('trigger');
|
||||
export const isManualTrigger = (type) => type === `${BASE}manualTrigger`;
|
||||
export const shortType = (type) => type.slice(type.lastIndexOf('.') + 1);
|
||||
|
||||
/** True when type matches a full type name, or its part after the last dot (case-insensitive). */
|
||||
export function typeMatches(type, pattern) {
|
||||
return type === pattern || shortType(type).toLowerCase() === pattern.toLowerCase();
|
||||
}
|
||||
|
||||
const AI_ROOT_PLACEHOLDER = {
|
||||
[`${LC}agent`]: (v) => ({ output: v }),
|
||||
[`${LC}openAiAssistant`]: (v) => ({ output: v }),
|
||||
[`${LC}informationExtractor`]: (v) => ({ output: v }),
|
||||
[`${LC}chainLlm`]: (v) => ({ text: v }),
|
||||
[`${LC}chainRetrievalQa`]: (v) => ({ response: v }),
|
||||
[`${LC}chainSummarization`]: (v) => ({ output: { output_text: v } }),
|
||||
[`${LC}sentimentAnalysis`]: (v) => ({ sentimentAnalysis: { category: v } }),
|
||||
[`${LC}googleGemini`]: (v) => ({ content: { parts: [{ text: v }], role: 'model' }, finishReason: 'STOP', index: 0 }),
|
||||
};
|
||||
|
||||
/** Vendor nodes that call a model themselves, without a model sub-node. */
|
||||
const AI_VENDOR_TYPES = new Set(['googleGemini', 'openAi', 'anthropic'].map((t) => LC + t));
|
||||
|
||||
// --- Graph ---------------------------------------------------------------------
|
||||
|
||||
/** Nodes, edges and lookups of a workflow. Sticky notes and disabled nodes are left out. */
|
||||
export function graph(workflow) {
|
||||
const nodes = (workflow.nodes ?? []).filter((n) => !isSticky(n.type) && !n.disabled);
|
||||
const byName = new Map(nodes.map((n) => [n.name, n]));
|
||||
const edges = [];
|
||||
for (const [from, outputs] of Object.entries(workflow.connections ?? {})) {
|
||||
for (const [type, branches] of Object.entries(outputs ?? {})) {
|
||||
(branches ?? []).forEach((targets, output) =>
|
||||
(targets ?? []).forEach((t) => {
|
||||
if (byName.has(from) && byName.has(t.node))
|
||||
edges.push({ from, to: t.node, type, output, input: t.index ?? 0 });
|
||||
}),
|
||||
);
|
||||
}
|
||||
}
|
||||
const subnodes = new Set(edges.filter((e) => e.type.startsWith('ai_')).map((e) => e.from));
|
||||
const mainParents = (name) => [...new Set(edges.filter((e) => e.type === 'main' && e.to === name).map((e) => e.from))];
|
||||
const mainChildren = (name) => edges.filter((e) => e.type === 'main' && e.from === name);
|
||||
const aiInputs = (name, type) => edges.filter((e) => e.to === name && e.type === type).map((e) => byName.get(e.from));
|
||||
const triggers = nodes.filter((n) => isTrigger(n.type) && !subnodes.has(n.name));
|
||||
return { nodes, byName, edges, subnodes, mainParents, mainChildren, aiInputs, triggers };
|
||||
}
|
||||
|
||||
/** Names of the nodes that a trigger reaches. AI sub-nodes count when they serve a reached node. */
|
||||
export function reachable(g) {
|
||||
const seen = new Set(g.triggers.map((t) => t.name));
|
||||
const queue = [...seen];
|
||||
while (queue.length) {
|
||||
const name = queue.shift();
|
||||
for (const e of g.edges) {
|
||||
if (e.type === 'main' && e.from === name && !seen.has(e.to)) {
|
||||
seen.add(e.to);
|
||||
queue.push(e.to);
|
||||
}
|
||||
}
|
||||
}
|
||||
// A sub-node can serve another sub-node, for example a model inside an auto-fixing parser.
|
||||
let grew = true;
|
||||
while (grew) {
|
||||
grew = false;
|
||||
for (const e of g.edges) {
|
||||
if (e.type.startsWith('ai_') && seen.has(e.to) && !seen.has(e.from)) {
|
||||
seen.add(e.from);
|
||||
grew = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
return seen;
|
||||
}
|
||||
|
||||
/** Names of the nodes downstream of name, through main connections. */
|
||||
export function downstream(g, name) {
|
||||
const seen = new Set();
|
||||
const queue = [name];
|
||||
while (queue.length) {
|
||||
for (const e of g.mainChildren(queue.shift())) {
|
||||
if (!seen.has(e.to)) {
|
||||
seen.add(e.to);
|
||||
queue.push(e.to);
|
||||
}
|
||||
}
|
||||
}
|
||||
return seen;
|
||||
}
|
||||
|
||||
// --- Parameters and expressions --------------------------------------------------
|
||||
|
||||
/** Every string in a parameter tree, with its dotted path. */
|
||||
export function strings(value, path = '') {
|
||||
if (typeof value === 'string') return [{ path, value }];
|
||||
if (Array.isArray(value)) return value.flatMap((v, i) => strings(v, `${path}[${i}]`));
|
||||
if (value && typeof value === 'object')
|
||||
return Object.entries(value).flatMap(([k, v]) => strings(v, path ? `${path}.${k}` : k));
|
||||
return [];
|
||||
}
|
||||
|
||||
const CODE_PARAMS = new Set(['jsCode', 'pythonCode', 'functionCode', 'code']);
|
||||
|
||||
/** The expression text of a parameter value, or null. Code fields count, because they use the same references. */
|
||||
export function expressionText(path, value) {
|
||||
if (value.startsWith('=')) return value.slice(1);
|
||||
const key = path.split('.').pop().replace(/\[\d+\]$/, '');
|
||||
return CODE_PARAMS.has(key) ? value : null;
|
||||
}
|
||||
|
||||
const IDENT = /^[A-Za-z_$][\w$]*/;
|
||||
|
||||
/** Read a property path such as `.a?.b['c'][0]` from text at index i. Returns { path, end, call }. */
|
||||
function readPath(text, i) {
|
||||
const path = [];
|
||||
let dynamic = false;
|
||||
for (;;) {
|
||||
if (text.startsWith('?.', i)) i += 2;
|
||||
else if (text[i] === '.') i += 1;
|
||||
else if (text[i] !== '[') break;
|
||||
if (text[i] === '[') {
|
||||
const close = text.indexOf(']', i);
|
||||
if (close < 0) break;
|
||||
const inner = text.slice(i + 1, close).trim();
|
||||
if (/^\d+$/.test(inner)) path.push(Number(inner));
|
||||
else if (/^(['"`])(.*)\1$/.test(inner)) path.push(inner.slice(1, -1));
|
||||
else {
|
||||
dynamic = true;
|
||||
i = close + 1;
|
||||
break;
|
||||
}
|
||||
i = close + 1;
|
||||
continue;
|
||||
}
|
||||
const match = text.slice(i).match(IDENT);
|
||||
if (!match) break;
|
||||
path.push(match[0]);
|
||||
i += match[0].length;
|
||||
}
|
||||
const call = /^\s*\(/.test(text.slice(i));
|
||||
// A trailing method call, such as `.toUpperCase()`, is not a field.
|
||||
if (call && path.length) path.pop();
|
||||
return { path, end: i, call, dynamic };
|
||||
}
|
||||
|
||||
const ACCESSORS = [
|
||||
{ re: /^\.item\b/, kind: 'item' },
|
||||
{ re: /^\.first\(\s*\)/, kind: 'first' },
|
||||
{ re: /^\.last\(\s*\)/, kind: 'last' },
|
||||
{ re: /^\.all\(\s*\)\[\s*\d+\s*\]/, kind: 'all-index' },
|
||||
{ re: /^\.all\(\s*\)/, kind: 'all' },
|
||||
{ re: /^\.itemMatching\([^)]*\)/, kind: 'item' },
|
||||
{ re: /^\.pairedItem\([^)]*\)/, kind: 'item' },
|
||||
];
|
||||
|
||||
/**
|
||||
* References in expression text: { node, kind, accessor, path, raw }.
|
||||
* node is null for `$json` and `$input`, which read the main input of the node.
|
||||
* path is null when the reference does not read item JSON, for example `$('A').params`.
|
||||
*/
|
||||
export function references(text) {
|
||||
const refs = [];
|
||||
const named = /\$\(\s*(['"`])((?:\\.|(?!\1).)+)\1\s*\)|\$node\[\s*(['"`])((?:\\.|(?!\3).)+)\3\s*\]|\$items\(\s*(['"`])((?:\\.|(?!\5).)+)\5/g;
|
||||
for (const m of text.matchAll(named)) {
|
||||
const node = (m[2] ?? m[4] ?? m[6]).replace(/\\(.)/g, '$1');
|
||||
let i = m.index + m[0].length;
|
||||
let accessor = m[4] !== undefined ? 'node' : m[6] !== undefined ? 'items' : null;
|
||||
if (accessor === null) {
|
||||
const rest = text.slice(i);
|
||||
const hit = ACCESSORS.find((a) => a.re.test(rest));
|
||||
if (hit) {
|
||||
accessor = hit.kind;
|
||||
i += rest.match(hit.re)[0].length;
|
||||
}
|
||||
}
|
||||
let path = null;
|
||||
if (text.startsWith('.json', i) && accessor !== 'all' && accessor !== 'items') {
|
||||
const read = readPath(text, i + '.json'.length);
|
||||
path = read.dynamic ? null : read.path;
|
||||
}
|
||||
refs.push({ node, kind: 'named', accessor, path, raw: text.slice(m.index, i + 40).split(/[}\n]/)[0] });
|
||||
}
|
||||
for (const m of text.matchAll(/\$json\b/g)) {
|
||||
const read = readPath(text, m.index + '$json'.length);
|
||||
refs.push({ node: null, kind: 'json', accessor: 'item', path: read.dynamic ? null : read.path,
|
||||
raw: text.slice(m.index, read.end) });
|
||||
}
|
||||
for (const m of text.matchAll(/\$input\.(item|first\(\s*\)|last\(\s*\)|all\(\s*\)\[\s*\d+\s*\])\.json\b/g)) {
|
||||
const read = readPath(text, m.index + m[0].length);
|
||||
const accessor = m[1].startsWith('first') ? 'first' : m[1].startsWith('last') ? 'last' : m[1] === 'item' ? 'item' : 'all-index';
|
||||
refs.push({ node: null, kind: 'input', accessor, path: read.dynamic ? null : read.path,
|
||||
raw: text.slice(m.index, read.end) });
|
||||
}
|
||||
return refs;
|
||||
}
|
||||
|
||||
/** Every reference in a workflow: { node (the node that holds it), param, ref, code }. */
|
||||
export function workflowReferences(g) {
|
||||
const out = [];
|
||||
for (const node of g.nodes) {
|
||||
for (const { path, value } of strings(node.parameters ?? {})) {
|
||||
const text = expressionText(path, value);
|
||||
if (text === null) continue;
|
||||
const code = !value.startsWith('=');
|
||||
for (const ref of references(text)) out.push({ node: node.name, param: path, ref, code });
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Look up a path in a value. Returns 'found', 'missing' or 'unknown'.
|
||||
* 'unknown' means that a null or an opaque value hides the rest of the path. With sample, the
|
||||
* value comes from a schema, which shows the shape of one array item and not the count, so an
|
||||
* index past the end of an array is 'unknown'. In real output, that index is 'missing'.
|
||||
*/
|
||||
export function lookup(value, path, { sample = false } = {}) {
|
||||
let current = value;
|
||||
for (const key of path) {
|
||||
if (current === null || current === undefined) return current === null ? 'unknown' : 'missing';
|
||||
if (Array.isArray(current)) {
|
||||
if (typeof key === 'number') {
|
||||
if (key >= current.length) return sample ? 'unknown' : 'missing';
|
||||
current = current[key];
|
||||
} else if (key === 'length') return 'found';
|
||||
else return 'missing';
|
||||
continue;
|
||||
}
|
||||
if (typeof current === 'string') return key === 'length' ? 'found' : 'missing';
|
||||
if (typeof current !== 'object') return 'missing';
|
||||
if (!(key in current)) return 'missing';
|
||||
current = current[key];
|
||||
}
|
||||
return current === undefined ? 'missing' : 'found';
|
||||
}
|
||||
|
||||
export const showPath = (path) =>
|
||||
path.map((k, i) => (typeof k === 'number' ? `[${k}]` : /^[A-Za-z_$][\w$]*$/.test(k) ? (i ? `.${k}` : k) : `['${k}']`)).join('');
|
||||
|
||||
// --- JSON Schema samples ---------------------------------------------------------
|
||||
|
||||
/** A deterministic sample value for a JSON Schema. key names the field, for readable strings. */
|
||||
export function sample(schema, key = 'value', depth = 0) {
|
||||
if (!schema || typeof schema !== 'object') return null;
|
||||
if (schema.example !== undefined) return schema.example;
|
||||
if (schema.const !== undefined) return schema.const;
|
||||
if (Array.isArray(schema.enum) && schema.enum.length) return schema.enum[0];
|
||||
if (schema.default !== undefined) return schema.default;
|
||||
const variant = schema.anyOf ?? schema.oneOf;
|
||||
if (Array.isArray(variant) && variant.length) return sample(variant.find((v) => v.type !== 'null') ?? variant[0], key, depth);
|
||||
let type = Array.isArray(schema.type) ? schema.type.find((t) => t !== 'null') : schema.type;
|
||||
if (!type) type = schema.properties ? 'object' : schema.items ? 'array' : 'string';
|
||||
switch (type) {
|
||||
case 'object': {
|
||||
if (depth > 6) return {};
|
||||
return Object.fromEntries(
|
||||
Object.entries(schema.properties ?? {}).map(([k, v]) => [k, sample(v, k, depth + 1)]),
|
||||
);
|
||||
}
|
||||
case 'array':
|
||||
return schema.items && depth <= 6 ? [sample(schema.items, key, depth + 1)] : [];
|
||||
case 'integer':
|
||||
case 'number':
|
||||
return 1;
|
||||
case 'boolean':
|
||||
return true;
|
||||
case 'null':
|
||||
return null;
|
||||
default: {
|
||||
const k = String(key);
|
||||
if (schema.format === 'date-time' || /(^|_)(date|time|at)$|At$/.test(k)) return '2026-01-01T09:00:00.000Z';
|
||||
if (schema.format === 'email' || /email/i.test(k)) return 'ada@example.com';
|
||||
if (schema.format === 'uri' || /url|link|href/i.test(k)) return `https://example.com/${k}`;
|
||||
if (/(^|_)id$|Id$/.test(k)) return `${k}-1`;
|
||||
return `sample ${k}`;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Set every boolean in a value to flag. */
|
||||
function setBooleans(value, flag) {
|
||||
if (Array.isArray(value)) return value.map((v) => setBooleans(v, flag));
|
||||
if (value && typeof value === 'object')
|
||||
return Object.fromEntries(Object.entries(value).map(([k, v]) => [k, setBooleans(v, flag)]));
|
||||
return typeof value === 'boolean' ? flag : value;
|
||||
}
|
||||
|
||||
function deepMerge(base, extra) {
|
||||
if (!base || typeof base !== 'object' || Array.isArray(base) || !extra || typeof extra !== 'object') return extra;
|
||||
const out = { ...base };
|
||||
for (const [k, v] of Object.entries(extra)) out[k] = deepMerge(base[k], v);
|
||||
return out;
|
||||
}
|
||||
|
||||
// --- Pin data ------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Trigger output for a scenario, in the shape that the trigger code emits.
|
||||
* WhatsApp: WhatsAppTrigger.node.ts emits `{ ...change.value, field }` for each change,
|
||||
* and change.value is the Cloud API webhook payload of a text message.
|
||||
*/
|
||||
export const TRIGGER_FIXTURES = {
|
||||
[`${BASE}whatsAppTrigger`]: [{
|
||||
messaging_product: 'whatsapp',
|
||||
metadata: { display_phone_number: '15550001111', phone_number_id: '106540352242922' },
|
||||
contacts: [{ profile: { name: 'Ada Lovelace' }, wa_id: '4915112345678' }],
|
||||
messages: [{ from: '4915112345678', id: 'wamid.HBgLNDkxNTExMjM0NTY3OBUCABIYFjNFQjA', timestamp: '1767258000',
|
||||
text: { body: 'What are your opening hours?' }, type: 'text' }],
|
||||
field: 'messages',
|
||||
}],
|
||||
[`${BASE}webhook`]: [{ headers: { 'content-type': 'application/json' }, params: {}, query: {},
|
||||
body: { message: 'Hello' }, webhookUrl: 'http://localhost:5678/webhook/test', executionMode: 'test' }],
|
||||
[`${LC}chatTrigger`]: [{ sessionId: 'session-1', action: 'sendMessage', chatInput: 'Hello' }],
|
||||
[`${BASE}manualTrigger`]: [{}],
|
||||
[`${BASE}scheduleTrigger`]: [{ timestamp: '2026-01-01T09:00:00.000+00:00', 'Readable date': 'January 1st 2026, 9:00:00 am',
|
||||
'Day of week': 'Thursday', Year: '2026', Month: 'January', 'Day of month': '01', Hour: '09', Minute: '00',
|
||||
Second: '00', Timezone: 'UTC' }],
|
||||
[`${BASE}telegramTrigger`]: [{ update_id: 1, message: { message_id: 1, date: 1767258000, text: 'Hello',
|
||||
chat: { id: 1001, type: 'private', first_name: 'Ada' }, from: { id: 1001, is_bot: false, first_name: 'Ada' } } }],
|
||||
[`${BASE}formTrigger`]: [{ submittedAt: '2026-01-01T09:00:00.000Z', formMode: 'test' }],
|
||||
};
|
||||
|
||||
const whatsAppMessage = (from, name, body, id) => ({
|
||||
messaging_product: 'whatsapp',
|
||||
metadata: { display_phone_number: '15550001111', phone_number_id: '106540352242922' },
|
||||
contacts: [{ profile: { name }, wa_id: from }],
|
||||
messages: [{ from, id, timestamp: '1767258000', text: { body }, type: 'text' }],
|
||||
field: 'messages',
|
||||
});
|
||||
|
||||
/**
|
||||
* Per trigger type: identity is the path of the value that names the sender in one item,
|
||||
* withMessage puts a scenario text into the fixture, and edges are the edge cases that every
|
||||
* workflow with this trigger must survive. derive 'each-item' means that each trigger item
|
||||
* must get its own outgoing request that names its sender.
|
||||
*/
|
||||
export const TRIGGER_KITS = {
|
||||
[`${BASE}whatsAppTrigger`]: {
|
||||
identity: ['messages', 0, 'from'],
|
||||
withMessage: (items, text) => items.map((i) => ({ ...i, messages: (i.messages ?? []).map((m) => ({ ...m, text: { body: text } })) })),
|
||||
edges: [
|
||||
{
|
||||
name: 'Two messages arrive in one webhook call',
|
||||
why: 'WhatsAppTrigger.node.ts emits one item for each change, and Meta can batch several changes in one call',
|
||||
derive: 'each-item',
|
||||
ai: { booleans: true },
|
||||
trigger: [
|
||||
whatsAppMessage('4915112345678', 'Ada Lovelace', 'What are your opening hours?', 'wamid.A1'),
|
||||
whatsAppMessage('4917699887766', 'Alan Turing', 'Do you ship abroad?', 'wamid.B2'),
|
||||
],
|
||||
},
|
||||
{
|
||||
name: 'A delivery status update arrives, not a customer message',
|
||||
why: 'the messages update also delivers sent, delivered and read statuses, with no messages array',
|
||||
expect: { noWrites: true },
|
||||
trigger: [{
|
||||
messaging_product: 'whatsapp',
|
||||
metadata: { display_phone_number: '15550001111', phone_number_id: '106540352242922' },
|
||||
statuses: [{ id: 'wamid.HBgLNDkxNTExMjM0NTY3OBUCABEYEjQ', status: 'delivered', timestamp: '1767258060', recipient_id: '4915112345678' }],
|
||||
field: 'messages',
|
||||
}],
|
||||
},
|
||||
],
|
||||
},
|
||||
};
|
||||
|
||||
/** The trigger items of a scenario: its own trigger, the case trigger, or the fixture with the scenario message. */
|
||||
export function triggerItems(node, scenario = {}, scenarios = {}) {
|
||||
const explicit = scenario.trigger ?? scenarios.trigger;
|
||||
if (explicit) return { items: Array.isArray(explicit) ? explicit : [explicit], source: 'case' };
|
||||
const fixture = TRIGGER_FIXTURES[node.type];
|
||||
if (!fixture) return null;
|
||||
const kit = TRIGGER_KITS[node.type];
|
||||
if (scenario.message !== undefined && kit) return { items: kit.withMessage(fixture, scenario.message), source: 'case' };
|
||||
return { items: fixture, source: 'fixture' };
|
||||
}
|
||||
|
||||
export const isAiRootNode = (g, node) => node.type.startsWith(LC) && !g.subnodes.has(node.name) && g.aiInputs(node.name, 'ai_languageModel').length > 0;
|
||||
|
||||
/** A node whose output comes from a model: an AI root node or a vendor node such as Google Gemini. */
|
||||
export const isAiNode = (g, node) => Boolean(node) && (isAiRootNode(g, node) || (AI_VENDOR_TYPES.has(node.type) && !g.subnodes.has(node.name)));
|
||||
|
||||
function parserSchema(parser) {
|
||||
const p = parser?.parameters ?? {};
|
||||
const parse = (text) => {
|
||||
try {
|
||||
return typeof text === 'string' ? JSON.parse(text) : text && typeof text === 'object' ? text : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
};
|
||||
const schemaType = p.schemaType ?? 'fromJson';
|
||||
if (schemaType === 'manual' && parse(p.inputSchema)) return { kind: 'schema', value: parse(p.inputSchema) };
|
||||
if (schemaType !== 'manual' && parse(p.jsonSchemaExample)) return { kind: 'example', value: parse(p.jsonSchemaExample) };
|
||||
if (parse(p.jsonSchema)) return { kind: 'schema', value: parse(p.jsonSchema) };
|
||||
return null;
|
||||
}
|
||||
|
||||
/** JSON objects that a prompt shows as examples, in order. Objects with a boolean come first. */
|
||||
function promptExamples(node) {
|
||||
const found = [];
|
||||
for (const { value } of strings(node.parameters ?? {})) {
|
||||
for (let start = value.indexOf('{'); start >= 0; start = value.indexOf('{', start + 1)) {
|
||||
let depth = 0;
|
||||
for (let i = start; i < value.length; i++) {
|
||||
if (value[i] === '{') depth++;
|
||||
else if (value[i] === '}' && --depth === 0) {
|
||||
try {
|
||||
const parsed = JSON.parse(value.slice(start, i + 1));
|
||||
if (parsed && typeof parsed === 'object' && !Array.isArray(parsed) && Object.keys(parsed).length)
|
||||
found.push(parsed);
|
||||
} catch {}
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return found.sort((a, b) => Number(hasBoolean(b)) - Number(hasBoolean(a)));
|
||||
}
|
||||
|
||||
const hasBoolean = (value) => Object.values(value).some((v) => typeof v === 'boolean');
|
||||
|
||||
/** Apply the ai settings of a scenario to a model result: every boolean, then explicit fields. */
|
||||
function applyAi(value, ai) {
|
||||
let out = typeof ai.booleans === 'boolean' ? setBooleans(value, ai.booleans) : value;
|
||||
if (ai.set && out && typeof out === 'object') out = deepMerge(out, ai.set);
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* The output of an AI root node for a scenario. Returns { json, provenance }.
|
||||
* With a structured output parser, the result follows the parser schema. Without one, the
|
||||
* model returns text: ai.text when the scenario sets it, else the first JSON example in the
|
||||
* prompt of the node (the contract that the builder declared), else a plain sentence.
|
||||
*/
|
||||
function aiRootOutput(g, node, ai = {}) {
|
||||
const parserNode = g.aiInputs(node.name, 'ai_outputParser')[0];
|
||||
const parser = parserNode ? parserSchema(parserNode) : null;
|
||||
if (parser) {
|
||||
const parsed = parser.kind === 'schema' ? sample(parser.value, 'output') : parser.value;
|
||||
return { json: { output: applyAi(parsed, ai) }, provenance: 'output-parser' };
|
||||
}
|
||||
const wrap = AI_ROOT_PLACEHOLDER[node.type] ?? ((v) => ({ output: v }));
|
||||
if (ai.text !== undefined) return { json: wrap(ai.text), provenance: 'case' };
|
||||
const example = promptExamples(node)[0];
|
||||
if (example) return { json: wrap(JSON.stringify(applyAi(example, ai))), provenance: 'prompt-example' };
|
||||
return { json: wrap('Sample answer from the model.'), provenance: AI_ROOT_PLACEHOLDER[node.type] ? 'ai-root' : 'placeholder' };
|
||||
}
|
||||
|
||||
/**
|
||||
* The pin data of each scenario. Logic nodes run for real. Triggers, service nodes and AI root
|
||||
* nodes get pin data, with the provenance of each node, so a check knows how far to trust it:
|
||||
* fixture, case the shape of real data (trigger code, or the case file)
|
||||
* node-schema the __schema__ file of the node
|
||||
* output-parser the schema of the attached structured output parser
|
||||
* prompt-example a JSON example in the prompt of an AI root node without a parser
|
||||
* ai-root the known output envelope of the AI root node
|
||||
* placeholder, empty nothing describes the output, so the shape is a guess or one empty item
|
||||
*/
|
||||
export function pinPlan(workflow, scenarios = {}, only = null) {
|
||||
const g = graph(workflow);
|
||||
const sdk = loadSdk()?.sdk;
|
||||
const list = only ?? (scenarios.scenarios?.length ? scenarios.scenarios : defaultScenarios(g));
|
||||
const plan = { pinned: {}, executed: [], subnodes: [...g.subnodes], scenarios: [] };
|
||||
for (const node of g.nodes) {
|
||||
if (!g.subnodes.has(node.name) && LOGIC_TYPES.has(node.type)) plan.executed.push(node.name);
|
||||
}
|
||||
for (const scenario of list) {
|
||||
const pinData = {};
|
||||
const provenance = {};
|
||||
const byType = mergeByType(scenarios.pinByType, scenario.pinByType);
|
||||
for (const node of g.nodes) {
|
||||
if (g.subnodes.has(node.name) || LOGIC_TYPES.has(node.type)) continue;
|
||||
let items;
|
||||
let source;
|
||||
const caseItems = Object.entries(byType).find(([pattern]) => typeMatches(node.type, pattern))?.[1];
|
||||
const explicit = scenario.trigger ?? scenarios.trigger;
|
||||
const trigger = isTrigger(node.type) && (explicit || !caseItems) ? triggerItems(node, scenario, scenarios) : null;
|
||||
if (trigger) {
|
||||
items = trigger.items;
|
||||
source = trigger.source;
|
||||
} else if (caseItems) {
|
||||
items = caseItems;
|
||||
source = 'case';
|
||||
} else if (isAiRootNode(g, node) || (isAiNode(g, node) && AI_ROOT_PLACEHOLDER[node.type])) {
|
||||
const out = aiRootOutput(g, node, scenario.ai);
|
||||
items = [out.json];
|
||||
source = out.provenance;
|
||||
} else {
|
||||
const schema = sdk?.discoverOutputSchemaForNode(node.type, node.typeVersion, node.parameters ?? {});
|
||||
items = [schema ? sample(schema, 'item') : {}];
|
||||
source = schema ? 'node-schema' : 'empty';
|
||||
}
|
||||
pinData[node.name] = (Array.isArray(items) ? items : [items]).map((json) => ({ json }));
|
||||
provenance[node.name] = source;
|
||||
}
|
||||
plan.scenarios.push({ name: scenario.name, expect: scenario.expect ?? {}, pinData, provenance, byType, triggerNode: scenarioTrigger(g, scenario) });
|
||||
Object.assign(plan.pinned, provenance);
|
||||
}
|
||||
return plan;
|
||||
}
|
||||
|
||||
/**
|
||||
* The trigger that a scenario starts from, or null when the workflow has one trigger. Without
|
||||
* scenario.triggerNode, it is the trigger whose downstream nodes match the most expect.reach types.
|
||||
*/
|
||||
function scenarioTrigger(g, scenario) {
|
||||
if (g.triggers.length < 2) return null;
|
||||
if (scenario.triggerNode && g.triggers.some((t) => t.name === scenario.triggerNode)) return scenario.triggerNode;
|
||||
const reach = scenario.expect?.reach ?? [];
|
||||
const score = (trigger) => {
|
||||
const types = [...downstream(g, trigger.name)].map((name) => g.byName.get(name)?.type ?? '');
|
||||
return reach.filter((pattern) => types.some((type) => typeMatches(type, pattern))).length;
|
||||
};
|
||||
return g.triggers.reduce((best, t) => (score(t) > score(best) ? t : best)).name;
|
||||
}
|
||||
|
||||
/** Case pin data by node type. A scenario entry replaces a case entry for the same short type. */
|
||||
function mergeByType(base = {}, extra = {}) {
|
||||
const out = { ...base };
|
||||
for (const [pattern, items] of Object.entries(extra ?? {})) {
|
||||
for (const key of Object.keys(out)) if (shortType(key).toLowerCase() === shortType(pattern).toLowerCase()) delete out[key];
|
||||
out[pattern] = items;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Without a case file: one scenario for each value of the booleans that an AI node returns. */
|
||||
function defaultScenarios(g) {
|
||||
const hasAi = g.nodes.some((n) => g.aiInputs(n.name, 'ai_languageModel').length && !g.subnodes.has(n.name));
|
||||
return hasAi
|
||||
? [{ name: 'AI flags true', ai: { booleans: true } }, { name: 'AI flags false', ai: { booleans: false } }]
|
||||
: [{ name: 'Default run' }];
|
||||
}
|
||||
|
||||
// --- CLI -----------------------------------------------------------------------
|
||||
|
||||
if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
const [command, workflowFile, scenariosFile] = process.argv.slice(2);
|
||||
if (command !== 'pin-plan' || !workflowFile) {
|
||||
console.error('usage: node sdk.mjs pin-plan <workflow.json> [scenarios.json]');
|
||||
process.exit(2);
|
||||
}
|
||||
const workflow = JSON.parse(readFileSync(workflowFile, 'utf8'));
|
||||
const scenarios = scenariosFile && existsSync(scenariosFile) ? JSON.parse(readFileSync(scenariosFile, 'utf8')) : {};
|
||||
process.stdout.write(JSON.stringify(pinPlan(workflow, scenarios)));
|
||||
}
|
||||
|
|
@ -1,215 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Print the evidence of each run for the judge step.
|
||||
|
||||
summarize_runs.py <run-dir> print the summary with the builder names
|
||||
summarize_runs.py <run-dir> --blind write judge/summary-blind.txt and judge/blind-map.json
|
||||
|
||||
The summary has the expected outcome criteria with their ids, and for each builder: the final
|
||||
agent message, every node with all its parameters and settings, the connections, and the facts
|
||||
from auto-checks.json. Nothing is truncated, so the judge can cite exact values.
|
||||
|
||||
In blind mode the builders are called A, B and C in a random order, and builder and model names
|
||||
are removed from the final messages. blind-map.json maps each letter back to its builder.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import random
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from build_report import load_case, parse_criteria # noqa: E402
|
||||
|
||||
NODE_SETTINGS = ('disabled', 'onError', 'retryOnFail', 'maxTries', 'waitBetweenTries', 'executeOnce',
|
||||
'alwaysOutputData', 'notesInFlow', 'notes')
|
||||
BUILDER_NAMES = re.compile(
|
||||
r'\b(Instance AI|n8n Assistant|Claude Code|Claude|Anthropic|Codex|OpenAI|ChatGPT|GPT-[\w.-]+|Sonnet|Opus|Haiku)\b',
|
||||
re.IGNORECASE)
|
||||
MARK = {True: '✓', False: '✗', None: '?'}
|
||||
REPO = Path(__file__).resolve().parents[4]
|
||||
LLM_CHECKS_DIR = REPO / 'packages/@n8n/instance-ai/evaluations/binaryChecks/checks'
|
||||
# The LLM binary checks of the eval suite that the judges answer as quality checks.
|
||||
QUALITY_CHECKS = ('valid-data-flow', 'correct-node-operations', 'handles-multiple-items', 'descriptive-node-names',
|
||||
'response-describes-changes-accurately')
|
||||
|
||||
|
||||
def quality_checks():
|
||||
"""[(name, description, rules)] read from the check sources, so the rules stay those of the eval suite."""
|
||||
out = []
|
||||
for file in QUALITY_CHECKS:
|
||||
path = LLM_CHECKS_DIR / f'{file}.ts'
|
||||
if not path.exists():
|
||||
continue
|
||||
text = path.read_text()
|
||||
name = re.search(r"name:\s*'([^']+)'", text)
|
||||
desc = re.search(r"description:\s*(['\"])(.*?)\1,", text, re.S)
|
||||
rules = re.search(r'systemPrompt:\s*`((?:\\`|[^`])*)`', text)
|
||||
if name and rules:
|
||||
out.append((name.group(1), desc.group(2) if desc else '', rules.group(1).replace('\\`', '`').strip()))
|
||||
return out
|
||||
|
||||
|
||||
def quality_text():
|
||||
checks = quality_checks()
|
||||
if not checks:
|
||||
return ''
|
||||
lines = ['QUALITY CHECKS (from the Instance AI eval suite). For each builder, answer each check once in "quality",',
|
||||
'with the check name as the id. The final message is the narration of the agent. Each build starts',
|
||||
'from an empty workflow, so the START workflow is empty.']
|
||||
for name, desc, rules in checks:
|
||||
lines += [f'--- [{name}] {desc}', rules]
|
||||
return '\n'.join(lines) + '\n'
|
||||
|
||||
|
||||
def session_final_message(sub, meta):
|
||||
"""Final agent message from the session file of an interactive CLI run."""
|
||||
if not meta.get('sessionFile'):
|
||||
return None
|
||||
text = None
|
||||
for line in (sub / meta['sessionFile']).read_text().splitlines():
|
||||
row = json.loads(line)
|
||||
payload = row.get('payload') or {}
|
||||
if row.get('type') == 'event_msg' and payload.get('type') == 'task_complete':
|
||||
text = payload.get('last_agent_message') or text
|
||||
if row.get('type') == 'assistant':
|
||||
blocks = (row.get('message') or {}).get('content') or []
|
||||
texts = [b['text'] for b in blocks if isinstance(b, dict) and b.get('type') == 'text' and b.get('text', '').strip()]
|
||||
text = '\n'.join(texts) or text
|
||||
return text
|
||||
|
||||
|
||||
def final_message(sub, meta):
|
||||
kind = meta.get('kind')
|
||||
if kind == 'instance-ai':
|
||||
trace = json.loads((sub / 'trace.json').read_text())
|
||||
rich = trace['rich'].get('data', trace['rich'])
|
||||
assistant = [m for m in rich['messages'] if m.get('role') == 'assistant']
|
||||
return (assistant[-1].get('agentTree') or {}).get('textContent') or assistant[-1].get('content') if assistant else None
|
||||
if not (sub / 'stream.jsonl').exists():
|
||||
return session_final_message(sub, meta)
|
||||
text = None
|
||||
for line in (sub / 'stream.jsonl').read_text().splitlines():
|
||||
event = json.loads(line)
|
||||
if event.get('type') == 'result':
|
||||
text = event.get('result')
|
||||
item = event.get('item') or {}
|
||||
if event.get('type') == 'item.completed' and item.get('type') == 'agent_message':
|
||||
text = item.get('text')
|
||||
return text
|
||||
|
||||
|
||||
def connections(workflow):
|
||||
lines = []
|
||||
for source, outputs in (workflow.get('connections') or {}).items():
|
||||
for kind, branches in outputs.items():
|
||||
for index, targets in enumerate(branches or []):
|
||||
for target in targets or []:
|
||||
branch = f'[{index}]' if len(branches) > 1 else ''
|
||||
# Merge and other multi-input nodes need the input index to show which side an edge feeds.
|
||||
target_index = target.get('index') or 0
|
||||
inlet = f'[in {target_index}]' if target_index else ''
|
||||
via = '' if kind == 'main' else f' ({kind})'
|
||||
lines.append(f' {source}{branch} -> {target["node"]}{inlet}{via}')
|
||||
return lines
|
||||
|
||||
|
||||
def criteria_text(expected):
|
||||
"""The expected outcome with the id of each criterion, as the judge must cite it."""
|
||||
groups = parse_criteria(expected or '')
|
||||
if not groups:
|
||||
return (expected or '(none)').strip()
|
||||
lines = []
|
||||
for heading, items in groups:
|
||||
if heading:
|
||||
lines.append(f'{heading}:')
|
||||
lines.extend(f' [{num}] {text}' for num, text in items)
|
||||
return '\n'.join(lines)
|
||||
|
||||
|
||||
def auto_check_lines(sub):
|
||||
path = sub / 'auto-checks.json'
|
||||
if not path.exists():
|
||||
return [' (none: derive_checks.mjs did not run)']
|
||||
lines = []
|
||||
for group in json.loads(path.read_text()).get('groups', []):
|
||||
score = group.get('score')
|
||||
suffix = f' · score {score["passed"]}/{score["total"]}' if score else ''
|
||||
lines.append(f' {group["title"]}{suffix} — {group.get("note", "")}')
|
||||
for c in group.get('checks', []):
|
||||
mark = 'i' if c.get('level') == 'info' else MARK.get(c.get('ok'), '?')
|
||||
lines.append(f' [{mark}] {c["id"]} {c["text"]}')
|
||||
lines.extend(f' · {d}' for d in c.get('details') or [])
|
||||
return lines
|
||||
|
||||
|
||||
def run_section(sub, meta, label, blind=False):
|
||||
out = [f'===== {label} workflow={meta.get("workflowId")} =====']
|
||||
try:
|
||||
message = (final_message(sub, meta) or '(none)').strip()
|
||||
out.append('FINAL MESSAGE:\n' + (scrub(message, meta) if blind else message))
|
||||
except Exception as e:
|
||||
out.append(f'FINAL MESSAGE: unreadable ({e})')
|
||||
wf_file = sub / 'workflow.json'
|
||||
if not wf_file.exists():
|
||||
out.append('WORKFLOW: none\n')
|
||||
return '\n'.join(out)
|
||||
workflow = json.loads(wf_file.read_text())
|
||||
out.append(f'WORKFLOW: {workflow.get("name")} nodes={len(workflow.get("nodes", []))}')
|
||||
out.append(f'SETTINGS: {json.dumps(workflow.get("settings") or {}, ensure_ascii=False)}')
|
||||
for node in workflow.get('nodes', []):
|
||||
creds = ', '.join((node.get('credentials') or {}).keys())
|
||||
out.append(f'- {node["name"]} | {node["type"]} v{node.get("typeVersion")}' + (f' | creds: {creds}' if creds else ''))
|
||||
settings = {k: node[k] for k in NODE_SETTINGS if k in node}
|
||||
if settings:
|
||||
out.append(f' (settings) {json.dumps(settings, ensure_ascii=False)}')
|
||||
for key, value in (node.get('parameters') or {}).items():
|
||||
text = value if isinstance(value, str) else json.dumps(value, ensure_ascii=False)
|
||||
out.append(f' {key}: {text}')
|
||||
out.append('CONNECTIONS:')
|
||||
out.append('\n'.join(connections(workflow)) or ' (none)')
|
||||
out.append('AUTO-CHECK FACTS (deterministic, from the node definitions, the n8n editor, pinned-data runs, '
|
||||
'in-process runs with recorded API requests, the eval binary checks and the final-message claims):')
|
||||
out.extend(auto_check_lines(sub))
|
||||
return '\n'.join(out) + '\n'
|
||||
|
||||
|
||||
def scrub(text, meta):
|
||||
"""Remove builder and model names from a final message. The workflow itself stays as built."""
|
||||
for name in filter(None, [meta.get('model'), meta.get('kind')]):
|
||||
text = re.sub(re.escape(name), 'the builder', text, flags=re.IGNORECASE)
|
||||
return BUILDER_NAMES.sub('the builder', text)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description='Print the evidence of each run for the judge step.')
|
||||
ap.add_argument('run_dir', nargs='?', default='.')
|
||||
ap.add_argument('--blind', action='store_true', help='Write a summary with the builders called A, B and C.')
|
||||
ap.add_argument('--seed', type=int, help='Seed of the blind order, for a reproducible test.')
|
||||
args = ap.parse_args()
|
||||
run_dir = Path(args.run_dir)
|
||||
case = load_case(run_dir)
|
||||
runs = [(f.parent, json.loads(f.read_text())) for f in sorted(run_dir.glob('*/run.json')) if f.parent.name != 'report']
|
||||
header = ('EXPECTED OUTCOME CRITERIA (cite each criterion by its id in brackets):\n'
|
||||
f'{criteria_text(case.get("expected"))}\n\n{quality_text()}')
|
||||
|
||||
if not args.blind:
|
||||
print(header)
|
||||
for sub, meta in runs:
|
||||
print(run_section(sub, meta, f'{sub.name} ({meta.get("kind")})'))
|
||||
return
|
||||
|
||||
order = runs[:]
|
||||
random.Random(args.seed).shuffle(order)
|
||||
letters = [chr(ord('A') + i) for i in range(len(order))]
|
||||
sections = [run_section(sub, meta, f'Builder {letter}', blind=True) for letter, (sub, meta) in zip(letters, order)]
|
||||
judge_dir = run_dir / 'judge'
|
||||
judge_dir.mkdir(exist_ok=True)
|
||||
(judge_dir / 'summary-blind.txt').write_text(header + '\n' + '\n'.join(sections))
|
||||
(judge_dir / 'blind-map.json').write_text(json.dumps({letter: sub.name for letter, (sub, _) in zip(letters, order)}, indent=2))
|
||||
print(judge_dir / 'summary-blind.txt')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -1,459 +0,0 @@
|
|||
// Run a workflow in this process with the real node code, and answer its HTTP requests here.
|
||||
//
|
||||
// The n8n execution engine and the nodes load from the compiled dist folders of the repository,
|
||||
// as in the execution evaluator of ai-workflow-builder.ee. n8n-core calls additionalData
|
||||
// .evalLlmMockHandler before it authenticates or sends a request (the Instance AI eval hook),
|
||||
// so every service request reaches respond() below and no request leaves the machine.
|
||||
// Credentials come from buildEvalMockCredentials, the same placeholders that the eval uses.
|
||||
// AI root nodes either get pinned data, or call their real model with a key from the settings.
|
||||
import { createRequire } from 'node:module';
|
||||
import { existsSync, readFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
|
||||
import { repo } from './sdk.mjs';
|
||||
|
||||
const coreRequire = createRequire(join(repo, 'packages/core/package.json'));
|
||||
const PACKAGES = [
|
||||
['n8n-nodes-base', join(repo, 'packages/nodes-base')],
|
||||
['@n8n/n8n-nodes-langchain', join(repo, 'packages/@n8n/nodes-langchain')],
|
||||
];
|
||||
const REQUIRED_DIST = [
|
||||
'packages/core/dist/execution-engine/index.js',
|
||||
'packages/nodes-base/dist/known/nodes.json',
|
||||
'packages/@n8n/nodes-langchain/dist/known/nodes.json',
|
||||
];
|
||||
|
||||
// --- Settings ------------------------------------------------------------------
|
||||
|
||||
// Same order as n8n_api.config(): .env.local, then .env.ai-surfaces, then the environment.
|
||||
const fileSettings = {};
|
||||
for (const file of [join(repo, '.env.local'), process.env.AI_SURFACES_ENV ?? join(repo, '.env.ai-surfaces')]) {
|
||||
if (!existsSync(file)) continue;
|
||||
for (const line of readFileSync(file, 'utf8').split('\n')) {
|
||||
const match = line.match(/^\s*(?:export\s+)?([A-Z0-9_]+)\s*=\s*(.*)$/);
|
||||
const value = match?.[2].trim().replace(/^(['"])(.*)\1$/, '$2');
|
||||
if (value) fileSettings[match[1]] = value;
|
||||
}
|
||||
}
|
||||
export const setting = (name) => process.env[name] || fileSettings[name] || '';
|
||||
|
||||
/** Model credential types, the settings that can hold their key, and the hosts of their API. */
|
||||
export const MODEL_CREDENTIALS = {
|
||||
googlePalmApi: { keys: ['GOOGLE_GENERATIVE_AI_API_KEY', 'GEMINI_API_KEY', 'GOOGLE_API_KEY'], field: 'apiKey', hosts: ['generativelanguage.googleapis.com'] },
|
||||
openAiApi: { keys: ['OPENAI_API_KEY'], field: 'apiKey', hosts: ['api.openai.com'] },
|
||||
anthropicApi: { keys: ['ANTHROPIC_API_KEY'], field: 'apiKey', hosts: ['api.anthropic.com'] },
|
||||
mistralCloudApi: { keys: ['MISTRAL_API_KEY'], field: 'apiKey', hosts: ['api.mistral.ai'] },
|
||||
groqApi: { keys: ['GROQ_API_KEY'], field: 'apiKey', hosts: ['api.groq.com'] },
|
||||
};
|
||||
export const modelKey = (type) => MODEL_CREDENTIALS[type]?.keys.map(setting).find(Boolean) ?? '';
|
||||
|
||||
// --- Engine --------------------------------------------------------------------
|
||||
|
||||
let engine;
|
||||
|
||||
/** The execution engine, node types and credential types. Returns { error } when a dist folder is missing. */
|
||||
export function loadEngine() {
|
||||
if (engine) return engine;
|
||||
const missing = REQUIRED_DIST.filter((p) => !existsSync(join(repo, p)));
|
||||
if (missing.length) return { error: `Build these first: ${missing.join(', ')}.` };
|
||||
coreRequire('reflect-metadata');
|
||||
const workflowLib = coreRequire('n8n-workflow');
|
||||
const core = coreRequire(join(repo, 'packages/core/dist/execution-engine/index.js'));
|
||||
const nodes = new Map();
|
||||
const credentials = new Map();
|
||||
for (const [prefix, dir] of PACKAGES) {
|
||||
const read = (file) => JSON.parse(readFileSync(join(dir, 'dist/known', file), 'utf8'));
|
||||
for (const [name, info] of Object.entries(read('nodes.json'))) nodes.set(`${prefix}.${name}`, { ...info, dir });
|
||||
for (const [name, info] of Object.entries(read('credentials.json'))) credentials.set(name, { ...info, dir });
|
||||
}
|
||||
const instances = new Map();
|
||||
const instance = (index, name) => {
|
||||
const key = `${index === nodes ? 'n' : 'c'}:${name}`;
|
||||
if (!instances.has(key)) {
|
||||
const info = index.get(name);
|
||||
if (!info) throw new Error(`Unknown type ${name}`);
|
||||
const mod = createRequire(join(info.dir, 'package.json'))(join(info.dir, info.sourcePath));
|
||||
instances.set(key, new mod[info.className]());
|
||||
}
|
||||
return instances.get(key);
|
||||
};
|
||||
const nodeTypes = {
|
||||
getByName: (type) => instance(nodes, type),
|
||||
getByNameAndVersion: (type, version) => {
|
||||
const loaded = instance(nodes, type);
|
||||
const versioned = 'nodeVersions' in loaded ? workflowLib.NodeHelpers.getVersionedNodeType(loaded, version) : loaded;
|
||||
// The same rule as shouldAssignExecuteMethod in packages/cli/src/utils.ts. Without it a
|
||||
// declarative node with a webhook (WhatsApp send and wait) gets no input data for $json.
|
||||
const declarative = versioned.description?.requestDefaults !== undefined;
|
||||
if (!versioned.execute && !versioned.supplyData && !versioned.poll && !versioned.trigger && (!versioned.webhook || declarative) && (!versioned.methods || declarative)) {
|
||||
versioned.execute = async function () {
|
||||
return (await new core.RoutingNode(this, versioned).runNode()) ?? [];
|
||||
};
|
||||
}
|
||||
return versioned;
|
||||
},
|
||||
getKnownTypes: () => ({}),
|
||||
};
|
||||
const credentialType = (name) => instance(credentials, name);
|
||||
const parentTypes = (name) => {
|
||||
try {
|
||||
return (credentialType(name).extends ?? []).flatMap((p) => [p, ...parentTypes(p)]);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
};
|
||||
const credentialProperties = (name) => [...(credentialType(name).extends ?? []).flatMap(credentialProperties), ...(credentialType(name).properties ?? [])];
|
||||
engine = { workflowLib, core, nodeTypes, nodes, credentialType, parentTypes, credentialProperties };
|
||||
return engine;
|
||||
}
|
||||
|
||||
/** Credential types that a node can use, from its node type description. */
|
||||
export function nodeCredentialTypes(node) {
|
||||
try {
|
||||
return (loadEngine().nodeTypes.getByNameAndVersion(node.type, node.typeVersion).description.credentials ?? []).map((c) => c.name);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
/** Apply the generic authenticate rule of a credential type, such as `qs.key` of Google Gemini. */
|
||||
function applyGenericAuth(credentialType, credentials, request) {
|
||||
const auth = credentialType.authenticate;
|
||||
if (!auth || typeof auth !== 'object' || auth.type !== 'generic') return request;
|
||||
const fill = (v) => (typeof v === 'string' ? v.replace(/^=/, '').replace(/\{\{\s*\$credentials\.(\w+)\s*\}\}/g, (_m, k) => credentials[k] ?? '') : v);
|
||||
for (const part of ['headers', 'qs', 'body']) {
|
||||
const props = auth.properties?.[part];
|
||||
if (props) request[part] = { ...(request[part] ?? {}), ...Object.fromEntries(Object.entries(props).map(([k, v]) => [k, fill(v)])) };
|
||||
}
|
||||
return request;
|
||||
}
|
||||
|
||||
function credentialsHelper(realModels) {
|
||||
const { core, parentTypes, credentialProperties, credentialType } = loadEngine();
|
||||
return {
|
||||
getParentTypes: parentTypes,
|
||||
getDecrypted: async (_data, _details, type) => {
|
||||
const mocked = core.buildEvalMockCredentials(credentialProperties(type));
|
||||
const model = MODEL_CREDENTIALS[type];
|
||||
if (realModels && model) mocked[model.field] = modelKey(type);
|
||||
return mocked;
|
||||
},
|
||||
// Vendor model nodes send their request through the n8n helpers, so the real key must be applied here.
|
||||
authenticate: async (credentials, type, request) => (realModels && MODEL_CREDENTIALS[type] ? applyGenericAuth(credentialType(type), credentials, request) : request),
|
||||
preAuthentication: async () => undefined,
|
||||
runPreAuthentication: async () => undefined,
|
||||
getCredentials: async () => ({}),
|
||||
getCredentialsProperties: (type) => credentialProperties(type),
|
||||
updateCredentials: async () => {},
|
||||
updateCredentialsOauthTokenData: async () => {},
|
||||
isCredentialUsableByNode: () => true,
|
||||
};
|
||||
}
|
||||
|
||||
// --- Setup values ----------------------------------------------------------------
|
||||
|
||||
/** Values that a node accepts for a parameter that the builder left for setup. */
|
||||
const SETUP_VALUES = {
|
||||
'googleSheets.sheetName': 'gid=0',
|
||||
};
|
||||
|
||||
const isEmptyValue = (v) => v === undefined || v === null || v === '';
|
||||
|
||||
/**
|
||||
* A copy of the workflow with placeholder values in the parameters that the builder left for
|
||||
* the user: empty resource locators, and empty parameters that the editor reports as required.
|
||||
* Returns { workflow, filled }.
|
||||
*/
|
||||
export function fillSetup(workflow, canvas) {
|
||||
const copy = structuredClone(workflow);
|
||||
const filled = [];
|
||||
const required = new Map();
|
||||
for (const node of canvas?.nodes ?? []) {
|
||||
for (const [param, messages] of Object.entries(node.issues?.parameters ?? {})) {
|
||||
if (messages.some((m) => /is required/i.test(m))) required.set(`${node.name}|${param}`, true);
|
||||
}
|
||||
}
|
||||
for (const node of copy.nodes ?? []) {
|
||||
const short = node.type.slice(node.type.lastIndexOf('.') + 1);
|
||||
for (const [param, value] of Object.entries(node.parameters ?? {})) {
|
||||
const placeholder = SETUP_VALUES[`${short}.${param}`] ?? `mock-${param}`;
|
||||
if (value && typeof value === 'object' && value.__rl && isEmptyValue(value.value)) {
|
||||
node.parameters[param] = { ...value, value: placeholder };
|
||||
filled.push(`${node.name} › ${param}`);
|
||||
} else if (isEmptyValue(value) && required.has(`${node.name}|${param}`)) {
|
||||
node.parameters[param] = placeholder;
|
||||
filled.push(`${node.name} › ${param}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
return { workflow: copy, filled };
|
||||
}
|
||||
|
||||
// --- Wire responses --------------------------------------------------------------
|
||||
|
||||
const WRITE_METHODS = new Set(['POST', 'PUT', 'PATCH', 'DELETE']);
|
||||
export const isWrite = (request) => WRITE_METHODS.has(request.method);
|
||||
|
||||
/** Rows for a read node: [{ Question, Answer }, ...] become the header row and the value rows. */
|
||||
function sheetValues(rows) {
|
||||
const headers = [...new Set(rows.flatMap((r) => Object.keys(r ?? {})))];
|
||||
if (!headers.length) return [['Question', 'Answer']];
|
||||
return [headers, ...rows.map((r) => headers.map((h) => (r?.[h] === undefined ? '' : String(r[h]))))];
|
||||
}
|
||||
|
||||
/** Page properties as Notion returns them: each with an id, a type, and plain_text in rich text. */
|
||||
function notionProperties(sent = {}) {
|
||||
const out = {};
|
||||
for (const [name, value] of Object.entries(sent ?? {})) {
|
||||
const type = value?.type ?? Object.keys(value ?? {}).find((k) => k !== 'id' && k !== 'type') ?? 'rich_text';
|
||||
let content = value?.[type];
|
||||
if (Array.isArray(content)) content = content.map((t) => ({ type: 'text', ...t, plain_text: t?.plain_text ?? t?.text?.content ?? '' }));
|
||||
out[name] = { id: `prop-${Object.keys(out).length}`, type, [type]: content ?? null };
|
||||
}
|
||||
if (!Object.values(out).some((p) => p.type === 'title')) out.Name = { id: 'title', type: 'title', title: [] };
|
||||
return out;
|
||||
}
|
||||
|
||||
const bodyOf = (request) => {
|
||||
if (typeof request.body !== 'string') return request.body ?? {};
|
||||
try {
|
||||
return JSON.parse(request.body);
|
||||
} catch {
|
||||
return {};
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Responses in the shape that each API documents. A fixture matches on host, method and path.
|
||||
* rows(node) gives the case rows for a read node (scenarios.json pinByType).
|
||||
*/
|
||||
const FIXTURES = [
|
||||
{
|
||||
service: 'Google Sheets',
|
||||
host: /(^|\.)sheets\.googleapis\.com$/,
|
||||
method: 'GET',
|
||||
path: /^\/v4\/spreadsheets\/[^/]+$/,
|
||||
respond: () => ({ spreadsheetId: 'mock-sheet', properties: { title: 'FAQ' }, sheets: [{ properties: { sheetId: 0, title: 'Sheet1', index: 0, gridProperties: { rowCount: 100, columnCount: 26 } } }] }),
|
||||
},
|
||||
{
|
||||
service: 'Google Sheets',
|
||||
host: /(^|\.)sheets\.googleapis\.com$/,
|
||||
method: 'GET',
|
||||
path: /^\/v4\/spreadsheets\/[^/]+\/values\//,
|
||||
respond: (_r, ctx, node) => ({ range: 'Sheet1!A1:Z100', majorDimension: 'ROWS', values: sheetValues(ctx.rows(node) ?? []) }),
|
||||
},
|
||||
{
|
||||
service: 'Google Sheets',
|
||||
host: /(^|\.)sheets\.googleapis\.com$/,
|
||||
method: 'POST',
|
||||
path: /:append$|values:batchUpdate$|:batchUpdate$/,
|
||||
respond: () => ({ spreadsheetId: 'mock-sheet', updates: { updatedRange: 'Sheet1!A2:B2', updatedRows: 1 }, replies: [] }),
|
||||
},
|
||||
{
|
||||
service: 'Notion',
|
||||
host: /(^|\.)api\.notion\.com$/,
|
||||
method: 'GET',
|
||||
path: /^\/v1\/(databases|data_sources)\/[^/]+$/,
|
||||
respond: (r) => ({
|
||||
object: r.url.includes('data_sources') ? 'data_source' : 'database',
|
||||
id: 'mock-database',
|
||||
title: [{ type: 'text', plain_text: 'Support tickets', text: { content: 'Support tickets' } }],
|
||||
properties: { Name: { id: 'title', name: 'Name', type: 'title', title: {} } },
|
||||
data_sources: [{ id: 'mock-data-source', name: 'Support tickets' }],
|
||||
}),
|
||||
},
|
||||
{
|
||||
service: 'Notion',
|
||||
host: /(^|\.)api\.notion\.com$/,
|
||||
method: 'POST',
|
||||
path: /^\/v1\/pages$/,
|
||||
respond: (r) => {
|
||||
const body = bodyOf(r);
|
||||
return { object: 'page', id: '59833787-2cf9-4fdf-8782-e53db20768a5', created_time: '2026-01-01T09:00:00.000Z', last_edited_time: '2026-01-01T09:00:00.000Z', archived: false, in_trash: false, parent: body.parent ?? {}, properties: notionProperties(body.properties), url: 'https://www.notion.so/Ticket-598337872cf94fdf8782e53db20768a5', public_url: null };
|
||||
},
|
||||
},
|
||||
{
|
||||
service: 'Notion',
|
||||
host: /(^|\.)api\.notion\.com$/,
|
||||
method: 'PATCH',
|
||||
path: /^\/v1\/blocks\/[^/]+\/children$/,
|
||||
respond: (r) => ({ object: 'list', results: (bodyOf(r).children ?? []).map((c, i) => ({ object: 'block', id: `mock-block-${i}`, ...c })), next_cursor: null, has_more: false }),
|
||||
},
|
||||
{
|
||||
service: 'Slack',
|
||||
host: /(^|\.)slack\.com$/,
|
||||
method: 'POST',
|
||||
path: /^\/api\/chat\.postMessage$/,
|
||||
respond: (r) => {
|
||||
const body = bodyOf(r);
|
||||
return { ok: true, channel: body.channel ?? 'C0000000000', ts: '1767258000.000100', message: { type: 'message', text: body.text ?? '', ts: '1767258000.000100', bot_id: 'B0000000000' } };
|
||||
},
|
||||
},
|
||||
{
|
||||
service: 'WhatsApp Cloud API',
|
||||
host: /(^|\.)graph\.facebook\.com$/,
|
||||
method: 'POST',
|
||||
path: /^\/v\d+(\.\d+)?\/[^/]+\/messages$/,
|
||||
respond: (r) => {
|
||||
const to = bodyOf(r).to ?? '';
|
||||
return { messaging_product: 'whatsapp', contacts: [{ input: to, wa_id: to }], messages: [{ id: 'wamid.HBgLMTU1NTAwMDExMTEVAgARGBI5QTNDQTVCM0Q0Q0Q2RTY3RTcA' }] };
|
||||
},
|
||||
},
|
||||
{
|
||||
service: 'Telegram Bot API',
|
||||
host: /(^|\.)api\.telegram\.org$/,
|
||||
method: 'POST',
|
||||
path: /\/sendMessage$/,
|
||||
respond: (r) => ({ ok: true, result: { message_id: 2, date: 1767258000, chat: { id: bodyOf(r).chat_id ?? 1001, type: 'private' }, text: bodyOf(r).text ?? '' } }),
|
||||
},
|
||||
{
|
||||
service: 'Gmail',
|
||||
host: /(^|\.)gmail\.googleapis\.com$|(^|\.)www\.googleapis\.com$/,
|
||||
method: 'POST',
|
||||
path: /\/messages\/send$/,
|
||||
respond: () => ({ id: 'mock-message', threadId: 'mock-thread', labelIds: ['SENT'] }),
|
||||
},
|
||||
];
|
||||
|
||||
/**
|
||||
* A request handler for evalLlmMockHandler. Every request is recorded in ctx.requests with the
|
||||
* fixture that answered it. A request with no fixture gets a generic answer: a read gets an
|
||||
* empty object, a write gets its own body back with an id. A request to a host in passHosts
|
||||
* goes to the network: these are the model APIs of the real-AI scenarios.
|
||||
*/
|
||||
export function wireHandler(ctx, passHosts = new Set()) {
|
||||
return async (request, node) => {
|
||||
const url = new URL(request.url ?? '', 'https://unknown.invalid');
|
||||
if (request.baseURL && !/^https?:/.test(request.url ?? '')) {
|
||||
const base = new URL(request.baseURL);
|
||||
url.host = base.host;
|
||||
url.pathname = `${base.pathname.replace(/\/$/, '')}/${(request.url ?? '').replace(/^\//, '')}`;
|
||||
}
|
||||
const method = String(request.method ?? 'GET').toUpperCase();
|
||||
if (passHosts.has(url.hostname)) {
|
||||
ctx.requests.push({ node: node.name, nodeType: node.type, method, url: url.toString(), body: request.body, answeredBy: 'real model' });
|
||||
return undefined;
|
||||
}
|
||||
const fixture = FIXTURES.find((f) => f.host.test(url.hostname) && f.method === method && f.path.test(url.pathname));
|
||||
const record = { node: node.name, nodeType: node.type, method, url: url.toString(), qs: request.qs, body: request.body, answeredBy: fixture?.service ?? 'generic' };
|
||||
let body;
|
||||
try {
|
||||
body = fixture ? fixture.respond({ ...request, url: url.toString() }, ctx, node) : method === 'GET' ? {} : { id: 'mock-1', ...(bodyOf(request) ?? {}) };
|
||||
} catch (error) {
|
||||
body = { error: String(error) };
|
||||
}
|
||||
ctx.requests.push({ ...record, response: body });
|
||||
return { statusCode: 200, headers: { 'content-type': 'application/json' }, body };
|
||||
};
|
||||
}
|
||||
|
||||
// --- Network guard ---------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Replace global fetch while a run is active. Model SDKs use fetch directly, so this is the
|
||||
* only path that could reach the network. Only the hosts in allow pass, the rest is blocked.
|
||||
*/
|
||||
function guardFetch(allow, blocked) {
|
||||
const original = globalThis.fetch;
|
||||
globalThis.fetch = async (input, init) => {
|
||||
const url = new URL(typeof input === 'string' ? input : input.url ?? String(input));
|
||||
if (allow.has(url.hostname)) return await original(input, init);
|
||||
blocked.push(`${init?.method ?? 'GET'} ${url.origin}${url.pathname}`);
|
||||
throw new Error(`Blocked network request to ${url.hostname} during a test run.`);
|
||||
};
|
||||
return () => {
|
||||
globalThis.fetch = original;
|
||||
};
|
||||
}
|
||||
|
||||
// --- Code nodes ------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* additionalData.startRunnerTask. The n8n server sends Code node jobs to a sandboxed task runner.
|
||||
* This process has no sandbox, so it never runs builder code. run_wire.mjs skips workflows with a
|
||||
* Code node. This refusal catches a node type that it does not know.
|
||||
*/
|
||||
async function startRunnerTask(_data, jobType) {
|
||||
return { ok: false, error: { message: `${jobType} code does not run in the in-process run.` } };
|
||||
}
|
||||
|
||||
// --- Run -------------------------------------------------------------------------
|
||||
|
||||
const CANCEL_GRACE_MS = 15_000;
|
||||
|
||||
/** The value of promise, or null after ms. */
|
||||
async function within(promise, ms) {
|
||||
let timer;
|
||||
try {
|
||||
return await Promise.race([promise, new Promise((resolve) => (timer = setTimeout(() => resolve(null), ms)))]);
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
/** The pinned trigger that a run starts from: triggerNode, or else the first pinned trigger. */
|
||||
export const startNode = (workflow, pinData, triggerNode) =>
|
||||
workflow.nodes.find((n) => (!triggerNode || n.name === triggerNode) && Object.hasOwn(pinData, n.name) && !n.disabled && Object.keys(workflow.connections ?? {}).includes(n.name) && /trigger|webhook/i.test(n.type));
|
||||
|
||||
/**
|
||||
* Run a workflow once. pinData holds the trigger and any pinned nodes. triggerNode names the
|
||||
* trigger to start from. With realModels, model sub-nodes get the real key and fetch can reach
|
||||
* their API hosts. Returns { status, error, runData, requests, blocked, ms }.
|
||||
*/
|
||||
export async function runWorkflow(workflow, { pinData, triggerNode, rows = () => undefined, realModels = false, timeoutMs = 90_000 }) {
|
||||
const { workflowLib, core, nodeTypes } = loadEngine();
|
||||
const started = Date.now();
|
||||
const ctx = { requests: [], rows };
|
||||
const blocked = [];
|
||||
const allow = new Set(realModels ? Object.values(MODEL_CREDENTIALS).flatMap((m) => m.hosts) : []);
|
||||
const restore = guardFetch(allow, blocked);
|
||||
let stillRunning = false;
|
||||
try {
|
||||
const instance = new workflowLib.Workflow({ id: 'wire-run', name: workflow.name, nodes: workflow.nodes, connections: workflow.connections, nodeTypes, active: false, settings: workflow.settings ?? {} });
|
||||
const start = startNode(workflow, pinData, triggerNode);
|
||||
if (!start) return { status: 'error', error: 'No pinned trigger to start from.', runData: {}, requests: [], blocked, ms: 0 };
|
||||
const hooks = new core.ExecutionLifecycleHooks('manual', 'wire-run', workflow);
|
||||
const finished = new Promise((resolve) => hooks.addHandler('workflowExecuteAfter', (run) => resolve(run)));
|
||||
const values = {
|
||||
executionId: 'wire-run',
|
||||
hooks,
|
||||
credentialsHelper: credentialsHelper(realModels),
|
||||
evalLlmMockHandler: wireHandler(ctx, allow),
|
||||
startRunnerTask,
|
||||
getRunnerStatus: () => ({ available: false }),
|
||||
variables: {},
|
||||
restApiUrl: 'http://localhost:5678/rest',
|
||||
instanceBaseUrl: 'http://localhost:5678',
|
||||
webhookBaseUrl: 'http://localhost:5678/webhook',
|
||||
webhookTestBaseUrl: 'http://localhost:5678/webhook-test',
|
||||
webhookWaitingBaseUrl: 'http://localhost:5678/webhook-waiting',
|
||||
formWaitingBaseUrl: 'http://localhost:5678/form-waiting',
|
||||
userId: 'wire-run-user',
|
||||
};
|
||||
// Unset fields stay undefined. Methods that n8n calls for logs and UI updates do nothing.
|
||||
const additionalData = new Proxy({}, {
|
||||
get: (_t, prop) => (prop in values ? values[prop] : typeof prop === 'string' && /^(log|send|set|on|emit|track)/.test(prop) ? () => undefined : undefined),
|
||||
});
|
||||
const runExecutionData = workflowLib.createRunExecutionData({
|
||||
executionData: { nodeExecutionStack: [{ node: start, data: { main: [pinData[start.name]] }, source: null }] },
|
||||
resultData: { pinData },
|
||||
});
|
||||
const execute = new core.WorkflowExecute(additionalData, 'manual', runExecutionData);
|
||||
const execution = execute.processRunExecutionData(instance);
|
||||
const run = await within(execution.then(() => finished), timeoutMs);
|
||||
if (!run) {
|
||||
// Cancel aborts the running node. Wait for the engine to stop, so no request runs after the guard is gone.
|
||||
execution.cancel();
|
||||
stillRunning = !(await within(execution.then(() => true, () => true), CANCEL_GRACE_MS));
|
||||
const error = `No result after ${timeoutMs / 1000} s.${stillRunning ? ' The run did not stop after the cancel.' : ''}`;
|
||||
return { status: 'timeout', error, runData: {}, requests: ctx.requests, blocked, ms: Date.now() - started };
|
||||
}
|
||||
const error = run.data?.resultData?.error;
|
||||
return { status: run.status, error: error ? error.message ?? String(error) : null, runData: run.data?.resultData?.runData ?? {}, requests: ctx.requests, blocked, ms: Date.now() - started };
|
||||
} catch (error) {
|
||||
return { status: 'error', error: `${error?.name ?? 'Error'}: ${error?.message ?? error}`, runData: {}, requests: ctx.requests, blocked, ms: Date.now() - started };
|
||||
} finally {
|
||||
// A run that did not stop keeps the guard. The process exits when all runs are done.
|
||||
if (!stillRunning) restore();
|
||||
}
|
||||
}
|
||||
|
|
@ -1 +0,0 @@
|
|||
../../../../.agents/skills/env-var-docs
|
||||
|
|
@ -1 +0,0 @@
|
|||
../../../../.agents/skills/testmax-assistant-ui
|
||||
|
|
@ -1 +0,0 @@
|
|||
../../../../.agents/skills/testmax-n8n-ai
|
||||
|
|
@ -1,75 +0,0 @@
|
|||
# Example env for comparing AI workflow builders (Instance AI, Claude Code + n8n MCP,
|
||||
# Codex + n8n MCP) on a local instance.
|
||||
#
|
||||
# Copy to `.env.ai-surfaces` (gitignored) and fill in the values that you need.
|
||||
# Never commit keys. The testmax-n8n-ai skill reads the files in this
|
||||
# order, from low to high priority: .env.local, .env.ai-surfaces, the shell
|
||||
# environment. Empty values are ignored, so keys that are already in .env.local
|
||||
# work without a change. Set AI_SURFACES_ENV to use a file in a different place.
|
||||
#
|
||||
# See .agents/skills/testmax-n8n-ai/SKILL.md for how the values are used.
|
||||
|
||||
# --- n8n ---------------------------------------------------------------------
|
||||
# Empty values use .env.local, then the defaults: http://localhost:5678,
|
||||
# nathan@n8n.io and PlaywrightTest123.
|
||||
N8N_BASE_URL=
|
||||
N8N_EVAL_EMAIL=
|
||||
N8N_EVAL_PASSWORD=
|
||||
# Optional. The MCP key of the user above (Settings > MCP). When set, the skill
|
||||
# does not need the cached token and never asks to rotate the key.
|
||||
N8N_MCP_TOKEN=
|
||||
|
||||
# --- Run defaults --------------------------------------------------------------
|
||||
COMPARE_BUILDERS=iai,claude,codex
|
||||
# container (Docker) or clean (Claude only, no Docker)
|
||||
COMPARE_ISOLATION=container
|
||||
COMPARE_RUNS_DIR=/tmp/builder-compare
|
||||
# 1 runs the builders at the same time. 0 runs them one at a time.
|
||||
COMPARE_PARALLEL=1
|
||||
# 1 opens a local page that shows the runs live and then the report. 0 turns it off.
|
||||
COMPARE_LIVE=1
|
||||
# 1 also opens the Instance AI thread in the browser and a terminal window for
|
||||
# each CLI builder.
|
||||
COMPARE_LIVE_WINDOWS=0
|
||||
# 1 records a video of the Instance AI thread for the replay on the page.
|
||||
COMPARE_RECORD=1
|
||||
# interactive records the real terminal screen of Claude Code and Codex.
|
||||
# headless uses `claude -p` and `codex exec` and has no screen recording.
|
||||
COMPARE_CLI_MODE=interactive
|
||||
|
||||
# --- Claude Code + n8n MCP -----------------------------------------------------
|
||||
# Empty model uses the Claude Code default.
|
||||
CLAUDE_MODEL=
|
||||
# none: Claude Code has only the n8n tools. default: it also has its built-in tools,
|
||||
# such as Bash, Read, Write and WebFetch, and uses all of them without a question,
|
||||
# like Codex in the same container. Clean isolation always uses none.
|
||||
COMPARE_CLAUDE_TOOLS=none
|
||||
# 1 runs each created workflow with pinned data through the n8n MCP server, once
|
||||
# for each scenario in scenarios.json. 0 turns it off.
|
||||
COMPARE_SCENARIOS=1
|
||||
# 1 runs each created workflow in process with recorded API fixtures, for the
|
||||
# scenarios, edge cases and questions in scenarios.json. 0 turns it off.
|
||||
COMPARE_WIRE=1
|
||||
# 1 runs the deterministic binary checks of the Instance AI eval suite.
|
||||
COMPARE_BINARY_CHECKS=1
|
||||
# Model keys for the real AI runs of the questions in scenarios.json. A workflow
|
||||
# model without a key skips those runs. Set the key of each model provider that
|
||||
# the builders can pick: GOOGLE_GENERATIVE_AI_API_KEY, OPENAI_API_KEY,
|
||||
# ANTHROPIC_API_KEY, MISTRAL_API_KEY or GROQ_API_KEY.
|
||||
GOOGLE_GENERATIVE_AI_API_KEY=
|
||||
# Falls back to ANTHROPIC_API_KEY.
|
||||
CLAUDE_API_KEY=
|
||||
|
||||
# --- Codex + n8n MCP -----------------------------------------------------------
|
||||
# Empty model uses the Codex default.
|
||||
CODEX_MODEL=
|
||||
# Falls back to OPENAI_API_KEY.
|
||||
CODEX_API_KEY=
|
||||
|
||||
# --- Instance AI cost (LangSmith) ----------------------------------------------
|
||||
# The n8n instance must trace to the same project (LANGSMITH_TRACING=true).
|
||||
LANGSMITH_API_KEY=
|
||||
# Use the same region as the instance. Empty uses https://api.smith.langchain.com.
|
||||
# For the EU region, set https://eu.api.smith.langchain.com.
|
||||
LANGSMITH_ENDPOINT=
|
||||
LANGSMITH_PROJECT=instance-ai
|
||||
1
.github/CI-TELEMETRY.md
vendored
1
.github/CI-TELEMETRY.md
vendored
|
|
@ -85,7 +85,6 @@ return 'blacksmith';
|
|||
| Accessibility buckets | `packages/quality/testing/playwright/reporters/a11y-reporter.ts` | Per-bucket axe score, violated rules, violating elements |
|
||||
| Build stats | `.github/scripts/send-build-stats.mjs` | Per-package build duration, cache hit/miss, run total |
|
||||
| Docker stats | `.github/scripts/send-docker-stats.mjs` | Image size per platform, docker build duration |
|
||||
| Dive image efficiency | `packages/quality/efficiency/images/image-efficiency.spec.ts` | Layer content size, estimated waste, efficiency score, and layer count through the shared Playwright reporter |
|
||||
| Container stack | `packages/quality/environments/containers/telemetry.ts` | E2E stack startup times per service |
|
||||
|
||||
## Secrets
|
||||
|
|
|
|||
4
.github/WORKFLOWS.md
vendored
4
.github/WORKFLOWS.md
vendored
|
|
@ -411,7 +411,6 @@ Run-workflow dropdown, which only a user with write access can pick.
|
|||
|-----------------------------|---------------------------------------------------------|
|
||||
| `util-codespace-preview.yml`| Wake, re-serve or delete a PR preview instance by hand |
|
||||
| `util-data-tooling.yml` | SQLite/PostgreSQL export/import validation (manual) |
|
||||
| `test-image-efficiency.yml` | Measure n8n and runners image size and layer waste with Dive |
|
||||
| `util-probe-registry.yml` | Diagnose slow npm metadata fetches (temporary) |
|
||||
| `sec-sync-bundle-branches.yml` | Sync both bundle branches on demand, including before a bundle cut |
|
||||
|
||||
|
|
@ -587,7 +586,7 @@ Push to master
|
|||
| Daily 03:00 | `sec-sync-bundle-branches.yml` | Merge the base into `bundle/*` |
|
||||
| Daily 00:00 | `docker-build-push.yml` | Nightly Docker images |
|
||||
| Daily 00:00 | `test-db.yml` | Database compatibility |
|
||||
| Daily 00:00 | `test-e2e-performance-reusable.yml`| Browser performance and idle memory baselines |
|
||||
| Daily 00:00 | `test-e2e-performance-reusable.yml`| Performance E2E |
|
||||
| Daily 00:00 | `release-storybook.yml` | Storybook deploy |
|
||||
| Daily 00:00 | `release-chromatic.yml` | Visual regression |
|
||||
| Daily 00:00 | `util-check-docs-urls.yml` | Doc link validation |
|
||||
|
|
@ -600,7 +599,6 @@ Push to master
|
|||
| Daily 08:00 | `build-v3-nightly.yml` | Nightly v3 Docker images |
|
||||
| Monday 00:00 | `util-update-node-popularity.yml` | Node usage stats |
|
||||
| Monday 02:00 | `test-e2e-coverage-weekly.yml` | Weekly E2E coverage |
|
||||
| Wednesday 04:00 | `util-update-get-n8n-versions.yml` | get-n8n stable versions |
|
||||
| Saturday 22:00 | `test-evals-ai.yml` | AI workflow evals |
|
||||
| 1st of month 04:00 | `util-refresh-cubic-schema.yml` | Refresh vendored cubic schema |
|
||||
|
||||
|
|
|
|||
8
.github/scripts/pnpm-lock.yaml
vendored
8
.github/scripts/pnpm-lock.yaml
vendored
|
|
@ -747,8 +747,8 @@ packages:
|
|||
fast-deep-equal@3.1.3:
|
||||
resolution: {integrity: sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==}
|
||||
|
||||
fast-uri@3.1.8:
|
||||
resolution: {integrity: sha512-GZMtZUTNRpOVIECoXwLNZS5xUGE+mVNbTB8h/7Rwh2TFWcBQiPzTgyZi05BF9UMZKkLJv8XBRJTlU7zg8+ZfMg==}
|
||||
fast-uri@3.1.5:
|
||||
resolution: {integrity: sha512-gHwA1O9LDIcKunMKhObS/HimwtehO1nPUECKAu5TpKgaO19fcWEl4bliWe1jWxVFvIXztJjjQ4L8XQ1EU9f7Jw==}
|
||||
|
||||
fd-package-json@2.0.0:
|
||||
resolution: {integrity: sha512-jKmm9YtsNXN789RS/0mSzOC1NUq9mkVd65vbSSVsKdjGvYXBuE4oWe2QOEoFeRmJg+lPuZxpmrfFclNhoRMneQ==}
|
||||
|
|
@ -1694,7 +1694,7 @@ snapshots:
|
|||
ajv@8.20.0:
|
||||
dependencies:
|
||||
fast-deep-equal: 3.1.3
|
||||
fast-uri: 3.1.8
|
||||
fast-uri: 3.1.5
|
||||
json-schema-traverse: 1.0.0
|
||||
require-from-string: 2.0.2
|
||||
|
||||
|
|
@ -1994,7 +1994,7 @@ snapshots:
|
|||
|
||||
fast-deep-equal@3.1.3: {}
|
||||
|
||||
fast-uri@3.1.8: {}
|
||||
fast-uri@3.1.5: {}
|
||||
|
||||
fd-package-json@2.0.0:
|
||||
dependencies:
|
||||
|
|
|
|||
2
.github/test-metrics/e2e-impact-map.json
vendored
2
.github/test-metrics/e2e-impact-map.json
vendored
File diff suppressed because one or more lines are too long
3
.github/workflows/build-v3-nightly.yml
vendored
3
.github/workflows/build-v3-nightly.yml
vendored
|
|
@ -147,8 +147,6 @@ jobs:
|
|||
DOCKER_BASE: ${{ secrets.DOCKER_USERNAME }}
|
||||
N8N_IMAGE: ${{ needs.build.outputs.n8n_image }}
|
||||
N8N_DIGEST: ${{ needs.build.outputs.n8n_digest }}
|
||||
N8N_PC_IMAGE: ${{ needs.build.outputs.n8n_pc_image }}
|
||||
N8N_PC_DIGEST: ${{ needs.build.outputs.n8n_pc_digest }}
|
||||
RUNNERS_IMAGE: ${{ needs.build.outputs.runners_image }}
|
||||
RUNNERS_DIGEST: ${{ needs.build.outputs.runners_digest }}
|
||||
DISTROLESS_IMAGE: ${{ needs.build.outputs.runners_distroless_image }}
|
||||
|
|
@ -197,6 +195,5 @@ jobs:
|
|||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
promote "$N8N_IMAGE" "$N8N_DIGEST" "${DOCKER_BASE}/n8n" ""
|
||||
promote "$N8N_PC_IMAGE" "$N8N_PC_DIGEST" "${DOCKER_BASE}/n8n" "-pc"
|
||||
promote "$RUNNERS_IMAGE" "$RUNNERS_DIGEST" "${DOCKER_BASE}/runners" ""
|
||||
promote "$DISTROLESS_IMAGE" "$DISTROLESS_DIGEST" "${DOCKER_BASE}/runners" "-distroless"
|
||||
|
|
|
|||
9
.github/workflows/ci-pull-requests.yml
vendored
9
.github/workflows/ci-pull-requests.yml
vendored
|
|
@ -80,7 +80,6 @@ jobs:
|
|||
!**/*.test.ts
|
||||
!**/*.spec.ts
|
||||
!packages/quality/testing/playwright/**
|
||||
!packages/quality/efficiency/**
|
||||
!packages/frontend/@n8n/storybook/**
|
||||
!.devcontainer/**
|
||||
!scripts/agent-setup.mjs
|
||||
|
|
@ -147,17 +146,10 @@ jobs:
|
|||
cubic.yaml
|
||||
.agents/review-rules/**
|
||||
e2e-performance:
|
||||
packages/quality/efficiency/memory/idle/**
|
||||
packages/quality/efficiency/playwright.config.ts
|
||||
packages/quality/efficiency/package.json
|
||||
packages/quality/testing/playwright/package.json
|
||||
packages/quality/testing/playwright/quality-*.ts
|
||||
packages/quality/testing/playwright/fixtures/**
|
||||
packages/quality/testing/playwright/tests/performance/**
|
||||
packages/quality/testing/playwright/utils/performance-helper.ts
|
||||
packages/quality/environments/containers/**
|
||||
.github/workflows/test-e2e-performance-reusable.yml
|
||||
.github/workflows/test-e2e-reusable.yml
|
||||
# Whole directories, not the `engine-*` files alone. Engine v2
|
||||
# routing lives in ordinary control plane code: `workflow-runner.ts`
|
||||
# and the webhook and trigger paths decide whether a run reaches the
|
||||
|
|
@ -198,7 +190,6 @@ jobs:
|
|||
packages/@n8n/db/**
|
||||
packages/@n8n/backend-test-utils/**
|
||||
packages/@n8n/scheduler/src/**
|
||||
packages/cli/src/scheduling/**
|
||||
packages/cli/src/**/*.integration.test.ts
|
||||
packages/cli/src/modules/instance-registry/**
|
||||
packages/cli/vitest.*
|
||||
|
|
|
|||
6
.github/workflows/docker-build-push.yml
vendored
6
.github/workflows/docker-build-push.yml
vendored
|
|
@ -55,12 +55,6 @@ on:
|
|||
n8n_digest:
|
||||
description: 'Multi-arch manifest digest of the n8n image'
|
||||
value: ${{ jobs.create_multi_arch_manifest.outputs.n8n_digest }}
|
||||
n8n_pc_image:
|
||||
description: 'GHCR repo for the pointer-compressed n8n image (no tag; same repo as n8n)'
|
||||
value: ${{ jobs.create_multi_arch_manifest.outputs.n8n_pc_image }}
|
||||
n8n_pc_digest:
|
||||
description: 'Multi-arch manifest digest of the pointer-compressed n8n image'
|
||||
value: ${{ jobs.create_multi_arch_manifest.outputs.n8n_pc_digest }}
|
||||
runners_image:
|
||||
description: 'GHCR repo for the runners image (no tag)'
|
||||
value: ${{ jobs.create_multi_arch_manifest.outputs.runners_image }}
|
||||
|
|
|
|||
10
.github/workflows/prepare-docker-reusable.yml
vendored
10
.github/workflows/prepare-docker-reusable.yml
vendored
|
|
@ -9,16 +9,6 @@ name: 'Prepare n8n Docker (reusable)'
|
|||
|
||||
on:
|
||||
workflow_call:
|
||||
secrets:
|
||||
QA_METRICS_WEBHOOK_URL:
|
||||
description: 'QA metrics webhook URL.'
|
||||
required: false
|
||||
QA_METRICS_WEBHOOK_USER:
|
||||
description: 'QA metrics webhook username.'
|
||||
required: false
|
||||
QA_METRICS_WEBHOOK_PASSWORD:
|
||||
description: 'QA metrics webhook password.'
|
||||
required: false
|
||||
inputs:
|
||||
build-variant:
|
||||
description: 'standard, coverage, or pc'
|
||||
|
|
|
|||
|
|
@ -1,8 +1,5 @@
|
|||
name: 'Test: E2E Performance'
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
workflow_dispatch:
|
||||
|
|
@ -27,17 +24,6 @@ jobs:
|
|||
artifact-prefix: performance
|
||||
secrets: inherit
|
||||
|
||||
idle-memory-baselines:
|
||||
needs: [prepare-docker]
|
||||
uses: ./.github/workflows/test-e2e-reusable.yml
|
||||
with:
|
||||
test-mode: docker-artifact
|
||||
test-command: pnpm --filter=@n8n/efficiency test:memory
|
||||
workers: '1'
|
||||
currents-project-id: 'O9BJaN'
|
||||
artifact-prefix: memory-performance
|
||||
secrets: inherit
|
||||
|
||||
canvas-perf-sentinels:
|
||||
name: Canvas Perf Sentinels
|
||||
# Sentinels are nightly only — wall-clock browser perf has 15-30% variance,
|
||||
|
|
|
|||
7
.github/workflows/test-e2e-reusable.yml
vendored
7
.github/workflows/test-e2e-reusable.yml
vendored
|
|
@ -195,10 +195,9 @@ jobs:
|
|||
with:
|
||||
name: ${{ inputs.artifact-prefix }}-shard-${{ matrix.shard }}
|
||||
path: |
|
||||
packages/quality/*/test-results*
|
||||
packages/quality/*/playwright-report/
|
||||
packages/quality/*/*/test-results*
|
||||
packages/quality/*/*/playwright-report/
|
||||
packages/quality/testing/playwright/test-results/
|
||||
packages/quality/testing/playwright/test-results.json
|
||||
packages/quality/testing/playwright/playwright-report/
|
||||
packages/quality/testing/playwright/coverage/lcov.info
|
||||
packages/quality/testing/playwright/coverage/by-spec/
|
||||
packages/quality/testing/playwright/coverage/index.html
|
||||
|
|
|
|||
71
.github/workflows/test-image-efficiency.yml
vendored
71
.github/workflows/test-image-efficiency.yml
vendored
|
|
@ -1,71 +0,0 @@
|
|||
name: 'Test: Image Efficiency'
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
prepare-docker:
|
||||
uses: ./.github/workflows/prepare-docker-reusable.yml
|
||||
secrets:
|
||||
QA_METRICS_WEBHOOK_URL: ${{ secrets.QA_METRICS_WEBHOOK_URL }}
|
||||
QA_METRICS_WEBHOOK_USER: ${{ secrets.QA_METRICS_WEBHOOK_USER }}
|
||||
QA_METRICS_WEBHOOK_PASSWORD: ${{ secrets.QA_METRICS_WEBHOOK_PASSWORD }}
|
||||
|
||||
image-efficiency:
|
||||
name: Image Efficiency
|
||||
needs: prepare-docker
|
||||
runs-on: ${{ vars.RUNNER_PROVIDER == 'github' && 'ubuntu-latest' || 'blacksmith-2vcpu-ubuntu-2204' }}
|
||||
timeout-minutes: 20
|
||||
env:
|
||||
CURRENTS_RECORD_KEY: ${{ secrets.CURRENTS_RECORD_KEY }}
|
||||
CURRENTS_PROJECT_ID: 'nHHLA5'
|
||||
CURRENTS_CI_BUILD_ID: ${{ github.repository }}-${{ github.run_id }}-${{ github.run_attempt }}-images
|
||||
QA_METRICS_WEBHOOK_URL: ${{ secrets.QA_METRICS_WEBHOOK_URL }}
|
||||
QA_METRICS_WEBHOOK_USER: ${{ secrets.QA_METRICS_WEBHOOK_USER }}
|
||||
QA_METRICS_WEBHOOK_PASSWORD: ${{ secrets.QA_METRICS_WEBHOOK_PASSWORD }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: useblacksmith/checkout@bcec731f1eb1367240608d1c889a75b96db6ec53 # v1.5.0
|
||||
|
||||
- name: Setup Environment
|
||||
uses: ./.github/actions/setup-nodejs
|
||||
with:
|
||||
build-command: pnpm turbo run build --filter=@n8n/utils...
|
||||
|
||||
- name: Load images from cache
|
||||
uses: ./.github/actions/load-n8n-docker
|
||||
|
||||
- name: Install Dive
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# Keep this version in sync with efficiency/images/dive-results.ts.
|
||||
VERSION=0.13.1
|
||||
DIR=$(mktemp -d "$RUNNER_TEMP/dive.XXXXXX")
|
||||
ARCHIVE="$DIR/dive.tar.gz"
|
||||
curl --fail --location --retry 3 \
|
||||
"https://github.com/wagoodman/dive/releases/download/v${VERSION}/dive_${VERSION}_linux_amd64.tar.gz" \
|
||||
--output "$ARCHIVE"
|
||||
echo "0970549eb4a306f8825a84145a2534153badb4d7dcf3febd1967c706367c3d0e $ARCHIVE" | sha256sum --check
|
||||
tar -xzf "$ARCHIVE" -C "$DIR" dive
|
||||
echo "$DIR" >> "$GITHUB_PATH"
|
||||
|
||||
- name: Measure image layer efficiency
|
||||
run: >-
|
||||
pnpm --filter=@n8n/efficiency exec playwright test
|
||||
--project=image-efficiency:n8n --project=image-efficiency:runners
|
||||
|
||||
- name: Upload image efficiency reports
|
||||
if: always()
|
||||
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
|
||||
with:
|
||||
name: image-efficiency-${{ github.run_attempt }}
|
||||
path: |
|
||||
packages/quality/efficiency/test-results/
|
||||
packages/quality/efficiency/test-results.json
|
||||
packages/quality/efficiency/playwright-report/
|
||||
retention-days: 7
|
||||
if-no-files-found: ignore
|
||||
118
.github/workflows/util-update-get-n8n-versions.yml
vendored
118
.github/workflows/util-update-get-n8n-versions.yml
vendored
|
|
@ -1,118 +0,0 @@
|
|||
name: 'Util: Update get-n8n Versions'
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Run every Wednesday at 04:00 UTC.
|
||||
- cron: '0 4 * * 3'
|
||||
workflow_dispatch:
|
||||
|
||||
permissions: {}
|
||||
|
||||
concurrency:
|
||||
group: update-get-n8n-versions
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
update-versions:
|
||||
name: Update get-n8n versions
|
||||
if: >-
|
||||
github.ref_name == github.event.repository.default_branch &&
|
||||
(github.event_name == 'workflow_dispatch' ||
|
||||
(github.event_name == 'schedule' && github.repository == 'n8n-io/n8n'))
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 25
|
||||
permissions:
|
||||
contents: read
|
||||
steps:
|
||||
- name: Generate GitHub App Token
|
||||
id: generate-token
|
||||
uses: actions/create-github-app-token@29824e69f54612133e76f7eaac726eef6c875baf # v2.2.1
|
||||
with:
|
||||
app-id: ${{ secrets.N8N_ASSISTANT_APP_ID }}
|
||||
private-key: ${{ secrets.N8N_ASSISTANT_PRIVATE_KEY }}
|
||||
permission-contents: write
|
||||
permission-pull-requests: write
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Resolve stable versions
|
||||
id: versions
|
||||
env:
|
||||
GH_TOKEN: ${{ steps.generate-token.outputs.token }}
|
||||
run: |
|
||||
n8n_version="$(gh api repos/n8n-io/n8n/releases/latest --jq '.tag_name | sub("^n8n@"; "")')"
|
||||
|
||||
registry_token="$(curl -fsSL --retry 3 'https://ghcr.io/token?scope=repository:n8n-io/n8n-sandbox-service-api:pull' | jq -er '.token')"
|
||||
stable_digest="$(docker buildx imagetools inspect \
|
||||
'ghcr.io/n8n-io/n8n-sandbox-service-api:stable' \
|
||||
--format '{{json .Manifest}}' | jq -er '.digest')"
|
||||
sandbox_version=''
|
||||
while IFS= read -r version; do
|
||||
version_digest="$(docker buildx imagetools inspect \
|
||||
"ghcr.io/n8n-io/n8n-sandbox-service-api:${version}" \
|
||||
--format '{{json .Manifest}}' | jq -er '.digest')"
|
||||
if [ "$version_digest" = "$stable_digest" ]; then
|
||||
sandbox_version="$version"
|
||||
break
|
||||
fi
|
||||
done < <(curl -fsSL --retry 3 \
|
||||
-H "Authorization: Bearer $registry_token" \
|
||||
'https://ghcr.io/v2/n8n-io/n8n-sandbox-service-api/tags/list?n=1000' |
|
||||
jq -r '.tags[]' |
|
||||
grep -E '^[0-9]+\.[0-9]+\.[0-9]+$' |
|
||||
sort -Vr)
|
||||
|
||||
printf '%s\n' "$n8n_version" | grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+$'
|
||||
printf '%s\n' "$sandbox_version" | grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+$'
|
||||
|
||||
for image in api runner-dind sandbox; do
|
||||
docker manifest inspect "ghcr.io/n8n-io/n8n-sandbox-service-${image}:${sandbox_version}" >/dev/null
|
||||
done
|
||||
|
||||
sed -i "s/^FALLBACK_N8N_VERSION=.*/FALLBACK_N8N_VERSION=\"${n8n_version}\"/" docker/get-n8n.sh
|
||||
sed -i "s/^SANDBOX_VERSION=.*/SANDBOX_VERSION=\"${sandbox_version}\"/" docker/get-n8n.sh
|
||||
|
||||
echo "n8n_version=$n8n_version" >> "$GITHUB_OUTPUT"
|
||||
echo "sandbox_version=$sandbox_version" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Check for changes
|
||||
id: changes
|
||||
run: |
|
||||
if git diff --quiet -- docker/get-n8n.sh; then
|
||||
echo 'has-changes=false' >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo 'has-changes=true' >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Test installer upgrade
|
||||
if: steps.changes.outputs.has-changes == 'true'
|
||||
run: |
|
||||
shellcheck docker/get-n8n.sh docker/test-get-n8n.sh
|
||||
dash -n docker/get-n8n.sh
|
||||
dash -n docker/test-get-n8n.sh
|
||||
sh docker/test-get-n8n.sh --e2e
|
||||
|
||||
- name: Create pull request
|
||||
if: steps.changes.outputs.has-changes == 'true'
|
||||
uses: peter-evans/create-pull-request@c0f553fe549906ede9cf27b5156039d195d2ece0 # v8.1.0
|
||||
with:
|
||||
token: ${{ steps.generate-token.outputs.token }}
|
||||
add-paths: docker/get-n8n.sh
|
||||
commit-message: 'chore: Update get-n8n fallback versions'
|
||||
labels: 'automation:scheduled-update'
|
||||
title: 'chore: Update get-n8n fallback versions'
|
||||
body: |
|
||||
This automated PR updates the get-n8n fallback release to ${{ steps.versions.outputs.n8n_version }}.
|
||||
It also updates the sandbox service release to ${{ steps.versions.outputs.sandbox_version }}.
|
||||
|
||||
The workflow verified that all sandbox images use the same release. It also upgraded a legacy installation to these versions and checked the running stack.
|
||||
|
||||
_Generated by the weekly get-n8n version update workflow._
|
||||
branch: update-get-n8n-versions
|
||||
base: master
|
||||
delete-branch: true
|
||||
author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
|
||||
committer: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
|
||||
1
.gitignore
vendored
1
.gitignore
vendored
|
|
@ -17,7 +17,6 @@ _START_PACKAGE
|
|||
.env
|
||||
.env.local
|
||||
.env.eval
|
||||
.env.ai-surfaces
|
||||
.vscode/*
|
||||
!.claude
|
||||
!.vscode/extensions.json
|
||||
|
|
|
|||
|
|
@ -14,11 +14,6 @@ frontend, and extensible node-based workflow engine.
|
|||
- Write all technical text (code comments, PR descriptions, issue and ticket
|
||||
descriptions, docs) in ASD-STE100 Simplified Technical English: use short
|
||||
sentences, the active voice, and one instruction for each sentence
|
||||
- **JSDoc on `@Env(...)` config properties is public documentation.** The
|
||||
docs team exports these comments into the environment variable reference on
|
||||
docs.n8n.io. Write them for a self-hosting operator and follow the
|
||||
`n8n:env-var-docs` skill, which applies the n8n docs style guide and
|
||||
overrides the ASD-STE100 rule above for these comments
|
||||
- **Secrets on the command line:** if a developer opted into anonymous dev
|
||||
metrics (`scripts/dev-metrics`), pnpm command arguments are recorded. Arguments
|
||||
of secret-carrying words (`config`, `login`, `publish`, `token`) — whether a
|
||||
|
|
|
|||
154
CHANGELOG.md
154
CHANGELOG.md
|
|
@ -1,151 +1,25 @@
|
|||
# [2.45.0](https://github.com/n8n-io/n8n/compare/n8n@2.44.0...n8n@2.45.0) (2026-10-20)
|
||||
## [2.44.3](https://github.com/n8n-io/n8n/compare/n8n@2.44.2...n8n@2.44.3) (2026-10-16)
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **AI Agent Node:** Replay Anthropic thinking turns with their tool arguments ([#41241](https://github.com/n8n-io/n8n/issues/41241)) ([ea98bd0](https://github.com/n8n-io/n8n/commit/ea98bd06003b5687882c94245df837331cfbb544))
|
||||
* Align data table and variable editing on protected instances ([#41258](https://github.com/n8n-io/n8n/issues/41258)) ([d4bde14](https://github.com/n8n-io/n8n/commit/d4bde146d6fa8f591a91f0d0ee44b56b96cb7019))
|
||||
* Allow skipping table initialization ([#40437](https://github.com/n8n-io/n8n/issues/40437)) ([d245b17](https://github.com/n8n-io/n8n/commit/d245b17b898651a50a524a2411d703509afd8339))
|
||||
* **API:** Apply projectId filter to executions for member API keys ([#40887](https://github.com/n8n-io/n8n/issues/40887)) ([3075b60](https://github.com/n8n-io/n8n/commit/3075b605766a73bee35d07b3afab02050c4e4a49))
|
||||
* **API:** Log errors before returning generic 500 in Public API ([#41186](https://github.com/n8n-io/n8n/issues/41186)) ([fe36279](https://github.com/n8n-io/n8n/commit/fe3627914a64e5d34b39fcd5e21ba48c8268b9b4))
|
||||
* Check provisioning license on Users settings page ([#41048](https://github.com/n8n-io/n8n/issues/41048)) ([066480d](https://github.com/n8n-io/n8n/commit/066480d8dbe5e3127a2cf099344a383492c7769a))
|
||||
* **Convert to File Node:** Decode data URIs and validate Base64 input ([#38554](https://github.com/n8n-io/n8n/issues/38554)) ([315e0a5](https://github.com/n8n-io/n8n/commit/315e0a5377b98f9d2e53f77160150a6655f43341))
|
||||
* **core:** Allow gateway model IDs and forward credential headers in agents ([#40470](https://github.com/n8n-io/n8n/issues/40470)) ([791c7bc](https://github.com/n8n-io/n8n/commit/791c7bc418bf789a9686859f10592b3980571b50))
|
||||
* **core:** Allow project promotion without an instance export ([#40939](https://github.com/n8n-io/n8n/issues/40939)) ([e0fe040](https://github.com/n8n-io/n8n/commit/e0fe04084e77d296e2daf1767423a56d66455cee))
|
||||
* **core:** Allow switching a node back to a credential the user cannot use after a save ([#41198](https://github.com/n8n-io/n8n/issues/41198)) ([e7f0607](https://github.com/n8n-io/n8n/commit/e7f0607bf13e5bc4a2f929caf7a2aaadf5701d26))
|
||||
* **core:** Clear agent runtime cache when credential is updated ([#40961](https://github.com/n8n-io/n8n/issues/40961)) ([4cb7a6e](https://github.com/n8n-io/n8n/commit/4cb7a6ee3da84c5258ee129285cafeaba5863728))
|
||||
* **core:** Close agent after run when invoked from workflow ([#41128](https://github.com/n8n-io/n8n/issues/41128)) ([f596162](https://github.com/n8n-io/n8n/commit/f596162dc7b7accdcd71bc544622b1cc13546c88))
|
||||
* **core:** Confirm project deletions before instance apply ([#40941](https://github.com/n8n-io/n8n/issues/40941)) ([f631112](https://github.com/n8n-io/n8n/commit/f63111202f5cfadb989051c02a87710aafbe33e8))
|
||||
* **core:** Create an XML parser for each webhook request body ([#41098](https://github.com/n8n-io/n8n/issues/41098)) ([1950434](https://github.com/n8n-io/n8n/commit/1950434a277d863c04c7f640b9dba5d69d6c4ac9))
|
||||
* **core:** Describe MCP servers only from user or registry text in Agent Builder ([#40974](https://github.com/n8n-io/n8n/issues/40974)) ([248c863](https://github.com/n8n-io/n8n/commit/248c86355b0eb30f0874f14f680f1986da5c9878))
|
||||
* **core:** Display an error message when failing to connect to oauth credentials ([#41058](https://github.com/n8n-io/n8n/issues/41058)) ([952b1e7](https://github.com/n8n-io/n8n/commit/952b1e73b281539b81a7d5461770b0a515a7add0))
|
||||
* **core:** Evict idle mcp-browser sessions to prevent connection leak ([#36850](https://github.com/n8n-io/n8n/issues/36850)) ([efd4e21](https://github.com/n8n-io/n8n/commit/efd4e21aadbe3dd826e8d1a3df6e7921559b204b))
|
||||
* **core:** Handle every escaped closing bracket in an expression ([#40487](https://github.com/n8n-io/n8n/issues/40487)) ([b09efb7](https://github.com/n8n-io/n8n/commit/b09efb706cc9f8b7f4d4328e5bfe13fee9e145eb))
|
||||
* **core:** Improve base URL check for declarative nodes ([#40778](https://github.com/n8n-io/n8n/issues/40778)) ([e45a65a](https://github.com/n8n-io/n8n/commit/e45a65a9bf1c1eaf1c9282c922753ec265641846))
|
||||
* **core:** Keep cancelled Assistant activity settled ([#40786](https://github.com/n8n-io/n8n/issues/40786)) ([d74f3f2](https://github.com/n8n-io/n8n/commit/d74f3f2cddabf0b732f4a22ae8a784ad66ed7f29))
|
||||
* **core:** Keep Observer detail lines instead of discarding the response ([#41134](https://github.com/n8n-io/n8n/issues/41134)) ([48edbef](https://github.com/n8n-io/n8n/commit/48edbef643cc2dcd9c24c18be56fa88fda06ee68))
|
||||
* **core:** Keep offline_access when a protected resource omits it ([#41384](https://github.com/n8n-io/n8n/issues/41384)) ([9675910](https://github.com/n8n-io/n8n/commit/967591041e508802f6cf77a1b1651ec48175f680))
|
||||
* **core:** Keep pending setup for nodes that n8n Assistant added in an earlier build ([#41070](https://github.com/n8n-io/n8n/issues/41070)) ([50c4ced](https://github.com/n8n-io/n8n/commit/50c4ced2241430e13fa7042e976ed33f07bf5009))
|
||||
* **core:** Keep the workflow source file when n8n Assistant sends blank source ([#41201](https://github.com/n8n-io/n8n/issues/41201)) ([b78913f](https://github.com/n8n-io/n8n/commit/b78913f75d2e7d93c22b380160e0b26d198097d9))
|
||||
* **core:** Key each OAuth authorization request to its own session ([#39930](https://github.com/n8n-io/n8n/issues/39930)) ([3c447b4](https://github.com/n8n-io/n8n/commit/3c447b42a7f019f8fa24cf4b1c599929fc9abb85))
|
||||
* **core:** Key queue job outcomes by job on shutdown resume ([#40569](https://github.com/n8n-io/n8n/issues/40569)) ([6e04287](https://github.com/n8n-io/n8n/commit/6e04287582ebb66d550faf166a865d0a27f936fa))
|
||||
* **core:** Let LLM failed-attempt handling retry and honour Retry-After ([#41002](https://github.com/n8n-io/n8n/issues/41002)) ([67c1981](https://github.com/n8n-io/n8n/commit/67c1981f2f2018e817706273d3a91a1d8bdf79ee))
|
||||
* **core:** Let preview chat agents make changes through their tools ([#41403](https://github.com/n8n-io/n8n/issues/41403)) ([8323dfd](https://github.com/n8n-io/n8n/commit/8323dfd1e1136e33a036ed7a2621cc1e05ca1e0e))
|
||||
* **core:** Log custom data truncation only above the 512-character limit ([#40899](https://github.com/n8n-io/n8n/issues/40899)) ([e74f1c1](https://github.com/n8n-io/n8n/commit/e74f1c19daa8b7784767f9332541f616aabd1d5f))
|
||||
* **core:** Pass reasoning effort on open router model node ([#41179](https://github.com/n8n-io/n8n/issues/41179)) ([683fd11](https://github.com/n8n-io/n8n/commit/683fd112594f91b5d90bba6720f0e1081977158a))
|
||||
* **core:** Redact a parent execution if any sub-workflows have redaction enabled ([#41129](https://github.com/n8n-io/n8n/issues/41129)) ([c70b5d4](https://github.com/n8n-io/n8n/commit/c70b5d49b322c4fbd7bebe940f255ec6e8510adb))
|
||||
* **core:** Redact the parent execution when a sub-workflow uses a credential the viewer cannot use ([#41085](https://github.com/n8n-io/n8n/issues/41085)) ([a57c8f3](https://github.com/n8n-io/n8n/commit/a57c8f3abaf92aadddd8d2798418bc7e0a53ab18))
|
||||
* **core:** Reflect failed publications in the active workflows endpoints ([#40641](https://github.com/n8n-io/n8n/issues/40641)) ([96eff1e](https://github.com/n8n-io/n8n/commit/96eff1efb92c85434c1324a68afa4a67e05c9276))
|
||||
* **core:** Remove unused N8N_SCHEDULER_MIN_INTERVAL setting ([#41194](https://github.com/n8n-io/n8n/issues/41194)) ([c839d6c](https://github.com/n8n-io/n8n/commit/c839d6cd490e355de1efa356327002b74d86ed75))
|
||||
* **core:** Reuse connections in the log streaming webhook destination ([#41261](https://github.com/n8n-io/n8n/issues/41261)) ([1581524](https://github.com/n8n-io/n8n/commit/158152496aca4f509a54c6af9710e0224f980be8))
|
||||
* **core:** Skip resource mapper validation when the mapped value is null ([#41069](https://github.com/n8n-io/n8n/issues/41069)) ([898701b](https://github.com/n8n-io/n8n/commit/898701b05b57c56cc2e88f5324b17adcc317d140))
|
||||
* **core:** Stop asking to set up a node when n8n Assistant only adds a step next to it ([#41067](https://github.com/n8n-io/n8n/issues/41067)) ([578eea4](https://github.com/n8n-io/n8n/commit/578eea403d406bd5e92db18067577e8145599c50))
|
||||
* **core:** Trace verification helpers and provider attempts ([#40936](https://github.com/n8n-io/n8n/issues/40936)) ([f73dc33](https://github.com/n8n-io/n8n/commit/f73dc337f8dc61a4fd44aebe41c55faafe4f53a4))
|
||||
* **core:** Turn off expression spans by default ([#41211](https://github.com/n8n-io/n8n/issues/41211)) ([842ec5b](https://github.com/n8n-io/n8n/commit/842ec5b62d9274bff50901d290ff02f837a2e904))
|
||||
* **core:** Use the module registry to decide if the Assistant has the Agents features ([#41026](https://github.com/n8n-io/n8n/issues/41026)) ([726a582](https://github.com/n8n-io/n8n/commit/726a5829c826f8419cf9ec2d9993c1cb19b5246a))
|
||||
* **core:** Web search misconfigs in agents as UserError ([#41257](https://github.com/n8n-io/n8n/issues/41257)) ([f844135](https://github.com/n8n-io/n8n/commit/f84413516a80c87f2dca194306f5c1ed18ac2d11))
|
||||
* **Databricks Node:** Bring the AI Agent panel to parity with AI Agent 2.2 ([#40989](https://github.com/n8n-io/n8n/issues/40989)) ([e859aba](https://github.com/n8n-io/n8n/commit/e859abaf189041f751fcee7333c79acde212f994))
|
||||
* **Databricks Node:** Rename Genie resource labels to Genie Agent and the MCP tile to Databricks Genie One ([#40760](https://github.com/n8n-io/n8n/issues/40760)) ([14c7790](https://github.com/n8n-io/n8n/commit/14c77900df54948c8ca5f566055e31ec417eae09))
|
||||
* **editor:** Apply the standard settings layout to the Context preferences page ([#40716](https://github.com/n8n-io/n8n/issues/40716)) ([b3e8634](https://github.com/n8n-io/n8n/commit/b3e8634c83cf660f71ea08319c8c19f975ad795f))
|
||||
* **editor:** Cancel the Assistant thread when the hand-off navigation fails ([#41056](https://github.com/n8n-io/n8n/issues/41056)) ([d26f914](https://github.com/n8n-io/n8n/commit/d26f914e3d4fb3fd09d3004a435d8dd92198a34f))
|
||||
* **editor:** Cleanup legacy button variants ([#41175](https://github.com/n8n-io/n8n/issues/41175)) ([ba97e46](https://github.com/n8n-io/n8n/commit/ba97e46afc579c48eacacb42832df524b49f790e))
|
||||
* **editor:** Display custom evaluation metrics correctly ([#40852](https://github.com/n8n-io/n8n/issues/40852)) ([f6b24d0](https://github.com/n8n-io/n8n/commit/f6b24d0ae93b55cceefa50045a0efb3cc9279a86))
|
||||
* **editor:** Improve destination validation and surface save errors ([#40897](https://github.com/n8n-io/n8n/issues/40897)) ([eaf47cf](https://github.com/n8n-io/n8n/commit/eaf47cfe39778fb8a85cae29358eeba6356d1e23))
|
||||
* **editor:** Improve NDV credential options performance on large instances ([#40198](https://github.com/n8n-io/n8n/issues/40198)) ([b98640f](https://github.com/n8n-io/n8n/commit/b98640f0abb2746983230412d85963030b0aaecb))
|
||||
* **editor:** Keep pending navigation when workflows list query is unchanged ([#41125](https://github.com/n8n-io/n8n/issues/41125)) ([116a81e](https://github.com/n8n-io/n8n/commit/116a81e06ec9f3b3478759003f0cb2aef380aec6))
|
||||
* **editor:** Keep the used-in rows on the credential sharing tab when personal-space sharing is disabled ([#41207](https://github.com/n8n-io/n8n/issues/41207)) ([ca9661c](https://github.com/n8n-io/n8n/commit/ca9661c091acd43b3dcd2408df22321d6662b07b))
|
||||
* **editor:** Keep workflow name in edit mode after Rename menu action ([#41120](https://github.com/n8n-io/n8n/issues/41120)) ([fc9f957](https://github.com/n8n-io/n8n/commit/fc9f9575e2cea830da1f2f1efccd73a49b43a26a))
|
||||
* **editor:** Load credential types on startup ([#41400](https://github.com/n8n-io/n8n/issues/41400)) ([9a025a1](https://github.com/n8n-io/n8n/commit/9a025a188f6164723f1e8cd928f1f21d30765587))
|
||||
* **editor:** Match the credential dropdown width to its field and remove horizontal scroll ([#41260](https://github.com/n8n-io/n8n/issues/41260)) ([51e53ad](https://github.com/n8n-io/n8n/commit/51e53ad9b77cc95c0fd8e683be4204ea9c7c85f7))
|
||||
* **editor:** Offer the managed OAuth choice when the app is supplied ([#40284](https://github.com/n8n-io/n8n/issues/40284)) ([14d2d30](https://github.com/n8n-io/n8n/commit/14d2d305630217271244c7fe962e36aab8f826f0))
|
||||
* **editor:** Place sticky notes and nodes at the right-click and clear the selection ([#41365](https://github.com/n8n-io/n8n/issues/41365)) ([28602ed](https://github.com/n8n-io/n8n/commit/28602ed56ea703c6f3d05652be00f9dc471f2828))
|
||||
* **editor:** Prevent doubled base path when opening folders with N8N_PATH set ([#33065](https://github.com/n8n-io/n8n/issues/33065)) ([a32c652](https://github.com/n8n-io/n8n/commit/a32c652c0957011bdea308439357b537d78ad1cd))
|
||||
* **editor:** Rename Admin Panel to Cloud console ([#41187](https://github.com/n8n-io/n8n/issues/41187)) ([8120328](https://github.com/n8n-io/n8n/commit/81203286613a2732301006cdd376f9a6ff3a8b1e))
|
||||
* **editor:** Restore From AI button corner radii ([#41176](https://github.com/n8n-io/n8n/issues/41176)) ([20ea671](https://github.com/n8n-io/n8n/commit/20ea6717a05c2bcef4d5541002cfe4b409618875))
|
||||
* **editor:** Restore the new session shortcut in agent chat ([#40907](https://github.com/n8n-io/n8n/issues/40907)) ([ebf078b](https://github.com/n8n-io/n8n/commit/ebf078b64c6f7578e0186300b5404804dc277feb))
|
||||
* **editor:** Return to the requested page after SSO sign-in ([#41151](https://github.com/n8n-io/n8n/issues/41151)) ([4a5fdd5](https://github.com/n8n-io/n8n/commit/4a5fdd5b68f676945e1e012963cafca64521058f))
|
||||
* **editor:** Save only the OpenTelemetry status when enabling or disabling ([#41142](https://github.com/n8n-io/n8n/issues/41142)) ([1903d53](https://github.com/n8n-io/n8n/commit/1903d53bc07f43729aa003fef4412a49c343e26f))
|
||||
* **editor:** Show follow-up run events in the Assistant after a reconnect ([#40883](https://github.com/n8n-io/n8n/issues/40883)) ([ffda639](https://github.com/n8n-io/n8n/commit/ffda639b3f5eac1ce1c58b0decff45b92682be04))
|
||||
* **editor:** Start the empty-state icon cards cycling on load and use them on the MCP and external secrets pages ([#40232](https://github.com/n8n-io/n8n/issues/40232)) ([1f223c8](https://github.com/n8n-io/n8n/commit/1f223c85f8a8fef43e785f86c27c9d4a75695b72))
|
||||
* **editor:** Stop auto-selecting the Evaluation Trigger over other triggers ([#38883](https://github.com/n8n-io/n8n/issues/38883)) ([8b5ea6d](https://github.com/n8n-io/n8n/commit/8b5ea6d9e0af5800c0a631e0a25665e0b1d3f3b9))
|
||||
* **editor:** Stop credential icon overlapping the credential name in the agent credential picker ([#41262](https://github.com/n8n-io/n8n/issues/41262)) ([70f0131](https://github.com/n8n-io/n8n/commit/70f0131be5a95e9d33bf4b542137b8d20ef59f32))
|
||||
* **editor:** Unify documentation links across settings pages ([#40289](https://github.com/n8n-io/n8n/issues/40289)) ([76cce99](https://github.com/n8n-io/n8n/commit/76cce9962685e585977fbcc3d400ae5654942bd3))
|
||||
* **editor:** Use the deployment name on the Az Foundry path ([#41103](https://github.com/n8n-io/n8n/issues/41103)) ([b3e8b7d](https://github.com/n8n-io/n8n/commit/b3e8b7d9ce0a00e97f5d03634a8928112019f060))
|
||||
* **Email Trigger (IMAP) Node:** Check the mailbox when the trigger activates ([#41013](https://github.com/n8n-io/n8n/issues/41013)) ([a6719b8](https://github.com/n8n-io/n8n/commit/a6719b8db71802ee6818f87266558417e302d11f))
|
||||
* **Email Trigger (IMAP) Node:** Find new mail on servers with a broken IMAP4rev2 mode ([#41018](https://github.com/n8n-io/n8n/issues/41018)) ([236220b](https://github.com/n8n-io/n8n/commit/236220b248bc6c553fd9a2f6f1734677b81a6a82))
|
||||
* **ERPNext Node:** Return each row once and tolerate empty Get Many options ([#36860](https://github.com/n8n-io/n8n/issues/36860)) ([3968a60](https://github.com/n8n-io/n8n/commit/3968a60bf74dbe7b72996fda53a5b5eaaed272a4))
|
||||
* **Execute Command Node:** Don't spawn console on Windows ([#40909](https://github.com/n8n-io/n8n/issues/40909)) ([52aed7a](https://github.com/n8n-io/n8n/commit/52aed7ac97c641f125e74bb68964bc31a29ef701))
|
||||
* **GitlabTrigger:** use NodeApiError.httpCode and safe description access in webhook checkExists ([#35486](https://github.com/n8n-io/n8n/issues/35486)) ([054288e](https://github.com/n8n-io/n8n/commit/054288e48855b5aa6372be2a4bf1864c9401f6c1))
|
||||
* **Gmail Node:** Keep the References header on a draft created in a thread ([#41392](https://github.com/n8n-io/n8n/issues/41392)) ([06ba85c](https://github.com/n8n-io/n8n/commit/06ba85cbb9bfeb7db6899c411e19bfb0269430af))
|
||||
* **HomeAssistant Node:** Only sent Home Assistant attributes on upsert if set ([#36298](https://github.com/n8n-io/n8n/issues/36298)) ([b8780d8](https://github.com/n8n-io/n8n/commit/b8780d8fee5e445a12d3583041a1e09a36213254))
|
||||
* Keep free-floating sticky notes clear of workflow nodes ([#41195](https://github.com/n8n-io/n8n/issues/41195)) ([131e9c8](https://github.com/n8n-io/n8n/commit/131e9c8679f96254873d401fbb7e9e57c5e976f2))
|
||||
* **MCP Trigger Node:** Pass through MCP content arrays instead of double-wrapping tool results ([#39131](https://github.com/n8n-io/n8n/issues/39131)) ([f483ebd](https://github.com/n8n-io/n8n/commit/f483ebd9d1b8bb032d05aab335413256aa590305))
|
||||
* **Microsoft Teams Node:** Add the TeamMember.Read.All scope and correct the Trigger scope notice ([#39819](https://github.com/n8n-io/n8n/issues/39819)) ([ddc9769](https://github.com/n8n-io/n8n/commit/ddc97694b5ad3d7888923940c2e9a20fe2fb042e))
|
||||
* **OpenAI Node:** Report token usage for image operations ([#38463](https://github.com/n8n-io/n8n/issues/38463)) ([a3f4711](https://github.com/n8n-io/n8n/commit/a3f47117cf6484184b4b4604319cf3d3191b0e16))
|
||||
* **OpenAI Node:** Send max_completion_tokens when it analyzes an image ([#38656](https://github.com/n8n-io/n8n/issues/38656)) ([e972309](https://github.com/n8n-io/n8n/commit/e9723095d6d0360d8d8afc1ea28d34dbce16feb6))
|
||||
* Pass structured inputs to OpenAI Responses tools ([#40638](https://github.com/n8n-io/n8n/issues/40638)) ([1d2c879](https://github.com/n8n-io/n8n/commit/1d2c879fabcfe100afc2cc66ceb1c66551a17b66))
|
||||
* **Postgres Node:** Keep comma-containing values as one query parameter ([#40240](https://github.com/n8n-io/n8n/issues/40240)) ([f5dc06f](https://github.com/n8n-io/n8n/commit/f5dc06f753e54de31b089c095f980057e95aa33b))
|
||||
* Restore command bar focus and standardize creation icons ([#41185](https://github.com/n8n-io/n8n/issues/41185)) ([9e3b52c](https://github.com/n8n-io/n8n/commit/9e3b52cbbb9336118e07e68a5bdec7e56047ec76))
|
||||
* **Send Email Node:** Send to every recipient when a recipient field is an array ([#36467](https://github.com/n8n-io/n8n/issues/36467)) ([660849d](https://github.com/n8n-io/n8n/commit/660849d93e3fa02fde0e12f33ce93a80f18643eb))
|
||||
* Skip dependency files and continue after custom node load ([#41117](https://github.com/n8n-io/n8n/issues/41117)) ([5a3fc39](https://github.com/n8n-io/n8n/commit/5a3fc3997284b3fdd7a9285b52f4a16230bd3b44))
|
||||
* **Splunk Node:** Preserve errors when continuing after failed items ([#40368](https://github.com/n8n-io/n8n/issues/40368)) ([e29c3a4](https://github.com/n8n-io/n8n/commit/e29c3a40856d17914649394036905ea9138fdda3))
|
||||
* Validate attendee values ([#40953](https://github.com/n8n-io/n8n/issues/40953)) ([21221ad](https://github.com/n8n-io/n8n/commit/21221adc78ff3bb5695c96b487418329e75484be))
|
||||
* **core:** Keep Observer detail lines instead of discarding the response ([#41157](https://github.com/n8n-io/n8n/issues/41157)) ([c09e0a9](https://github.com/n8n-io/n8n/commit/c09e0a94f57d7b046232b78f6cf2c9066784f87e))
|
||||
|
||||
|
||||
## [2.44.2](https://github.com/n8n-io/n8n/compare/n8n@2.44.1...n8n@2.44.2) (2026-10-15)
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **core:** Create an XML parser for each webhook request body ([#41112](https://github.com/n8n-io/n8n/issues/41112)) ([c52f181](https://github.com/n8n-io/n8n/commit/c52f1817b38cb03758fee45b1109caae12411ffa))
|
||||
|
||||
|
||||
## [2.44.1](https://github.com/n8n-io/n8n/compare/n8n@2.44.0...n8n@2.44.1) (2026-10-14)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **API:** Honor Idempotency-Key on Public API POST requests ([#40978](https://github.com/n8n-io/n8n/issues/40978)) ([b7b3d72](https://github.com/n8n-io/n8n/commit/b7b3d7273756ae137e7ba45007f09735d76f6362))
|
||||
* **API:** Return the parent folder on workflow read endpoints ([#40981](https://github.com/n8n-io/n8n/issues/40981)) ([a906d2f](https://github.com/n8n-io/n8n/commit/a906d2fdfcc5481799a571b4b666eea864fa07a1))
|
||||
* **Azure OpenAI Chat Model Node:** Add an optional Project field to the Foundry credentials ([#41181](https://github.com/n8n-io/n8n/issues/41181)) ([0c1a787](https://github.com/n8n-io/n8n/commit/0c1a78793a5546851d0585607c06767790d25ccb))
|
||||
* **core:** Add an endpoint that notifies workflow owners about migration findings ([#40952](https://github.com/n8n-io/n8n/issues/40952)) ([d8a0666](https://github.com/n8n-io/n8n/commit/d8a06666daf8b9ddda768a5ed6d419bd48b420fc))
|
||||
* **core:** Add an optional model for n8n Assistant memory tasks ([#41137](https://github.com/n8n-io/n8n/issues/41137)) ([f34c093](https://github.com/n8n-io/n8n/commit/f34c093f082978f29e44fdb71620f0047ef75ca5))
|
||||
* **core:** Add create-with-values credential missing mode to package import ([#36767](https://github.com/n8n-io/n8n/issues/36767)) ([d3bc275](https://github.com/n8n-io/n8n/commit/d3bc2754b44b7a350927bf3b0d9dcfe6e377a155))
|
||||
* **core:** Add self-healing results to Inbox ([#40741](https://github.com/n8n-io/n8n/issues/40741)) ([1405d4a](https://github.com/n8n-io/n8n/commit/1405d4a351db542b363f5f2c8675e948b353c3ee))
|
||||
* **core:** Add the blocking project and opt-in state to policy violations ([#40985](https://github.com/n8n-io/n8n/issues/40985)) ([4419fc4](https://github.com/n8n-io/n8n/commit/4419fc4ba237fa825dbeb90d9e697994c96b58aa))
|
||||
* **core:** Add typing indicator, media download and media retention … ([#40920](https://github.com/n8n-io/n8n/issues/40920)) ([adedbbf](https://github.com/n8n-io/n8n/commit/adedbbf5608e70acca75e6481ae5b98c518a9f6d))
|
||||
* **core:** Add WhatsApp credential check and webhook handshake status ([#40921](https://github.com/n8n-io/n8n/issues/40921)) ([9a524e7](https://github.com/n8n-io/n8n/commit/9a524e7c21ab5ae09c74d9fbae9f2ab5a8f95fe6))
|
||||
* **core:** Append node tool attributions to agent replies and add Genie attribution and deep links to the Databricks node ([#40765](https://github.com/n8n-io/n8n/issues/40765)) ([44c8a31](https://github.com/n8n-io/n8n/commit/44c8a312c37e15ba226c05fd675b7702da25639a))
|
||||
* **core:** Apply budget alerts and stops to sub-agent run ([#40884](https://github.com/n8n-io/n8n/issues/40884)) ([9498088](https://github.com/n8n-io/n8n/commit/949808825637633d6114afb06bdaa28b023107d7))
|
||||
* **core:** Enable node and credential type policies by default ([#41022](https://github.com/n8n-io/n8n/issues/41022)) ([dd79a61](https://github.com/n8n-io/n8n/commit/dd79a610c5afa1daa07aa548c795d8cad98e6187))
|
||||
* **core:** Enable the durable scheduler by default ([#40891](https://github.com/n8n-io/n8n/issues/40891)) ([f6861fe](https://github.com/n8n-io/n8n/commit/f6861fe4948567ff792c71aedaba52624ab48fdc))
|
||||
* **core:** Implement automated migrations for removed nodes ([#40877](https://github.com/n8n-io/n8n/issues/40877)) ([739c59e](https://github.com/n8n-io/n8n/commit/739c59e9913a27328256a10b6570d48402ef5779))
|
||||
* **core:** Only show execution input and output for agent subworkflows in timeline ([#41189](https://github.com/n8n-io/n8n/issues/41189)) ([4d292a6](https://github.com/n8n-io/n8n/commit/4d292a6009d14a8c634d802ef09c54c4cab5acf0))
|
||||
* **core:** Refuse revoked access tokens of the local server on the inbound pipeline ([#41266](https://github.com/n8n-io/n8n/issues/41266)) ([0936dfd](https://github.com/n8n-io/n8n/commit/0936dfddc32280a45e270ef1ec12c33f31c54795))
|
||||
* **core:** Remove the startup pass from workflow history compaction ([#40886](https://github.com/n8n-io/n8n/issues/40886)) ([45eea1b](https://github.com/n8n-io/n8n/commit/45eea1bc35355305409988bc3b1ccb0ac74bf649))
|
||||
* **core:** Report the CPU and memory limits n8n was given in telemetry ([#40421](https://github.com/n8n-io/n8n/issues/40421)) ([d6d24f2](https://github.com/n8n-io/n8n/commit/d6d24f267c9658aae7a565d739cdda11f8d580e7))
|
||||
* **core:** Report the runtime n8n runs on in telemetry ([#40420](https://github.com/n8n-io/n8n/issues/40420)) ([b2edcb5](https://github.com/n8n-io/n8n/commit/b2edcb5727ea07c08428056ec9a36f84b486c244))
|
||||
* **core:** Run the Form and Chat trigger redirect login against a trusted source ([#41123](https://github.com/n8n-io/n8n/issues/41123)) ([862c7ec](https://github.com/n8n-io/n8n/commit/862c7eceab3bfe6b3e757a37c4806124c3d1be99))
|
||||
* **core:** Store promotion exports in n8n directory ([#40863](https://github.com/n8n-io/n8n/issues/40863)) ([333a010](https://github.com/n8n-io/n8n/commit/333a0101121352f032c93e6b10c268eb59bb4dfc))
|
||||
* **core:** Surface the responding model name in LLM sub-node run data ([#41161](https://github.com/n8n-io/n8n/issues/41161)) ([b10c97e](https://github.com/n8n-io/n8n/commit/b10c97ed453b8d6fff043677e67b0a7e82866eba))
|
||||
* **core:** Suspend in-flight production executions at worker shutdown ([#40568](https://github.com/n8n-io/n8n/issues/40568)) ([f3e2bd8](https://github.com/n8n-io/n8n/commit/f3e2bd83b3a18308324971e5275b39f7a88e4395))
|
||||
* **core:** Treat notified migration findings as unresolved ([#40951](https://github.com/n8n-io/n8n/issues/40951)) ([347385b](https://github.com/n8n-io/n8n/commit/347385b07b449a26ac214f69d0d198385a108581))
|
||||
* **Databricks Node:** Show the Vector Store, Embeddings and AI Agent nodes in the node picker ([#41206](https://github.com/n8n-io/n8n/issues/41206)) ([191ea98](https://github.com/n8n-io/n8n/commit/191ea984824ba92ab33e9d2019a125af23b4712c))
|
||||
* Don't save MCP connections without user approval ([#39996](https://github.com/n8n-io/n8n/issues/39996)) ([d27de60](https://github.com/n8n-io/n8n/commit/d27de60b955fcc4afcda4319873431f2ce1b2bd7))
|
||||
* **editor:** Align the WhatsApp channel setup and settings with the … ([#41283](https://github.com/n8n-io/n8n/issues/41283)) ([275f7eb](https://github.com/n8n-io/n8n/commit/275f7ebf685b67cfcfe0c006f0116b04a8249d4b))
|
||||
* **editor:** Extend N8nCopyInput with accessibility, redaction, loading and actions ([#34942](https://github.com/n8n-io/n8n/issues/34942)) ([c084d99](https://github.com/n8n-io/n8n/commit/c084d996afd41406ca5a21b4fcdf2f9aa0f43a0c))
|
||||
* **editor:** Let admins notify workflow owners from the migration rule detail ([#40949](https://github.com/n8n-io/n8n/issues/40949)) ([a972eeb](https://github.com/n8n-io/n8n/commit/a972eebe513b0e43e2fa32eb3c726626c1f9cdc0))
|
||||
* **editor:** Let admins notify workflow owners in bulk from the migration rule detail ([#41060](https://github.com/n8n-io/n8n/issues/41060)) ([de42c95](https://github.com/n8n-io/n8n/commit/de42c9515d43a7357bb196f224c65e22bf173a9a))
|
||||
* **editor:** Link to the credential sharing tab when execution data is redacted ([#40976](https://github.com/n8n-io/n8n/issues/40976)) ([da3491f](https://github.com/n8n-io/n8n/commit/da3491fc8786525670e24d35a2c136a0482b32cc))
|
||||
* **editor:** Publish agent modal and modal for warning about unpublished workflows ([#40987](https://github.com/n8n-io/n8n/issues/40987)) ([e47940e](https://github.com/n8n-io/n8n/commit/e47940eecd3523105c7c097e949c4da91770328e))
|
||||
* **editor:** Rework the WhatsApp channel setup to match the new design ([#40923](https://github.com/n8n-io/n8n/issues/40923)) ([af0202d](https://github.com/n8n-io/n8n/commit/af0202d840b4f48a31011e4c28fd9759df17a7a1))
|
||||
* **editor:** Show agent personalisation icon in subagent modal ([#41094](https://github.com/n8n-io/n8n/issues/41094)) ([6160624](https://github.com/n8n-io/n8n/commit/616062487ae56e85a31002d7116917b9c7815069))
|
||||
* **editor:** Show multiple workflows in the review detail view ([#40695](https://github.com/n8n-io/n8n/issues/40695)) ([f6d70c1](https://github.com/n8n-io/n8n/commit/f6d70c140b9bdd9cb4f777311610b8a8909bbbfe))
|
||||
* **editor:** Show policy-blocked items in the source control pull modal ([#40604](https://github.com/n8n-io/n8n/issues/40604)) ([1ea18e9](https://github.com/n8n-io/n8n/commit/1ea18e9ef9e95cead7fe91ad4a7cb3fa46b3dfd7))
|
||||
* **editor:** Show the model that answered in the AI log header ([#40988](https://github.com/n8n-io/n8n/issues/40988)) ([caeaff7](https://github.com/n8n-io/n8n/commit/caeaff72300b4af739a6256046faa637e72627ca))
|
||||
* **editor:** Show the policy reason when an agent tool call is refused ([#40673](https://github.com/n8n-io/n8n/issues/40673)) ([bb45bca](https://github.com/n8n-io/n8n/commit/bb45bca5ff7ef7d2cd2b2f2e98812d96a5971b01))
|
||||
* **editor:** Update budget panel when agent executes ([#40971](https://github.com/n8n-io/n8n/issues/40971)) ([632d8b9](https://github.com/n8n-io/n8n/commit/632d8b9eb0bb1f088856520b7770e75e115ffd60))
|
||||
* **editor:** Update UI with n8n rebrand ([#40342](https://github.com/n8n-io/n8n/issues/40342)) ([bffe7d8](https://github.com/n8n-io/n8n/commit/bffe7d8392d614b542beef65867a88aa671baa89))
|
||||
* Implements workflow previews above Assistant empty state experiment ([#40913](https://github.com/n8n-io/n8n/issues/40913)) ([eac2f46](https://github.com/n8n-io/n8n/commit/eac2f46b0c7a0e82c60d4391986c80865367124b))
|
||||
* **Kafka Node:** Make version 2 the default for new Kafka nodes and triggers ([#40744](https://github.com/n8n-io/n8n/issues/40744)) ([e5866ac](https://github.com/n8n-io/n8n/commit/e5866ac6c17c61cd7bfdfc9b49294c936af3d566))
|
||||
* **Monday.com Node:** Add V2 with full V1 parity as the default ([#35515](https://github.com/n8n-io/n8n/issues/35515)) ([8977833](https://github.com/n8n-io/n8n/commit/8977833ee2f76df7f92a7764589d5b3164706f46))
|
||||
* Overhaul MCP tool permissions for Agents ([#39878](https://github.com/n8n-io/n8n/issues/39878)) ([bda9118](https://github.com/n8n-io/n8n/commit/bda9118168ce47b709355aca12bfb0e0046ba479))
|
||||
* Show Gemini models by name and list prioritized models first in model pickers ([#40965](https://github.com/n8n-io/n8n/issues/40965)) ([acf5291](https://github.com/n8n-io/n8n/commit/acf5291d800aeffec50911f9d63215e5c12dc134))
|
||||
* **Text Classifier Node:** Add opt-in confidence scores to the output ([#40743](https://github.com/n8n-io/n8n/issues/40743)) ([3800bce](https://github.com/n8n-io/n8n/commit/3800bce8126cebbff80536e45851e7f21d9ad8ec))
|
||||
|
||||
|
||||
### Performance Improvements
|
||||
|
||||
* **editor:** Load prettier only when the user formats code ([#41014](https://github.com/n8n-io/n8n/issues/41014)) ([e2529a8](https://github.com/n8n-io/n8n/commit/e2529a849432bdb3082615c04b1702df61cb91d5))
|
||||
* Show Gemini models by name and list prioritized models first in model pickers ([#41015](https://github.com/n8n-io/n8n/issues/41015)) ([321b020](https://github.com/n8n-io/n8n/commit/321b0203b73e3351c766cbece6a8f79c800d453f))
|
||||
|
||||
|
||||
# [2.44.0](https://github.com/n8n-io/n8n/compare/n8n@2.43.0...n8n@2.44.0) (2026-10-13)
|
||||
|
|
|
|||
143
OWNERS
143
OWNERS
|
|
@ -30,7 +30,6 @@ packages/@n8n/backend-network/ @n8n-io/catalysts
|
|||
packages/@n8n/backend-services/ @n8n-io/catalysts
|
||||
packages/@n8n/backend-services/src/services/role.service.ts @n8n-io/iam
|
||||
packages/@n8n/backend-services/src/services/role-cache.service.ts @n8n-io/iam
|
||||
packages/@n8n/backend-services/src/services/jwt.service.ts @n8n-io/iam
|
||||
packages/@n8n/backend-test-utils/ @n8n-io/catalysts
|
||||
packages/@n8n/di/ @n8n-io/catalysts
|
||||
packages/@n8n/errors/ @n8n-io/catalysts
|
||||
|
|
@ -85,6 +84,7 @@ packages/cli/src/security-audit/ @n8n-io/catalysts
|
|||
packages/cli/src/modules/workflow-index/ @n8n-io/catalysts
|
||||
packages/cli/src/modules/breaking-changes/ @n8n-io/catalysts
|
||||
packages/cli/src/workflows/publication/ @n8n-io/catalysts required
|
||||
packages/cli/src/modules/otel/ @n8n-io/ligo
|
||||
|
||||
packages/cli/src/auth/ @n8n-io/iam
|
||||
packages/cli/src/credentials/ @n8n-io/iam
|
||||
|
|
@ -114,19 +114,17 @@ packages/cli/src/collaboration/ @n8n-io/catalysts
|
|||
packages/cli/src/binary-data/ @n8n-io/catalysts
|
||||
packages/cli/src/posthog/ @n8n-io/growth
|
||||
packages/cli/src/modules/data-table/ @n8n-io/knowledge
|
||||
packages/cli/src/modules/knowledge/ @n8n-io/knowledge
|
||||
|
||||
packages/cli/src/evaluation.ee/ @n8n-io/ai-trust
|
||||
packages/cli/src/chat/ @n8n-io/ai-agents
|
||||
packages/cli/src/modules/agents/ @n8n-io/ai-agents
|
||||
packages/cli/src/evaluation.ee/ @n8n-io/ai
|
||||
packages/cli/src/chat/ @n8n-io/ai
|
||||
packages/cli/src/tool-generation/ @n8n-io/ai
|
||||
packages/modules/workflow-builder/ @n8n-io/ai-assistant
|
||||
packages/cli/src/modules/workflow-builder/ @n8n-io/ai-assistant
|
||||
packages/cli/src/modules/mcp/ @n8n-io/context
|
||||
packages/cli/src/modules/skills/ @n8n-io/context
|
||||
packages/cli/test/integration/skills/ @n8n-io/context
|
||||
packages/cli/src/modules/quick-connect/ @n8n-io/ai
|
||||
packages/cli/src/modules/chat-hub/ @n8n-io/ai-assistant
|
||||
packages/cli/src/modules/instance-ai/ @n8n-io/ai-assistant
|
||||
packages/modules/inbox/ @n8n-io/ligo
|
||||
|
||||
packages/cli/src/modules/community-packages/ @n8n-io/nodes
|
||||
|
||||
# CLI controllers
|
||||
packages/cli/src/controllers/auth.controller.ts @n8n-io/iam
|
||||
|
|
@ -156,6 +154,7 @@ packages/cli/src/controllers/dynamic-node-parameters.controller.ts @n8n-io/node
|
|||
packages/cli/src/controllers/e2e.controller.ts @n8n-io/qa-dx
|
||||
|
||||
# CLI services
|
||||
packages/cli/src/services/jwt.service.ts @n8n-io/iam
|
||||
packages/cli/src/services/user.service.ts @n8n-io/iam
|
||||
packages/cli/src/services/password.utility.ts @n8n-io/iam
|
||||
packages/cli/src/services/public-api-key.service.ts @n8n-io/iam
|
||||
|
|
@ -167,7 +166,7 @@ packages/cli/src/services/ai-usage.service.ts @n8n-io/ai-assistant
|
|||
packages/cli/src/services/__tests__/ai-usage.service.test.ts @n8n-io/ai-assistant
|
||||
packages/cli/src/controllers/__tests__/ai.controller.test.ts @n8n-io/ai-assistant
|
||||
packages/cli/src/services/ai-workflow-builder.service.ts @n8n-io/ai-assistant
|
||||
packages/cli/src/services/annotation-tag.service.ee.ts @n8n-io/context
|
||||
packages/cli/src/services/annotation-tag.service.ee.ts @n8n-io/ai
|
||||
packages/cli/src/services/folder.service.ts @n8n-io/context
|
||||
packages/cli/src/services/tag.service.ts @n8n-io/context
|
||||
packages/cli/src/services/cta.service.ts @n8n-io/knowledge
|
||||
|
|
@ -197,42 +196,15 @@ packages/frontend/@n8n/storybook/ @n8n-io/design-engineeri
|
|||
packages/frontend/@n8n/i18n/ @n8n-io/frontend
|
||||
packages/@n8n/stylelint-config/ @n8n-io/qa-dx
|
||||
|
||||
packages/frontend/editor-ui/src/features/ai/gateway/ @n8n-io/relay
|
||||
packages/frontend/editor-ui/src/app/composables/useAiGatewayTopUp.ts @n8n-io/relay
|
||||
packages/frontend/editor-ui/src/app/composables/useFreeAiCredits.ts @n8n-io/relay
|
||||
|
||||
packages/frontend/editor-ui/src/features/settings/communityNodes/ @n8n-io/nodes
|
||||
packages/frontend/editor-ui/src/features/credentials/quickConnect/ @n8n-io/nodes required
|
||||
packages/frontend/editor-ui/src/features/ndv/runData/schemaPreview.api.ts @n8n-io/nodes
|
||||
packages/frontend/editor-ui/src/features/ndv/runData/schemaPreview.store.ts @n8n-io/nodes
|
||||
packages/frontend/editor-ui/src/features/agents/channels/ @n8n-io/nodes
|
||||
|
||||
# AI
|
||||
|
||||
packages/@n8n/instance-ai/ @n8n-io/ai-assistant
|
||||
packages/@n8n/nodes-langchain/ @n8n-io/ai
|
||||
packages/@n8n/nodes-langchain/nodes/agents/ @n8n-io/ai-agents
|
||||
packages/@n8n/nodes-langchain/nodes/chains/ @n8n-io/ai-agents
|
||||
packages/@n8n/nodes-langchain/nodes/code/ @n8n-io/ai-agents
|
||||
packages/@n8n/nodes-langchain/nodes/output_parser/ @n8n-io/ai-agents
|
||||
packages/@n8n/nodes-langchain/nodes/llms/ @n8n-io/ai-agents
|
||||
packages/@n8n/nodes-langchain/nodes/vendors/ @n8n-io/ai-agents
|
||||
packages/@n8n/nodes-langchain/nodes/ModelSelector/ @n8n-io/ai-agents
|
||||
packages/@n8n/nodes-langchain/nodes/embeddings/ @n8n-io/ai-agents
|
||||
packages/@n8n/nodes-langchain/nodes/vector_store/ @n8n-io/ai-agents
|
||||
packages/@n8n/nodes-langchain/nodes/document_loaders/ @n8n-io/ai-agents
|
||||
packages/@n8n/nodes-langchain/nodes/text_splitters/ @n8n-io/ai-agents
|
||||
packages/@n8n/nodes-langchain/nodes/retrievers/ @n8n-io/ai-agents
|
||||
packages/@n8n/nodes-langchain/nodes/rerankers/ @n8n-io/ai-agents
|
||||
packages/@n8n/nodes-langchain/nodes/memory/ @n8n-io/ai-agents
|
||||
packages/@n8n/nodes-langchain/nodes/tools/ @n8n-io/ai-agents
|
||||
packages/@n8n/nodes-langchain/nodes/ToolExecutor/ @n8n-io/ai-agents
|
||||
packages/@n8n/nodes-langchain/nodes/trigger/ @n8n-io/ai-agents
|
||||
packages/@n8n/ai-utilities/ @n8n-io/ai-agents
|
||||
packages/@n8n/ai-utilities/ @n8n-io/ai
|
||||
packages/@n8n/ai-node-sdk/ @n8n-io/ai
|
||||
packages/@n8n/ai-workflow-builder.ee/ @n8n-io/ai-assistant
|
||||
packages/@n8n/agents/ @n8n-io/ai-agents
|
||||
packages/frontend/@n8n/chat/ @n8n-io/ai-agents
|
||||
packages/frontend/editor-ui/src/features/agents/ @n8n-io/ai-agents
|
||||
packages/@n8n/agents/ @n8n-io/ai
|
||||
packages/frontend/@n8n/chat/ @n8n-io/ai
|
||||
|
||||
# Chat Hub
|
||||
|
||||
|
|
@ -247,7 +219,6 @@ packages/@n8n/nodes-langchain/nodes/vector_store/shared/chatHub.ts @n8n-io/ai-as
|
|||
packages/@n8n/nodes-langchain/credentials/ChatHubVectorStorePGVectorApi.credentials.ts @n8n-io/ai-assistant
|
||||
packages/@n8n/nodes-langchain/credentials/ChatHubVectorStorePineconeApi.credentials.ts @n8n-io/ai-assistant
|
||||
packages/@n8n/nodes-langchain/credentials/ChatHubVectorStoreQdrantApi.credentials.ts @n8n-io/ai-assistant
|
||||
packages/@n8n/nodes-langchain/nodes/mcp/ @n8n-io/nodes required
|
||||
packages/frontend/editor-ui/src/features/shared/commandBar/composables/useChatHubCommands.ts @n8n-io/ai-assistant
|
||||
packages/cli/src/modules/breaking-changes/rules/v3/chat-hub-deprecated.rule.ts @n8n-io/ai-assistant
|
||||
|
||||
|
|
@ -257,89 +228,33 @@ packages/@n8n/codemirror-lang/ @n8n-io/nodes
|
|||
packages/@n8n/codemirror-lang-html/ @n8n-io/nodes
|
||||
packages/@n8n/codemirror-lang-sql/ @n8n-io/nodes
|
||||
packages/nodes-base/ @n8n-io/nodes
|
||||
packages/nodes-base/nodes/Code/ @n8n-io/nodes required
|
||||
packages/nodes-base/nodes/ExecuteCommand/ @n8n-io/nodes required
|
||||
packages/nodes-base/nodes/Ssh/ @n8n-io/nodes required
|
||||
packages/nodes-base/nodes/HttpRequest/ @n8n-io/nodes required
|
||||
packages/nodes-base/nodes/Webhook/ @n8n-io/nodes required
|
||||
packages/nodes-base/nodes/RespondToWebhook/ @n8n-io/nodes required
|
||||
packages/nodes-base/nodes/Form/ @n8n-io/nodes required
|
||||
packages/nodes-base/nodes/Wait/ @n8n-io/nodes required
|
||||
packages/nodes-base/utils/sendAndWait/ @n8n-io/nodes required
|
||||
packages/nodes-base/nodes/Git/ @n8n-io/nodes required
|
||||
packages/@n8n/decorators/ @n8n-io/catalysts
|
||||
packages/node-dev/ @n8n-io/nodes
|
||||
packages/@n8n/imap/ @n8n-io/nodes
|
||||
packages/@n8n/computer-use/ @n8n-io/nodes required
|
||||
packages/@n8n/local-gateway/ @n8n-io/nodes required
|
||||
packages/@n8n/mcp-browser/ @n8n-io/nodes required
|
||||
packages/@n8n/mcp-browser-extension/ @n8n-io/nodes required
|
||||
packages/modules/quick-connect/ @n8n-io/nodes required
|
||||
packages/@n8n/api-types/src/dto/quick-connect/ @n8n-io/nodes required
|
||||
packages/@n8n/api-types/src/quick-connect.ts @n8n-io/nodes required
|
||||
packages/cli/src/modules/community-packages/ @n8n-io/nodes
|
||||
packages/cli/src/modules/mcp-registry/ @n8n-io/nodes required
|
||||
packages/cli/src/modules/agents/integrations/ @n8n-io/nodes
|
||||
packages/@n8n/api-types/src/schemas/mcp-registry.schema.ts @n8n-io/nodes required
|
||||
packages/cli/templates/send-and-wait-no-action-required.handlebars @n8n-io/nodes required
|
||||
packages/@n8n/ai-node-sdk/ @n8n-io/nodes required
|
||||
|
||||
# Relay (AI gateway & credits)
|
||||
packages/cli/src/services/ai-gateway.service.ts @n8n-io/relay required
|
||||
packages/cli/src/services/ai-gateway-eligibility.ts @n8n-io/relay required
|
||||
packages/cli/src/services/free-ai-credits.service.ts @n8n-io/relay required
|
||||
packages/@n8n/config/src/configs/ai-gateway.config.ts @n8n-io/relay
|
||||
packages/@n8n/api-types/src/dto/ai/ai-gateway-config-response.dto.ts @n8n-io/relay
|
||||
packages/@n8n/api-types/src/dto/ai/ai-gateway-usage-response.dto.ts @n8n-io/relay
|
||||
packages/@n8n/api-types/src/dto/ai/ai-gateway-wallet-response.dto.ts @n8n-io/relay
|
||||
packages/cli/src/modules/instance-ai/instance-ai-gateway.service.ts @n8n-io/relay
|
||||
packages/cli/src/modules/mcp/mcp-ai-gateway.helper.ts @n8n-io/relay
|
||||
packages/cli/src/modules/agents/json-config/node-tool-ai-gateway.service.ts @n8n-io/relay
|
||||
packages/frontend/editor-ui/src/app/stores/aiGateway.store.ts @n8n-io/relay
|
||||
packages/frontend/editor-ui/src/app/composables/useAiGateway.ts @n8n-io/relay
|
||||
packages/frontend/editor-ui/src/features/credentials/gatewayCreditsPromotion/ @n8n-io/relay
|
||||
packages/frontend/editor-ui/src/features/shared/nodeCreator/components/SectionHeaderCreditsTag.vue @n8n-io/relay
|
||||
packages/@n8n/create-node/ @n8n-io/nodes
|
||||
packages/@n8n/node-cli/ @n8n-io/community-engineering
|
||||
packages/@n8n/imap/ @n8n-io/iam
|
||||
packages/@n8n/syslog-client/ @n8n-io/iam
|
||||
packages/@n8n/scan-community-package/ @n8n-io/community-engineering
|
||||
packages/@n8n/eslint-plugin-community-nodes/ @n8n-io/community-engineering
|
||||
packages/@n8n/computer-use/ @n8n-io/nodes
|
||||
packages/@n8n/local-gateway/ @n8n-io/nodes
|
||||
packages/@n8n/mcp-browser/ @n8n-io/nodes
|
||||
packages/@n8n/mcp-browser-extension/ @n8n-io/nodes
|
||||
|
||||
# Community engineering
|
||||
.agents/skills/create-community-node-lint-rule/ @n8n-io/community-engineering
|
||||
.agents/skills/community-pr-readiness-check/ @n8n-io/community-engineering
|
||||
packages/@n8n/scan-community-package/ @n8n-io/community-engineering
|
||||
packages/@n8n/eslint-plugin-community-nodes/ @n8n-io/community-engineering
|
||||
packages/@n8n/node-cli/ @n8n-io/community-engineering
|
||||
packages/@n8n/create-node/ @n8n-io/community-engineering
|
||||
|
||||
# IAM
|
||||
|
||||
packages/@n8n/permissions/ @n8n-io/iam
|
||||
packages/@n8n/client-oauth2/ @n8n-io/iam
|
||||
packages/@n8n/inbound-auth/ @n8n-io/iam
|
||||
packages/@n8n/syslog-client/ @n8n-io/iam
|
||||
|
||||
# Governance & Observability
|
||||
|
||||
packages/modules/type-availability-policies/ @n8n-io/governance-and-observability
|
||||
packages/cli/src/modules/type-availability-policies/ @n8n-io/governance-and-observability
|
||||
packages/modules/otel/ @n8n-io/governance-and-observability
|
||||
packages/cli/src/modules/otel/ @n8n-io/governance-and-observability
|
||||
packages/cli/src/public-api/v1/controllers/otel.public.controller.ts @n8n-io/governance-and-observability
|
||||
packages/cli/src/public-api/v1/shared/otel.mapper.ts @n8n-io/governance-and-observability
|
||||
packages/cli/src/public-api/v1/controllers/node-type-policies.public.controller.ts @n8n-io/governance-and-observability
|
||||
packages/cli/src/public-api/v1/controllers/credential-type-policies.public.controller.ts @n8n-io/governance-and-observability
|
||||
packages/cli/src/public-api/v1/handlers/node-type-policies/ @n8n-io/governance-and-observability
|
||||
packages/cli/src/public-api/v1/handlers/credential-type-policies/ @n8n-io/governance-and-observability
|
||||
packages/cli/src/modules/policy-infrastructure/ @n8n-io/governance-and-observability
|
||||
packages/cli/src/policy/ @n8n-io/governance-and-observability
|
||||
packages/@n8n/decorators/src/policy-check/ @n8n-io/governance-and-observability
|
||||
packages/@n8n/api-types/src/dto/otel/ @n8n-io/governance-and-observability
|
||||
packages/@n8n/api-types/src/dto/type-availability-policies/ @n8n-io/governance-and-observability
|
||||
packages/@n8n/api-types/src/node-type-availability.ts @n8n-io/governance-and-observability
|
||||
packages/@n8n/api-types/src/credential-type-availability.ts @n8n-io/governance-and-observability
|
||||
packages/@n8n/api-types/src/schemas/policy-check-failure.schema.ts @n8n-io/governance-and-observability
|
||||
packages/@n8n/api-types/src/schemas/policy-violation.schema.ts @n8n-io/governance-and-observability
|
||||
packages/@n8n/telemetry/src/events/policies.ts @n8n-io/governance-and-observability
|
||||
packages/@n8n/telemetry/src/events/type-availability-policies.ts @n8n-io/governance-and-observability
|
||||
packages/workflow/src/errors/policy-refusal.ts @n8n-io/governance-and-observability
|
||||
packages/frontend/editor-ui/src/app/composables/usePolicyViolationToast.ts @n8n-io/governance-and-observability
|
||||
|
||||
# CI/CD
|
||||
|
||||
|
|
@ -367,15 +282,3 @@ packages/@n8n/db/src/entities/deployment-key.ts @n8n-io/iam required
|
|||
packages/@n8n/db/src/repositories/deployment-key.repository.ts @n8n-io/iam required
|
||||
packages/@n8n/eslint-config/src/configs/encryption-boundary.ts @n8n-io/iam required
|
||||
packages/@n8n/eslint-config/src/rules/no-encryption-guardrail-disable.ts @n8n-io/iam required
|
||||
|
||||
# AI Trust (evaluations)
|
||||
# Placed last so these rules override the broader package rules above.
|
||||
|
||||
packages/@n8n/instance-ai/evaluations/ @n8n-io/ai-trust
|
||||
packages/cli/src/modules/instance-ai/eval/ @n8n-io/ai-trust
|
||||
packages/quality/testing/playwright/tests/evals/instance-ai/ @n8n-io/ai-trust
|
||||
.agents/skills/create-instance-ai-eval/ @n8n-io/ai-trust
|
||||
.agents/skills/create-agent-builder-eval/ @n8n-io/ai-trust
|
||||
packages/cli/src/modules/agent-evals/ @n8n-io/ai-trust
|
||||
packages/@n8n/agents/src/evals/ @n8n-io/ai-trust
|
||||
packages/frontend/editor-ui/src/features/ai/evaluation.ee/ @n8n-io/ai-trust
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
|
||||
|
||||
|
||||
# n8n – The Platform for AI Agents and Workflow Automation
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"repository": "n8n-io/cat-bot",
|
||||
"releaseTag": "harness-v0.6.0",
|
||||
"version": "0.6.0",
|
||||
"assetName": "n8n-opencode-harness-0.6.0.tgz"
|
||||
"releaseTag": "harness-v0.3.0",
|
||||
"version": "0.3.0",
|
||||
"assetName": "n8n-opencode-harness-0.3.0.tgz"
|
||||
}
|
||||
|
|
|
|||
Binary file not shown.
|
Before Width: | Height: | Size: 355 KiB After Width: | Height: | Size: 98 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 289 KiB |
|
|
@ -15,11 +15,11 @@ set -eu
|
|||
SCRIPT_VERSION="1.4.0"
|
||||
# The version to install is derived from the latest stable GitHub release in
|
||||
# resolve_n8n_version(); this fallback only applies when that lookup fails.
|
||||
FALLBACK_N8N_VERSION="2.43.4"
|
||||
FALLBACK_N8N_VERSION="2.32.0"
|
||||
# Sandbox service release (API, runner and sandbox images ship together).
|
||||
# Pinned here, not resolved at run time: the stack definition must match it.
|
||||
# --upgrade moves existing installs to this pin.
|
||||
SANDBOX_VERSION="1.8.1"
|
||||
SANDBOX_VERSION="1.3.4"
|
||||
N8N_DIR="${N8N_DIR:-./n8n}"
|
||||
N8N_PORT=5678
|
||||
SOURCE_URL="https://github.com/n8n-io/n8n/blob/master/docker/get-n8n.sh"
|
||||
|
|
|
|||
199
docs/generated/postgres-schema/README.md
generated
199
docs/generated/postgres-schema/README.md
generated
|
|
@ -9,7 +9,7 @@ Auto-generated from the PostgreSQL migrations in @n8n/db. Do not edit by hand.
|
|||
| Name | Columns | Comment | Type |
|
||||
| ---- | ------- | ------- | ---- |
|
||||
| [public.activity_event](public.activity_event.md) | 11 | | BASE TABLE |
|
||||
| [public.agent_background_job](public.agent_background_job.md) | 23 | | BASE TABLE |
|
||||
| [public.agent_background_job](public.agent_background_job.md) | 20 | | BASE TABLE |
|
||||
| [public.agent_budget_applied_call](public.agent_budget_applied_call.md) | 2 | | BASE TABLE |
|
||||
| [public.agent_budget_spend](public.agent_budget_spend.md) | 4 | | BASE TABLE |
|
||||
| [public.agent_channel_status](public.agent_channel_status.md) | 11 | | BASE TABLE |
|
||||
|
|
@ -23,20 +23,18 @@ Auto-generated from the PostgreSQL migrations in @n8n/db. Do not edit by hand.
|
|||
| [public.agent_eval_run](public.agent_eval_run.md) | 14 | | BASE TABLE |
|
||||
| [public.agent_execution](public.agent_execution.md) | 23 | | BASE TABLE |
|
||||
| [public.agent_execution_message_links](public.agent_execution_message_links.md) | 5 | | BASE TABLE |
|
||||
| [public.agent_execution_threads](public.agent_execution_threads.md) | 20 | | BASE TABLE |
|
||||
| [public.agent_execution_threads](public.agent_execution_threads.md) | 19 | | BASE TABLE |
|
||||
| [public.agent_files](public.agent_files.md) | 10 | | BASE TABLE |
|
||||
| [public.agent_history](public.agent_history.md) | 9 | | BASE TABLE |
|
||||
| [public.agent_message_queue](public.agent_message_queue.md) | 10 | | BASE TABLE |
|
||||
| [public.agent_plan](public.agent_plan.md) | 8 | | BASE TABLE |
|
||||
| [public.agent_plan_history](public.agent_plan_history.md) | 6 | | BASE TABLE |
|
||||
| [public.agent_skill_dependency](public.agent_skill_dependency.md) | 4 | | BASE TABLE |
|
||||
| [public.agent_task_definition](public.agent_task_definition.md) | 8 | | BASE TABLE |
|
||||
| [public.agent_task_run_lock](public.agent_task_run_lock.md) | 6 | | BASE TABLE |
|
||||
| [public.agent_task_snapshot](public.agent_task_snapshot.md) | 9 | | BASE TABLE |
|
||||
| [public.agent_thread_grants](public.agent_thread_grants.md) | 4 | | BASE TABLE |
|
||||
| [public.agent_version_skill_dependency](public.agent_version_skill_dependency.md) | 4 | | BASE TABLE |
|
||||
| [public.agent_workflow_dependency](public.agent_workflow_dependency.md) | 3 | | BASE TABLE |
|
||||
| [public.agents](public.agents.md) | 15 | | BASE TABLE |
|
||||
| [public.agents](public.agents.md) | 14 | | BASE TABLE |
|
||||
| [public.agents_memory_entries](public.agents_memory_entries.md) | 13 | | BASE TABLE |
|
||||
| [public.agents_memory_entry_candidates](public.agents_memory_entry_candidates.md) | 14 | | BASE TABLE |
|
||||
| [public.agents_memory_entry_locks](public.agents_memory_entry_locks.md) | 6 | | BASE TABLE |
|
||||
|
|
@ -50,9 +48,6 @@ Auto-generated from the PostgreSQL migrations in @n8n/db. Do not edit by hand.
|
|||
| [public.ai_builder_temporary_workflow](public.ai_builder_temporary_workflow.md) | 4 | | BASE TABLE |
|
||||
| [public.ai_preference](public.ai_preference.md) | 8 | | BASE TABLE |
|
||||
| [public.annotation_tag_entity](public.annotation_tag_entity.md) | 4 | | BASE TABLE |
|
||||
| [public.app](public.app.md) | 9 | | BASE TABLE |
|
||||
| [public.app_artifact](public.app_artifact.md) | 8 | | BASE TABLE |
|
||||
| [public.app_version](public.app_version.md) | 10 | | BASE TABLE |
|
||||
| [public.auth_identity](public.auth_identity.md) | 5 | | BASE TABLE |
|
||||
| [public.auth_provider_sync_history](public.auth_provider_sync_history.md) | 11 | | BASE TABLE |
|
||||
| [public.binary_data](public.binary_data.md) | 9 | | BASE TABLE |
|
||||
|
|
@ -120,11 +115,10 @@ Auto-generated from the PostgreSQL migrations in @n8n/db. Do not edit by hand.
|
|||
| [public.project_pool_settings](public.project_pool_settings.md) | 4 | | BASE TABLE |
|
||||
| [public.project_relation](public.project_relation.md) | 5 | | BASE TABLE |
|
||||
| [public.project_secrets_provider_access](public.project_secrets_provider_access.md) | 5 | | BASE TABLE |
|
||||
| [public.promotion_environment](public.promotion_environment.md) | 9 | | BASE TABLE |
|
||||
| [public.promotion_environment_project](public.promotion_environment_project.md) | 5 | | BASE TABLE |
|
||||
| [public.promotion_config](public.promotion_config.md) | 7 | | BASE TABLE |
|
||||
| [public.promotion_connection](public.promotion_connection.md) | 7 | | BASE TABLE |
|
||||
| [public.promotion_connection_project](public.promotion_connection_project.md) | 4 | | BASE TABLE |
|
||||
| [public.promotion_provider](public.promotion_provider.md) | 8 | | BASE TABLE |
|
||||
| [public.promotion_review](public.promotion_review.md) | 10 | | BASE TABLE |
|
||||
| [public.promotion_run](public.promotion_run.md) | 8 | | BASE TABLE |
|
||||
| [public.role](public.role.md) | 7 | | BASE TABLE |
|
||||
| [public.role_mapping_rule](public.role_mapping_rule.md) | 7 | | BASE TABLE |
|
||||
| [public.role_mapping_rule_project](public.role_mapping_rule_project.md) | 2 | | BASE TABLE |
|
||||
|
|
@ -133,14 +127,10 @@ Auto-generated from the PostgreSQL migrations in @n8n/db. Do not edit by hand.
|
|||
| [public.scheduled_task](public.scheduled_task.md) | 19 | | BASE TABLE |
|
||||
| [public.scope](public.scope.md) | 3 | | BASE TABLE |
|
||||
| [public.secrets_provider_connection](public.secrets_provider_connection.md) | 7 | | BASE TABLE |
|
||||
| [public.self_healing_investigation](public.self_healing_investigation.md) | 12 | | BASE TABLE |
|
||||
| [public.self_healing_result](public.self_healing_result.md) | 15 | | BASE TABLE |
|
||||
| [public.settings](public.settings.md) | 3 | | BASE TABLE |
|
||||
| [public.shared_credentials](public.shared_credentials.md) | 5 | | BASE TABLE |
|
||||
| [public.shared_workflow](public.shared_workflow.md) | 5 | | BASE TABLE |
|
||||
| [public.skill](public.skill.md) | 8 | | BASE TABLE |
|
||||
| [public.skill_file](public.skill_file.md) | 5 | | BASE TABLE |
|
||||
| [public.skill_version](public.skill_version.md) | 11 | | BASE TABLE |
|
||||
| [public.tag_entity](public.tag_entity.md) | 4 | | BASE TABLE |
|
||||
| [public.test_case_execution](public.test_case_execution.md) | 14 | | BASE TABLE |
|
||||
| [public.test_run](public.test_run.md) | 16 | | BASE TABLE |
|
||||
|
|
@ -203,8 +193,6 @@ erDiagram
|
|||
"public.activity_event" }o--o| "public.user" : "FOREIGN KEY (#quot;userId#quot;) REFERENCES #quot;user#quot;(id) ON DELETE SET NULL"
|
||||
"public.activity_event" }o--|| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
"public.agent_background_job" }o--|| "public.agents" : "FOREIGN KEY (#quot;parentAgentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agent_background_job" }o--o| "public.agent_execution" : "FOREIGN KEY (#quot;sourceExecutionId#quot;) REFERENCES agent_execution(id) ON DELETE SET NULL"
|
||||
"public.agent_background_job" }o--o| "public.agent_plan" : "FOREIGN KEY (#quot;planId#quot;) REFERENCES agent_plan(id) ON DELETE SET NULL"
|
||||
"public.agent_channel_status" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agent_chat_attachments" }o--|| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
"public.agent_chat_attachments" }o--o| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
|
|
@ -235,19 +223,13 @@ erDiagram
|
|||
"public.agent_message_queue" }o--|| "public.agent_execution_threads" : "FOREIGN KEY (#quot;threadId#quot;) REFERENCES agent_execution_threads(id) ON DELETE CASCADE"
|
||||
"public.agent_plan" }o--|| "public.agent_execution_threads" : "FOREIGN KEY (#quot;threadId#quot;) REFERENCES agent_execution_threads(id) ON DELETE CASCADE"
|
||||
"public.agent_plan_history" }o--|| "public.agent_plan" : "FOREIGN KEY (#quot;planId#quot;) REFERENCES agent_plan(id) ON DELETE CASCADE"
|
||||
"public.agent_skill_dependency" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agent_skill_dependency" }o--|| "public.skill" : "FOREIGN KEY (#quot;skillId#quot;) REFERENCES skill(id) ON DELETE CASCADE"
|
||||
"public.agent_skill_dependency" }o--o| "public.skill_version" : "FOREIGN KEY (#quot;skillVersionId#quot;) REFERENCES skill_version(id)"
|
||||
"public.agent_task_definition" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agent_task_run_lock" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agent_task_snapshot" }o--|| "public.agent_history" : "FOREIGN KEY (#quot;versionId#quot;) REFERENCES agent_history(#quot;versionId#quot;) ON DELETE CASCADE"
|
||||
"public.agent_thread_grants" }o--|| "public.agent_execution_threads" : "FOREIGN KEY (#quot;threadId#quot;) REFERENCES agent_execution_threads(id) ON DELETE CASCADE"
|
||||
"public.agent_version_skill_dependency" }o--|| "public.agent_history" : "FOREIGN KEY (#quot;agentVersionId#quot;) REFERENCES agent_history(#quot;versionId#quot;) ON DELETE CASCADE"
|
||||
"public.agent_version_skill_dependency" }o--|| "public.skill" : "FOREIGN KEY (#quot;skillId#quot;) REFERENCES skill(id) ON DELETE CASCADE"
|
||||
"public.agent_version_skill_dependency" }o--|| "public.skill_version" : "FOREIGN KEY (#quot;skillVersionId#quot;) REFERENCES skill_version(id)"
|
||||
"public.agent_workflow_dependency" }o--|| "public.workflow_entity" : "FOREIGN KEY (#quot;workflowId#quot;) REFERENCES workflow_entity(id) ON DELETE CASCADE"
|
||||
"public.agent_workflow_dependency" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agents" }o--o| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
"public.agents" }o--|| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
"public.agents" }o--o| "public.agent_history" : "FOREIGN KEY (#quot;activeVersionId#quot;) REFERENCES agent_history(#quot;versionId#quot;) ON DELETE SET NULL"
|
||||
"public.agents_memory_entries" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agents_memory_entries" }o--|| "public.agents_resources" : "FOREIGN KEY (#quot;resourceId#quot;) REFERENCES agents_resources(id) ON DELETE CASCADE"
|
||||
|
|
@ -277,12 +259,6 @@ erDiagram
|
|||
"public.ai_preference" }o--o| "public.user" : "FOREIGN KEY (#quot;createdById#quot;) REFERENCES #quot;user#quot;(id) ON DELETE SET NULL"
|
||||
"public.ai_preference" }o--o| "public.user" : "FOREIGN KEY (#quot;userId#quot;) REFERENCES #quot;user#quot;(id) ON DELETE CASCADE"
|
||||
"public.ai_preference" }o--o| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
"public.app" }o--|| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
"public.app" }o--o| "public.app_version" : "FOREIGN KEY (#quot;draftVersionId#quot;) REFERENCES app_version(id) ON DELETE SET NULL"
|
||||
"public.app" }o--o| "public.app_version" : "FOREIGN KEY (#quot;publishedVersionId#quot;) REFERENCES app_version(id) ON DELETE SET NULL"
|
||||
"public.app_version" }o--|| "public.app" : "FOREIGN KEY (#quot;appId#quot;) REFERENCES app(id) ON DELETE CASCADE"
|
||||
"public.app_version" }o--o| "public.app_artifact" : "FOREIGN KEY (#quot;bundleArtifactId#quot;) REFERENCES app_artifact(id) ON DELETE SET NULL"
|
||||
"public.app_version" }o--|| "public.app_artifact" : "FOREIGN KEY (#quot;sourceArtifactId#quot;) REFERENCES app_artifact(id) ON DELETE RESTRICT"
|
||||
"public.auth_identity" }o--o| "public.user" : "FOREIGN KEY (#quot;userId#quot;) REFERENCES #quot;user#quot;(id)"
|
||||
"public.chat_hub_agent_tools" }o--|| "public.chat_hub_agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES chat_hub_agents(id) ON DELETE CASCADE"
|
||||
"public.chat_hub_agent_tools" }o--|| "public.chat_hub_tools" : "FOREIGN KEY (#quot;toolId#quot;) REFERENCES chat_hub_tools(id) ON DELETE CASCADE"
|
||||
|
|
@ -374,25 +350,16 @@ erDiagram
|
|||
"public.project_relation" }o--|| "public.role" : "FOREIGN KEY (role) REFERENCES role(slug)"
|
||||
"public.project_secrets_provider_access" }o--|| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
"public.project_secrets_provider_access" }o--|| "public.secrets_provider_connection" : "FOREIGN KEY (#quot;secretsProviderConnectionId#quot;) REFERENCES secrets_provider_connection(id) ON DELETE CASCADE"
|
||||
"public.promotion_environment" }o--|| "public.promotion_provider" : "FOREIGN KEY (#quot;providerId#quot;) REFERENCES promotion_provider(id) ON DELETE RESTRICT"
|
||||
"public.promotion_environment_project" }o--|| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
"public.promotion_environment_project" }o--|| "public.promotion_environment" : "FOREIGN KEY (#quot;environmentId#quot;) REFERENCES promotion_environment(id) ON DELETE CASCADE"
|
||||
"public.promotion_review" }o--o| "public.user" : "FOREIGN KEY (#quot;approvedById#quot;) REFERENCES #quot;user#quot;(id) ON DELETE SET NULL"
|
||||
"public.promotion_review" }o--o| "public.user" : "FOREIGN KEY (#quot;createdById#quot;) REFERENCES #quot;user#quot;(id) ON DELETE SET NULL"
|
||||
"public.promotion_review" }o--o| "public.promotion_environment" : "FOREIGN KEY (#quot;environmentId#quot;) REFERENCES promotion_environment(id) ON DELETE SET NULL"
|
||||
"public.promotion_run" }o--o| "public.user" : "FOREIGN KEY (#quot;createdById#quot;) REFERENCES #quot;user#quot;(id) ON DELETE SET NULL"
|
||||
"public.promotion_run" }o--o| "public.promotion_environment" : "FOREIGN KEY (#quot;environmentId#quot;) REFERENCES promotion_environment(id) ON DELETE SET NULL"
|
||||
"public.promotion_run" }o--o| "public.promotion_review" : "FOREIGN KEY (#quot;reviewId#quot;) REFERENCES promotion_review(id) ON DELETE SET NULL"
|
||||
"public.promotion_config" }o--|| "public.promotion_connection" : "FOREIGN KEY (#quot;connectionId#quot;) REFERENCES promotion_connection(id) ON DELETE CASCADE"
|
||||
"public.promotion_connection" }o--|| "public.promotion_provider" : "FOREIGN KEY (#quot;providerId#quot;) REFERENCES promotion_provider(id) ON DELETE RESTRICT"
|
||||
"public.promotion_connection_project" |o--|| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
"public.promotion_connection_project" }o--|| "public.promotion_connection" : "FOREIGN KEY (#quot;connectionId#quot;) REFERENCES promotion_connection(id) ON DELETE CASCADE"
|
||||
"public.role_mapping_rule" }o--|| "public.role" : "FOREIGN KEY (role) REFERENCES role(slug) ON UPDATE CASCADE ON DELETE CASCADE"
|
||||
"public.role_mapping_rule_project" }o--|| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
"public.role_mapping_rule_project" }o--|| "public.role_mapping_rule" : "FOREIGN KEY (#quot;roleMappingRuleId#quot;) REFERENCES role_mapping_rule(id) ON DELETE CASCADE"
|
||||
"public.role_scope" }o--|| "public.scope" : "FOREIGN KEY (#quot;scopeSlug#quot;) REFERENCES scope(slug) ON UPDATE CASCADE ON DELETE CASCADE"
|
||||
"public.role_scope" }o--|| "public.role" : "FOREIGN KEY (#quot;roleSlug#quot;) REFERENCES role(slug) ON UPDATE CASCADE ON DELETE CASCADE"
|
||||
"public.scheduled_task" }o--|| "public.scheduled_job" : "FOREIGN KEY (#quot;jobId#quot;) REFERENCES scheduled_job(id) ON DELETE CASCADE"
|
||||
"public.self_healing_investigation" }o--|| "public.user" : "FOREIGN KEY (#quot;backgroundUserId#quot;) REFERENCES #quot;user#quot;(id) ON DELETE CASCADE"
|
||||
"public.self_healing_investigation" }o--|| "public.workflow_entity" : "FOREIGN KEY (#quot;workflowId#quot;) REFERENCES workflow_entity(id) ON DELETE CASCADE"
|
||||
"public.self_healing_investigation" }o--|| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
"public.self_healing_investigation" }o--o| "public.self_healing_result" : "FOREIGN KEY (#quot;resultId#quot;) REFERENCES self_healing_result(id) ON DELETE CASCADE"
|
||||
"public.self_healing_result" }o--|| "public.user" : "FOREIGN KEY (#quot;backgroundUserId#quot;) REFERENCES #quot;user#quot;(id) ON DELETE CASCADE"
|
||||
"public.self_healing_result" }o--o| "public.user" : "FOREIGN KEY (#quot;dismissedById#quot;) REFERENCES #quot;user#quot;(id) ON DELETE SET NULL"
|
||||
"public.self_healing_result" }o--|| "public.workflow_entity" : "FOREIGN KEY (#quot;workflowId#quot;) REFERENCES workflow_entity(id) ON DELETE CASCADE"
|
||||
|
|
@ -402,12 +369,6 @@ erDiagram
|
|||
"public.shared_credentials" }o--|| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
"public.shared_workflow" }o--|| "public.workflow_entity" : "FOREIGN KEY (#quot;workflowId#quot;) REFERENCES workflow_entity(id) ON DELETE CASCADE"
|
||||
"public.shared_workflow" }o--|| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
"public.skill" }o--o| "public.user" : "FOREIGN KEY (#quot;createdById#quot;) REFERENCES #quot;user#quot;(id) ON DELETE SET NULL"
|
||||
"public.skill" }o--o| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
"public.skill" }o--o| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.skill_file" }o--|| "public.skill_version" : "FOREIGN KEY (#quot;skillVersionId#quot;) REFERENCES skill_version(id) ON DELETE CASCADE"
|
||||
"public.skill_version" }o--o| "public.user" : "FOREIGN KEY (#quot;createdById#quot;) REFERENCES #quot;user#quot;(id) ON DELETE SET NULL"
|
||||
"public.skill_version" }o--|| "public.skill" : "FOREIGN KEY (#quot;skillId#quot;) REFERENCES skill(id) ON DELETE CASCADE"
|
||||
"public.test_case_execution" }o--o| "public.execution_entity" : "FOREIGN KEY (#quot;executionId#quot;) REFERENCES execution_entity(id) ON DELETE SET NULL"
|
||||
"public.test_case_execution" }o--|| "public.test_run" : "FOREIGN KEY (#quot;testRunId#quot;) REFERENCES test_run(id) ON DELETE CASCADE"
|
||||
"public.test_run" }o--|| "public.workflow_entity" : "FOREIGN KEY (#quot;workflowId#quot;) REFERENCES workflow_entity(id) ON DELETE CASCADE"
|
||||
|
|
@ -488,11 +449,8 @@ erDiagram
|
|||
varchar_255_ parentResourceId
|
||||
varchar_128_ parentThreadId
|
||||
uuid pauseRequestId
|
||||
uuid planId FK
|
||||
uuid planTaskId
|
||||
text result
|
||||
timestamp_3__with_time_zone settledAt
|
||||
varchar_36_ sourceExecutionId FK
|
||||
varchar_16_ status
|
||||
varchar_36_ subAgentId
|
||||
timestamp_3__with_time_zone timeoutAt
|
||||
|
|
@ -660,7 +618,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer sessionNumber
|
||||
varchar_32_ taskId
|
||||
json taskStop
|
||||
varchar_36_ taskVersionId FK
|
||||
varchar_255_ title
|
||||
integer totalCompletionTokens
|
||||
|
|
@ -722,12 +679,6 @@ erDiagram
|
|||
uuid planId FK
|
||||
integer revision
|
||||
}
|
||||
"public.agent_skill_dependency" {
|
||||
varchar_36_ agentId FK
|
||||
timestamp_3__with_time_zone createdAt
|
||||
varchar_36_ skillId FK
|
||||
uuid skillVersionId FK
|
||||
}
|
||||
"public.agent_task_definition" {
|
||||
varchar_36_ agentId FK
|
||||
timestamp_3__with_time_zone createdAt
|
||||
|
|
@ -763,12 +714,6 @@ erDiagram
|
|||
varchar_128_ threadId FK
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.agent_version_skill_dependency" {
|
||||
varchar_36_ agentVersionId FK
|
||||
timestamp_3__with_time_zone createdAt
|
||||
varchar_36_ skillId FK
|
||||
uuid skillVersionId FK
|
||||
}
|
||||
"public.agent_workflow_dependency" {
|
||||
varchar_36_ agentId FK
|
||||
timestamp_3__with_time_zone createdAt
|
||||
|
|
@ -784,7 +729,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
@ -922,39 +866,6 @@ erDiagram
|
|||
varchar_24_ name
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.app" {
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid draftVersionId FK
|
||||
uuid id
|
||||
varchar_128_ name
|
||||
varchar_128_ namespace
|
||||
varchar_36_ projectId FK
|
||||
uuid publishedVersionId FK
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
varchar_16_ visibility
|
||||
}
|
||||
"public.app_artifact" {
|
||||
uuid appId
|
||||
timestamp_3__with_time_zone createdAt
|
||||
integer id
|
||||
integer sizeBytes
|
||||
text storageKey
|
||||
varchar_2_ storedAt
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
varchar_16_ uploadStatus
|
||||
}
|
||||
"public.app_version" {
|
||||
uuid appId FK
|
||||
integer buildAttempts
|
||||
varchar_16_ buildStatus
|
||||
timestamp_3__with_time_zone buildStatusUpdatedAt
|
||||
integer bundleArtifactId FK
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid id
|
||||
varchar_255_ label
|
||||
integer sourceArtifactId FK
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.auth_identity" {
|
||||
timestamp_3__with_time_zone createdAt
|
||||
varchar_255_ providerId
|
||||
|
|
@ -1582,21 +1493,27 @@ erDiagram
|
|||
integer secretsProviderConnectionId FK
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.promotion_environment" {
|
||||
"public.promotion_config" {
|
||||
varchar_36_ connectionId FK
|
||||
timestamp_3__with_time_zone createdAt
|
||||
varchar_16_ direction
|
||||
uuid id
|
||||
varchar_36_ id
|
||||
varchar_128_ name
|
||||
varchar_36_ providerId FK
|
||||
json remote
|
||||
varchar_16_ scope
|
||||
json settings
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.promotion_environment_project" {
|
||||
"public.promotion_connection" {
|
||||
timestamp_3__with_time_zone createdAt
|
||||
varchar_36_ id
|
||||
varchar_128_ name
|
||||
varchar_36_ providerId FK
|
||||
varchar_16_ scope
|
||||
json target
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.promotion_connection_project" {
|
||||
varchar_36_ connectionId FK
|
||||
timestamp_3__with_time_zone createdAt
|
||||
varchar_16_ direction
|
||||
uuid environmentId FK
|
||||
varchar_36_ projectId FK
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
|
|
@ -1610,28 +1527,6 @@ erDiagram
|
|||
varchar_32_ type
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.promotion_review" {
|
||||
timestamp_3__with_time_zone approvedAt
|
||||
uuid approvedById FK
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid createdById FK
|
||||
uuid environmentId FK
|
||||
uuid id
|
||||
timestamp_3__with_time_zone lastStateChangeAt
|
||||
varchar_255_ remoteReviewId
|
||||
varchar_32_ state
|
||||
varchar_16_ type
|
||||
}
|
||||
"public.promotion_run" {
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid createdById FK
|
||||
json details
|
||||
varchar_16_ direction
|
||||
uuid environmentId FK
|
||||
uuid id
|
||||
uuid reviewId FK
|
||||
varchar_16_ type
|
||||
}
|
||||
"public.role" {
|
||||
timestamp_3__with_time_zone createdAt
|
||||
text description
|
||||
|
|
@ -1720,20 +1615,6 @@ erDiagram
|
|||
varchar_36_ type
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.self_healing_investigation" {
|
||||
uuid backgroundUserId FK
|
||||
json baseline
|
||||
timestamp_3__with_time_zone createdAt
|
||||
varchar_36_ executionId
|
||||
timestamp_3__with_time_zone finishedAt
|
||||
varchar_36_ id
|
||||
varchar_36_ projectId FK
|
||||
varchar_36_ resultId FK
|
||||
uuid runId
|
||||
uuid threadId
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
varchar_36_ workflowId FK
|
||||
}
|
||||
"public.self_healing_result" {
|
||||
uuid backgroundUserId FK
|
||||
timestamp_3__with_time_zone completedAt
|
||||
|
|
@ -1770,36 +1651,6 @@ erDiagram
|
|||
timestamp_3__with_time_zone updatedAt
|
||||
varchar_36_ workflowId FK
|
||||
}
|
||||
"public.skill" {
|
||||
varchar_36_ agentId FK
|
||||
boolean availableToAssistant
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid createdById FK
|
||||
varchar_36_ id
|
||||
varchar_36_ projectId FK
|
||||
varchar_16_ source
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.skill_file" {
|
||||
text content
|
||||
timestamp_3__with_time_zone createdAt
|
||||
varchar_512_ path
|
||||
uuid skillVersionId FK
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.skill_version" {
|
||||
varchar_64_ contentHash
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid createdById FK
|
||||
varchar_1024_ description
|
||||
json frontmatter
|
||||
uuid id
|
||||
text instructions
|
||||
varchar_128_ name
|
||||
varchar_36_ skillId FK
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
integer version
|
||||
}
|
||||
"public.tag_entity" {
|
||||
timestamp_3__with_time_zone createdAt
|
||||
varchar_36_ id
|
||||
|
|
|
|||
|
|
@ -16,11 +16,8 @@
|
|||
| parentResourceId | varchar(255) | | false | | | Memory resource of the parent agent run |
|
||||
| parentThreadId | varchar(128) | | false | | | |
|
||||
| pauseRequestId | uuid | | true | | | Groups jobs stopped by one user request |
|
||||
| planId | uuid | | true | | [public.agent_plan](public.agent_plan.md) | Plan that owns this work |
|
||||
| planTaskId | uuid | | true | | | Task ID in the saved plan document |
|
||||
| result | text | | true | | | Final answer of a settled sub-agent job |
|
||||
| settledAt | timestamp(3) with time zone | | true | | | |
|
||||
| sourceExecutionId | varchar(36) | | true | | [public.agent_execution](public.agent_execution.md) | Execution that dispatched this job |
|
||||
| status | varchar(16) | | false | | | running: child works; suspended: child awaits approval; paused: user stopped the child; completed, failed, cancelled: terminal |
|
||||
| subAgentId | varchar(36) | | true | | | Sub-agent jobs only |
|
||||
| timeoutAt | timestamp(3) with time zone | | true | | | When reconciliation fails the job as timed out; NULL means no timeout |
|
||||
|
|
@ -34,8 +31,6 @@
|
|||
| ---- | ---- | ---------- |
|
||||
| CHK_agent_background_job_kind | CHECK | CHECK (((kind)::text = ANY ((ARRAY['subagent'::character varying, 'workflow'::character varying])::text[]))) |
|
||||
| CHK_agent_background_job_status | CHECK | CHECK (((status)::text = ANY ((ARRAY['running'::character varying, 'suspended'::character varying, 'paused'::character varying, 'completed'::character varying, 'failed'::character varying, 'cancelled'::character varying])::text[]))) |
|
||||
| FK_4db3b7ebea4ed4f69649982018c | FOREIGN KEY | FOREIGN KEY ("sourceExecutionId") REFERENCES agent_execution(id) ON DELETE SET NULL |
|
||||
| FK_ce131bff629d89015893e350e9c | FOREIGN KEY | FOREIGN KEY ("planId") REFERENCES agent_plan(id) ON DELETE SET NULL |
|
||||
| FK_d46c6f00730c2ef8bcb6ee24b67 | FOREIGN KEY | FOREIGN KEY ("parentAgentId") REFERENCES agents(id) ON DELETE CASCADE |
|
||||
| PK_6e0db58281aa2b4c956dc0d58e9 | PRIMARY KEY | PRIMARY KEY (id) |
|
||||
| agent_background_job_createdAt_not_null | n | NOT NULL "createdAt" |
|
||||
|
|
@ -56,8 +51,6 @@
|
|||
| IDX_93d62baabe9858816b5adafb44 | CREATE INDEX "IDX_93d62baabe9858816b5adafb44" ON public.agent_background_job USING btree ("parentThreadId", status) |
|
||||
| IDX_agent_background_job_childExecutionId | CREATE UNIQUE INDEX "IDX_agent_background_job_childExecutionId" ON public.agent_background_job USING btree ("childExecutionId") WHERE ("childExecutionId" IS NOT NULL) |
|
||||
| IDX_agent_background_job_parentThreadId | CREATE INDEX "IDX_agent_background_job_parentThreadId" ON public.agent_background_job USING btree ("parentThreadId") WHERE (((status)::text <> 'running'::text) AND ("notifiedAt" IS NULL)) |
|
||||
| IDX_agent_background_job_planId_planTaskId | CREATE INDEX "IDX_agent_background_job_planId_planTaskId" ON public.agent_background_job USING btree ("planId", "planTaskId") |
|
||||
| IDX_agent_background_job_sourceExecutionId | CREATE INDEX "IDX_agent_background_job_sourceExecutionId" ON public.agent_background_job USING btree ("sourceExecutionId") |
|
||||
| IDX_agent_background_job_timeoutAt | CREATE INDEX "IDX_agent_background_job_timeoutAt" ON public.agent_background_job USING btree ("timeoutAt") WHERE ((status)::text = ANY ((ARRAY['running'::character varying, 'suspended'::character varying])::text[])) |
|
||||
| IDX_d46c6f00730c2ef8bcb6ee24b6 | CREATE INDEX "IDX_d46c6f00730c2ef8bcb6ee24b6" ON public.agent_background_job USING btree ("parentAgentId") |
|
||||
| IDX_e43e630272995a93dfeb94ab3e | CREATE INDEX "IDX_e43e630272995a93dfeb94ab3e" ON public.agent_background_job USING btree ("settledAt") |
|
||||
|
|
@ -69,8 +62,6 @@
|
|||
erDiagram
|
||||
|
||||
"public.agent_background_job" }o--|| "public.agents" : "FOREIGN KEY (#quot;parentAgentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agent_background_job" }o--o| "public.agent_plan" : "FOREIGN KEY (#quot;planId#quot;) REFERENCES agent_plan(id) ON DELETE SET NULL"
|
||||
"public.agent_background_job" }o--o| "public.agent_execution" : "FOREIGN KEY (#quot;sourceExecutionId#quot;) REFERENCES agent_execution(id) ON DELETE SET NULL"
|
||||
|
||||
"public.agent_background_job" {
|
||||
varchar_36_ childExecutionId
|
||||
|
|
@ -85,11 +76,8 @@ erDiagram
|
|||
varchar_255_ parentResourceId
|
||||
varchar_128_ parentThreadId
|
||||
uuid pauseRequestId
|
||||
uuid planId FK
|
||||
uuid planTaskId
|
||||
text result
|
||||
timestamp_3__with_time_zone settledAt
|
||||
varchar_36_ sourceExecutionId FK
|
||||
varchar_16_ status
|
||||
varchar_36_ subAgentId
|
||||
timestamp_3__with_time_zone timeoutAt
|
||||
|
|
@ -107,48 +95,12 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
varchar_36_ versionId
|
||||
}
|
||||
"public.agent_plan" {
|
||||
timestamp_3__with_time_zone closedAt
|
||||
timestamp_3__with_time_zone createdAt
|
||||
json data
|
||||
integer formatVersion
|
||||
uuid id
|
||||
integer revision
|
||||
varchar_128_ threadId FK
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.agent_execution" {
|
||||
boolean acceptsSteering
|
||||
json attachments
|
||||
json author
|
||||
integer completionTokens
|
||||
double_precision cost
|
||||
timestamp_3__with_time_zone createdAt
|
||||
integer duration
|
||||
text error
|
||||
json failureSummary
|
||||
varchar_16_ hitlStatus
|
||||
varchar_36_ id
|
||||
varchar_255_ model
|
||||
integer promptTokens
|
||||
varchar_32_ source
|
||||
timestamp_3__with_time_zone startedAt
|
||||
varchar_16_ status
|
||||
timestamp_3__with_time_zone stoppedAt
|
||||
varchar_2_ storedAt
|
||||
varchar_128_ threadId FK
|
||||
json timeline
|
||||
integer totalTokens
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
text userMessage
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -70,7 +70,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
|
|
@ -75,7 +75,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
|
|
@ -56,7 +56,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
|
|
@ -58,7 +58,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
|
|
@ -49,7 +49,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
|
|
@ -69,7 +69,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@
|
|||
| error | text | | true | | | |
|
||||
| failureSummary | json | | true | | | Execution failure projection as {count, latest} for session list queries |
|
||||
| hitlStatus | varchar(16) | | true | | | |
|
||||
| id | varchar(36) | | false | [public.agent_background_job](public.agent_background_job.md) [public.agent_execution_message_links](public.agent_execution_message_links.md) [public.agent_message_queue](public.agent_message_queue.md) | | |
|
||||
| id | varchar(36) | | false | [public.agent_execution_message_links](public.agent_execution_message_links.md) [public.agent_message_queue](public.agent_message_queue.md) | | |
|
||||
| model | varchar(255) | | true | | | |
|
||||
| promptTokens | integer | | true | | | |
|
||||
| source | varchar(32) | | true | | | |
|
||||
|
|
@ -59,7 +59,6 @@
|
|||
```mermaid
|
||||
erDiagram
|
||||
|
||||
"public.agent_background_job" }o--o| "public.agent_execution" : "FOREIGN KEY (#quot;sourceExecutionId#quot;) REFERENCES agent_execution(id) ON DELETE SET NULL"
|
||||
"public.agent_execution_message_links" }o--|| "public.agent_execution" : "FOREIGN KEY (#quot;executionId#quot;) REFERENCES agent_execution(id) ON DELETE CASCADE"
|
||||
"public.agent_message_queue" }o--o| "public.agent_execution" : "FOREIGN KEY (#quot;executionId#quot;) REFERENCES agent_execution(id)"
|
||||
"public.agent_message_queue" }o--o| "public.agent_execution" : "FOREIGN KEY (#quot;steeringExecutionId#quot;) REFERENCES agent_execution(id)"
|
||||
|
|
@ -90,31 +89,6 @@ erDiagram
|
|||
timestamp_3__with_time_zone updatedAt
|
||||
text userMessage
|
||||
}
|
||||
"public.agent_background_job" {
|
||||
varchar_36_ childExecutionId
|
||||
varchar_128_ childThreadId
|
||||
timestamp_3__with_time_zone createdAt
|
||||
text error
|
||||
varchar_36_ id
|
||||
varchar_16_ kind
|
||||
timestamp_3__with_time_zone notifiedAt
|
||||
varchar_36_ parentAgentId FK
|
||||
varchar_64_ parentPrincipalHash
|
||||
varchar_255_ parentResourceId
|
||||
varchar_128_ parentThreadId
|
||||
uuid pauseRequestId
|
||||
uuid planId FK
|
||||
uuid planTaskId
|
||||
text result
|
||||
timestamp_3__with_time_zone settledAt
|
||||
varchar_36_ sourceExecutionId FK
|
||||
varchar_16_ status
|
||||
varchar_36_ subAgentId
|
||||
timestamp_3__with_time_zone timeoutAt
|
||||
varchar_255_ title
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
varchar_36_ workflowId
|
||||
}
|
||||
"public.agent_execution_message_links" {
|
||||
timestamp_3__with_time_zone createdAt
|
||||
varchar_6_ direction
|
||||
|
|
@ -147,7 +121,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer sessionNumber
|
||||
varchar_32_ taskId
|
||||
json taskStop
|
||||
varchar_36_ taskVersionId FK
|
||||
varchar_255_ title
|
||||
integer totalCompletionTokens
|
||||
|
|
|
|||
|
|
@ -16,7 +16,6 @@
|
|||
| projectId | varchar(255) | | false | | [public.project](public.project.md) | |
|
||||
| sessionNumber | integer | 0 | false | | | |
|
||||
| taskId | varchar(32) | | true | | | Published task ID that triggered this session; not an FK because published runs can outlive draft task definition rows |
|
||||
| taskStop | json | | true | | | Latest task stop boundary and unresolved stop errors |
|
||||
| taskVersionId | varchar(36) | | true | | [public.agent_history](public.agent_history.md) | Published agent_history version that supplied the task snapshot |
|
||||
| title | varchar(255) | | true | | | |
|
||||
| totalCompletionTokens | integer | 0 | false | | | |
|
||||
|
|
@ -85,7 +84,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer sessionNumber
|
||||
varchar_32_ taskId
|
||||
json taskStop
|
||||
varchar_36_ taskVersionId FK
|
||||
varchar_255_ title
|
||||
integer totalCompletionTokens
|
||||
|
|
@ -104,7 +102,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
|
|
@ -70,7 +70,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@
|
|||
| skills | json | | true | | | Frozen map of `skillId → AgentSkill` at publish time |
|
||||
| tools | json | | true | | | Frozen map of `toolId → { code, descriptor }` at publish time |
|
||||
| updatedAt | timestamp(3) with time zone | CURRENT_TIMESTAMP(3) | false | | | |
|
||||
| versionId | varchar(36) | | false | [public.agent_execution_threads](public.agent_execution_threads.md) [public.agent_task_snapshot](public.agent_task_snapshot.md) [public.agent_version_skill_dependency](public.agent_version_skill_dependency.md) [public.agents](public.agents.md) | | |
|
||||
| versionId | varchar(36) | | false | [public.agent_execution_threads](public.agent_execution_threads.md) [public.agent_task_snapshot](public.agent_task_snapshot.md) [public.agents](public.agents.md) | | |
|
||||
|
||||
## Constraints
|
||||
|
||||
|
|
@ -43,7 +43,6 @@ erDiagram
|
|||
"public.agent_history" }o--o| "public.user" : "FOREIGN KEY (#quot;publishedById#quot;) REFERENCES #quot;user#quot;(id) ON DELETE SET NULL"
|
||||
"public.agent_execution_threads" }o--o| "public.agent_history" : "FOREIGN KEY (#quot;taskVersionId#quot;) REFERENCES agent_history(#quot;versionId#quot;) ON DELETE SET NULL"
|
||||
"public.agent_task_snapshot" }o--|| "public.agent_history" : "FOREIGN KEY (#quot;versionId#quot;) REFERENCES agent_history(#quot;versionId#quot;) ON DELETE CASCADE"
|
||||
"public.agent_version_skill_dependency" }o--|| "public.agent_history" : "FOREIGN KEY (#quot;agentVersionId#quot;) REFERENCES agent_history(#quot;versionId#quot;) ON DELETE CASCADE"
|
||||
"public.agents" }o--o| "public.agent_history" : "FOREIGN KEY (#quot;activeVersionId#quot;) REFERENCES agent_history(#quot;versionId#quot;) ON DELETE SET NULL"
|
||||
|
||||
"public.agent_history" {
|
||||
|
|
@ -67,7 +66,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
@ -104,7 +102,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer sessionNumber
|
||||
varchar_32_ taskId
|
||||
json taskStop
|
||||
varchar_36_ taskVersionId FK
|
||||
varchar_255_ title
|
||||
integer totalCompletionTokens
|
||||
|
|
@ -124,12 +121,6 @@ erDiagram
|
|||
timestamp_3__with_time_zone updatedAt
|
||||
varchar_36_ versionId FK
|
||||
}
|
||||
"public.agent_version_skill_dependency" {
|
||||
varchar_36_ agentVersionId FK
|
||||
timestamp_3__with_time_zone createdAt
|
||||
varchar_36_ skillId FK
|
||||
uuid skillVersionId FK
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -118,7 +118,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer sessionNumber
|
||||
varchar_32_ taskId
|
||||
json taskStop
|
||||
varchar_36_ taskVersionId FK
|
||||
varchar_255_ title
|
||||
integer totalCompletionTokens
|
||||
|
|
|
|||
29
docs/generated/postgres-schema/public.agent_plan.md
generated
29
docs/generated/postgres-schema/public.agent_plan.md
generated
|
|
@ -8,7 +8,7 @@
|
|||
| createdAt | timestamp(3) with time zone | CURRENT_TIMESTAMP(3) | false | | | |
|
||||
| data | json | | false | | | Opaque plan document |
|
||||
| formatVersion | integer | | false | | | Version of the plan JSON format |
|
||||
| id | uuid | | false | [public.agent_background_job](public.agent_background_job.md) [public.agent_plan_history](public.agent_plan_history.md) | | Plan ID generated by the caller |
|
||||
| id | uuid | | false | [public.agent_plan_history](public.agent_plan_history.md) | | Plan ID generated by the caller |
|
||||
| revision | integer | 1 | false | | | Current document revision |
|
||||
| threadId | varchar(128) | | false | | [public.agent_execution_threads](public.agent_execution_threads.md) | Session that owns the plan |
|
||||
| updatedAt | timestamp(3) with time zone | CURRENT_TIMESTAMP(3) | false | | | |
|
||||
|
|
@ -42,7 +42,6 @@
|
|||
```mermaid
|
||||
erDiagram
|
||||
|
||||
"public.agent_background_job" }o--o| "public.agent_plan" : "FOREIGN KEY (#quot;planId#quot;) REFERENCES agent_plan(id) ON DELETE SET NULL"
|
||||
"public.agent_plan_history" }o--|| "public.agent_plan" : "FOREIGN KEY (#quot;planId#quot;) REFERENCES agent_plan(id) ON DELETE CASCADE"
|
||||
"public.agent_plan" }o--|| "public.agent_execution_threads" : "FOREIGN KEY (#quot;threadId#quot;) REFERENCES agent_execution_threads(id) ON DELETE CASCADE"
|
||||
|
||||
|
|
@ -56,31 +55,6 @@ erDiagram
|
|||
varchar_128_ threadId FK
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.agent_background_job" {
|
||||
varchar_36_ childExecutionId
|
||||
varchar_128_ childThreadId
|
||||
timestamp_3__with_time_zone createdAt
|
||||
text error
|
||||
varchar_36_ id
|
||||
varchar_16_ kind
|
||||
timestamp_3__with_time_zone notifiedAt
|
||||
varchar_36_ parentAgentId FK
|
||||
varchar_64_ parentPrincipalHash
|
||||
varchar_255_ parentResourceId
|
||||
varchar_128_ parentThreadId
|
||||
uuid pauseRequestId
|
||||
uuid planId FK
|
||||
uuid planTaskId
|
||||
text result
|
||||
timestamp_3__with_time_zone settledAt
|
||||
varchar_36_ sourceExecutionId FK
|
||||
varchar_16_ status
|
||||
varchar_36_ subAgentId
|
||||
timestamp_3__with_time_zone timeoutAt
|
||||
varchar_255_ title
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
varchar_36_ workflowId
|
||||
}
|
||||
"public.agent_plan_history" {
|
||||
timestamp_3__with_time_zone closedAt
|
||||
timestamp_3__with_time_zone createdAt
|
||||
|
|
@ -102,7 +76,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer sessionNumber
|
||||
varchar_32_ taskId
|
||||
json taskStop
|
||||
varchar_36_ taskVersionId FK
|
||||
varchar_255_ title
|
||||
integer totalCompletionTokens
|
||||
|
|
|
|||
|
|
@ -1,90 +0,0 @@
|
|||
# public.agent_skill_dependency
|
||||
|
||||
## Columns
|
||||
|
||||
| Name | Type | Default | Nullable | Children | Parents | Comment |
|
||||
| ---- | ---- | ------- | -------- | -------- | ------- | ------- |
|
||||
| agentId | varchar(36) | | false | | [public.agents](public.agents.md) | |
|
||||
| createdAt | timestamp(3) with time zone | CURRENT_TIMESTAMP(3) | false | | | |
|
||||
| skillId | varchar(36) | | false | | [public.skill](public.skill.md) | |
|
||||
| skillVersionId | uuid | | true | | [public.skill_version](public.skill_version.md) | Set when the agent draft pins a saved version. NULL = follows the latest saved version |
|
||||
|
||||
## Constraints
|
||||
|
||||
| Name | Type | Definition |
|
||||
| ---- | ---- | ---------- |
|
||||
| FK_068f21aefba31ae621482f396dc | FOREIGN KEY | FOREIGN KEY ("skillId") REFERENCES skill(id) ON DELETE CASCADE |
|
||||
| FK_72325bb13511a51847c53fcb377 | FOREIGN KEY | FOREIGN KEY ("agentId") REFERENCES agents(id) ON DELETE CASCADE |
|
||||
| FK_7676571bd6f41d9e32f0d066fe9 | FOREIGN KEY | FOREIGN KEY ("skillVersionId") REFERENCES skill_version(id) |
|
||||
| PK_18a54f60491d3e2aa2200b70f4a | PRIMARY KEY | PRIMARY KEY ("agentId", "skillId") |
|
||||
| agent_skill_dependency_agentId_not_null | n | NOT NULL "agentId" |
|
||||
| agent_skill_dependency_createdAt_not_null | n | NOT NULL "createdAt" |
|
||||
| agent_skill_dependency_skillId_not_null | n | NOT NULL "skillId" |
|
||||
|
||||
## Indexes
|
||||
|
||||
| Name | Definition |
|
||||
| ---- | ---------- |
|
||||
| IDX_068f21aefba31ae621482f396d | CREATE INDEX "IDX_068f21aefba31ae621482f396d" ON public.agent_skill_dependency USING btree ("skillId") |
|
||||
| PK_18a54f60491d3e2aa2200b70f4a | CREATE UNIQUE INDEX "PK_18a54f60491d3e2aa2200b70f4a" ON public.agent_skill_dependency USING btree ("agentId", "skillId") |
|
||||
|
||||
## Relations
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
|
||||
"public.agent_skill_dependency" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agent_skill_dependency" }o--|| "public.skill" : "FOREIGN KEY (#quot;skillId#quot;) REFERENCES skill(id) ON DELETE CASCADE"
|
||||
"public.agent_skill_dependency" }o--o| "public.skill_version" : "FOREIGN KEY (#quot;skillVersionId#quot;) REFERENCES skill_version(id)"
|
||||
|
||||
"public.agent_skill_dependency" {
|
||||
varchar_36_ agentId FK
|
||||
timestamp_3__with_time_zone createdAt
|
||||
varchar_36_ skillId FK
|
||||
uuid skillVersionId FK
|
||||
}
|
||||
"public.agents" {
|
||||
varchar_36_ activeVersionId FK
|
||||
boolean availableInMCP
|
||||
timestamp_3__with_time_zone createdAt
|
||||
varchar_36_ id
|
||||
json integrations
|
||||
varchar_128_ name
|
||||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
varchar_36_ versionId
|
||||
}
|
||||
"public.skill" {
|
||||
varchar_36_ agentId FK
|
||||
boolean availableToAssistant
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid createdById FK
|
||||
varchar_36_ id
|
||||
varchar_36_ projectId FK
|
||||
varchar_16_ source
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.skill_version" {
|
||||
varchar_64_ contentHash
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid createdById FK
|
||||
varchar_1024_ description
|
||||
json frontmatter
|
||||
uuid id
|
||||
text instructions
|
||||
varchar_128_ name
|
||||
varchar_36_ skillId FK
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
integer version
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> Generated by [tbls](https://github.com/k1LoW/tbls)
|
||||
|
|
@ -61,7 +61,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
|
|
@ -55,7 +55,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
|
|
@ -52,7 +52,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer sessionNumber
|
||||
varchar_32_ taskId
|
||||
json taskStop
|
||||
varchar_36_ taskVersionId FK
|
||||
varchar_255_ title
|
||||
integer totalCompletionTokens
|
||||
|
|
|
|||
|
|
@ -1,86 +0,0 @@
|
|||
# public.agent_version_skill_dependency
|
||||
|
||||
## Columns
|
||||
|
||||
| Name | Type | Default | Nullable | Children | Parents | Comment |
|
||||
| ---- | ---- | ------- | -------- | -------- | ------- | ------- |
|
||||
| agentVersionId | varchar(36) | | false | | [public.agent_history](public.agent_history.md) | |
|
||||
| createdAt | timestamp(3) with time zone | CURRENT_TIMESTAMP(3) | false | | | |
|
||||
| skillId | varchar(36) | | false | | [public.skill](public.skill.md) | |
|
||||
| skillVersionId | uuid | | false | | [public.skill_version](public.skill_version.md) | |
|
||||
|
||||
## Constraints
|
||||
|
||||
| Name | Type | Definition |
|
||||
| ---- | ---- | ---------- |
|
||||
| FK_97472baf9196d914d9cd9de6201 | FOREIGN KEY | FOREIGN KEY ("agentVersionId") REFERENCES agent_history("versionId") ON DELETE CASCADE |
|
||||
| FK_c494292243a7e510acd0331c3d1 | FOREIGN KEY | FOREIGN KEY ("skillId") REFERENCES skill(id) ON DELETE CASCADE |
|
||||
| FK_df63badfd7d3f2aa622ece471c4 | FOREIGN KEY | FOREIGN KEY ("skillVersionId") REFERENCES skill_version(id) |
|
||||
| PK_df0a2577371b94f3c7008d81822 | PRIMARY KEY | PRIMARY KEY ("agentVersionId", "skillId") |
|
||||
| agent_version_skill_dependency_agentVersionId_not_null | n | NOT NULL "agentVersionId" |
|
||||
| agent_version_skill_dependency_createdAt_not_null | n | NOT NULL "createdAt" |
|
||||
| agent_version_skill_dependency_skillId_not_null | n | NOT NULL "skillId" |
|
||||
| agent_version_skill_dependency_skillVersionId_not_null | n | NOT NULL "skillVersionId" |
|
||||
|
||||
## Indexes
|
||||
|
||||
| Name | Definition |
|
||||
| ---- | ---------- |
|
||||
| IDX_c494292243a7e510acd0331c3d | CREATE INDEX "IDX_c494292243a7e510acd0331c3d" ON public.agent_version_skill_dependency USING btree ("skillId") |
|
||||
| IDX_df63badfd7d3f2aa622ece471c | CREATE INDEX "IDX_df63badfd7d3f2aa622ece471c" ON public.agent_version_skill_dependency USING btree ("skillVersionId") |
|
||||
| PK_df0a2577371b94f3c7008d81822 | CREATE UNIQUE INDEX "PK_df0a2577371b94f3c7008d81822" ON public.agent_version_skill_dependency USING btree ("agentVersionId", "skillId") |
|
||||
|
||||
## Relations
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
|
||||
"public.agent_version_skill_dependency" }o--|| "public.agent_history" : "FOREIGN KEY (#quot;agentVersionId#quot;) REFERENCES agent_history(#quot;versionId#quot;) ON DELETE CASCADE"
|
||||
"public.agent_version_skill_dependency" }o--|| "public.skill" : "FOREIGN KEY (#quot;skillId#quot;) REFERENCES skill(id) ON DELETE CASCADE"
|
||||
"public.agent_version_skill_dependency" }o--|| "public.skill_version" : "FOREIGN KEY (#quot;skillVersionId#quot;) REFERENCES skill_version(id)"
|
||||
|
||||
"public.agent_version_skill_dependency" {
|
||||
varchar_36_ agentVersionId FK
|
||||
timestamp_3__with_time_zone createdAt
|
||||
varchar_36_ skillId FK
|
||||
uuid skillVersionId FK
|
||||
}
|
||||
"public.agent_history" {
|
||||
varchar_36_ agentId FK
|
||||
varchar_255_ author
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid publishedById FK
|
||||
json schema
|
||||
json skills
|
||||
json tools
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
varchar_36_ versionId
|
||||
}
|
||||
"public.skill" {
|
||||
varchar_36_ agentId FK
|
||||
boolean availableToAssistant
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid createdById FK
|
||||
varchar_36_ id
|
||||
varchar_36_ projectId FK
|
||||
varchar_16_ source
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.skill_version" {
|
||||
varchar_64_ contentHash
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid createdById FK
|
||||
varchar_1024_ description
|
||||
json frontmatter
|
||||
uuid id
|
||||
text instructions
|
||||
varchar_128_ name
|
||||
varchar_36_ skillId FK
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
integer version
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> Generated by [tbls](https://github.com/k1LoW/tbls)
|
||||
|
|
@ -49,7 +49,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
34
docs/generated/postgres-schema/public.agents.md
generated
34
docs/generated/postgres-schema/public.agents.md
generated
|
|
@ -7,13 +7,12 @@
|
|||
| activeVersionId | varchar(36) | | true | | [public.agent_history](public.agent_history.md) | |
|
||||
| availableInMCP | boolean | false | false | | | Whether MCP clients granted agent scopes may operate on this agent |
|
||||
| createdAt | timestamp(3) with time zone | CURRENT_TIMESTAMP(3) | false | | | |
|
||||
| id | varchar(36) | | false | [public.agent_background_job](public.agent_background_job.md) [public.agent_channel_status](public.agent_channel_status.md) [public.agent_chat_attachments](public.agent_chat_attachments.md) [public.agent_chat_subscriptions](public.agent_chat_subscriptions.md) [public.agent_checkpoints](public.agent_checkpoints.md) [public.agent_credential_dependency](public.agent_credential_dependency.md) [public.agent_eval_dataset](public.agent_eval_dataset.md) [public.agent_execution_threads](public.agent_execution_threads.md) [public.agent_files](public.agent_files.md) [public.agent_history](public.agent_history.md) [public.agent_skill_dependency](public.agent_skill_dependency.md) [public.agent_task_definition](public.agent_task_definition.md) [public.agent_task_run_lock](public.agent_task_run_lock.md) [public.agent_workflow_dependency](public.agent_workflow_dependency.md) [public.agents_memory_entries](public.agents_memory_entries.md) [public.agents_memory_entry_candidates](public.agents_memory_entry_candidates.md) [public.agents_memory_entry_locks](public.agents_memory_entry_locks.md) [public.agents_memory_entry_sources](public.agents_memory_entry_sources.md) [public.agents_observation_cursors](public.agents_observation_cursors.md) [public.agents_observation_locks](public.agents_observation_locks.md) [public.agents_observations](public.agents_observations.md) [public.skill](public.skill.md) | | |
|
||||
| id | varchar(36) | | false | [public.agent_background_job](public.agent_background_job.md) [public.agent_channel_status](public.agent_channel_status.md) [public.agent_chat_attachments](public.agent_chat_attachments.md) [public.agent_chat_subscriptions](public.agent_chat_subscriptions.md) [public.agent_checkpoints](public.agent_checkpoints.md) [public.agent_credential_dependency](public.agent_credential_dependency.md) [public.agent_eval_dataset](public.agent_eval_dataset.md) [public.agent_execution_threads](public.agent_execution_threads.md) [public.agent_files](public.agent_files.md) [public.agent_history](public.agent_history.md) [public.agent_task_definition](public.agent_task_definition.md) [public.agent_task_run_lock](public.agent_task_run_lock.md) [public.agent_workflow_dependency](public.agent_workflow_dependency.md) [public.agents_memory_entries](public.agents_memory_entries.md) [public.agents_memory_entry_candidates](public.agents_memory_entry_candidates.md) [public.agents_memory_entry_locks](public.agents_memory_entry_locks.md) [public.agents_memory_entry_sources](public.agents_memory_entry_sources.md) [public.agents_observation_cursors](public.agents_observation_cursors.md) [public.agents_observation_locks](public.agents_observation_locks.md) [public.agents_observations](public.agents_observations.md) | | |
|
||||
| integrations | json | '[]'::json | false | | | |
|
||||
| name | varchar(128) | | false | | | |
|
||||
| projectId | varchar(255) | | true | | [public.project](public.project.md) | |
|
||||
| projectId | varchar(255) | | false | | [public.project](public.project.md) | |
|
||||
| revision | integer | 0 | false | | | |
|
||||
| schema | json | | true | | | |
|
||||
| scope | varchar(16) | 'project'::character varying | false | | | project: belongs to projectId; instance: code-defined, belongs to no project (projectId is null) |
|
||||
| setupCompletedAt | timestamp(3) with time zone | | true | | | When this agent first reached a complete, publishable setup |
|
||||
| skills | json | '{}'::json | false | | | |
|
||||
| tools | json | '{}'::json | false | | | |
|
||||
|
|
@ -24,8 +23,6 @@
|
|||
|
||||
| Name | Type | Definition |
|
||||
| ---- | ---- | ---------- |
|
||||
| CHK_agents_scope | CHECK | CHECK (((scope)::text = ANY ((ARRAY['project'::character varying, 'instance'::character varying])::text[]))) |
|
||||
| CHK_agents_scope_project | CHECK | CHECK ((((scope)::text = 'instance'::text) = ("projectId" IS NULL))) |
|
||||
| FK_940597dfe9753375309ce6aeea0 | FOREIGN KEY | FOREIGN KEY ("activeVersionId") REFERENCES agent_history("versionId") ON DELETE SET NULL |
|
||||
| FK_a30d560207c4071d98aa03c179c | FOREIGN KEY | FOREIGN KEY ("projectId") REFERENCES project(id) ON DELETE CASCADE |
|
||||
| PK_9c653f28ae19c5884d5baf6a1d9 | PRIMARY KEY | PRIMARY KEY (id) |
|
||||
|
|
@ -34,8 +31,8 @@
|
|||
| agents_id_not_null | n | NOT NULL id |
|
||||
| agents_integrations_not_null | n | NOT NULL integrations |
|
||||
| agents_name_not_null | n | NOT NULL name |
|
||||
| agents_projectId_not_null | n | NOT NULL "projectId" |
|
||||
| agents_revision_not_null | n | NOT NULL revision |
|
||||
| agents_scope_not_null | n | NOT NULL scope |
|
||||
| agents_skills_not_null | n | NOT NULL skills |
|
||||
| agents_tools_not_null | n | NOT NULL tools |
|
||||
| agents_updatedAt_not_null | n | NOT NULL "updatedAt" |
|
||||
|
|
@ -64,7 +61,6 @@ erDiagram
|
|||
"public.agent_execution_threads" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agent_files" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agent_history" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agent_skill_dependency" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agent_task_definition" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agent_task_run_lock" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agent_workflow_dependency" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
|
|
@ -75,8 +71,7 @@ erDiagram
|
|||
"public.agents_observation_cursors" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agents_observation_locks" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agents_observations" }o--|| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.skill" }o--o| "public.agents" : "FOREIGN KEY (#quot;agentId#quot;) REFERENCES agents(id) ON DELETE CASCADE"
|
||||
"public.agents" }o--o| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
"public.agents" }o--|| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
|
||||
"public.agents" {
|
||||
varchar_36_ activeVersionId FK
|
||||
|
|
@ -88,7 +83,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
@ -119,11 +113,8 @@ erDiagram
|
|||
varchar_255_ parentResourceId
|
||||
varchar_128_ parentThreadId
|
||||
uuid pauseRequestId
|
||||
uuid planId FK
|
||||
uuid planTaskId
|
||||
text result
|
||||
timestamp_3__with_time_zone settledAt
|
||||
varchar_36_ sourceExecutionId FK
|
||||
varchar_16_ status
|
||||
varchar_36_ subAgentId
|
||||
timestamp_3__with_time_zone timeoutAt
|
||||
|
|
@ -205,7 +196,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer sessionNumber
|
||||
varchar_32_ taskId
|
||||
json taskStop
|
||||
varchar_36_ taskVersionId FK
|
||||
varchar_255_ title
|
||||
integer totalCompletionTokens
|
||||
|
|
@ -226,12 +216,6 @@ erDiagram
|
|||
varchar_2_ storedAt
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.agent_skill_dependency" {
|
||||
varchar_36_ agentId FK
|
||||
timestamp_3__with_time_zone createdAt
|
||||
varchar_36_ skillId FK
|
||||
uuid skillVersionId FK
|
||||
}
|
||||
"public.agent_task_definition" {
|
||||
varchar_36_ agentId FK
|
||||
timestamp_3__with_time_zone createdAt
|
||||
|
|
@ -336,16 +320,6 @@ erDiagram
|
|||
integer tokenCount
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.skill" {
|
||||
varchar_36_ agentId FK
|
||||
boolean availableToAssistant
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid createdById FK
|
||||
varchar_36_ id
|
||||
varchar_36_ projectId FK
|
||||
varchar_16_ source
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.project" {
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid creatorId FK
|
||||
|
|
|
|||
|
|
@ -82,7 +82,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
|
|
@ -91,7 +91,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
|
|
@ -58,7 +58,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
|
|
@ -80,7 +80,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
|
|
@ -58,7 +58,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
|
|
@ -62,7 +62,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
|
|
@ -81,7 +81,6 @@ erDiagram
|
|||
varchar_255_ projectId FK
|
||||
integer revision
|
||||
json schema
|
||||
varchar_16_ scope
|
||||
timestamp_3__with_time_zone setupCompletedAt
|
||||
json skills
|
||||
json tools
|
||||
|
|
|
|||
92
docs/generated/postgres-schema/public.app.md
generated
92
docs/generated/postgres-schema/public.app.md
generated
|
|
@ -1,92 +0,0 @@
|
|||
# public.app
|
||||
|
||||
## Columns
|
||||
|
||||
| Name | Type | Default | Nullable | Children | Parents | Comment |
|
||||
| ---- | ---- | ------- | -------- | -------- | ------- | ------- |
|
||||
| createdAt | timestamp(3) with time zone | CURRENT_TIMESTAMP(3) | false | | | |
|
||||
| draftVersionId | uuid | | true | | [public.app_version](public.app_version.md) | Current working app version |
|
||||
| id | uuid | | false | [public.app_version](public.app_version.md) | | |
|
||||
| name | varchar(128) | | false | | | |
|
||||
| namespace | varchar(128) | | false | | | Unique URL path segment for the app |
|
||||
| projectId | varchar(36) | | false | | [public.project](public.project.md) | |
|
||||
| publishedVersionId | uuid | | true | | [public.app_version](public.app_version.md) | Published app version; null when unpublished |
|
||||
| updatedAt | timestamp(3) with time zone | CURRENT_TIMESTAMP(3) | false | | | |
|
||||
| visibility | varchar(16) | 'project'::character varying | false | | | App access: everyone, project members, or instance users |
|
||||
|
||||
## Constraints
|
||||
|
||||
| Name | Type | Definition |
|
||||
| ---- | ---- | ---------- |
|
||||
| CHK_app_visibility | CHECK | CHECK (((visibility)::text = ANY ((ARRAY['public'::character varying, 'project'::character varying, 'instance'::character varying])::text[]))) |
|
||||
| FK_f84dd7eb539e46e0c233fa09b20 | FOREIGN KEY | FOREIGN KEY ("projectId") REFERENCES project(id) ON DELETE CASCADE |
|
||||
| PK_9478629fc093d229df09e560aea | PRIMARY KEY | PRIMARY KEY (id) |
|
||||
| app_createdAt_not_null | n | NOT NULL "createdAt" |
|
||||
| app_draftVersionId_foreign | FOREIGN KEY | FOREIGN KEY ("draftVersionId") REFERENCES app_version(id) ON DELETE SET NULL |
|
||||
| app_id_not_null | n | NOT NULL id |
|
||||
| app_name_not_null | n | NOT NULL name |
|
||||
| app_namespace_not_null | n | NOT NULL namespace |
|
||||
| app_projectId_not_null | n | NOT NULL "projectId" |
|
||||
| app_publishedVersionId_foreign | FOREIGN KEY | FOREIGN KEY ("publishedVersionId") REFERENCES app_version(id) ON DELETE SET NULL |
|
||||
| app_updatedAt_not_null | n | NOT NULL "updatedAt" |
|
||||
| app_visibility_not_null | n | NOT NULL visibility |
|
||||
|
||||
## Indexes
|
||||
|
||||
| Name | Definition |
|
||||
| ---- | ---------- |
|
||||
| IDX_a73a1fa47c9e5097660f1161b3 | CREATE UNIQUE INDEX "IDX_a73a1fa47c9e5097660f1161b3" ON public.app USING btree (namespace) |
|
||||
| IDX_ca0a32194dd73ba54acb0adabf | CREATE INDEX "IDX_ca0a32194dd73ba54acb0adabf" ON public.app USING btree ("draftVersionId") |
|
||||
| IDX_e1c99aaf677c86879562203218 | CREATE INDEX "IDX_e1c99aaf677c86879562203218" ON public.app USING btree ("publishedVersionId") |
|
||||
| IDX_f84dd7eb539e46e0c233fa09b2 | CREATE INDEX "IDX_f84dd7eb539e46e0c233fa09b2" ON public.app USING btree ("projectId") |
|
||||
| PK_9478629fc093d229df09e560aea | CREATE UNIQUE INDEX "PK_9478629fc093d229df09e560aea" ON public.app USING btree (id) |
|
||||
|
||||
## Relations
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
|
||||
"public.app" }o--o| "public.app_version" : "FOREIGN KEY (#quot;draftVersionId#quot;) REFERENCES app_version(id) ON DELETE SET NULL"
|
||||
"public.app_version" }o--|| "public.app" : "FOREIGN KEY (#quot;appId#quot;) REFERENCES app(id) ON DELETE CASCADE"
|
||||
"public.app" }o--|| "public.project" : "FOREIGN KEY (#quot;projectId#quot;) REFERENCES project(id) ON DELETE CASCADE"
|
||||
"public.app" }o--o| "public.app_version" : "FOREIGN KEY (#quot;publishedVersionId#quot;) REFERENCES app_version(id) ON DELETE SET NULL"
|
||||
|
||||
"public.app" {
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid draftVersionId FK
|
||||
uuid id
|
||||
varchar_128_ name
|
||||
varchar_128_ namespace
|
||||
varchar_36_ projectId FK
|
||||
uuid publishedVersionId FK
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
varchar_16_ visibility
|
||||
}
|
||||
"public.app_version" {
|
||||
uuid appId FK
|
||||
integer buildAttempts
|
||||
varchar_16_ buildStatus
|
||||
timestamp_3__with_time_zone buildStatusUpdatedAt
|
||||
integer bundleArtifactId FK
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid id
|
||||
varchar_255_ label
|
||||
integer sourceArtifactId FK
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.project" {
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid creatorId FK
|
||||
json customTelemetryTags
|
||||
varchar_512_ description
|
||||
json icon
|
||||
varchar_36_ id
|
||||
varchar_255_ name
|
||||
varchar_36_ type
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> Generated by [tbls](https://github.com/k1LoW/tbls)
|
||||
|
|
@ -1,72 +0,0 @@
|
|||
# public.app_artifact
|
||||
|
||||
## Columns
|
||||
|
||||
| Name | Type | Default | Nullable | Children | Parents | Comment |
|
||||
| ---- | ---- | ------- | -------- | -------- | ------- | ------- |
|
||||
| appId | uuid | | false | | | No foreign key so the owning app ID remains available for cleanup after app deletion. |
|
||||
| createdAt | timestamp(3) with time zone | CURRENT_TIMESTAMP(3) | false | | | |
|
||||
| id | integer | | false | [public.app_version](public.app_version.md) | | |
|
||||
| sizeBytes | integer | | false | | | |
|
||||
| storageKey | text | | false | | | Reference to the object within its storage backend |
|
||||
| storedAt | varchar(2) | | false | | | Storage backend: database, filesystem, S3, or Azure |
|
||||
| updatedAt | timestamp(3) with time zone | CURRENT_TIMESTAMP(3) | false | | | |
|
||||
| uploadStatus | varchar(16) | 'pending'::character varying | false | | | Upload state: pending or completed |
|
||||
|
||||
## Constraints
|
||||
|
||||
| Name | Type | Definition |
|
||||
| ---- | ---- | ---------- |
|
||||
| CHK_app_artifact_storedAt | CHECK | CHECK ((("storedAt")::text = ANY ((ARRAY['db'::character varying, 'fs'::character varying, 's3'::character varying, 'az'::character varying])::text[]))) |
|
||||
| CHK_app_artifact_uploadStatus | CHECK | CHECK ((("uploadStatus")::text = ANY ((ARRAY['pending'::character varying, 'completed'::character varying])::text[]))) |
|
||||
| PK_307f7470d2b759f7c4bc056d3a5 | PRIMARY KEY | PRIMARY KEY (id) |
|
||||
| app_artifact_appId_not_null | n | NOT NULL "appId" |
|
||||
| app_artifact_createdAt_not_null | n | NOT NULL "createdAt" |
|
||||
| app_artifact_id_not_null | n | NOT NULL id |
|
||||
| app_artifact_sizeBytes_not_null | n | NOT NULL "sizeBytes" |
|
||||
| app_artifact_storageKey_not_null | n | NOT NULL "storageKey" |
|
||||
| app_artifact_storedAt_not_null | n | NOT NULL "storedAt" |
|
||||
| app_artifact_updatedAt_not_null | n | NOT NULL "updatedAt" |
|
||||
| app_artifact_uploadStatus_not_null | n | NOT NULL "uploadStatus" |
|
||||
|
||||
## Indexes
|
||||
|
||||
| Name | Definition |
|
||||
| ---- | ---------- |
|
||||
| PK_307f7470d2b759f7c4bc056d3a5 | CREATE UNIQUE INDEX "PK_307f7470d2b759f7c4bc056d3a5" ON public.app_artifact USING btree (id) |
|
||||
|
||||
## Relations
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
|
||||
"public.app_version" }o--o| "public.app_artifact" : "FOREIGN KEY (#quot;bundleArtifactId#quot;) REFERENCES app_artifact(id) ON DELETE SET NULL"
|
||||
"public.app_version" }o--|| "public.app_artifact" : "FOREIGN KEY (#quot;sourceArtifactId#quot;) REFERENCES app_artifact(id) ON DELETE RESTRICT"
|
||||
|
||||
"public.app_artifact" {
|
||||
uuid appId
|
||||
timestamp_3__with_time_zone createdAt
|
||||
integer id
|
||||
integer sizeBytes
|
||||
text storageKey
|
||||
varchar_2_ storedAt
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
varchar_16_ uploadStatus
|
||||
}
|
||||
"public.app_version" {
|
||||
uuid appId FK
|
||||
integer buildAttempts
|
||||
varchar_16_ buildStatus
|
||||
timestamp_3__with_time_zone buildStatusUpdatedAt
|
||||
integer bundleArtifactId FK
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid id
|
||||
varchar_255_ label
|
||||
integer sourceArtifactId FK
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> Generated by [tbls](https://github.com/k1LoW/tbls)
|
||||
92
docs/generated/postgres-schema/public.app_version.md
generated
92
docs/generated/postgres-schema/public.app_version.md
generated
|
|
@ -1,92 +0,0 @@
|
|||
# public.app_version
|
||||
|
||||
## Columns
|
||||
|
||||
| Name | Type | Default | Nullable | Children | Parents | Comment |
|
||||
| ---- | ---- | ------- | -------- | -------- | ------- | ------- |
|
||||
| appId | uuid | | false | | [public.app](public.app.md) | |
|
||||
| buildAttempts | integer | 0 | false | | | |
|
||||
| buildStatus | varchar(16) | 'queued'::character varying | false | | | Build state: queued, running, succeeded, or failed |
|
||||
| buildStatusUpdatedAt | timestamp(3) with time zone | | true | | | |
|
||||
| bundleArtifactId | integer | | true | | [public.app_artifact](public.app_artifact.md) | |
|
||||
| createdAt | timestamp(3) with time zone | CURRENT_TIMESTAMP(3) | false | | | |
|
||||
| id | uuid | | false | [public.app](public.app.md) | | |
|
||||
| label | varchar(255) | | true | | | |
|
||||
| sourceArtifactId | integer | | false | | [public.app_artifact](public.app_artifact.md) | |
|
||||
| updatedAt | timestamp(3) with time zone | CURRENT_TIMESTAMP(3) | false | | | |
|
||||
|
||||
## Constraints
|
||||
|
||||
| Name | Type | Definition |
|
||||
| ---- | ---- | ---------- |
|
||||
| CHK_app_version_buildStatus | CHECK | CHECK ((("buildStatus")::text = ANY ((ARRAY['queued'::character varying, 'running'::character varying, 'succeeded'::character varying, 'failed'::character varying])::text[]))) |
|
||||
| FK_2ab536cbf97d71bad59517afee5 | FOREIGN KEY | FOREIGN KEY ("bundleArtifactId") REFERENCES app_artifact(id) ON DELETE SET NULL |
|
||||
| FK_9f1db6f02f5a0d1ae2f2b41a327 | FOREIGN KEY | FOREIGN KEY ("sourceArtifactId") REFERENCES app_artifact(id) ON DELETE RESTRICT |
|
||||
| FK_e9aeab5b1db8dc77708231ae44e | FOREIGN KEY | FOREIGN KEY ("appId") REFERENCES app(id) ON DELETE CASCADE |
|
||||
| PK_f2573b981a7eac664875e7483ac | PRIMARY KEY | PRIMARY KEY (id) |
|
||||
| app_version_appId_not_null | n | NOT NULL "appId" |
|
||||
| app_version_buildAttempts_not_null | n | NOT NULL "buildAttempts" |
|
||||
| app_version_buildStatus_not_null | n | NOT NULL "buildStatus" |
|
||||
| app_version_createdAt_not_null | n | NOT NULL "createdAt" |
|
||||
| app_version_id_not_null | n | NOT NULL id |
|
||||
| app_version_sourceArtifactId_not_null | n | NOT NULL "sourceArtifactId" |
|
||||
| app_version_updatedAt_not_null | n | NOT NULL "updatedAt" |
|
||||
|
||||
## Indexes
|
||||
|
||||
| Name | Definition |
|
||||
| ---- | ---------- |
|
||||
| IDX_2ab536cbf97d71bad59517afee | CREATE INDEX "IDX_2ab536cbf97d71bad59517afee" ON public.app_version USING btree ("bundleArtifactId") |
|
||||
| IDX_9f1db6f02f5a0d1ae2f2b41a32 | CREATE INDEX "IDX_9f1db6f02f5a0d1ae2f2b41a32" ON public.app_version USING btree ("sourceArtifactId") |
|
||||
| IDX_e9aeab5b1db8dc77708231ae44 | CREATE INDEX "IDX_e9aeab5b1db8dc77708231ae44" ON public.app_version USING btree ("appId") |
|
||||
| PK_f2573b981a7eac664875e7483ac | CREATE UNIQUE INDEX "PK_f2573b981a7eac664875e7483ac" ON public.app_version USING btree (id) |
|
||||
|
||||
## Relations
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
|
||||
"public.app_version" }o--|| "public.app" : "FOREIGN KEY (#quot;appId#quot;) REFERENCES app(id) ON DELETE CASCADE"
|
||||
"public.app_version" }o--o| "public.app_artifact" : "FOREIGN KEY (#quot;bundleArtifactId#quot;) REFERENCES app_artifact(id) ON DELETE SET NULL"
|
||||
"public.app" }o--o| "public.app_version" : "FOREIGN KEY (#quot;draftVersionId#quot;) REFERENCES app_version(id) ON DELETE SET NULL"
|
||||
"public.app" }o--o| "public.app_version" : "FOREIGN KEY (#quot;publishedVersionId#quot;) REFERENCES app_version(id) ON DELETE SET NULL"
|
||||
"public.app_version" }o--|| "public.app_artifact" : "FOREIGN KEY (#quot;sourceArtifactId#quot;) REFERENCES app_artifact(id) ON DELETE RESTRICT"
|
||||
|
||||
"public.app_version" {
|
||||
uuid appId FK
|
||||
integer buildAttempts
|
||||
varchar_16_ buildStatus
|
||||
timestamp_3__with_time_zone buildStatusUpdatedAt
|
||||
integer bundleArtifactId FK
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid id
|
||||
varchar_255_ label
|
||||
integer sourceArtifactId FK
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
}
|
||||
"public.app" {
|
||||
timestamp_3__with_time_zone createdAt
|
||||
uuid draftVersionId FK
|
||||
uuid id
|
||||
varchar_128_ name
|
||||
varchar_128_ namespace
|
||||
varchar_36_ projectId FK
|
||||
uuid publishedVersionId FK
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
varchar_16_ visibility
|
||||
}
|
||||
"public.app_artifact" {
|
||||
uuid appId
|
||||
timestamp_3__with_time_zone createdAt
|
||||
integer id
|
||||
integer sizeBytes
|
||||
text storageKey
|
||||
varchar_2_ storedAt
|
||||
timestamp_3__with_time_zone updatedAt
|
||||
varchar_16_ uploadStatus
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> Generated by [tbls](https://github.com/k1LoW/tbls)
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user