1cc28ee2e3
Pass/fail is the wrong primary metric for a context or memory change. An agent with weaker memory can compensate by re-reading instance state — which for some cases is the desired behaviour — so two approaches can tie on pass rate while differing sharply in what they spent. That difference only shows up in tokens, and the harness reported none. Sum cache-aware token totals and a tool-call count per iteration into `eval-results.json`. Nothing new needed collecting: run debug is already captured per build and already reaches the run result, so this is extraction plus emission. Cache reads are kept separate from uncached input rather than folded together. They are an order of magnitude cheaper, and cache hit rate is usually the term a context change moves most, so combining them would hide the effect. `build-cost-report` gains a third cost source that prices these counts locally, preferred over the existing LangSmith thread join. A two-arm quality-per-token comparison now reads straight from two result files with no credentials, no network and no trace-retention window; the join remains the fallback for runs recorded before these fields existed. Rates default to Sonnet-class list prices and are overridable per token class. Tool calls count every non-narration step rather than only `kind: 'tool-call'`. Several tools render as their own step kind and never fall through — a `create-tasks` call becomes `plan`, an `ask-user` call becomes `ask-user` — so the narrower filter dropped orchestration calls most multi-step conversations make. This matches `seededTurnCounters`, which produces the per-turn count reported in the same artifact, so the two numbers cannot disagree. Seeded turns are excluded: restored fixture is not work the evaluated run did. The new fields are omitted rather than emitted all-null when a run captured nothing, since this file is a cross-repo contract and the report already treats an all-null array as absent. |
||
|---|---|---|
| .agents/skills | ||
| .claude | ||
| .devcontainer | ||
| .github | ||
| .opencode/skills/setup-mcps | ||
| .vscode | ||
| assets | ||
| docker | ||
| docs | ||
| packages | ||
| patches | ||
| scripts | ||
| security | ||
| .actrc | ||
| .aikido | ||
| .boundaries-baseline.json | ||
| .code-health-baseline.json | ||
| .dockerignore | ||
| .editorconfig | ||
| .env.eval.example | ||
| .env.local.example | ||
| .git-blame-ignore-revs | ||
| .gitattributes | ||
| .gitignore | ||
| .npmignore | ||
| .npmrc | ||
| .poutine.yml | ||
| .prettierignore | ||
| .prettierrc.js | ||
| .tbls.postgres.yml | ||
| .tbls.sqlite.yml | ||
| AGENTS.md | ||
| biome.jsonc | ||
| CHANGELOG.md | ||
| CLAUDE.md | ||
| CODE_OF_CONDUCT.md | ||
| codecov.yml | ||
| CONTRIBUTING.md | ||
| CONTRIBUTOR_LICENSE_AGREEMENT.md | ||
| cubic.yaml | ||
| lefthook.yml | ||
| LICENSE_EE.md | ||
| LICENSE.md | ||
| package.json | ||
| pnpm-lock.yaml | ||
| pnpm-workspace.yaml | ||
| README.md | ||
| SECURITY.md | ||
| tsconfig.configs.json | ||
| tsconfig.json | ||
| turbo.json | ||
| vitest.workspace.ts | ||
n8n – The Platform for AI Agents and Workflow Automation
Fair-code platform to build and deploy AI agents and workflows. Combine a visual canvas with custom code, run it self-hosted or in the cloud, and connect to 1500+ integrations. AI automation you can trust with real work, from prototype to production.
Key Capabilities
- AI-Native Automation Platform: Build and operationalize AI workflows and multi-step agents using your own data, models, and tools
- Model Flexibility, No Lock-In: Connect to OpenAI, Anthropic, Google, or open-source models and switch providers without changing your architecture
- From Prototype to Production: Design multi-step AI workflows with logic, tool use, human approvals, and full observability
- Code When You Need It: Combine visual building with JavaScript, Python, and npm packages for advanced AI workflows
- Enterprise-Ready AI: Self-host or deploy securely with role-based access, audit trails, and support for sensitive data
- Leverage What Already Exists: 1500+ integrations and 9,000+ workflow templates to connect AI with your existing systems
Quick Start
Try n8n instantly with npx (requires Node.js):
npx n8n
Or deploy with Docker:
docker volume create n8n_data
docker run -it --rm --name n8n -p 5678:5678 -v n8n_data:/home/node/.n8n docker.n8n.io/n8nio/n8n
Access the editor at http://localhost:5678
Resources
- 📚 Documentation
- 🔧 1500+ Integrations
- 💡 Example Workflows
- 🤖 AI & LangChain Guide
- 👥 Community Forum
- 📖 Community Tutorials
Support
Need help? Our community forum is the place to get support and connect with other users: community.n8n.io
License
n8n is fair-code distributed under the Sustainable Use License and n8n Enterprise License.
- Source Available: Always visible source code
- Self-Hostable: Deploy anywhere
- Extensible: Add your own nodes and functionality
Enterprise Licenses available for additional features and support.
Additional information about the license model can be found in the docs.
Contributing
Found a bug 🐛 or have a feature idea ✨? Check our Contributing Guide for a setup guide & best practices.
Join the Team
Want to shape the future of automation? Check out our job posts and join our team!
What does n8n mean?
Short answer: It means "nodemation" and is pronounced as n-eight-n.
Long answer: "I get that question quite often (more often than I expected) so I decided it is probably best to answer it here. While looking for a good name for the project with a free domain I realized very quickly that all the good ones I could think of were already taken. So, in the end, I chose nodemation. 'node-' in the sense that it uses a Node-View and that it uses Node.js and '-mation' for 'automation' which is what the project is supposed to help with. However, I did not like how long the name was and I could not imagine writing something that long every time in the CLI. That is when I then ended up on 'n8n'." - Jan Oberhauser, Founder and CEO, n8n.io