n8n/packages/@n8n/instance-ai/evaluations/cli
Claude d3bc90a777
feat(instance-ai): Record errored MCP build tool calls with messages (no-changelog)
Correlate tool_result events back to their tool_use by id and record every
is_error result as {tool, message} (message truncated to 500 chars; string
and text-block-array contents both handled). The failure list rides next to
toolCalls through the whole spend pipeline: buildToolErrorsPerRun per case
and per-tool error counts in summary.mcpBuild.toolErrors in
eval-results.json, a build_tool_errors count as LangSmith row feedback plus
per-tool totals in experiment metadata, and per-build lists + totals in the
manifest builder's stats sidecar. Timed-out builds keep contributing the
errors their partial stream recorded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GA23ytEUfHpzkv7meRdVAJ
2026-07-31 13:32:37 +03:00
..
args.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
build-cost-report.ts feat(ai-builder): Report per-case claude build cost for MCP eval builds (no-changelog) (#34777) 2026-07-23 18:42:09 +00:00
build-mcp-manifest.ts feat(instance-ai): Record errored MCP build tool calls with messages (no-changelog) 2026-07-31 13:32:37 +03:00
compare-pairwise.ts refactor: Remove @n8n/utils barrel export (no-changelog) (#33082) 2026-06-30 16:05:24 +03:00
index.ts fix(ai-builder): Route eval report artifacts through --output-dir (no-changelog) (#35083) 2026-07-29 11:04:42 +00:00
langtracer-push.ts feat(ai-builder): Push agent intent eval cases from data/agents and drop synced case files (no-changelog) (#34957) 2026-07-28 07:25:30 +00:00
mcp-builder.ts feat(instance-ai): Record errored MCP build tool calls with messages (no-changelog) 2026-07-31 13:32:37 +03:00
pairwise.ts refactor(ai-builder): Split the eval harness runner into domain modules (no-changelog) (#34834) 2026-07-27 09:01:09 +00:00
report.ts feat(core): Decouple Instance AI workflow verification from planning (#31340) 2026-06-05 10:44:47 +00:00