n8n/packages/@n8n/instance-ai/evaluations/data/workflows/linear-bq-leaderboard.json
José Braulio González Valido 81ea56fa6b
test(ai-builder): Add multi-turn capability for IAI evals (no-changelog) (#30586)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:03:35 +00:00

37 lines
3.3 KiB
JSON

{
"conversation": [
{
"role": "user",
"text": "I want a workflow that posts a bi-weekly Slack message ranking my team by how engaged they are with our product."
},
{
"role": "assistant",
"text": "How should engagement be measured, and where should the ranking post?"
},
{
"role": "user",
"text": "Each person's score combines their Linear bugs filed (any team, `bug` label, case-sensitive, matched by name) and their n8n usage hours from BigQuery. Order the list by bugs filed (desc), then by execs (desc). Team is David Roberts (id 1), David Arens (id 2), Niklas Hatje (id 3). Post to Slack channel D034WT7G4CW."
},
{
"role": "assistant",
"text": "What format should the Slack message use, and do you have a BigQuery query I should reuse?"
},
{
"role": "user",
"text": "Format like this:\n\n```\nUsage in the last two weeks:\n\nJonathan Clift: 7 tickets (5 execs, 3 hours)\nFabian Puehringer: 7 tickets (4 execs, 1 hours)\nTuukka Kantola: 6 tickets (16 execs, 6 hours)\n\nTickets = Linear bug tickets created\nExecs = Manual execs on registered accounts\n```\n\nBigQuery:\n\nwith\nsettings as (\n select\n timestamp('<start_cutoff>') as start_cutoff,\n timestamp('<end_cutoff>') as end_cutoff,\n),\nuser_accounts as (\n select * from unnest([\n struct<name string, user_id string>\n ...\n ])\n),\nnode_exec as (\n select\n timestamp,\n timestamp_trunc(timestamp, hour) as timestamp_hour,\n name,\n f.user_id,\n f.instance_id,\n status,\n from rudder_schema.manual_node_exec_finished f\n inner join user_accounts a on a.user_id = f.user_id\n cross join settings\n where f.timestamp between start_cutoff and end_cutoff\n),\nworkflow_exec as (\n select\n timestamp,\n timestamp_trunc(timestamp, hour) as timestamp_hour,\n name,\n f.user_id,\n f.instance_id,\n status,\n from rudder_schema.manual_workflow_exec_finished f\n inner join user_accounts a on a.user_id = f.user_id\n cross join settings\n where f.timestamp between start_cutoff and end_cutoff\n),\nexec as (\n select * from node_exec\n union all\n select * from workflow_exec\n),\nexec_summary as (\n select\n name,\n count(distinct instance_id) as instances,\n count(distinct timestamp_hour) as hours,\n count(*) as manual_execs,\n from exec\n group by 1\n)\n\nselect * from exec_summary"
}
],
"messageBudget": 6,
"complexity": "simple",
"tags": ["build", "schedule", "http-request", "bigquery"],
"triggerType": "schedule",
"executionScenarios": [
{
"name": "happy-path",
"description": "Leaderboard is posted to Slack",
"dataSetup": "Linear issues created in time period:\n - Issue1 by David Arens (has bug label)\n - Issue2 by David Arens (has no bug label)\n - Issue3 by David Roberts (has bug label)\n - Issue4 by David Arens (has bug label)\n\nResult of BQ query on n8n usage:\n - David Arens: 72 execs, 122 hours\n - David Roberts: 556 execs, 2 hours\n - Niklas Hatje: 0 execs, 0 hours",
"successCriteria": "The workflow executes without errors. A Slack message is posted in the channel specified using the format specified, and it contains the same data that was specified in the dataSetup. Niklas Hatje is included in the Slack message, and is listed as reporting 0 bugs."
}
]
}