mirror of
https://github.com/Crosstalk-Solutions/project-nomad.git
synced 2026-10-17 05:14:19 +02:00
a915cdcfc9
The post-rerank relevance floor rejects noise but passes coherent off-topic questions, and it gets worse as the library grows. Adding Gutenberg prose, the Python tutorial and MDN pages to the eval corpus took the off-topic leak rate from 22% to 67% at the 0.62 floor. Score cutoff does not fix it. The weakest correct top hit scores 0.666 while "write me a haiku about autumn leaves" pulls Walden at 0.705. A cosine floor and a top-vs-median contrast test both cost recall faster than they cut leaks. - Add rag.relevanceCheck (default off). After the floor, the tasks model, or the chat model when none is set, gives one grammar-constrained on_topic verdict for the whole retrieved set. Off topic means no context and no citations. Any failure keeps the chunks unjudged. - Add a Settings > Models toggle recommending an 8B+ tasks model. On llama3:8b the check cuts off-topic leaks from 67% to 6% for about 5 points of core recall@5 and ~0.6s per turn. On 3B models it rejects most valid questions, which is why it is opt-in. - Add 6 distractor documents and an off_topic suite (18 unanswerable questions, 7 distractor controls) to the eval corpus. - Report per-tag leak rates and leaked cases with top scores in eval:retrieval. Add --judge-model to measure the check and --dump to export pre-floor candidate pools for offline gate simulation. - Pin the check off in eval:generation so a local setting cannot move scores. - Record the distractor sweep on RAG_MIN_FINAL_SCORE. The 0.62 default stays: 0.65 halves leaks at no measured recall cost but leaves only 0.016 of margin. The corpus fingerprint changes to b4ebe5dce8699f5d. Retrieval reports from before this commit are not comparable, and a new baseline is committed. Refs #1341
33 lines
6.6 KiB
JSON
33 lines
6.6 KiB
JSON
// #1341: coherent questions the corpus cannot answer. Every refusal case here must retrieve NOTHING.
|
|
// The distractor-* documents (Gutenberg prose, Python tutorial, MDN) sit near everything in embedding
|
|
// space, which is what makes the relevance floor fire on a well-stocked library and not on a small one.
|
|
// Retrieval-only for now: generation scores expectRefusal as "the model declined", and "Paris" is the
|
|
// right answer to the capital question as long as nothing is cited for it.
|
|
{"id": "offtopic-issue-gibberish-01", "query": "zxqw plorbnat frunkle wibbet 7734 quozz", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "gibberish", "issue-1341"]}
|
|
{"id": "offtopic-issue-banana-01", "query": "Reply with exactly the single word: banana", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "instruction", "issue-1341"]}
|
|
{"id": "offtopic-issue-france-01", "query": "What is the capital city of France?", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "general-knowledge", "issue-1341"]}
|
|
{"id": "offtopic-issue-birthday-01", "query": "My sister's birthday is next Tuesday. What should I get her?", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "personal", "issue-1341"]}
|
|
{"id": "offtopic-issue-raised-bed-01", "query": "How deep should soil be in a raised garden bed?", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "missing-domain", "issue-1341"]}
|
|
{"id": "offtopic-hamlet-01", "query": "Who wrote Hamlet?", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "general-knowledge"]}
|
|
{"id": "offtopic-planets-01", "query": "How many planets are in the solar system?", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "general-knowledge"]}
|
|
{"id": "offtopic-celsius-01", "query": "What is 30 degrees Celsius in Fahrenheit?", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "general-knowledge"]}
|
|
{"id": "offtopic-haiku-01", "query": "Write me a haiku about autumn leaves.", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "instruction"]}
|
|
{"id": "offtopic-joke-01", "query": "Tell me a joke.", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "instruction"]}
|
|
{"id": "offtopic-greeting-01", "query": "Hi there, how are you doing today?", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "instruction"]}
|
|
{"id": "offtopic-translate-01", "query": "Translate 'good morning' into Spanish.", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "instruction"]}
|
|
{"id": "offtopic-puppy-name-01", "query": "What should I name my new golden retriever puppy?", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "personal"]}
|
|
{"id": "offtopic-resume-01", "query": "Can you help me write a cover letter for a marketing job?", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "personal"]}
|
|
{"id": "offtopic-tomatoes-01", "query": "When should I transplant tomato seedlings outdoors?", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "missing-domain"]}
|
|
{"id": "offtopic-oil-change-01", "query": "How often should I change the oil in my car?", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "missing-domain"]}
|
|
{"id": "offtopic-sourdough-01", "query": "How do I feed a sourdough starter?", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "missing-domain"]}
|
|
{"id": "offtopic-mortgage-01", "query": "Should I refinance my mortgage if rates drop by half a percent?", "turns": [], "relevantDocIds": [], "mustInclude": [], "mustNotInclude": [], "expectRefusal": true, "tags": ["off-topic", "missing-domain"]}
|
|
// Controls: questions the distractors DO answer. A gate that fixes the cases above by rejecting these
|
|
// has traded one bug for another, so these must keep retrieving their document.
|
|
{"id": "distractor-python-listcomp-01", "query": "How do list comprehensions work in Python?", "turns": [], "relevantDocIds": ["distractor-python-tutorial"], "mustInclude": [], "mustNotInclude": [], "expectRefusal": false, "tags": ["distractor-control"]}
|
|
{"id": "distractor-python-exceptions-01", "query": "How do I catch an exception and still run cleanup code in Python?", "turns": [], "relevantDocIds": ["distractor-python-tutorial"], "mustInclude": [], "mustNotInclude": [], "expectRefusal": false, "tags": ["distractor-control"]}
|
|
{"id": "distractor-js-promises-01", "query": "How do I chain promises in JavaScript and handle errors?", "turns": [], "relevantDocIds": ["distractor-mdn-javascript-guide"], "mustInclude": [], "mustNotInclude": [], "expectRefusal": false, "tags": ["distractor-control"]}
|
|
{"id": "distractor-css-flex-01", "query": "How do I center items along the main axis of a flex container?", "turns": [], "relevantDocIds": ["distractor-mdn-css-flexbox"], "mustInclude": [], "mustNotInclude": [], "expectRefusal": false, "tags": ["distractor-control"]}
|
|
{"id": "distractor-walden-beans-01", "query": "What did Thoreau write about hoeing his bean field at Walden?", "turns": [], "relevantDocIds": ["distractor-gutenberg-walden"], "mustInclude": [], "mustNotInclude": [], "expectRefusal": false, "tags": ["distractor-control"]}
|
|
{"id": "distractor-holmes-redheaded-01", "query": "What was the Red-Headed League in the Sherlock Holmes story?", "turns": [], "relevantDocIds": ["distractor-gutenberg-sherlock-holmes"], "mustInclude": [], "mustNotInclude": [], "expectRefusal": false, "tags": ["distractor-control"]}
|
|
{"id": "distractor-pride-darcy-01", "query": "What did Mr. Darcy say about Elizabeth Bennet at the ball when he refused to dance with her?", "turns": [], "relevantDocIds": ["distractor-gutenberg-pride-and-prejudice"], "mustInclude": [], "mustNotInclude": [], "expectRefusal": false, "tags": ["distractor-control"]}
|