mirror of
https://github.com/Crosstalk-Solutions/project-nomad.git
synced 2026-10-17 13:19:18 +02:00
a915cdcfc9
The post-rerank relevance floor rejects noise but passes coherent off-topic questions, and it gets worse as the library grows. Adding Gutenberg prose, the Python tutorial and MDN pages to the eval corpus took the off-topic leak rate from 22% to 67% at the 0.62 floor. Score cutoff does not fix it. The weakest correct top hit scores 0.666 while "write me a haiku about autumn leaves" pulls Walden at 0.705. A cosine floor and a top-vs-median contrast test both cost recall faster than they cut leaks. - Add rag.relevanceCheck (default off). After the floor, the tasks model, or the chat model when none is set, gives one grammar-constrained on_topic verdict for the whole retrieved set. Off topic means no context and no citations. Any failure keeps the chunks unjudged. - Add a Settings > Models toggle recommending an 8B+ tasks model. On llama3:8b the check cuts off-topic leaks from 67% to 6% for about 5 points of core recall@5 and ~0.6s per turn. On 3B models it rejects most valid questions, which is why it is opt-in. - Add 6 distractor documents and an off_topic suite (18 unanswerable questions, 7 distractor controls) to the eval corpus. - Report per-tag leak rates and leaked cases with top scores in eval:retrieval. Add --judge-model to measure the check and --dump to export pre-floor candidate pools for offline gate simulation. - Pin the check off in eval:generation so a local setting cannot move scores. - Record the distractor sweep on RAG_MIN_FINAL_SCORE. The 0.62 default stays: 0.65 halves leaks at no measured recall cost but leaves only 0.016 of margin. The corpus fingerprint changes to b4ebe5dce8699f5d. Retrieval reports from before this commit are not comparable, and a new baseline is committed. Refs #1341 |
||
|---|---|---|
| .. | ||
| distractor-gutenberg-pride-and-prejudice.md | ||
| distractor-gutenberg-sherlock-holmes.md | ||
| distractor-gutenberg-walden.md | ||
| distractor-mdn-css-flexbox.md | ||
| distractor-mdn-javascript-guide.md | ||
| distractor-python-tutorial.md | ||
| equipment-tr88-pump.md | ||
| firstaid-bleeding-control.md | ||
| firstaid-burns.md | ||
| firstaid-hypothermia.md | ||
| firstaid-ifak-contents.md | ||
| food-canning-pressure.md | ||
| food-canning-waterbath.md | ||
| food-dehydrating.md | ||
| food-mylar-oxygen.md | ||
| food-storage-rotation.md | ||
| gear-bug-out-bag.md | ||
| gear-edc.md | ||
| navigation-map-compass.md | ||
| power-battery-chemistry.md | ||
| power-emp-protection.md | ||
| power-solar-sizing.md | ||
| protocol-thornfield-cache.md | ||
| radio-antenna-basics.md | ||
| radio-gmrs-frs.md | ||
| radio-ham-bands.md | ||
| regions-elevation-table.md | ||
| sanitation-waste.md | ||
| shelter-cold-layering.md | ||
| water-boiling.md | ||
| water-chemical-treatment.md | ||
| water-filtration.md | ||
| water-river-song.md | ||
| water-storage.md | ||