mirror of
https://github.com/Crosstalk-Solutions/project-nomad.git
synced 2026-10-17 05:14:19 +02:00
a915cdcfc9
The post-rerank relevance floor rejects noise but passes coherent off-topic questions, and it gets worse as the library grows. Adding Gutenberg prose, the Python tutorial and MDN pages to the eval corpus took the off-topic leak rate from 22% to 67% at the 0.62 floor. Score cutoff does not fix it. The weakest correct top hit scores 0.666 while "write me a haiku about autumn leaves" pulls Walden at 0.705. A cosine floor and a top-vs-median contrast test both cost recall faster than they cut leaks. - Add rag.relevanceCheck (default off). After the floor, the tasks model, or the chat model when none is set, gives one grammar-constrained on_topic verdict for the whole retrieved set. Off topic means no context and no citations. Any failure keeps the chunks unjudged. - Add a Settings > Models toggle recommending an 8B+ tasks model. On llama3:8b the check cuts off-topic leaks from 67% to 6% for about 5 points of core recall@5 and ~0.6s per turn. On 3B models it rejects most valid questions, which is why it is opt-in. - Add 6 distractor documents and an off_topic suite (18 unanswerable questions, 7 distractor controls) to the eval corpus. - Report per-tag leak rates and leaked cases with top scores in eval:retrieval. Add --judge-model to measure the check and --dump to export pre-floor candidate pools for offline gate simulation. - Pin the check off in eval:generation so a local setting cannot move scores. - Record the distractor sweep on RAG_MIN_FINAL_SCORE. The 0.62 default stays: 0.65 halves leaks at no measured recall cost but leaves only 0.016 of margin. The corpus fingerprint changes to b4ebe5dce8699f5d. Retrieval reports from before this commit are not comparable, and a new baseline is committed. Refs #1341 |
||
|---|---|---|
| .. | ||
| eval | ||
| affirmative_remedies.ts | ||
| amd_hsa_override.ts | ||
| chat_images.ts | ||
| content_auto_update_backoff.ts | ||
| content_reindex_decision.ts | ||
| context_budget.ts | ||
| context_window.ts | ||
| downloads.ts | ||
| drug_install_row_reconcile.ts | ||
| fs.ts | ||
| gpu_model.ts | ||
| hosted_content_auth.ts | ||
| hosted_content.ts | ||
| image_disk_preflight.ts | ||
| kb_ingest_decision.ts | ||
| kb_job_health.ts | ||
| kb_orphan_decision.ts | ||
| kb_ratio_lookup.ts | ||
| kb_warning_decision.ts | ||
| misc.ts | ||
| model_capabilities.ts | ||
| platform_metadata.ts | ||
| queue_stall_options.ts | ||
| rag_prompt.ts | ||
| rag_relevance.ts | ||
| relevance_judge.ts | ||
| response_style.ts | ||
| sampler.ts | ||
| service_catalog_merge.ts | ||
| structured_output.ts | ||
| superseded_resource.ts | ||
| tasks_model.ts | ||
| think_stream.ts | ||
| token_estimate.ts | ||
| update_window.ts | ||
| version.ts | ||
| zim_batch_decision.ts | ||
| zim_download_resolution.ts | ||
| zim_filename.ts | ||
| zim_html.ts | ||