Commit Graph

4 Commits

Author SHA1 Message Date
jakeaturner
ae06a295de feat(AI): let retrieval decline instead of making the prompt paper over it
There was one cutoff, Qdrant's score_threshold at 0.3, and nomic scores
unrelated text well above it, so every question retrieved *something*
(nonEmptyRateOnRefusal 1.0) and rag_context wasted four lines asking a 3B
model to "silently judge" relevance. That is a classification a number
should decide, not a micro model.

Adds `RAG_MIN_FINAL_SCORE`, a floor on the post-rerank score, applied before
source diversity. The floor judges relevance, the 0.85^n penalty judges
redundancy. Nothing clears it and nothing is injected. `score_threshold` stays
at 0.3 as the candidate net. Exposed as `rag.minRelevance` with a module-
scoped TTL cache invalidated from `updateSetting`.

The floor is calibrated and the harness was describing the wrong number:
aggregate() bucketed semanticScore, so its percentiles measured raw cosine
while the new knob filters the reranked score. Both axes now, and the gap
matters: recall falls at 0.60 on the semantic axis, 0.66 on the final one.

  floor  recall@5  precision@5  emptyOnAnswerable  nonEmptyOnRefusal
  0.00     0.991      0.221            0%               100%
  0.62     0.991      0.524            0%                60%
  0.66     0.991      0.695            0%                60%
  0.67     0.972      0.704            1%                60%

Default 0.62, not the last-lossless 0.66: the refusal win is complete at
0.62 and 28 documents is a thin basis for spending margin. 0.60 is the
honest floor. Three of five refusal goldens are near-corpus by
construction, where declining is the generation tier's job.

Both eval tiers pin the floor to the constant and ignore the setting, for
the same reason the harness never sets skipRetrieval.

4 metrics improved against baseline, no regressions.
2026-09-29 15:54:41 -07:00
Jake Turner
86ef96b200 feat(AI): specify a model for ancillary tasks (#1244)
Allows users to optionally specify a model to use for ancillary tasks (chat suggestions, chat titles, etc.) instead of strictly using the last used model. This means aesthetic/non-critical work can be passed to a small, lightweight model instead of a heavy reasoning model used for chats.

If a model is not selected in the AI Assistant settings (`/settings/models`), the existing behavior of using the last used model is retained.
2026-09-29 15:54:41 -07:00
Jake Turner
1923cd4cde feat(AI): chat suggestions and assistant settings 2026-02-01 07:24:21 +00:00
Jake Turner
12a6f2230d
feat: [wip] new maps system 2025-11-30 22:29:16 -08:00