mirror of
https://github.com/Crosstalk-Solutions/project-nomad.git
synced 2026-10-17 13:19:18 +02:00
ae06a295de
There was one cutoff, Qdrant's score_threshold at 0.3, and nomic scores unrelated text well above it, so every question retrieved *something* (nonEmptyRateOnRefusal 1.0) and rag_context wasted four lines asking a 3B model to "silently judge" relevance. That is a classification a number should decide, not a micro model. Adds `RAG_MIN_FINAL_SCORE`, a floor on the post-rerank score, applied before source diversity. The floor judges relevance, the 0.85^n penalty judges redundancy. Nothing clears it and nothing is injected. `score_threshold` stays at 0.3 as the candidate net. Exposed as `rag.minRelevance` with a module- scoped TTL cache invalidated from `updateSetting`. The floor is calibrated and the harness was describing the wrong number: aggregate() bucketed semanticScore, so its percentiles measured raw cosine while the new knob filters the reranked score. Both axes now, and the gap matters: recall falls at 0.60 on the semantic axis, 0.66 on the final one. floor recall@5 precision@5 emptyOnAnswerable nonEmptyOnRefusal 0.00 0.991 0.221 0% 100% 0.62 0.991 0.524 0% 60% 0.66 0.991 0.695 0% 60% 0.67 0.972 0.704 1% 60% Default 0.62, not the last-lossless 0.66: the refusal win is complete at 0.62 and 28 documents is a thin basis for spending margin. 0.60 is the honest floor. Three of five refusal goldens are near-corpus by construction, where declining is the generation tier's job. Both eval tiers pin the floor to the constant and ignore the setting, for the same reason the harness never sets skipRetrieval. 4 metrics improved against baseline, no regressions.
50 lines
1.9 KiB
TypeScript
50 lines
1.9 KiB
TypeScript
import logger from '@adonisjs/core/services/logger'
|
|
import KVStore from '#models/kv_store'
|
|
import { RAG_MIN_FINAL_SCORE } from '../../constants/ollama.js'
|
|
import { parseMinRelevance } from './misc.js'
|
|
|
|
/**
|
|
* Resolves the user's retrieval relevance floor (`rag.minRelevance`), cached.
|
|
*
|
|
* Read on every chat turn, changed roughly never, so a KV round-trip per turn
|
|
* would be pure overhead on the critical path. Cached at module scope rather
|
|
* than on a service instance: NOMAD's services are `@inject()` but are not
|
|
* registered as singletons, so instance state lives and dies with a single
|
|
* request — which is why ContextWindowService's "memoized for the process
|
|
* lifetime" memo isn't, and why its invalidate() never had a caller to matter to.
|
|
*
|
|
* Belt and braces, mirroring the assistant-name cache in config/inertia.ts: the
|
|
* TTL bounds how stale this can get if an invalidation is ever missed, and
|
|
* SystemService.updateSetting calls invalidateMinRelevanceCache() so the common
|
|
* case takes effect immediately.
|
|
*/
|
|
const CACHE_TTL_MS = 60_000
|
|
|
|
let cache: { value: number; expiresAt: number } | null = null
|
|
|
|
export function invalidateMinRelevanceCache(): void {
|
|
cache = null
|
|
}
|
|
|
|
/** The relevance floor to apply to this turn's retrieval. */
|
|
export async function resolveMinFinalScore(): Promise<number> {
|
|
const now = Date.now()
|
|
if (cache && now < cache.expiresAt) return cache.value
|
|
|
|
let raw: string | null = null
|
|
try {
|
|
raw = await KVStore.getValue('rag.minRelevance')
|
|
} catch (error) {
|
|
// A KV read failure must not take retrieval out, and must not be cached —
|
|
// fall back to the default and let the next turn try again.
|
|
logger.warn(
|
|
`[RAG] Failed to read rag.minRelevance, using ${RAG_MIN_FINAL_SCORE}: ${error instanceof Error ? error.message : error}`
|
|
)
|
|
return RAG_MIN_FINAL_SCORE
|
|
}
|
|
|
|
const value = parseMinRelevance(raw, RAG_MIN_FINAL_SCORE)
|
|
cache = { value, expiresAt: now + CACHE_TTL_MS }
|
|
return value
|
|
}
|