mirror of
https://github.com/Crosstalk-Solutions/project-nomad.git
synced 2026-10-17 13:19:18 +02:00
27be71b46e
Chat sent num_ctx, num_predict and nothing else, so every turn decoded at whatever the backend picked: temperature 0.8, top_p 0.9, no min_p. On a heavily quantized 1-3B model that is the wandering, repetitive output people read as "the local model is bad". Adds `ai.responseStyle` with four profiles. min_p is the lever, not temperature: it floors relative to the top token, so it truncates hard on a confident distribution and barely at all on a flat one, which is where small models start to wander. top_p stays at 1.0 rather than stacked on top of it. repeat_penalty moves with the style, because a grounded answer legitimately repeats a part number inside the 64-token window: `focused` relaxes it to 1.05, `creative` tightens it to 1.15. `auto` is the only style that defers to the model. Plenty of library models ship author-tuned samplers (qwen3 sets 0.6/0.95/20), so those win key by key and NOMAD fills only the gap, which in practice is min_p. A style the user picked by name overrides them, or it would silently do nothing on half their models. The compat transport drops min_p, so a profile carries a separate compatTopP. Passing top_p 1.0 to a backend with no min_p behind it would sample the raw distribution, which is worse than the 0.9 it replaced. Resolved in the controller, so the grammar-constrained task calls keep their temperature 0. An explicit temperature still outranks a profile's, which is what keeps the eval harness deterministic. `eval:generation --style` is new and load-bearing: temperature 0 makes min_p, top_p and top_k inert, so a default run cannot see this change at all. `--style=off` sends no samplers and no temperature, and is the control. Verified against Ollama 0.32.9, min_p accepted. 22 unit tests.
48 lines
1.8 KiB
TypeScript
48 lines
1.8 KiB
TypeScript
import logger from '@adonisjs/core/services/logger'
|
|
import KVStore from '#models/kv_store'
|
|
import { DEFAULT_RESPONSE_STYLE } from '../../constants/ollama.js'
|
|
import type { ResponseStyle } from '../../types/ollama.js'
|
|
import { parseResponseStyle } from './sampler.js'
|
|
|
|
/**
|
|
* Resolves the user's Response Style (`ai.responseStyle`), cached.
|
|
*
|
|
* Same shape and same reasoning as the relevance-floor cache next door: read on
|
|
* every chat turn, changed roughly never, and NOMAD's services are `@inject()`
|
|
* but not registered as singletons, so caching on a service instance would give
|
|
* a cache that lives and dies inside one request.
|
|
*
|
|
* The TTL bounds how stale this gets if an invalidation is ever missed;
|
|
* SystemService.updateSetting calls invalidateResponseStyleCache() so the common
|
|
* case takes effect on the next turn rather than a minute later.
|
|
*/
|
|
const CACHE_TTL_MS = 60_000
|
|
|
|
let cache: { value: ResponseStyle; expiresAt: number } | null = null
|
|
|
|
export function invalidateResponseStyleCache(): void {
|
|
cache = null
|
|
}
|
|
|
|
/** The response style to sample this turn under. */
|
|
export async function resolveResponseStyle(): Promise<ResponseStyle> {
|
|
const now = Date.now()
|
|
if (cache && now < cache.expiresAt) return cache.value
|
|
|
|
let raw: string | null = null
|
|
try {
|
|
raw = await KVStore.getValue('ai.responseStyle')
|
|
} catch (error) {
|
|
// A KV read failure must not take chat out, and must not be cached — fall
|
|
// back to the default and let the next turn try again.
|
|
logger.warn(
|
|
`[Ollama] Failed to read ai.responseStyle, using ${DEFAULT_RESPONSE_STYLE}: ${error instanceof Error ? error.message : error}`
|
|
)
|
|
return DEFAULT_RESPONSE_STYLE
|
|
}
|
|
|
|
const value = parseResponseStyle(raw)
|
|
cache = { value, expiresAt: now + CACHE_TTL_MS }
|
|
return value
|
|
}
|