mirror of
https://github.com/Crosstalk-Solutions/project-nomad.git
synced 2026-10-17 13:19:18 +02:00
9a95f7b2b8
Chat was run through Ollama's OpenAI-compatible /v1 endpoint, which has no context-size field, so the num_ctx we sent was silently discarded and the entire NUM_CTX_LADDER was dead code. Every conversation ran at the backend default and was truncated from the middle (where history and retrieved context actually sit). Measured with identical ~7k-token prompts: 2,060 tokens processed against 7,049 on the native endpoint, and with the needed fact in the discarded middle the old path hallucinated a plausible wrong number. Transport: add a native /api/chat path using the already-installed ollama package, sending a real options object (num_ctx, num_predict, temperature, seed) plus keep_alive. The OpenAI path is still used for LM Studio and llama.cpp, minus the ineffective num_ctx and with include_usage so both paths return prompt token counts. /api/show is now read for trained context, parameter size, quantization and any modelfile num_ctx. The <think> stream normalizer moves to app/utils/think_stream.ts so both transports share it; it gains a flush() that fixes a dangling partial tag being dropped at end of stream. Token accounting: replace chars/N with a structural segmenter scaled by a per-model factor learned by EWMA from the prompt_eval_count every response already returned and threw away. End-to-end estimator error is 4.6% against 18-20% for a fixed divisor — qwen2.5 converges to k=1.26 where llama3 sits at 1.00, so no single global constant serves both. Context window: resolved once per model from trained context, the exact KV cost derived from GGUF metadata (head_count_kv, not head_count — GQA makes them differ 4x), and available VRAM, then memoized. Memoization is load-bearing: asking for a different num_ctx makes Ollama unload and reload the model, stalling the turn and discarding the KV cache. The floor never overrides the model's trained ceiling — tinyllama trains to 2048, and a blind 4096 floor pushes RoPE past anything the weights saw. Prompt budget: planPrompt allocates explicitly instead of delegating overflow to silent backend truncation — response reserve, then fixed blocks (system and current query, never dropped), then RAG chunks whole and best-first, then history in whole user+assistant turns, evicted newest-first in blocks with an elision marker. Block eviction keeps the prefix stable for several turns rather than shifting on every one. KV-cache ordering: the per-turn retrieved block moves behind history to the tail, so the stable prefix stays byte-identical turn to turn. At turn 14 of a long conversation, prefill is 231ms at the tail against 539ms at the front (~1ms/turn growth against ~25ms). The win is conditional — with a large retrieved block and short history the two measure the same — so both orderings ship behind RAG_PLACEMENT. The per-turn query rewrite also moves to the tasks model; on a single-slot Ollama it was evicting the chat model's cached prefix on every single turn. Also: OLLAMA_CONTEXT_LENGTH and OLLAMA_KV_CACHE_TYPE=q8_0 on the container (q8_0 is treated as headroom, never as a budgeting assumption, since unsupported architectures fall back to f16 silently), a Context Window setting in Settings > Models, and budget fields on PipelineTrace. Eval: golden suites (--suite) so the new long_context goldens cannot move the aggregates the committed baselines were recorded against, plus budget and estimator-error metrics through the report path. Retrieval is unchanged and eval:compare reports no metric outside the tolerance band; the long_context suite scores correctness 1.000 with historyElided 0.500 (the two eviction cases, by design). Unit tests 337 -> 386 passing, same 7 pre-existing failures. Ingest-side CHAR_TO_TOKEN_RATIO is deliberately untouched: changing it re-chunks the corpus and invalidates retrieval baselines, which belongs in its own measured change.
65 lines
2.2 KiB
TypeScript
65 lines
2.2 KiB
TypeScript
import logger from '@adonisjs/core/services/logger'
|
|
import KVStore from '#models/kv_store'
|
|
import type { OllamaService } from '#services/ollama_service'
|
|
import { pickTasksModel } from './misc.js'
|
|
|
|
/**
|
|
* Resolve the model to use for ancillary work — chat titles, suggestions, and
|
|
* the per-turn retrieval query rewrite.
|
|
*
|
|
* Lifted out of ChatService so the RAG pipeline can use it too. The rewrite is
|
|
* the ancillary call that matters most and was the one still running on the main
|
|
* chat model: it happens on *every* turn, on the critical path, and it sends a
|
|
* completely different prompt. With Ollama's default OLLAMA_NUM_PARALLEL=1 there
|
|
* is a single slot, so that call evicted the chat model's cached prefix every
|
|
* turn and guaranteed a full re-prefill of the conversation. Routing it to a
|
|
* small dedicated model keeps the chat model's KV cache intact.
|
|
*
|
|
* Falls back to `fallback` when the setting is unset (preserving the previous
|
|
* behaviour) or when the configured model is no longer installed. `installed` is
|
|
* passed by callers that already listed models, to avoid a second round-trip.
|
|
*/
|
|
export async function resolveTasksModel(
|
|
ollamaService: OllamaService,
|
|
fallback: string | null,
|
|
installed?: { name: string }[],
|
|
logPrefix = '[TasksModel]'
|
|
): Promise<string | null> {
|
|
let configured: string | null = null
|
|
try {
|
|
configured = await KVStore.getValue('ai.tasksModel')
|
|
} catch (error) {
|
|
logger.error(
|
|
`${logPrefix} Failed to read ai.tasksModel: ${error instanceof Error ? error.message : error}`
|
|
)
|
|
return fallback
|
|
}
|
|
if (!configured?.trim()) {
|
|
return fallback
|
|
}
|
|
|
|
let models = installed
|
|
if (!models) {
|
|
try {
|
|
models = await ollamaService.getModels()
|
|
} catch (error) {
|
|
logger.error(
|
|
`${logPrefix} Failed to list models while resolving the tasks model: ${error instanceof Error ? error.message : error}`
|
|
)
|
|
return fallback
|
|
}
|
|
}
|
|
|
|
const { model, staleConfigured } = pickTasksModel(
|
|
configured,
|
|
(models ?? []).map((m) => m.name),
|
|
fallback
|
|
)
|
|
if (staleConfigured) {
|
|
logger.warn(
|
|
`${logPrefix} Configured tasks model "${staleConfigured}" is not installed; falling back to "${fallback ?? 'none'}"`
|
|
)
|
|
}
|
|
return model
|
|
}
|