mirror of
https://github.com/Crosstalk-Solutions/project-nomad.git
synced 2026-10-17 05:14:19 +02:00
9a95f7b2b8
Chat was run through Ollama's OpenAI-compatible /v1 endpoint, which has no context-size field, so the num_ctx we sent was silently discarded and the entire NUM_CTX_LADDER was dead code. Every conversation ran at the backend default and was truncated from the middle (where history and retrieved context actually sit). Measured with identical ~7k-token prompts: 2,060 tokens processed against 7,049 on the native endpoint, and with the needed fact in the discarded middle the old path hallucinated a plausible wrong number. Transport: add a native /api/chat path using the already-installed ollama package, sending a real options object (num_ctx, num_predict, temperature, seed) plus keep_alive. The OpenAI path is still used for LM Studio and llama.cpp, minus the ineffective num_ctx and with include_usage so both paths return prompt token counts. /api/show is now read for trained context, parameter size, quantization and any modelfile num_ctx. The <think> stream normalizer moves to app/utils/think_stream.ts so both transports share it; it gains a flush() that fixes a dangling partial tag being dropped at end of stream. Token accounting: replace chars/N with a structural segmenter scaled by a per-model factor learned by EWMA from the prompt_eval_count every response already returned and threw away. End-to-end estimator error is 4.6% against 18-20% for a fixed divisor — qwen2.5 converges to k=1.26 where llama3 sits at 1.00, so no single global constant serves both. Context window: resolved once per model from trained context, the exact KV cost derived from GGUF metadata (head_count_kv, not head_count — GQA makes them differ 4x), and available VRAM, then memoized. Memoization is load-bearing: asking for a different num_ctx makes Ollama unload and reload the model, stalling the turn and discarding the KV cache. The floor never overrides the model's trained ceiling — tinyllama trains to 2048, and a blind 4096 floor pushes RoPE past anything the weights saw. Prompt budget: planPrompt allocates explicitly instead of delegating overflow to silent backend truncation — response reserve, then fixed blocks (system and current query, never dropped), then RAG chunks whole and best-first, then history in whole user+assistant turns, evicted newest-first in blocks with an elision marker. Block eviction keeps the prefix stable for several turns rather than shifting on every one. KV-cache ordering: the per-turn retrieved block moves behind history to the tail, so the stable prefix stays byte-identical turn to turn. At turn 14 of a long conversation, prefill is 231ms at the tail against 539ms at the front (~1ms/turn growth against ~25ms). The win is conditional — with a large retrieved block and short history the two measure the same — so both orderings ship behind RAG_PLACEMENT. The per-turn query rewrite also moves to the tasks model; on a single-slot Ollama it was evicting the chat model's cached prefix on every single turn. Also: OLLAMA_CONTEXT_LENGTH and OLLAMA_KV_CACHE_TYPE=q8_0 on the container (q8_0 is treated as headroom, never as a budgeting assumption, since unsupported architectures fall back to f16 silently), a Context Window setting in Settings > Models, and budget fields on PipelineTrace. Eval: golden suites (--suite) so the new long_context goldens cannot move the aggregates the committed baselines were recorded against, plus budget and estimator-error metrics through the report path. Retrieval is unchanged and eval:compare reports no metric outside the tolerance band; the long_context suite scores correctness 1.000 with historyElided 0.500 (the two eviction cases, by design). Unit tests 337 -> 386 passing, same 7 pre-existing failures. Ingest-side CHAR_TO_TOKEN_RATIO is deliberately untouched: changing it re-chunks the corpus and invalidates retrieval baselines, which belongs in its own measured change.
104 lines
3.9 KiB
TypeScript
104 lines
3.9 KiB
TypeScript
/**
|
|
* Characterization tests for the prompt-assembly helpers extracted out of
|
|
* OllamaController into RagPipelineService.
|
|
*
|
|
* These lock in the *current* behaviour so any later deliberate change shows up
|
|
* as a failing test rather than a silent shift in answer quality.
|
|
*
|
|
* Token budgeting moved out of this module: the estimate now lives in
|
|
* `token_estimate.spec.ts` and the allocation in `context_budget.spec.ts`. What
|
|
* remains here is model-size tiering and context-block rendering.
|
|
*
|
|
* Pure functions only — no MySQL, Redis, Qdrant, or Ollama needed:
|
|
* npm run test:unit
|
|
*/
|
|
import * as assert from 'node:assert/strict'
|
|
import { test } from 'node:test'
|
|
|
|
import {
|
|
buildContextBlock,
|
|
getContextLimitsForModel,
|
|
type ContextLimitTier,
|
|
} from '../../app/utils/rag_prompt.js'
|
|
|
|
/** Mirrors RAG_CONTEXT_LIMITS in constants/ollama.ts. Passed in explicitly so
|
|
* these tests need no AdonisJS-flavoured imports, per the kb_ratio_lookup
|
|
* precedent — and so a change to the shipped tiers shows up here as a
|
|
* deliberate edit rather than a silently-passing test. */
|
|
const TIERS: ContextLimitTier[] = [
|
|
{ maxParams: 3, maxResults: 2, maxTokens: 1000 },
|
|
{ maxParams: 8, maxResults: 4, maxTokens: 2500 },
|
|
{ maxParams: Infinity, maxResults: 5, maxTokens: 0 },
|
|
]
|
|
|
|
const limits = (model: string) => getContextLimitsForModel(model, TIERS)
|
|
|
|
const chunk = (text: string, metadata: Record<string, any> = {}) => ({
|
|
text,
|
|
score: 0.5,
|
|
metadata,
|
|
})
|
|
|
|
// --- getContextLimitsForModel ------------------------------------------------
|
|
|
|
test('context limits: 1-3B models get the tightest budget', () => {
|
|
assert.deepEqual(limits('llama3.2:1b'), { maxResults: 2, maxTokens: 1000 })
|
|
assert.deepEqual(limits('qwen2.5:3b'), { maxResults: 2, maxTokens: 1000 })
|
|
})
|
|
|
|
test('context limits: fractional sizes parse correctly', () => {
|
|
// "1.5b" must read as 1.5, not 1 or 15 — it decides which tier the model lands in.
|
|
assert.deepEqual(limits('qwen2.5:1.5b'), { maxResults: 2, maxTokens: 1000 })
|
|
})
|
|
|
|
test('context limits: 4-8B tier', () => {
|
|
assert.deepEqual(limits('llama3.1:8b'), { maxResults: 4, maxTokens: 2500 })
|
|
})
|
|
|
|
test('context limits: 13B+ is uncapped', () => {
|
|
assert.deepEqual(limits('llama2:70b'), { maxResults: 5, maxTokens: 0 })
|
|
})
|
|
|
|
test('context limits: unparseable model name is assumed to be 8B', () => {
|
|
// Documented quirk, not an endorsement: "phi3" has no size token, so it is
|
|
// handed the 4-8B budget regardless of what it actually is.
|
|
assert.deepEqual(limits('phi3'), { maxResults: 4, maxTokens: 2500 })
|
|
})
|
|
|
|
test('context limits: quantization suffixes do not confuse the size parse', () => {
|
|
assert.deepEqual(limits('llama3.1:8b-text-q4_1'), {
|
|
maxResults: 4,
|
|
maxTokens: 2500,
|
|
})
|
|
})
|
|
|
|
// --- buildContextBlock -------------------------------------------------------
|
|
|
|
test('context block: numbers each chunk from 1', () => {
|
|
const out = buildContextBlock([chunk('first'), chunk('second')])
|
|
assert.equal(out, '[Context 1]\nfirst\n\n[Context 2]\nsecond')
|
|
})
|
|
|
|
test('context block: labels with full_title when present', () => {
|
|
const out = buildContextBlock([chunk('body', { full_title: 'Water - Boiling' })])
|
|
assert.equal(out, '[Context 1 — Water - Boiling]\nbody')
|
|
})
|
|
|
|
test('context block: falls back to article_title', () => {
|
|
const out = buildContextBlock([chunk('body', { article_title: 'Water' })])
|
|
assert.equal(out, '[Context 1 — Water]\nbody')
|
|
})
|
|
|
|
test('context block: full_title wins over article_title', () => {
|
|
const out = buildContextBlock([chunk('body', { full_title: 'A - B', article_title: 'A' })])
|
|
assert.equal(out, '[Context 1 — A - B]\nbody')
|
|
})
|
|
|
|
test('context block: never leaks the relevance score to the model', () => {
|
|
// Deliberate: nomic cosine scores for genuinely relevant passages sit around
|
|
// 0.4-0.6, and showing the model "42%" primes it to distrust correct context.
|
|
const out = buildContextBlock([chunk('body', { source: 'f.md', semantic_score: 0.42 })])
|
|
assert.ok(!out.includes('0.42'))
|
|
assert.ok(!out.includes('42'))
|
|
})
|