1 Commits

Author	SHA1	Message	Date
Jake Turner	9a95f7b2b8	feat(AI): budget the prompt and actually set the context window (#1253) ... Chat was run through Ollama's OpenAI-compatible /v1 endpoint, which has no context-size field, so the num_ctx we sent was silently discarded and the entire NUM_CTX_LADDER was dead code. Every conversation ran at the backend default and was truncated from the middle (where history and retrieved context actually sit). Measured with identical ~7k-token prompts: 2,060 tokens processed against 7,049 on the native endpoint, and with the needed fact in the discarded middle the old path hallucinated a plausible wrong number. Transport: add a native /api/chat path using the already-installed ollama package, sending a real options object (num_ctx, num_predict, temperature, seed) plus keep_alive. The OpenAI path is still used for LM Studio and llama.cpp, minus the ineffective num_ctx and with include_usage so both paths return prompt token counts. /api/show is now read for trained context, parameter size, quantization and any modelfile num_ctx. The <think> stream normalizer moves to app/utils/think_stream.ts so both transports share it; it gains a flush() that fixes a dangling partial tag being dropped at end of stream. Token accounting: replace chars/N with a structural segmenter scaled by a per-model factor learned by EWMA from the prompt_eval_count every response already returned and threw away. End-to-end estimator error is 4.6% against 18-20% for a fixed divisor — qwen2.5 converges to k=1.26 where llama3 sits at 1.00, so no single global constant serves both. Context window: resolved once per model from trained context, the exact KV cost derived from GGUF metadata (head_count_kv, not head_count — GQA makes them differ 4x), and available VRAM, then memoized. Memoization is load-bearing: asking for a different num_ctx makes Ollama unload and reload the model, stalling the turn and discarding the KV cache. The floor never overrides the model's trained ceiling — tinyllama trains to 2048, and a blind 4096 floor pushes RoPE past anything the weights saw. Prompt budget: planPrompt allocates explicitly instead of delegating overflow to silent backend truncation — response reserve, then fixed blocks (system and current query, never dropped), then RAG chunks whole and best-first, then history in whole user+assistant turns, evicted newest-first in blocks with an elision marker. Block eviction keeps the prefix stable for several turns rather than shifting on every one. KV-cache ordering: the per-turn retrieved block moves behind history to the tail, so the stable prefix stays byte-identical turn to turn. At turn 14 of a long conversation, prefill is 231ms at the tail against 539ms at the front (~1ms/turn growth against ~25ms). The win is conditional — with a large retrieved block and short history the two measure the same — so both orderings ship behind RAG_PLACEMENT. The per-turn query rewrite also moves to the tasks model; on a single-slot Ollama it was evicting the chat model's cached prefix on every single turn. Also: OLLAMA_CONTEXT_LENGTH and OLLAMA_KV_CACHE_TYPE=q8_0 on the container (q8_0 is treated as headroom, never as a budgeting assumption, since unsupported architectures fall back to f16 silently), a Context Window setting in Settings > Models, and budget fields on PipelineTrace. Eval: golden suites (--suite) so the new long_context goldens cannot move the aggregates the committed baselines were recorded against, plus budget and estimator-error metrics through the report path. Retrieval is unchanged and eval:compare reports no metric outside the tolerance band; the long_context suite scores correctness 1.000 with historyElided 0.500 (the two eviction cases, by design). Unit tests 337 -> 386 passing, same 7 pre-existing failures. Ingest-side CHAR_TO_TOKEN_RATIO is deliberately untouched: changing it re-chunks the corpus and invalidates retrieval baselines, which belongs in its own measured change.	2026-09-29 15:54:41 -07:00