2 Commits

Author	SHA1	Message	Date
Jake Turner	5c97bf605a	fix(AI): stop <think> output leaking into titles, chips, and the RAG query (#1254) ... ThinkTagSplitter was wired into both streaming paths but neither non-streaming one, and every ancillary call goes through non-streaming chat(). So a reasoning tasks model put its reasoning transcript into the sidebar title, into the suggestion chips, and — worst — into the string that gets embedded and sent to Qdrant, silently corrupting retrieval for the rest of the conversation. splitThinkTags() already existed and was already unit tested; it just had no production callers. Fixed in three layers: 1. Strip at the transport. New normalizeNonStreamed() in think_stream.ts merges structured reasoning (native `thinking`, /v1 `reasoning`) with anything the splitter pulls out of inline tags, and both _chatNative and _chatCompat now return through it. That covers every caller at once, including the legacy controller path and the eval harness. 2. Suppress at the source. _chatNative and _chatStreamNative forwarded `think` only when truthy, so `think: false` was never sent and Ollama left reasoning on for capable models. Guard is now definedness. This also fixes the ai.autoThinking=off setting, which had no effect on the native streaming transport. 3. Guard the call sites. Title, suggestions, and query rewrite now pass think:false plus thinkingCapable (from the memoized capability check, so no extra round-trip), and handle an empty-after-strip result. The rewrite matters most: QUERY_REWRITE_MAX_TOKENS caps it at 120, which a reasoning model burns entirely on thinking, so stripping alone would have turned a corrupt query into an empty one. It now falls back to the raw user message instead of embedding "". Verified: think_stream.spec.ts 17/17 (6 new cases, including the truncated-mid-thought case that drives the fallback), typecheck clean, full test:unit failures byte-identical to the pre-change baseline.	2026-09-29 15:54:41 -07:00
Jake Turner	9a95f7b2b8	feat(AI): budget the prompt and actually set the context window (#1253) ... Chat was run through Ollama's OpenAI-compatible /v1 endpoint, which has no context-size field, so the num_ctx we sent was silently discarded and the entire NUM_CTX_LADDER was dead code. Every conversation ran at the backend default and was truncated from the middle (where history and retrieved context actually sit). Measured with identical ~7k-token prompts: 2,060 tokens processed against 7,049 on the native endpoint, and with the needed fact in the discarded middle the old path hallucinated a plausible wrong number. Transport: add a native /api/chat path using the already-installed ollama package, sending a real options object (num_ctx, num_predict, temperature, seed) plus keep_alive. The OpenAI path is still used for LM Studio and llama.cpp, minus the ineffective num_ctx and with include_usage so both paths return prompt token counts. /api/show is now read for trained context, parameter size, quantization and any modelfile num_ctx. The <think> stream normalizer moves to app/utils/think_stream.ts so both transports share it; it gains a flush() that fixes a dangling partial tag being dropped at end of stream. Token accounting: replace chars/N with a structural segmenter scaled by a per-model factor learned by EWMA from the prompt_eval_count every response already returned and threw away. End-to-end estimator error is 4.6% against 18-20% for a fixed divisor — qwen2.5 converges to k=1.26 where llama3 sits at 1.00, so no single global constant serves both. Context window: resolved once per model from trained context, the exact KV cost derived from GGUF metadata (head_count_kv, not head_count — GQA makes them differ 4x), and available VRAM, then memoized. Memoization is load-bearing: asking for a different num_ctx makes Ollama unload and reload the model, stalling the turn and discarding the KV cache. The floor never overrides the model's trained ceiling — tinyllama trains to 2048, and a blind 4096 floor pushes RoPE past anything the weights saw. Prompt budget: planPrompt allocates explicitly instead of delegating overflow to silent backend truncation — response reserve, then fixed blocks (system and current query, never dropped), then RAG chunks whole and best-first, then history in whole user+assistant turns, evicted newest-first in blocks with an elision marker. Block eviction keeps the prefix stable for several turns rather than shifting on every one. KV-cache ordering: the per-turn retrieved block moves behind history to the tail, so the stable prefix stays byte-identical turn to turn. At turn 14 of a long conversation, prefill is 231ms at the tail against 539ms at the front (~1ms/turn growth against ~25ms). The win is conditional — with a large retrieved block and short history the two measure the same — so both orderings ship behind RAG_PLACEMENT. The per-turn query rewrite also moves to the tasks model; on a single-slot Ollama it was evicting the chat model's cached prefix on every single turn. Also: OLLAMA_CONTEXT_LENGTH and OLLAMA_KV_CACHE_TYPE=q8_0 on the container (q8_0 is treated as headroom, never as a budgeting assumption, since unsupported architectures fall back to f16 silently), a Context Window setting in Settings > Models, and budget fields on PipelineTrace. Eval: golden suites (--suite) so the new long_context goldens cannot move the aggregates the committed baselines were recorded against, plus budget and estimator-error metrics through the report path. Retrieval is unchanged and eval:compare reports no metric outside the tolerance band; the long_context suite scores correctness 1.000 with historyElided 0.500 (the two eviction cases, by design). Unit tests 337 -> 386 passing, same 7 pre-existing failures. Ingest-side CHAR_TO_TOKEN_RATIO is deliberately untouched: changing it re-chunks the corpus and invalidates retrieval baselines, which belongs in its own measured change.	2026-09-29 15:54:41 -07:00