project-nomad/admin/app/controllers/settings_controller.ts
Jake Turner 9a95f7b2b8 feat(AI): budget the prompt and actually set the context window (#1253)
Chat was run through Ollama's OpenAI-compatible /v1 endpoint, which has no
context-size field, so the num_ctx we sent was silently discarded and the
entire NUM_CTX_LADDER was dead code. Every conversation ran at the backend
default and was truncated from the middle (where history and
retrieved context actually sit). Measured with identical ~7k-token prompts:
2,060 tokens processed against 7,049 on the native endpoint, and with the
needed fact in the discarded middle the old path hallucinated a plausible
wrong number.

Transport: add a native /api/chat path using the already-installed ollama
package, sending a real options object (num_ctx, num_predict, temperature,
seed) plus keep_alive. The OpenAI path is still used for LM Studio and llama.cpp,
minus the ineffective num_ctx and with include_usage so both paths return
prompt token counts. /api/show is now read for trained context, parameter
size, quantization and any modelfile num_ctx. The <think> stream normalizer
moves to app/utils/think_stream.ts so both transports share it; it gains a
flush() that fixes a dangling partial tag being dropped at end of stream.

Token accounting: replace chars/N with a structural segmenter scaled by a
per-model factor learned by EWMA from the prompt_eval_count every response
already returned and threw away. End-to-end estimator error is 4.6% against
18-20% for a fixed divisor — qwen2.5 converges to k=1.26 where llama3 sits
at 1.00, so no single global constant serves both.

Context window: resolved once per model from trained context, the exact KV
cost derived from GGUF metadata (head_count_kv, not head_count — GQA makes
them differ 4x), and available VRAM, then memoized. Memoization is
load-bearing: asking for a different num_ctx makes Ollama unload and reload
the model, stalling the turn and discarding the KV cache. The floor never
overrides the model's trained ceiling — tinyllama trains to 2048, and a
blind 4096 floor pushes RoPE past anything the weights saw.

Prompt budget: planPrompt allocates explicitly instead of delegating
overflow to silent backend truncation — response reserve, then fixed blocks
(system and current query, never dropped), then RAG chunks whole and
best-first, then history in whole user+assistant turns, evicted newest-first
in blocks with an elision marker. Block eviction keeps the prefix stable for
several turns rather than shifting on every one.

KV-cache ordering: the per-turn retrieved block moves behind history to the
tail, so the stable prefix stays byte-identical turn to turn. At turn 14 of
a long conversation, prefill is 231ms at the tail against 539ms at the front
(~1ms/turn growth against ~25ms). The win is conditional — with a large
retrieved block and short history the two measure the same — so both
orderings ship behind RAG_PLACEMENT. The per-turn query rewrite also moves
to the tasks model; on a single-slot Ollama it was evicting the chat model's
cached prefix on every single turn.

Also: OLLAMA_CONTEXT_LENGTH and OLLAMA_KV_CACHE_TYPE=q8_0 on the container
(q8_0 is treated as headroom, never as a budgeting assumption, since
unsupported architectures fall back to f16 silently), a Context Window
setting in Settings > Models, and budget fields on PipelineTrace.

Eval: golden suites (--suite) so the new long_context goldens cannot move
the aggregates the committed baselines were recorded against, plus budget
and estimator-error metrics through the report path. Retrieval is unchanged
and eval:compare reports no metric outside the tolerance band; the
long_context suite scores correctness 1.000 with historyElided 0.500 (the
two eviction cases, by design). Unit tests 337 -> 386 passing, same 7
pre-existing failures.

Ingest-side CHAR_TO_TOKEN_RATIO is deliberately untouched: changing it
re-chunks the corpus and invalidates retrieval baselines, which belongs in
its own measured change.
2026-09-29 15:54:41 -07:00

175 lines
6.2 KiB
TypeScript

import KVStore from '#models/kv_store'
import { BenchmarkService } from '#services/benchmark_service'
import { ContextWindowService } from '#services/context_window_service'
import { MapService } from '#services/map_service'
import { OllamaService } from '#services/ollama_service'
import { SystemService } from '#services/system_service'
import { getSettingSchema, updateSettingSchema, validateSettingValue } from '#validators/settings'
import { inject } from '@adonisjs/core'
import type { HttpContext } from '@adonisjs/core/http'
import env from '#start/env'
@inject()
export default class SettingsController {
constructor(
private systemService: SystemService,
private mapService: MapService,
private benchmarkService: BenchmarkService,
private ollamaService: OllamaService,
private contextWindowService: ContextWindowService
) {}
async system({ inertia }: HttpContext) {
const systemInfo = await this.systemService.getSystemInfo()
return inertia.render('settings/system', {
system: {
info: systemInfo,
},
})
}
async apps({ inertia }: HttpContext) {
const services = await this.systemService.getServices({ installedOnly: false })
return inertia.render('settings/apps', {
system: {
services,
},
})
}
async legal({ inertia }: HttpContext) {
return inertia.render('settings/legal')
}
async support({ inertia }: HttpContext) {
return inertia.render('settings/support')
}
async maps({ inertia }: HttpContext) {
const baseAssetsCheck = await this.mapService.ensureBaseAssets()
const [regionFiles, worldBasemapExists] = await Promise.all([
this.mapService.listRegions(),
this.mapService.checkWorldBasemapExists(),
])
return inertia.render('settings/maps', {
maps: {
baseAssetsExist: baseAssetsCheck,
worldBasemapExists,
regionFiles: regionFiles.files,
},
})
}
async models({ inertia }: HttpContext) {
const availableModels = await this.ollamaService.getAvailableModels({
sort: 'pulls',
recommendedOnly: false,
query: null,
limit: 15,
})
const installedModels = await this.ollamaService.getModels().catch(() => [])
const chatSuggestionsEnabled = await KVStore.getValue('chat.suggestionsEnabled')
const aiAssistantCustomName = await KVStore.getValue('ai.assistantCustomName')
const remoteOllamaUrl = await KVStore.getValue('ai.remoteOllamaUrl')
const ollamaFlashAttention = await KVStore.getValue('ai.ollamaFlashAttention')
const autoThinking = await KVStore.getValue('ai.autoThinking')
const tasksModel = await KVStore.getValue('ai.tasksModel')
const ragEnabled = await KVStore.getValue('rag.enabled')
const contextWindow = await KVStore.getValue('ai.contextWindow')
// Resolved window per installed model, so the setting shows what "Auto"
// actually produced rather than leaving the user to guess. Best-effort:
// a model whose metadata can't be read simply doesn't get a badge.
const resolvedContextWindows: Record<string, number> = {}
await Promise.all(
(installedModels || []).map(async (model) => {
try {
resolvedContextWindows[model.name] = await this.contextWindowService.windowFor(model.name)
} catch {
/* leave unset */
}
})
)
return inertia.render('settings/models', {
models: {
availableModels: availableModels?.models || [],
installedModels: installedModels || [],
settings: {
chatSuggestionsEnabled: chatSuggestionsEnabled ?? false,
aiAssistantCustomName: aiAssistantCustomName ?? '',
remoteOllamaUrl: remoteOllamaUrl ?? '',
ollamaFlashAttention: ollamaFlashAttention ?? true,
autoThinking: autoThinking ?? false,
tasksModel: tasksModel ?? '',
ragEnabled: ragEnabled ?? true,
contextWindow: contextWindow ?? 'auto',
},
resolvedContextWindows,
},
})
}
async update({ inertia }: HttpContext) {
const updateInfo = await this.systemService.checkLatestVersion()
return inertia.render('settings/update', {
system: {
updateAvailable: updateInfo.updateAvailable,
latestVersion: updateInfo.latestVersion,
currentVersion: updateInfo.currentVersion,
},
})
}
async zim({ inertia }: HttpContext) {
return inertia.render('settings/zim/index')
}
async zimRemote({ inertia }: HttpContext) {
return inertia.render('settings/zim/remote-explorer')
}
async creatorPacks({ inertia }: HttpContext) {
return inertia.render('settings/creator-packs')
}
async benchmark({ inertia }: HttpContext) {
const latestResult = await this.benchmarkService.getLatestResult()
const status = this.benchmarkService.getStatus()
return inertia.render('settings/benchmark', {
benchmark: {
latestResult,
status: status.status,
currentBenchmarkId: status.benchmarkId,
},
})
}
async advanced({ inertia }: HttpContext) {
// When the env var is set it always takes precedence over the stored value,
// so surface that to the UI to disable the field and explain the override.
const envOverride = Boolean(env.get('INTERNET_STATUS_TEST_URL')?.trim())
const internetStatusTestUrl = await KVStore.getValue('system.internetStatusTestUrl')
return inertia.render('settings/advanced', {
advanced: {
internetStatusTestUrl: internetStatusTestUrl ?? '',
internetStatusTestUrlEnvOverride: envOverride,
},
})
}
async getSetting({ request, response }: HttpContext) {
const { key } = await getSettingSchema.validate({ key: request.qs().key });
const value = await KVStore.getValue(key);
return response.status(200).send({ key, value });
}
async updateSetting({ request, response }: HttpContext) {
const reqData = await request.validateUsing(updateSettingSchema)
const valueError = validateSettingValue(reqData.key, reqData.value)
if (valueError) {
return response.status(422).send({ success: false, message: valueError })
}
await this.systemService.updateSetting(reqData.key, reqData.value)
return response.status(200).send({ success: true, message: 'Setting updated successfully' })
}
}