mirror of
https://github.com/Crosstalk-Solutions/project-nomad.git
synced 2026-10-17 05:14:19 +02:00
Chat was run through Ollama's OpenAI-compatible /v1 endpoint, which has no context-size field, so the num_ctx we sent was silently discarded and the entire NUM_CTX_LADDER was dead code. Every conversation ran at the backend default and was truncated from the middle (where history and retrieved context actually sit). Measured with identical ~7k-token prompts: 2,060 tokens processed against 7,049 on the native endpoint, and with the needed fact in the discarded middle the old path hallucinated a plausible wrong number. Transport: add a native /api/chat path using the already-installed ollama package, sending a real options object (num_ctx, num_predict, temperature, seed) plus keep_alive. The OpenAI path is still used for LM Studio and llama.cpp, minus the ineffective num_ctx and with include_usage so both paths return prompt token counts. /api/show is now read for trained context, parameter size, quantization and any modelfile num_ctx. The <think> stream normalizer moves to app/utils/think_stream.ts so both transports share it; it gains a flush() that fixes a dangling partial tag being dropped at end of stream. Token accounting: replace chars/N with a structural segmenter scaled by a per-model factor learned by EWMA from the prompt_eval_count every response already returned and threw away. End-to-end estimator error is 4.6% against 18-20% for a fixed divisor — qwen2.5 converges to k=1.26 where llama3 sits at 1.00, so no single global constant serves both. Context window: resolved once per model from trained context, the exact KV cost derived from GGUF metadata (head_count_kv, not head_count — GQA makes them differ 4x), and available VRAM, then memoized. Memoization is load-bearing: asking for a different num_ctx makes Ollama unload and reload the model, stalling the turn and discarding the KV cache. The floor never overrides the model's trained ceiling — tinyllama trains to 2048, and a blind 4096 floor pushes RoPE past anything the weights saw. Prompt budget: planPrompt allocates explicitly instead of delegating overflow to silent backend truncation — response reserve, then fixed blocks (system and current query, never dropped), then RAG chunks whole and best-first, then history in whole user+assistant turns, evicted newest-first in blocks with an elision marker. Block eviction keeps the prefix stable for several turns rather than shifting on every one. KV-cache ordering: the per-turn retrieved block moves behind history to the tail, so the stable prefix stays byte-identical turn to turn. At turn 14 of a long conversation, prefill is 231ms at the tail against 539ms at the front (~1ms/turn growth against ~25ms). The win is conditional — with a large retrieved block and short history the two measure the same — so both orderings ship behind RAG_PLACEMENT. The per-turn query rewrite also moves to the tasks model; on a single-slot Ollama it was evicting the chat model's cached prefix on every single turn. Also: OLLAMA_CONTEXT_LENGTH and OLLAMA_KV_CACHE_TYPE=q8_0 on the container (q8_0 is treated as headroom, never as a budgeting assumption, since unsupported architectures fall back to f16 silently), a Context Window setting in Settings > Models, and budget fields on PipelineTrace. Eval: golden suites (--suite) so the new long_context goldens cannot move the aggregates the committed baselines were recorded against, plus budget and estimator-error metrics through the report path. Retrieval is unchanged and eval:compare reports no metric outside the tolerance band; the long_context suite scores correctness 1.000 with historyElided 0.500 (the two eviction cases, by design). Unit tests 337 -> 386 passing, same 7 pre-existing failures. Ingest-side CHAR_TO_TOKEN_RATIO is deliberately untouched: changing it re-chunks the corpus and invalidates retrieval baselines, which belongs in its own measured change.
175 lines
6.2 KiB
TypeScript
175 lines
6.2 KiB
TypeScript
import KVStore from '#models/kv_store'
|
|
import { BenchmarkService } from '#services/benchmark_service'
|
|
import { ContextWindowService } from '#services/context_window_service'
|
|
import { MapService } from '#services/map_service'
|
|
import { OllamaService } from '#services/ollama_service'
|
|
import { SystemService } from '#services/system_service'
|
|
import { getSettingSchema, updateSettingSchema, validateSettingValue } from '#validators/settings'
|
|
import { inject } from '@adonisjs/core'
|
|
import type { HttpContext } from '@adonisjs/core/http'
|
|
import env from '#start/env'
|
|
|
|
@inject()
|
|
export default class SettingsController {
|
|
constructor(
|
|
private systemService: SystemService,
|
|
private mapService: MapService,
|
|
private benchmarkService: BenchmarkService,
|
|
private ollamaService: OllamaService,
|
|
private contextWindowService: ContextWindowService
|
|
) {}
|
|
|
|
async system({ inertia }: HttpContext) {
|
|
const systemInfo = await this.systemService.getSystemInfo()
|
|
return inertia.render('settings/system', {
|
|
system: {
|
|
info: systemInfo,
|
|
},
|
|
})
|
|
}
|
|
|
|
async apps({ inertia }: HttpContext) {
|
|
const services = await this.systemService.getServices({ installedOnly: false })
|
|
return inertia.render('settings/apps', {
|
|
system: {
|
|
services,
|
|
},
|
|
})
|
|
}
|
|
|
|
async legal({ inertia }: HttpContext) {
|
|
return inertia.render('settings/legal')
|
|
}
|
|
|
|
async support({ inertia }: HttpContext) {
|
|
return inertia.render('settings/support')
|
|
}
|
|
|
|
async maps({ inertia }: HttpContext) {
|
|
const baseAssetsCheck = await this.mapService.ensureBaseAssets()
|
|
const [regionFiles, worldBasemapExists] = await Promise.all([
|
|
this.mapService.listRegions(),
|
|
this.mapService.checkWorldBasemapExists(),
|
|
])
|
|
return inertia.render('settings/maps', {
|
|
maps: {
|
|
baseAssetsExist: baseAssetsCheck,
|
|
worldBasemapExists,
|
|
regionFiles: regionFiles.files,
|
|
},
|
|
})
|
|
}
|
|
|
|
async models({ inertia }: HttpContext) {
|
|
const availableModels = await this.ollamaService.getAvailableModels({
|
|
sort: 'pulls',
|
|
recommendedOnly: false,
|
|
query: null,
|
|
limit: 15,
|
|
})
|
|
const installedModels = await this.ollamaService.getModels().catch(() => [])
|
|
const chatSuggestionsEnabled = await KVStore.getValue('chat.suggestionsEnabled')
|
|
const aiAssistantCustomName = await KVStore.getValue('ai.assistantCustomName')
|
|
const remoteOllamaUrl = await KVStore.getValue('ai.remoteOllamaUrl')
|
|
const ollamaFlashAttention = await KVStore.getValue('ai.ollamaFlashAttention')
|
|
const autoThinking = await KVStore.getValue('ai.autoThinking')
|
|
const tasksModel = await KVStore.getValue('ai.tasksModel')
|
|
const ragEnabled = await KVStore.getValue('rag.enabled')
|
|
const contextWindow = await KVStore.getValue('ai.contextWindow')
|
|
// Resolved window per installed model, so the setting shows what "Auto"
|
|
// actually produced rather than leaving the user to guess. Best-effort:
|
|
// a model whose metadata can't be read simply doesn't get a badge.
|
|
const resolvedContextWindows: Record<string, number> = {}
|
|
await Promise.all(
|
|
(installedModels || []).map(async (model) => {
|
|
try {
|
|
resolvedContextWindows[model.name] = await this.contextWindowService.windowFor(model.name)
|
|
} catch {
|
|
/* leave unset */
|
|
}
|
|
})
|
|
)
|
|
return inertia.render('settings/models', {
|
|
models: {
|
|
availableModels: availableModels?.models || [],
|
|
installedModels: installedModels || [],
|
|
settings: {
|
|
chatSuggestionsEnabled: chatSuggestionsEnabled ?? false,
|
|
aiAssistantCustomName: aiAssistantCustomName ?? '',
|
|
remoteOllamaUrl: remoteOllamaUrl ?? '',
|
|
ollamaFlashAttention: ollamaFlashAttention ?? true,
|
|
autoThinking: autoThinking ?? false,
|
|
tasksModel: tasksModel ?? '',
|
|
ragEnabled: ragEnabled ?? true,
|
|
contextWindow: contextWindow ?? 'auto',
|
|
},
|
|
resolvedContextWindows,
|
|
},
|
|
})
|
|
}
|
|
|
|
async update({ inertia }: HttpContext) {
|
|
const updateInfo = await this.systemService.checkLatestVersion()
|
|
return inertia.render('settings/update', {
|
|
system: {
|
|
updateAvailable: updateInfo.updateAvailable,
|
|
latestVersion: updateInfo.latestVersion,
|
|
currentVersion: updateInfo.currentVersion,
|
|
},
|
|
})
|
|
}
|
|
|
|
async zim({ inertia }: HttpContext) {
|
|
return inertia.render('settings/zim/index')
|
|
}
|
|
|
|
async zimRemote({ inertia }: HttpContext) {
|
|
return inertia.render('settings/zim/remote-explorer')
|
|
}
|
|
|
|
async creatorPacks({ inertia }: HttpContext) {
|
|
return inertia.render('settings/creator-packs')
|
|
}
|
|
|
|
async benchmark({ inertia }: HttpContext) {
|
|
const latestResult = await this.benchmarkService.getLatestResult()
|
|
const status = this.benchmarkService.getStatus()
|
|
return inertia.render('settings/benchmark', {
|
|
benchmark: {
|
|
latestResult,
|
|
status: status.status,
|
|
currentBenchmarkId: status.benchmarkId,
|
|
},
|
|
})
|
|
}
|
|
|
|
async advanced({ inertia }: HttpContext) {
|
|
// When the env var is set it always takes precedence over the stored value,
|
|
// so surface that to the UI to disable the field and explain the override.
|
|
const envOverride = Boolean(env.get('INTERNET_STATUS_TEST_URL')?.trim())
|
|
const internetStatusTestUrl = await KVStore.getValue('system.internetStatusTestUrl')
|
|
return inertia.render('settings/advanced', {
|
|
advanced: {
|
|
internetStatusTestUrl: internetStatusTestUrl ?? '',
|
|
internetStatusTestUrlEnvOverride: envOverride,
|
|
},
|
|
})
|
|
}
|
|
|
|
async getSetting({ request, response }: HttpContext) {
|
|
const { key } = await getSettingSchema.validate({ key: request.qs().key });
|
|
const value = await KVStore.getValue(key);
|
|
return response.status(200).send({ key, value });
|
|
}
|
|
|
|
async updateSetting({ request, response }: HttpContext) {
|
|
const reqData = await request.validateUsing(updateSettingSchema)
|
|
const valueError = validateSettingValue(reqData.key, reqData.value)
|
|
if (valueError) {
|
|
return response.status(422).send({ success: false, message: valueError })
|
|
}
|
|
await this.systemService.updateSetting(reqData.key, reqData.value)
|
|
return response.status(200).send({ success: true, message: 'Setting updated successfully' })
|
|
}
|
|
}
|