The dry-run that sizes a country selection had a 60 s limit. Its runtime
scales with land area, so large selections went over it on ordinary
hardware: Canada alone takes 41 s on a fast box, US+CA+MX 61 s. The user
got a generic 500 with an empty stderr, indistinguishable from a real
failure because pmtiles buffers its output and loses it on SIGTERM.
- Raise the limit to 10 minutes.
- Detect a timeout from execFile's killed/signal/code (isExecTimeout) and
return a 504 with a plain message instead of a 500.
- Let the country picker show the server's message inline; the call no
longer goes through catchInternal's generic toast.
Closes#1258
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ollama 0.33.3 gives two answers for gemma3: /api/tags lists it as
['completion'], while /api/show for the same model reports
['completion', 'vision']. getModelCapabilities trusted the list outright
and cached it for the life of the process, so image upload stayed disabled
for gemma3, one of the most common small vision models, even though the
model downloader correctly shows it as text + image. Re-pulling does not
help: the local digest matches the registry.
A list that reports vision is still trusted without probing. A list that
omits it is no longer conclusive: the model is probed through /api/show
(then llama.cpp /props), and the list's answer stands only when the probes
have nothing better, which keeps hybrid routers without /api/show working.
The service's own copy of the short-circuit is removed so the rule lives in
the one tested function.
Verified on NOMAD3 (v1.35.0 + this patch): installed-models moves gemma3:4b
from unsupported to supported, every other model unchanged, and a chat with
an attached image is answered correctly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
embedAndStoreText read the file's active flag, then upserted its
chunks. A toggle landing between those two steps set its payload
before the new points existed, so the new points kept the old value.
Retrieval and the KB panel then disagreed.
Both toggles (setFileActive and setKnowledgeCollectionActive) now write
kb_ingest_state before calling Qdrant setPayload, and restore the rows
they changed if setPayload fails. embedAndStoreText re-reads the flag
after its upsert and sets active on its own point IDs if the flag
changed. Whichever order the two run in, one of them writes the final
value to the new points.
Also coerce the flag to a boolean before stamping it into the payload.
MySQL returns tinyint(1) as 0/1, and a raw 0 got past the search
filter's must_not active == false, so an inactive file that was
re-embedded became searchable again.
setFileActive updated the Qdrant payload but skipped the DB write when
a source had no kb_ingest_state row. ZIMs hit this during embedding
(markIndexed runs only after the final batch), after a lost row, or on
pre-RFC installs. The API returned success, getStoredFiles still showed
the file as active, and the switch could never turn it back on. All
chunks stayed excluded from retrieval.
When the row is missing, create it the same way the scanner backfills:
indexed if Qdrant has chunks for the source, pending_decision if not.
Then save the flag.
Creating the row does not affect retrieval, since search filters only
on the Qdrant payload. It also leaves failed or partial embeds no worse
off:
- A failed job already leaves a row via markFailed. Only active changes.
- A crashed worker leaves partial chunks and no row. The next scan
writes the same indexed row through backfill_indexed.
- For an in-flight job, markIndexed/markFailed reuse the row and keep
active, so later batches stop stamping chunks active: true.
- The UI already treats a null state as indexed, and stall/zero-chunk
warnings come from Qdrant counts, so Re-embed still appears.
A partial ZIM with no row is still labeled indexed, as before.
markStalled has no callers, so stalls are not tracked in the row.
Boot runs ensureDirectoryExists() on the zim root, so a separate volume
that failed to mount leaves an empty mountpoint behind. The scan walks it
without error and reports it as scanned. One upload in kb_uploads keeps
the overall scan non-empty, so the next manual sync purged the vectors
for every ZIM under that root.
decideOrphans() now decides per scanned root and returns
{ orphans, withheld }. A root keeps its indexed sources when:
- it holds no embeddable files (kiwix-library.xml does not count)
- the purge would remove more than 50% of its indexed sources and at
least 5 of them, which points to a wrong or stale mount rather than
hand deletions
scanAndSyncStorage() logs each withheld root and appends a note to the
sync result message, so the user sees a likely unmounted drive instead
of losing hours of embeddings.
Replaces the unit test that asserted the old empty-root purge with
regression coverage for #1378 and both guards.
Closes#1378
Upgraded AMD iGPU installs sometimes carry a working
HSA_OVERRIDE_GFX_VERSION on nomad_ollama, set by hand or by an older
install path, but have no ai.amdHsaOverride KV and no .nomad-amd-gfx
marker. The CPU-fallback banner offered Fix: Reinstall, and the
reinstall rebuilt the container without the override, so a 780M
(gfx1103) stayed on CPU.
- Add pickAmdHsaOverride to utils/amd_hsa_override.ts with the
resolution order KV, marker, existing container override, none.
A marker still wins even when it maps to none. KV "none" still
disables the override.
- forceReinstall inspects nomad_ollama before removing it and saves a
container-only override to ai.amdHsaOverride. Saving it keeps the
value through a reinstall that fails after the removal and shows it
in the Set GFX Override modal.
- The update path passes the old container env to the resolver, so a
routine Update keeps the override too.
- The GPU health check passes the container env, so
diffAmdOllamaConfig compares against what a reinstall would build.
On the reported box, the only drift is OLLAMA_IGPU_ENABLE=1.
- diagnoseAmdCpuFallback suggests the container's current override
when one is set. With an override applied, Ollama logs the coerced
target (gfx1100), which maps to none and would suggest removing a
working value.
- Remove the unused _mapGfxToHsaOverride wrapper.
parseAmdGfxTargetFromLogs is unchanged. It still matches only
gpu_type=. The "dropping integrated GPU ... compute=gfx..." line needs
a full sample from an affected box before we can write a regex for it.
We'll address in a later PR.
GpuPassthroughRemediationProvider returned early unless the NVIDIA
runtime was registered with Docker. AMD doesn't register one, so the
log-based CPU-fallback detection added in v1.35.0-rc.1 never ran on
AMD boxes. On a Phoenix1 iGPU running ollama:rocm with /dev/kfd and
/dev/dri passed through, Ollama logged library=cpu while gpuHealth
reported status "ok".
The gpuHealth side had its own bug. hasNvidiaRuntime and
hasRocmRuntime were set only inside the lspci probe branch. When lspci
returned a plausible controller, the else branch reported "ok" without
probing and left both flags false. A blind reinstall would not have
helped either: gfx1103 needs HSA_OVERRIDE_GFX_VERSION=11.0.0, and
without the gfx marker or ai.amdHsaOverride a reinstall rebuilds the
same CPU-bound container.
- Add resolveExpectedGpuVendor(). It reads the nomad_ollama container
config first (nvidia DeviceRequest, ROCm image tag, /dev/kfd), then
the nvidia runtime, then the AMD GPU type when no container exists.
/dev/dri alone is not treated as AMD. A CPU container on an AMD box
is trusted as deliberate (#1232 fallback, acceleration disabled).
- Gate the provider on the expected vendor. NVIDIA auto-reinstall is
unchanged but now skips when the container requests a GPU with no
runtime registered. AMD is detect-only: the provider logs whether a
reinstall would change the container and never reinstalls itself.
- In gpuHealth, set the runtime flags up front and let Ollama's
startup log decide health whenever a GPU is expected, whatever lspci
returned. A CPU-only log now reports passthrough_failed and skips
the nvidia-smi exec. lspci stays the display source when clean.
- Parse the first GPU "inference compute" line instead of the last
line, which could be the CPU fallback device, and recognize Vulkan.
Move the startup-log reader into ollama_compute so both callers
share it.
- Report amdReinstallWouldChange, amdGfxTarget, currentHsaOverride and
suggestedHsaOverride on AMD passthrough_failed. The gfx target is
parsed from Ollama's "amdgpu is not supported" line (best-effort).
- Add GpuPassthroughAlert, shared by Settings > System and Settings >
Models. When a reinstall would rebuild the same AMD container, it
offers "Fix: Set GFX Override" instead of a reinstall. The dialog
saves ai.amdHsaOverride and then reinstalls.
- Expose ai.amdHsaOverride through the settings API. The value lands
in the container env, so only x.y.z, "none" or empty is accepted.
- Silence the HSA override resolver's logging on the gpuHealth poll.
The #1325 case where no GPU runtime is expected and lspci reports a
controller still returns "ok" unprobed.
Fixes#1344
Refs #1325, #1232
The post-rerank relevance floor rejects noise but passes coherent
off-topic questions, and it gets worse as the library grows. Adding
Gutenberg prose, the Python tutorial and MDN pages to the eval corpus
took the off-topic leak rate from 22% to 67% at the 0.62 floor.
Score cutoff does not fix it. The weakest correct top hit scores 0.666
while "write me a haiku about autumn leaves" pulls Walden at 0.705.
A cosine floor and a top-vs-median contrast test both cost recall
faster than they cut leaks.
- Add rag.relevanceCheck (default off). After the floor, the tasks
model, or the chat model when none is set, gives one grammar-constrained
on_topic verdict for the whole retrieved set. Off topic means no
context and no citations. Any failure keeps the chunks unjudged.
- Add a Settings > Models toggle recommending an 8B+ tasks model.
On llama3:8b the check cuts off-topic leaks from 67% to 6% for about
5 points of core recall@5 and ~0.6s per turn. On 3B models it rejects
most valid questions, which is why it is opt-in.
- Add 6 distractor documents and an off_topic suite (18 unanswerable
questions, 7 distractor controls) to the eval corpus.
- Report per-tag leak rates and leaked cases with top scores in
eval:retrieval. Add --judge-model to measure the check and --dump to
export pre-floor candidate pools for offline gate simulation.
- Pin the check off in eval:generation so a local setting cannot move
scores.
- Record the distractor sweep on RAG_MIN_FINAL_SCORE. The 0.62 default
stays: 0.65 halves leaks at no measured recall cost but leaves only
0.016 of margin.
The corpus fingerprint changes to b4ebe5dce8699f5d. Retrieval reports
from before this commit are not comparable, and a new baseline is
committed.
Refs #1341
planPrompt() returned the response reserve as num_predict, and the
reserve was clamped by MAX_RESPONSE_RESERVE = 1024. Chat answers
stopped at 1024 output tokens regardless of window size, often
mid-sentence, and the normalized stream chunks dropped Ollama's
done_reason, so neither the UI nor API clients could discern between
a cut-off answer and a finished one.
- Decouple the generation cap from the reserve. num_predict is now the
space left in the window after the prompt, less a 5% (min 64 token)
margin for estimator error, with the reserve as its floor. Raise
MAX_RESPONSE_RESERVE to 8192 so it only bounds prompt hold-back.
- Carry done_reason through all four OllamaService chat paths. Native
forwards Ollama's done_reason; compat maps OpenAI's finish_reason.
- Log length-limited stops in OllamaController with num_predict,
num_ctx and tokens generated.
- Show a truncation notice on cut-off answers, with a Continue button
on the latest one that sends a normal follow-up turn.
Generation evals before and after this change are not length
comparable.
Closes#1342
decideOrphans guards on the disk scan coming back empty, but the failure
mode it needs to defend against is per-root. _discoverKbFiles() walks
kb_uploads and zim in a loop and skips a missing root rather than
failing, because a fresh install legitimately has no kb_uploads yet.
So if the zim root is relocated, renamed, or not yet mounted (#1050),
the scan still returns a non-empty list from kb_uploads. The empty-scan
guard never fires, filterOrphanCandidates has no way to tell a root that
scanned clean from one that was never there, and every ZIM in the index
is classified as an orphan and purged in a single batch. On a
Wikipedia-scale knowledge base that is millions of vectors and a
multi-hour re-embed. It is reachable from the UI: scanAndSyncStorage is
POST /rag/sync, so the button pressed when the knowledge base looks
wrong is the button that empties it.
_discoverKbFilesWithRoots() now reports which roots were walked, and
filterOrphanCandidates allowlists against that list instead of the
configured roots. A root that wasn't there contributes no candidates; a
root that scanned clean and held no files still yields its orphans, so
the legitimate purge path is unchanged. The empty-scan guard becomes the
degenerate case of the same rule rather than a separate one.
Also builds the unit-test paths with join() instead of POSIX literals.
Written as '/data/...' the filterOrphanCandidates cases pass on Linux CI
and fail on Windows, where sep is backslash and no source ever matches a
prefix, which is exactly where they would be run while changing this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Self-review pass over the last few commits:
- Extracted the kb_uploads/zim allowlist check out of scanAndSyncStorage()
into a new filterOrphanCandidates() in kb_orphan_decision.ts, alongside
the existing decideOrphans(). It was untested logic sitting inline in a
large method; pulling it out next to decideOrphans() makes it a pure,
I/O-free function the same way decideOrphans() already is, and lets it
be covered directly instead of only indirectly through scanAndSyncStorage.
- Added unit tests for it: keeps sources under the scanned roots, excludes
sources outside them (e.g. bundled docs), and a dedicated case for the
sibling-directory false-positive a naive prefix check (no trailing
separator) would let through.
- Also fixed two prettier violations introduced earlier in this PR (an
over-length logger.info call and an unwrapped ternary in
scanAndSyncStorage) that weren't caught by the per-edit lint checks I
ran at the time.
scanAndSyncStorage()'s orphan sweep purged each orphaned source in its
own sequential await, re-running _ensureCollection and a separate
Qdrant delete call per orphan. Add purgeIndexedSources() /
_deletePointsBySources(), which use Qdrant's match.any filter and a
single whereIn() query to purge a whole batch in one round trip each,
and point the sweep at it. purgeIndexedSource() now delegates to the
batched form for a single source, so there's one implementation
instead of two.
purgeIndexedSource() deleted Qdrant points before removing the
KbIngestState row. If the row-delete failed or the process died in
between, the source vanished from Qdrant's orphan-detection signal but
the ghost state row stayed forever — nothing else ever revisits a row
with no matching file and no matching Qdrant source. Reversing the
order means a failure instead leaves points in Qdrant, which the next
sync's reverse orphan sweep will find and retry; both steps are
already idempotent, so a retry from either partial state is safe.
Also switch scanAndSyncStorage()'s orphan-candidate filter from
denylisting known non-scanned roots (README.md/docs) to allowlisting
the two roots _discoverKbFiles() actually scans (kb_uploads, zim),
via a new shared _kbScanRoots() helper. A denylist silently treats any
future source root outside kb_uploads/zim as an orphan the first time
it's added — the same class of bug the denylist itself was written to
fix for docs.
ZimService.delete() was calling purgeIndexedSource() unconditionally,
so installs without the AI Assistant (Qdrant) feature logged a false
"offline" error on every ZIM delete. Guard it on Qdrant being
installed, same as reconcileReplacedContentFile() already does.
Also point reconcileReplacedContentFile() at purgeIndexedSource()
instead of its own inline _deletePointsBySource + KbIngestState.remove
pair, so there's one shared implementation instead of two.
delete() removed the file, the kiwix library entry, and the
InstalledResource row, but never touched Qdrant or KbIngestState. A
deleted ZIM kept surfacing as a search citation indefinitely. The sync
sweep added in the previous commit would eventually catch these too,
but no reason to wait for the next sync when we already know exactly
what needs cleaning up right here.
scanAndSyncStorage builds sourcesInQdrant and embeddableFiles in the
same pass but only ever checks one direction - is this on-disk file
already embedded? It never checks the reverse, so points left behind
by a deleted or replaced ZIM just sit in Qdrant forever and keep
showing up as search hits for content that doesn't exist anymore.
Added a reverse sweep that diffs the two sets and purges anything left
in Qdrant with no file backing it. Guarded so it's a no-op when the
disk scan comes back empty, so a filesystem hiccup can't get misread
as "delete everything."
Also has to exclude README.md/docs, since those get embedded by
discoverNomadDocs and live outside the kb_uploads/zim paths the
scanner actually walks. First pass without that exclusion would have
wiped the docs KB on every single sync - caught it by testing against
a live install where those sources were already indexed.
Verified live: found exactly one real orphan out of 29 indexed
sources (an old devdocs_en_bash zim superseded by a newer download),
purged it, and left the other 28 - including all the docs - alone.
On 1.34.0 every FDA ingest finished with its labels loaded but lost the
installed_resources 'dataset' row, because resource_type was still
enum('zim','map'). The enum migration in #1243 fixes new writes but can't recreate
the lost row. Re-selecting the tier doesn't fix it either, because ZimService
skips a dataset that already has rows. So Medicine never resolved its
Standard tier.
- Add DrugInstallRowProvider. On each boot it backfills the row when an
ingest has finished (rows present, export date set, no download
marker, no drug job running) and no row exists.
- Put the decision in decideDrugRowReconcile() so it has unit tests.
- Add DrugReferenceService.recordInstalledRow() as the one writer used
by the ingest job, the provider, and ZimService.
- ZimService now writes the row when a Medicine tier re-select finds a
finished ingest with no row.
Manual ingests and reset-and-reingest runs, which dispatch without
resourceMeta, also get the row on the next boot.
Chat sent num_ctx, num_predict and nothing else, so every turn decoded at
whatever the backend picked: temperature 0.8, top_p 0.9, no min_p. On a
heavily quantized 1-3B model that is the wandering, repetitive output people
read as "the local model is bad".
Adds `ai.responseStyle` with four profiles. min_p is the lever, not
temperature: it floors relative to the top token, so it truncates hard on a
confident distribution and barely at all on a flat one, which is where small
models start to wander. top_p stays at 1.0 rather than stacked on top of it.
repeat_penalty moves with the style, because a grounded answer legitimately
repeats a part number inside the 64-token window: `focused` relaxes it to
1.05, `creative` tightens it to 1.15.
`auto` is the only style that defers to the model. Plenty of library models
ship author-tuned samplers (qwen3 sets 0.6/0.95/20), so those win key by key
and NOMAD fills only the gap, which in practice is min_p. A style the user
picked by name overrides them, or it would silently do nothing on half
their models.
The compat transport drops min_p, so a profile carries a separate compatTopP.
Passing top_p 1.0 to a backend with no min_p behind it would sample the raw
distribution, which is worse than the 0.9 it replaced.
Resolved in the controller, so the grammar-constrained task calls keep their
temperature 0. An explicit temperature still outranks a profile's, which is
what keeps the eval harness deterministic.
`eval:generation --style` is new and load-bearing: temperature 0 makes min_p,
top_p and top_k inert, so a default run cannot see this change at all.
`--style=off` sends no samplers and no temperature, and is the control.
Verified against Ollama 0.32.9, min_p accepted. 22 unit tests.
getModels() and the _isNativeBackend() probe both set isOllamaNative =
false in their catch blocks for ANY failure of GET /api/tags - including
a plain 5s timeout while Ollama is busy or still starting, or a refused
connection while the container is down. The probe memoized that verdict
for the process lifetime. Every model download then failed with 'Model
is not available in your AI host... (model pulling is only supported
for Ollama backends)' on stock single-Ollama installs (#1349, two
reporters on GA).
The fix separates 'the backend answered and is not Ollama' from 'the
request failed'. Only an actual HTTP response - a non-2xx status, or a
200 whose body fails the Ollama shape check - proves a live non-Ollama
server and classifies false. No response at all leaves the
classification untouched (null, or a previously proven value), the
probe unmemoized for re-probing, and the download path falls through to
the pull attempt, whose existing error handling reports the real
network error as a retryable failure.
Verified live (v1.35.0-rc.1 + this patch, Ollama container stopped):
- stock behavior reproduced verbatim: dispatch fails with the
'only supported for Ollama backends' message
- patched: the same dispatch fails with 'Failed to download model:
getaddrinfo ENOTFOUND nomad_ollama' and stays retryable
- after starting Ollama again, the retrying BullMQ job recovered on
its own and installed the model with no user action
Fixes#1349
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
applyUpdate guarded with the raw RunDownloadJob.getByUrl and only bailed
out for active/waiting/delayed states, letting failed/completed states
fall through to dispatch. But dispatch adds the job under a deterministic
jobId (sha256 of the URL) with no removeOnFail, so once a download
exhausts its retries the failed job persists in Redis. The next apply
then hits "job already exists", dispatch returns the stale failed job,
and applyUpdate reports success while nothing is re-downloaded — the
resource is stuck forever.
Use getActiveByUrl instead, which removes any terminal job for the URL
before returning, so a fresh job is dispatched. This matches every other
dispatch site (map_service, zim_service, creator_pack_service).
The GpuPassthroughRemediationProvider probed GPU passthrough by exec'ing
`nvidia-smi` inside nomad_ollama. When Ollama was created CPU-only — most
commonly because the NVIDIA container runtime was registered with Docker
only AFTER the AI Assistant was first installed (DockerService attaches a
GPU DeviceRequest only when 'nvidia' is in docker.info().Runtimes at
install time) — the container ships no nvidia-smi, so the exec returned an
error string whose letters the alphabetic-output check mistook for
"passthrough healthy". That false negative suppressed all auto-remediation,
leaving Ollama silently stuck on CPU (models run at a fraction of a token
per second) until the user manually clicked "Fix: Reinstall AI Assistant".
Replace the nvidia-smi probe with Ollama's own "inference compute" startup
log line — the ground truth for whether a GPU backend actually loaded — via
a new, unit-tested classifyOllamaComputeBackend() helper. One signal covers
both the "created CPU-only" case and the original "DeviceRequests present
but the toolkit binding tore" case. Add a cooldown on gpu.autoRemediatedAt
so a GPU that genuinely cannot be accelerated (an architecture Ollama has
no kernels for) does not trigger a reinstall on every admin boot.
Co-Authored-By: Claude <noreply@anthropic.com>
Content Explorer hardcoded `lang: 'eng'`, so it could only ever show
about 1,300 of the catalog's ~10,900 books. Adds a language selector
sourced live from the catalog's own language feed, with book counts,
defaulting to English and remembered per browser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
* feat(kb): add per-file active/inactive toggle for search inclusion
Implements the per-file toggle agreed on in #1119, in place of disabling
ZIM auto-discovery by default. Any indexed file (ZIM, upload, or other)
can now be excluded from RAG search results instantly, without deleting
or re-embedding its vectors.
- Vectors stay in Qdrant permanently; each point's payload carries an
active boolean. Toggling flips this payload in place, both directions,
with no re-embedding cost.
- Search filter is a denylist (must_not: active == false), not an
allowlist, so a point missing the field entirely is still found by
default -- search correctness never depends on backfill timing.
- embedAndStoreText() looks up a file's existing active state before
writing new points, so retries, force re-embeds, and replaced-content
reindexing never silently reactivate a file the user explicitly
toggled off.
- Applies uniformly to all non-admin_docs buckets (ZIM, uploads, other),
matching how the rest of this pipeline already treats buckets, rather
than special-casing ZIM.
New POST /api/rag/files/active endpoint and KB panel toggle switch,
following the existing updateFileCollection pattern.
Real vector deletion for genuinely deleted/replaced files remains
#1170's separate concern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* feat(kb): group personal KB uploads by collection with bulk toggle
Groups personal KB uploads into collapsible per-collection header
rows (with an "Uncategorized" bucket for files with no collection) to
reduce clutter in the file table. Each collection header also exposes
a bulk active/inactive toggle to flip every file in that collection at
once.
Also fixes three pre-existing bugs found in the collection-assignment
dropdown while building this: the popover was clipped by table cell
truncation, clipped again by the table wrapper's overflow, and the
combobox query was stale-filtering results on open.
Refs #1193
* fix(kb): widen the modal for the Active column and coerce active to a boolean
Both found by browser-verifying on NOMAD3.
The Active column adds ~57px, which put the table back over the modal width and
reintroduced the exact defect #1198 fixed: the Delete button falls off the right
edge at every viewport. Measured on the box at 1018px of table against 961px of
usable width at max-w-5xl. Widened to 6xl.
MySQL returns tinyint(1) as 0/1 rather than a boolean, and the raw 1 reached the
DOM as aria-checked="1". That is not a valid ARIA value, so a screen reader is
told nothing useful about whether the toggle is on. Coerced in the service so
every consumer sees a real boolean, with a !! guard in Switch as well.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
* fix(kb): validate the bulk collection toggle's input before sanitising
sanitizeCollectionName ran before the type check, so a form-encoded request
sending collection[] as an array reached .trim() on a non-string and produced
an uncaught TypeError, surfacing as a 500. The vision controller already
guards this the same way.
Also documents why requiring `active` to be a real boolean is load-bearing
rather than mere input hygiene: a cross-site HTML form can only send the string
"true", so that check is what keeps a bulk Qdrant write across every point in
the index out of reach of a form POST from another origin. Worth a comment,
because coercing it would look like a harmless usability fix.
Found in a security review of the release PRs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
---------
Co-authored-by: John Cortright <jcortright@zscaler.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
`q=` searches an entry's title and summary, never its filename, so it
cannot match a resource id. It cost up to 5 paged catalog requests per
unresolved book on every check and resolved none. Verified on a test box:
identical results with and without it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
OPDS indexes a book under its base name with the flavour in a separate
element, so `name=wikibooks_en_all_nopic` matched nothing and no
maxi/mini/nopic book could ever resolve an update. Retry against the
base. Also drop the `lang: 'eng'` filter, which made every non-English
installed ZIM permanently unresolvable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
browse.library.kiwix.org grew an anti-crawler interstitial on 2026-08-29
that answers HTTP 200 with an HTML confirmation page. Content Explorer's
remote listing 500s on it and ZIM auto-update silently reports "no
updates". opds.library.kiwix.org serves the same OPDS API ungated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
The catalog-sync loop overwrote container_config and ui_location for every
non-user-modified curated service, resetting the published host port of an
install that had been moved off the catalog default. The row then disagreed
with the running container: the Open link pointed at a dead port and the
next recreate collided.
Preserve the live host ports for installed rows and apply every other
catalog change, so these installs keep receiving catalog updates instead of
being opted out of them.
Diagnosis and the original fix are @johno10661's work in #1005.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
"Address" was vague for a field that takes a URL. "No scheme? We'll default
to http://" named a concept most people do not have a word for, and it was not
needed: the live preview already shows http:// appearing the moment someone
types a bare address, which demonstrates the behaviour rather than describing
it. The preview stays, the sentence goes.
The two link tile validation messages follow the same wording. The custom URL
override message is left alone; it belongs to a different feature.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
A link tile rendered on a flat surface colour, which read as a stark white card
against the filled green app tiles. Tiles now take one of six colours from the
brand palette, chosen when creating or editing one, and the card is tinted from
it.
Tiles stay visually distinct from managed apps regardless of colour: the dashed
border and the external-link marker carry that, not the hue, and the tint is
deliberately light so the distinction survives every option.
Class names are written out in full rather than interpolated, because Tailwind
scans source text for literal class strings and would silently drop a computed
one, leaving the tile untinted. A test asserts that.
Added as a separate migration from the one introducing is_link_tile, since that
has already run on the test box and editing it in place would leave fresh and
upgraded installs with different schemas.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
_syncContainersWithDatabase flips any service marked installed to not-installed
when no container exists for it. A link tile has no container by definition, so
every tile was silently unset on the next sync and disappeared from the
dashboard within seconds of being created.
Caught by creating one in the browser and watching it not appear.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
Users want things they already run to appear on the dashboard: a NAS, a router
UI, a printer, a Pi, a container they started themselves. Adds a tile they
address themselves, with a name, one URL field, an optional description and an
icon from the curated set.
Chosen over adopting containers from the Docker daemon, which is what community
PR #1239 proposed. Adoption only reaches containers on this Docker host and has
to guess: it took the lowest published host port and assumed http, which picks
the wrong port for anything publishing several (Portainer 8000 against 9443,
Jellyfin 8096 against 8920) and breaks entirely for HTTPS. A tile the user
addresses has no guessing in it, covers the whole LAN rather than one Docker
host, and has no lifecycle that can break.
One URL field rather than separate host, port and path, because people paste
URLs and it gets https and sub-paths for free. A bare "192.168.1.50:8080" gains
http automatically, which is what most LAN devices need. normalizeCustomUrl and
getServiceLink already did exactly this work for the custom app URL override, so
both are reused rather than duplicated.
Stored as a service row with no container behind it: is_link_tile marks it,
custom_url carries the destination, and the service_name is namespaced
nomad_link_* so it can never collide with a curated entry. The seeder only
iterates its own DEFAULT_SERVICES and never prunes unknown rows, so a reseed
cannot touch or remove a tile.
The tile is deliberately styled differently from a managed app, outlined and
dashed with an external-link marker rather than filled, because if it looked the
same users would expect Start, Stop and Update and file bugs when those controls
are not there. Removing a tile arms a confirm first, since it is user-entered
data with no undo.
Icons come from a fixed 36 shared by the picker and the server-side validator,
so the two cannot disagree about what is selectable. DynamicIcon renders nothing
at all for a name it does not know, which would leave a silent hole in a tile.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
The UI enforces 500 characters but the validator did not, so the API accepted
up to the MySQL TEXT ceiling. The API needs no authentication by design, so an
uncapped free-text column reachable from any page a user visits is a cheap way
to bloat the database. Matches the limit the form already shows.
Found in a security review of the release PRs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
Salvages @kennethbrewer3's work from #836, which had drifted onto a stale base
(GitHub showed 231 files; the real change is the map UI plus a migration).
Marker visibility (individually and Hide All), per-marker icon and custom hex
colour with contrast-adjusted icon colour, search, sorting and a tree view over
the marker list, coordinate search and fly-to with URL parameters
(?lat=&lng=&zoom=), creating a marker from typed coordinates, and markers
rendered as SVGs anchored to true coordinates rather than approximately.
Name, notes and preset colour already existed; the new columns are
custom_color, icon, icon_color and visible.
Dropped from the original as stale-base noise rather than his work: unrelated
duplicate changes to the Ollama, Docker, download, RAG and ZIM services, the
API reference and getting-started docs, README/FAQ/CONTRIBUTING, the installer
script, and a revert of AppLayout to a version predating the compact sidebar.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
Closes#1176. Salvages @gabegraves's work from #1178, rebased onto the
RagPipelineService refactor that landed after it was written.
Adds JPEG/PNG/WebP attachments to chat over multipart, leaving text-only JSON
requests on their existing path. Images are normalized server-side to bounded
JPEG with metadata stripped, capped on count/size/pixels/output, and the
temporary uploads are cleaned up on every validation path. They are attached
only to the latest user message, are excluded from RAG, titles, logs and
persisted history, and are never written to storage.
Vision support is resolved per model from what the backend actually reports --
OpenAI-compatible /v1/models metadata, then Ollama /api/show, then llama.cpp
/props -- never inferred from a model name. A model that reports no vision
support has attachments disabled with a visible explanation; a backend that
reports nothing is allowed with a warning and its rejection is translated into
plain compatibility guidance rather than a raw error.
Changes from the original:
- Prompt assembly moved into RagPipelineService in the meantime, so the
controller half is rebuilt against the trace and the multipart-aware
collection filter is now the only one (dev's request.input() version would
have silently dropped the collection on every image request).
- Added a native-path converter. dev split chat into native and OpenAI-compat
paths after this was written; Ollama's native /api/chat does not take OpenAI
content parts, it takes a sibling `images` array of bare base64. Without this
every image request on a native Ollama backend would fail.
- Dropped the model-recommendation change (selectRecommendedModels and the
Qwen3-VL fallback entry). Displacing a recommended model at first-time setup
is a product decision and is being taken separately. The "Supports images"
label in Easy Setup is kept: it describes what is already on offer rather
than changing it.
- Dropped the DEFAULT_QUERY_REWRITE_MODEL download prompt his branch carried,
which dev deleted in d022c2d.
- Removed a duplicate checkModelHasThinking; kept dev's memoized getModelInfo
path, since thinking detection is not part of this feature.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
Closes#1179.
Surfaces the documents an answer was drawn from as a "Sources" list under the
assistant message, with the archive date where one is known, and persists it so
reopening a past conversation still shows what backed each answer.
Offline there is no second opinion to check an answer against, so provenance is
the only signal the user has that a claim came from a real document rather than
from the model. This is the other half of retrieval being allowed to decline:
one says when nothing relevant was found, this says what was found.
Citations are built from the chunks actually injected into the prompt, not from
everything retrieval returned. A chunk dropped by the budget planner never
reached the model, so citing it would credit the answer to a document it was
not based on -- the exact failure the list exists to prevent.
The archive title and date have been written into the vector payload by the ZIM
ingestion path all along and were simply never read back, so this works on
already-embedded content with no re-embed.
Rebased onto the RagPipelineService refactor, which landed after this was
written: the citation list now comes off the pipeline trace rather than the
inline retrieval block that used to live in the controller. Entries with
neither a path nor a title are skipped rather than shown as "Unknown source" --
a row the user cannot act on makes an answer look sourced when it is not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
#1165 taught the benchmark path to reject an unresolved PCI id as a GPU
model, but isUnresolvedGpuModel lived inside benchmark_service.ts and the
Settings > System display path never learned the same thing. It still shows
whatever lspci handed back.
The two symptoms of a stale pci.ids database are independent, which is why
the existing guard misses this. system_service already re-probes when a
discrete-GPU controller reports an implausible BAR0 reading as VRAM, and
that happens to fix the name as a side effect. But lspci can return a
perfectly plausible VRAM figure alongside a name it could not resolve, and
then nothing fires.
Confirmed live on an AMD box. The Radeon 780M reports:
model: "Device 1900", vram: 512
512 MiB clears the 256 MiB bogus-VRAM threshold, so no probe runs and the
System page renders "Device 1900". The same box's Ollama startup log carries
description="AMD Radeon 780M Graphics", so the probe the guard declined to
run had the real name available the whole time. NVIDIA hides this because
its path tends to be reached for other reasons; AMD has no equivalent.
Adds an unresolved-name trigger alongside the VRAM one, and moves
isUnresolvedGpuModel to app/utils/gpu_model.ts so the submission path and the
display path share one definition instead of diverging again. Tests cover
both placeholder shapes and, more importantly, a set of real product names
that must never be rejected.
Stacked on #1277, so the shared helper carries the WSL placeholders too.
Closes#1196
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_embedWithFallback() always tried Ollama's native /api/embed first and only
reached /v1/embeddings from the catch block. Against a backend that only
speaks the OpenAI embeddings API, every batch therefore paid a
guaranteed-failing round trip plus a warn line before doing the real work.
Invisible on a handful of documents. Expensive on a bulk file-embeddings
job: the reporter is indexing several million chunks through an
OpenAI-compatible proxy, which doubles request volume and log noise against
their embedding host for the entire lifetime of the ingest.
Gates the native attempt on _isNativeBackend(), matching how _doDownloadModel
already skips /api/pull. Used the probe rather than reading isOllamaNative
directly: the field is null until something populates it, and an embed job
can easily be the first thing to touch the service after a restart, which is
exactly when a long ingest is starting and the saving matters most. The probe
memoizes for the process lifetime, so this costs one request overall rather
than one per batch.
Behaviour is otherwise unchanged. A native backend still tries /api/embed
first and still falls back if it fails for a non-context reason, and
context-length errors still bubble to embed() for the truncated retry. The
response-shape validation stays: the probe establishes that /api/tags
answered like Ollama, not that /api/embed will.
Reported by @horsleyb.
Closes#1279
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Under WSL2 the GPU is reached through /dev/dxg rather than the real adapter, so
si.graphics() returns Microsoft's generic placeholder even while CUDA work runs
on a physical card. isUnresolvedGpuModel() caught blanks, "Device <hex>" and
"Unknown" but not this, so the placeholder was accepted as a legitimate model
name and the authoritative detection path was skipped.
The reporter on #1218 has an RTX 3090 doing real work (92% utilisation, 11.6 GB
VRAM, 352 W, 127.9 tok/s) that reaches the public leaderboard labelled
"Microsoft Basic Render Driver". A wrong label is worse than none: it fragments
per-hardware grouping, which is the same failure #1165 fixed for raw PCI ids.
Adds Microsoft's two placeholder adapter names to the same rejection list.
Scoped so it cannot affect anyone off WSL: these are Microsoft's own placeholder
names and appear nowhere outside a Windows/WSL graphics stack, so on a native
Linux host the function returns exactly what it returns today. Checked against
real product names from all four vendors, including "Microsoft Basic Renderer
Pro 9000" as a near-miss, to confirm the anchored pattern cannot swallow one.
Note this makes the model resolve to unknown rather than to "RTX 3090": the
admin container has no nvidia-smi and WSL's copy lives at /usr/lib/wsl/lib,
outside the container. Honest-unknown is the correct outcome here; naming the
card under WSL is a separate problem.
Refs #1218
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LtGWCVMPYn6qPWVYiycK74
When an AMD GPU is detected, the install switches to the ROCm image and passes
/dev/kfd and /dev/dri into the container. _discoverAMDDevices() returns both
unconditionally, with no check that they exist. Detection reads PCI data, which
says nothing about whether the ROCm kernel interface was ever loaded, so on a
pre-ROCm card or a host without amdgpu the container is rejected:
(HTTP code 500) server error - error gathering device information while
adding custom device "/dev/kfd": no such file or directory
and the install dies. There is an opt-out, ai.amdGpuAcceleration, but it is a
backend-only KV key with no settings toggle, no route and no UI, so an affected
user cannot install the AI Assistant at all through the interface even though
CPU-only would work fine on their machine.
Fall back to the CPU image instead, with a message explaining why.
Two things about the shape of this fix, both measured rather than assumed:
- It hooks start(), not createContainer(). The daemon accepts a container whose
--device path does not exist and only resolves devices on start. Verified
against Docker 29.6.2 over the socket: create returns 201, start returns 500
with the message above.
- It retries rather than pre-checking. The admin runs in its own container, so
stat()ing /dev/kfd describes the admin's namespace, not the host's. Verified
on NOMAD6, a working ROCm box (Ollama running ollama/ollama:rocm with both
devices attached): /dev/kfd is PRESENT on the host and ABSENT inside the admin
container. A pre-check would therefore have stripped GPU passthrough from a
machine where AMD acceleration works.
The matcher keys on the device-gathering phrase rather than a bare "no such file
or directory", which Docker also emits for missing bind-mount sources; falling
back to CPU on one of those would hide a real failure behind a degraded install.
Checked against all three formats the daemon/dockerode produce, and against a
port-conflict message, to confirm it discriminates.
The success path is unchanged: the create payload is built by one helper used by
both attempts, and the fallback only runs after a start() failure that matches.
Fixes#1232
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LtGWCVMPYn6qPWVYiycK74
Ollama's /api/pull returns HTTP 200 and reports failures IN-BAND, as a JSON
line inside the NDJSON stream body:
HTTP_STATUS=200
{"status":"pulling manifest"}
{"error":"pull model manifest: file does not exist"}
The stream handler only ever read completed/total/digest off each line and
silently dropped everything else, including that error line. The stream then
ended normally, on('end') resolved unconditionally, and we logged the model as
downloaded successfully. on('error') never fires for this, because at the HTTP
layer the request succeeded.
Every failed model pull, for any reason, was reported as a success. On #1071
that surfaced as an ~800 GB fp16 model "downloading" in two seconds and nothing
being installed, which had been triaged twice as disk space or a stale blob.
Those explanations produce workarounds that cannot work, since the pull never
reaches the disk at all.
Three changes:
- Check parsed.error in the data handler and destroy the stream with it, so the
existing catch reports a real failure.
- Verify the model is actually in getModels() before returning success. This is
the load-bearing half: it catches silent failures we have not anticipated,
not only the ones Ollama labels. includeEmbeddings must be true or
nomic-embed-text fails verification after a good pull, and names are
normalised because Ollama resolves a bare name to ":latest".
- Treat an in-band "model does not exist" as permanent. It was retryable, so a
bad model name spun through attempts: 10 on exponential backoff and the user
watched a delayed job for hours instead of being told the name was wrong.
Verified on NOMAD3 against the shipped v1.34.0 build, which carries the same
defect:
- bogus tag, before: 'Model "..." downloaded successfully.'
- bogus tag, after: 'Failed to download model "...": pull model manifest:
file does not exist', and the job stops at 1 attempt instead of retrying
- real model (qwen2.5:0.5b): still succeeds, passes verification, and is
present in `ollama list`
Fixes#1273
Refs #1071
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LtGWCVMPYn6qPWVYiycK74
Three dispatch sites pin a deterministic jobId. `queue.add` with an existing
custom jobId returns the existing job rather than throwing, so any retained
terminal record makes later dispatches a silent no-op while still reporting
success to the caller.
- download_model_job: only `failed` records were cleared, so once a model
downloaded successfully its retained completed job blocked every later
request for that model. Deleting the model and reinstalling it did nothing,
and a restart did not help because the record is persisted in Redis. Adds
getActiveByModelName, mirroring RunDownloadJob.getActiveByUrl.
- download_service.retryFailedJob: the model branch dispatched before removing,
so remove() targeted the job just enqueued under the same id instead of the
old record. Flipped to remove-then-dispatch, matching the file branch.
- embed_file_job: a retained failed record made re-indexing that file a no-op,
returning 202 "Indexing queued" with nothing enqueued.
In-flight jobs are still returned as-is, so re-clicking during an active
download or embed stays idempotent.
Diagnosis by @caweis in #1214, including the retry-ordering race and the embed
case. Reimplemented in-house rather than ported, as noted there.
Closes#1214
Fixes#1269. Fixes#1203.
`file-embeddings` and `downloads` both routinely outlive the default
5 minute lock, so BullMQ calls healthy jobs stalled on both. They need
opposite recovery policies, which is why neither was covered by the
existing drug-queue override.
Embedding is a self-continuing chain, so re-queueing a stalled job while
the original is alive runs two chains over the same file. Both write to
Qdrant with freshly minted point ids, so the second cannot overwrite the
first, and the queue's concurrency of 2 means the fork never contends
for a slot. maxStalledCount 0 fails on the first stall instead.
A download is resumable, so recovery is safe there; the default of 1 was
failing jobs outright and bypassing their own attempts: 10 policy.
The decision moves to a pure, unit-tested util alongside the other
decision helpers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LtGWCVMPYn6qPWVYiycK74
