project-nomad/admin/app/utils
John Cortright 18986e0527 fix(rag): reap orphaned qdrant sources during sync (#1170)
scanAndSyncStorage builds sourcesInQdrant and embeddableFiles in the
same pass but only ever checks one direction - is this on-disk file
already embedded? It never checks the reverse, so points left behind
by a deleted or replaced ZIM just sit in Qdrant forever and keep
showing up as search hits for content that doesn't exist anymore.

Added a reverse sweep that diffs the two sets and purges anything left
in Qdrant with no file backing it. Guarded so it's a no-op when the
disk scan comes back empty, so a filesystem hiccup can't get misread
as "delete everything."

Also has to exclude README.md/docs, since those get embedded by
discoverNomadDocs and live outside the kb_uploads/zim paths the
scanner actually walks. First pass without that exclusion would have
wiped the docs KB on every single sync - caught it by testing against
a live install where those sources were already indexed.

Verified live: found exactly one real orphan out of 29 indexed
sources (an old devdocs_en_bash zim superseded by a newer download),
purged it, and left the other 28 - including all the docs - alone.
2026-09-29 15:54:41 -07:00
..
eval feat(AI): let retrieval decline instead of making the prompt paper over it 2026-09-29 15:54:41 -07:00
affirmative_remedies.ts Add offline FDA drug reference (labels, interaction view, conditions, remedies) (#1040) 2026-08-04 03:38:33 +00:00
amd_hsa_override.ts fix(amd): coerce gfx1103 (780M) to HSA_OVERRIDE 11.0.0 so it stays on GPU (#1134) 2026-08-04 03:38:36 +00:00
chat_images.ts feat(ai): support local vision attachments 2026-09-29 15:54:41 -07:00
content_auto_update_backoff.ts feat(content): opt-in automatic updates for installed ZIM & map content 2026-06-23 04:47:01 +00:00
content_reindex_decision.ts feat(content): opt-in automatic updates for installed ZIM & map content 2026-06-23 04:47:01 +00:00
context_budget.ts feat(AI): budget the prompt and actually set the context window (#1253) 2026-09-29 15:54:41 -07:00
context_window.ts feat(AI): budget the prompt and actually set the context window (#1253) 2026-09-29 15:54:41 -07:00
downloads.ts fix(downloads): don't retry a rejected entitlement for four hours (#1205) 2026-08-04 03:40:02 +00:00
drug_install_row_reconcile.ts fix(drug-reference): backfill the missing dataset install row on boot 2026-09-29 15:54:41 -07:00
fs.ts fix(rag): add proper .docx text extraction via mammoth (#1100) 2026-08-04 03:37:42 +00:00
gpu_model.ts fix(system): resolve a placeholder GPU name on the System page too 2026-09-29 15:54:41 -07:00
hosted_content_auth.ts feat: support gated downloads for self-hosted curated content (#1172) 2026-08-04 03:39:55 +00:00
hosted_content.ts feat: support gated downloads for self-hosted curated content (#1172) 2026-08-04 03:39:55 +00:00
image_disk_preflight.ts feat(supply-depot): opt-in automatic updates for installed apps 2026-06-23 04:46:52 +00:00
kb_ingest_decision.ts feat(KB): Always/Manual ingest policy toggle (RFC #883 §1/§4) (#894) 2026-05-20 10:16:00 -07:00
kb_job_health.ts feat(KB): status pill + last-activity timestamp on Processing Queue (RFC #883 §5/§10) 2026-05-20 10:16:00 -07:00
kb_orphan_decision.ts fix(rag): reap orphaned qdrant sources during sync (#1170) 2026-09-29 15:54:41 -07:00
kb_ratio_lookup.ts fix(KB): stop partial_stall warning firing on atypical ZIMs (link-out/PDF-heavy) 2026-06-23 04:46:54 +00:00
kb_warning_decision.ts refactor(KB): move FileWarning to shared types/rag following existing convention 2026-05-20 10:16:00 -07:00
misc.ts feat(AI): let retrieval decline instead of making the prompt paper over it 2026-09-29 15:54:41 -07:00
model_capabilities.ts feat(ai): support local vision attachments 2026-09-29 15:54:41 -07:00
platform_metadata.ts feat(benchmark): official multi-arch sysbench, resolved digest, platform metadata (#1158) 2026-08-04 03:39:53 +00:00
queue_stall_options.ts fix(queue): stop stalled-job recovery forking the embedding chain 2026-09-29 15:54:41 -07:00
rag_prompt.ts feat: show source document and date as citations under RAG chat answers 2026-09-29 15:54:41 -07:00
rag_relevance.ts feat(AI): let retrieval decline instead of making the prompt paper over it 2026-09-29 15:54:41 -07:00
response_style.ts feat(AI): give chat a sampler instead of the backend's defaults 2026-09-29 15:54:41 -07:00
sampler.ts feat(AI): give chat a sampler instead of the backend's defaults 2026-09-29 15:54:41 -07:00
service_catalog_merge.ts fix(seeder): derive ui_location from the merged config, not the live row 2026-09-29 15:54:41 -07:00
structured_output.ts fix(AI): tell a broken grammar apart from a backend that never had one 2026-09-29 15:54:41 -07:00
superseded_resource.ts fix(content): remove superseded curated map/ZIM files when a new version installs 2026-06-23 04:47:01 +00:00
tasks_model.ts feat(AI): budget the prompt and actually set the context window (#1253) 2026-09-29 15:54:41 -07:00
think_stream.ts fix(AI): stop <think> output leaking into titles, chips, and the RAG query (#1254) 2026-09-29 15:54:41 -07:00
token_estimate.ts feat(AI): budget the prompt and actually set the context window (#1253) 2026-09-29 15:54:41 -07:00
update_window.ts feat(supply-depot): opt-in automatic updates for installed apps 2026-06-23 04:46:52 +00:00
version.ts feat: support for updating services 2026-03-11 14:08:09 -07:00
zim_batch_decision.ts fix(rag): don't abandon a ZIM after one sparse batch 2026-09-29 15:54:41 -07:00
zim_download_resolution.ts feat: support gated downloads for self-hosted curated content (#1172) 2026-08-04 03:39:55 +00:00
zim_filename.ts fix(ZIM): preserve co-existing Wikipedia corpora on cleanup (#884) 2026-05-20 10:16:00 -07:00
zim_html.ts chore(KB): filter non-content sections + render tables in ZIM extraction (#1044) 2026-08-04 03:37:30 +00:00