github-mirror/project-nomad
1
0
Fork 0
mirror of https://github.com/Crosstalk-Solutions/project-nomad.git synced 2026-07-28 19:24:39 +02:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity Graph
724 Commits 38 Branches 75 Tags 50 MiB
docs/drug-reference
Commit Graph

25 Commits

Author SHA1 Message Date
chriscrosstalk
0891d176e5
feat(benchmark): official multi-arch sysbench, resolved digest, platform metadata (#1158) 
Three changes that together let ARM hardware appear on the leaderboard honestly.
Shipping them separately would leave ARM half-supported either way: without the
image a Pi cannot submit at all, and without the architecture field it submits
but is indistinguishable from x86.

1. PIN THE OFFICIAL MULTI-ARCH SYSBENCH IMAGE

severalnines/sysbench publishes amd64 only, so ARM hosts could not run the
System Benchmark at all — not a graceful failure, the container simply cannot
execute. Apple Silicon could only run it under Rosetta emulation, which distorts
the measurement it is taking, and that is what drove a community macOS fork to
substitute a different benchmark and submit incomparable numbers.

Swaps to ghcr.io/crosstalk-solutions/nomad-sysbench (Debian 12 + sysbench
1.0.20+ds-5, built for linux/amd64 + linux/arm64). One digest covers both
architectures; verified that pulling the pinned manifest-list digest resolves to
arm64 on a Raspberry Pi 5 and amd64 on x86, and that RepoDigests reports the
same manifest-list digest on both — so a single allowlist entry serves both.

No rescoring: 1.0.17 -> 1.0.20 measured 1.25% apart on identical hardware with
identical flags (7170.18 vs 7259.56 events/sec), inside run-to-run noise and
~0.3% on a composite. Both digests are allowlisted server-side, so the fleet can
cross over gradually.

2. REPORT THE DIGEST ACTUALLY RESOLVED

The submission previously sent SYSBENCH_DIGEST, the constant the client was
compiled with. The leaderboard validates that field, but a constant attests to
how a client was BUILT rather than what it RAN, so any build inherits a valid
value simply by carrying the same source.

Now reads it back from the image. Uses RepoDigests (the manifest digest we
pulled by), never Id — Id is the config digest, differs per architecture, and
would never match the allowlist. Falls back to the constant if inspection yields
nothing usable, so a benchmark never fails over provenance metadata.

Still forgeable, and always will be with an open-source client. It moves the bar
from "no effort" to "deliberate", which is the distinction that matters when
judging whether a submission is a mistake or a choice.

3. RECORD CPU ARCHITECTURE AND OS

The leaderboard is a single board across instruction sets by design, with
disclosure as the fairness mechanism. Without an architecture field an ARM result
sits unlabelled beside x86 — exactly what the disclosure exists to prevent.

All three fields come from the Docker daemon, reusing the docker.info() call
_detectRunEnvironment already makes. That is deliberate: inside the admin
container os.arch() and si.osInfo() describe the CONTAINER, not the host being
benchmarked.

  cpu_architecture  Architecture       x86_64 -> amd64, aarch64 -> arm64
  os_version        OSVersion          '24.04' (already structured, no parsing)
  os_name           OperatingSystem    'Ubuntu 24.04.4 LTS' minus the version

run_environment is kept rather than replaced: "which distro" and "is this
virtualised" are different questions, and WSL2 is a real performance factor.

String handling lives in app/utils/platform_metadata.ts with unit tests, matching
the amd_hsa_override convention, so it is testable without a Docker daemon.
Unknown architectures pass through verbatim rather than being guessed at, and
os_name falls back to the full description whenever the version is missing or
absent from it — an over-long name is harmless, a wrong one is not.

Columns are nullable and the submission fields optional, so results recorded
before this shipped remain submittable.

Closes #1156
Refs #1151
 2026-07-27 10:26:38 -07:00
Jake Turner
c37496bd47
fix(amd): coerce gfx1103 (780M) to HSA_OVERRIDE 11.0.0 so it stays on GPU (#1134) 
PR #1076 stopped forcing HSA_OVERRIDE_GFX_VERSION=11.0.0 on
"natively-supported" AMD iGPUs. That was correct for gfx1150/gfx1151
(Strix 890M / Strix Halo, which are in the bundled rocblas allowlist)
but wrong for gfx1103 (Phoenix/Hawk Point 780M/760M), which is NOT in
that list. Without the override, ollama drops the 780M with "no rocblas
support for gfx target" and falls back to CPU on a fresh AI provision.

Extract the gfx→HSA mapping into a pure, unit-tested util and map
gfx1103 → 11.0.0 (gfx1100 kernels), the value that worked on v1.33.0 and
that restores full GPU offload in the field. gfx1150/1151 stay native.

Also harden the installer's 780M detection (Hawk Point / "Radeon 780M/
760M" strings) so the gfx marker isn't silently deleted, and upgrade the
no-marker fallback log from info to warn since it can mask CPU fallback.
 2026-07-22 11:07:42 -07:00
caweis
77c36b3fef
Add offline FDA drug reference (labels, interaction view, conditions, remedies) (#1040) 
* feat(drug-reference): offline FDA drug labels, conditions, and remedies

Adds an offline medical-reference feature with three coupled layers:

- Drug Reference: full-text search over openFDA drug-label indications,
  a detail page per label, and a side-by-side single-drug comparison view.
  A two-phase background pipeline downloads the openFDA label parts to the
  storage volume (resumable) and ingests them into the search table.
- Conditions ("When to use what"): a curated spine of first-aid situations
  that maps each situation to matching OTC drugs, each linking back to its
  Drug Reference detail page.
- Curated remedies: hand-authored natural and home-remedy entries drawn from
  US-government public-domain sources (NCCIH, CDC, MedlinePlus, FDA), shown
  with their source links and the same safety disclaimers as the rest of the
  feature.

The drug-reference and conditions layers are intentionally coupled: the drug
detail page shows the situations a drug treats, and the conditions controller
reads the same drug_labels table.

Safety surfaces ship as written. The amber SafetyBanner ("informational only,
not medical advice, not an FDA endorsement, not a drug-interaction checker, in
an emergency call emergency services") renders on the condition pages and the
search page; the detail and comparison pages carry their own "not a cross-drug
interaction checker" callout; and every page carries the openFDA CC0 source
citation and no-FDA-affiliation footer.

Wiring on this branch:
- start/routes.ts: the /drug-reference and /conditions page GETs plus their
  /api/* groups.
- commands/queue/work.ts: the drug-download and drug-ingest queues, both at
  concurrency 1. The two drug queues get a per-queue stall override
  (lockDuration 1_800_000, maxStalledCount 3) because each part is one long
  stream; every other queue keeps the existing 300000 default.
- inertia/pages/home.tsx: Drug Reference and "When to use what" tiles. The
  icon and display_order are a starting point, open to change.
- types/kv_store.ts: the two drugReference.* keys the pipeline reads and writes.
- package.json: yauzl and stream-json (plus their @types), used by the ingest
  job to stream the label JSON out of the downloaded zips.

The app reads the conditions and remedy data from the compiled TS constants in
app/data/; the repo-root collections/*.json files are the browseable mirrors.
The natural-remedies standalone test reads collections/natural_remedies.json to
assert the two stay in sync, so that file is also a test fixture.

The four standalone tests pass (drug_interactions, drug_ingest_status,
conditions, natural_remedies). tsc reports no errors in the feature code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore(drug-reference): drop a fork-internal comment reference

* fix(drug-reference): address review — defer FULLTEXT, fallback remedies prop, strip fork refs

- migration: wrap the FULLTEXT ALTER in this.defer so it runs after the
  deferred createTable (was silently swallowed, index never created)
- controller: add remedies:[] to the index() error fallback (required prop)
- strip fork-internal issue/spec references from ported comments
- tsconfig: exclude tests/standalone (node --experimental-strip-types only)
- correct the varchar(768) byte-math comment; extend remedy-spine test

* fix(drug-reference): make the interaction comparison readable at five drugs

The comparison view laid its columns out on an equal-fraction CSS grid
(repeat(N, minmax(0, 1fr))), so each added drug shrank every column; at the
five-drug maximum the FDA interaction text was squeezed into unreadable slivers.

Lay the columns out with flex instead: full-width and stacked on phones, then
fixed-width columns that scroll sideways from the sm: breakpoint up, so they
never shrink below a readable width. Theme the columns with the same palette as
the rest of the page (they were on stock gray), and give the headers a fixed
min-height so columns line up when drug names wrap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(collections): route a 'dataset' tier resource to the drug pipeline

Add an optional `type` discriminator to SpecResource ('zim' | 'dataset',
absent == 'zim'), so the tier installer can carry a DB-ingested resource
alongside ZIM files. ZimService.downloadCategoryTier branches on it: a
'dataset' resource dispatches the existing FDA download+ingest pipeline
instead of RunDownloadJob, guarded against duplicate dispatch by the drug
ingest status. Every existing manifest entry has no `type` and keeps the
exact ZIM path.

Widen InstalledResource.resource_type to include 'dataset' and exclude
dataset rows from the ZIM/map catalog-update scan (datasets aren't
filename-versioned; their freshness path is separate). No dataset rows are
written yet: the InstalledResource 'dataset' row on ingest-ready, the
manifest entry, install-gating, and the downloads-aggregator integration
are follow-up commits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(drug-reference): rework into an opt-in medicine-standard tier dataset

Reshapes the offline FDA drug reference from a built-in feature into curated
content installed by selecting the Medicine / Standard tier, per maintainer
direction.

- install-state: the ingest writes an installed_resources 'dataset' row on
  ready (version = the openFDA export_date), threaded installer to download to
  ingest; the tier-status math and the home-tile gate read it. Manual ingests
  write no row, so install-state stays tied to the curated path.
- manifest: declare the dataset in the medicine-standard tier (runtime fetches
  the remote manifest, so this also needs to land upstream).
- install-gating: the drug-reference home tiles render only when installed.
- uninstall: DrugReferenceService.uninstall() stops the two drug queues, deletes
  the on-disk parts, truncates drug_labels (schema kept), clears the KV markers,
  and drops the install row. Best-effort, logged, scoped to drug data only.
- downloads: the download phase reports the canonical {percent, downloadedBytes,
  totalBytes} shape as one drug-data card in the Active Downloads aggregator with
  cancel/remove; the heavy ingest stays in the IngestStatus surface with an
  Indexing handoff on the card.
- auto-update: a daily DrugAutoUpdateJob compares the manifest export_date and
  re-downloads when newer, gated on installed + no active job.

typecheck clean; the drug standalone suites pass. Three points are flagged in
code for the maintainer: the InstalledResource 'dataset' approach, the tier
home, and the export_date string format.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(downloads): keep the drug download card live across all parts

The Active Downloads card filtered to the deterministic jobId, but the
download's continuations run under auto-generated jobIds (only part 0 uses the
deterministic one). So the card tracked part 0 and then vanished while parts
2..N kept downloading. The queue is concurrency 1, so collapse to whichever
single part is in flight and report the deterministic jobId: one card tracks
aggregate progress through the whole download and cancel/remove still routes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(drug-reference): co-locate a persistent safety note with affirmative remedy guidance

Add RemedySafetyNote at the head of every natural-remedy section — the two on
Drug Reference and the one on "When to use what" — so the "informational only,
not medical advice, seek real medical care in an emergency" framing appears with
the guidance itself, not only in the page-top banner. Replaces the terse
per-section caveat with the same amber alert language as SafetyBanner.

Addresses the upstream #1040 review request that the disclaimer be unmistakable
and present wherever affirmative self-care guidance appears, not a one-time banner.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(drug-reference): fold dataset freshness into the content-auto-update path

The drug dataset auto-updated on its own daily cron that ignored the
contentAutoUpdate.* master switch, so it would refresh even with content
auto-update turned off, and it didn't ride the content-update path the way
ZIMs and maps do.

Move the export_date freshness/apply orchestration onto
DrugReferenceService.attemptAutoUpdate(), add
ContentAutoUpdateService.attemptDrugDataset() gated on the same enabled +
window config, and have the hourly ContentAutoUpdateJob drive both. Retire
the standalone DrugAutoUpdateJob. The ZIM/map attempt() path is unchanged.

Addresses the upstream #1040 request to wire the openFDA export_date check
into the content updater so the dataset updates alongside ZIMs and maps.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(drug-reference): strengthen the remedy note to consult a clinician before combining with meds

Widen the affirmative-remedy safety note from "talk to a clinician before use"
to explicitly cover using a remedy AND combining one with a medication the user
already takes — the interaction case is the higher-risk path for an off-grid
user self-treating.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(drug-reference): gate affirmative remedy content behind an off-by-default flag

Add drugReference.remediesEnabled (default off), independent of the tier
install. When off, the server emits no remedy data at any boundary — the
drug-reference page prop, conditions show, and the /api/conditions/drugs
situation search — and the "Natural" filter is hidden, so installing the
medicine-standard tier lights up the verbatim FDA label search and the
condition-to-OTC matching but not the hand-authored self-care and herbal
sections. No user-facing toggle: it is flipped on after a clinician content-pass.

Implements the upstream #1040 split-by-risk request: the regulated label content
ships with the tier; the authored remedy guidance stays gated until sign-off.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com>
 2026-07-21 11:06:18 -07:00
NgoQuocViet2001
c6bc40e767
fix(content): resolve current ZIM URL before download (#1091)
Some checks failed
Validate Collection URLs / validate-urls (push) Has been cancelled
Details 
* fix(content): resolve current ZIM URL before download
* fix: compare ZIM catalog versions numerically
 2026-07-20 19:36:53 -07:00
just-jbc
4f4cc5bc65
feat(rag): add subject/collection organization to knowledge base (#1063) 
* feat(rag): add subject/collection organization to knowledge base

- Add nullable collection field to KbIngestState, propagated through the
  embed job, RAG service, and Qdrant point payloads (indexed for filtering)
- Add upload-time category selection and per-file collection reassignment
  in the Knowledge Base modal, with a filterable Stored Files table
- Add a 'Search in' collection filter to the chat interface, threaded
  through to searchSimilarDocuments as an optional Qdrant filter
- Fix .docx extraction: previously routed through raw-text extraction
  (garbage output for a ZIP-based XML format); adds a proper mammoth-based
  extractor and a dedicated 'docx' file-type case

* feat(rag): support dynamic KB collection creation, rename, and removal

Extends collection organization with a Manage Collections UI: collections
are created on the fly when a file is assigned to a new name, can be
renamed (bulk-updates every tagged file and Qdrant point), and can be
removed (reassigns tagged files back to Uncategorized rather than
deleting anything).

* fix(rag): use dynamic collections query in chat search filter

chat/index.tsx still imported the static KB_COLLECTIONS constant for its
'Search in' dropdown, inconsistent with KnowledgeBaseModal.tsx which already
uses the live getKnowledgeCollections() query. Renamed/added collections
via the new Manage Collections UI weren't reflected in the chat filter.

* feat(rag): broaden preset tags and add creatable collection combobox

Replaces the survival-specific preset list with general-purpose starter
tags (recipes, diy, health, technology, finance, travel, hobbies,
reference, survival, energy) so the Knowledge Base reads well for
home-lab/reference use, not just prepping.

Adds sanitizeCollectionName() (trim, lowercase, length cap) applied on
every write path server-side, and a dependency-free CollectionCombobox
component replacing the plain <select> + window.prompt pattern for
tagging — autocompletes against presets + tags already in use, with a
'+ Create' option for anything new.

* chore(rag): remove .docx fix from this branch, split into #1100

Per review feedback, the .docx extraction fix is unrelated to the
collections feature and can merge independently. Moved to a standalone
PR (Crosstalk-Solutions/project-nomad#1100) off dev.

* chore: remove unrelated diff noise (lockfile, comments, indentation)

---------

Co-authored-by: John Cortright <jcortright@zscaler.com>
 2026-07-20 15:09:11 -07:00
chriscrosstalk
8f56d76fe7
chore(KB): filter non-content sections + render tables in ZIM extraction (#1044) 
Closes #902.

Two gaps in the structured ZIM extraction path:

1. NON_CONTENT_HEADING_PATTERNS was only used by the structure heuristic to
   count meaningful sections, never at section-emit time. Sections under
   "See also" / "References" / "External links" / etc. were still chunked and
   embedded. They're now flagged when the heading opens and dropped.

2. <table> elements were run through cheerio's `.text()`, concatenating every
   cell with no separators ("AgeDoseAdult500mg") into unsearchable word salad.
   New tableToText() joins cells with " | " and rows with newlines so
   row/column structure survives into the chunk.

Refactor: moved extractStructuredContent out of ZIMExtractionService into a
pure, cheerio-only util (app/utils/zim_html.ts) so it can be unit-tested
without the native @openzim/libzim binding. Service delegates to it; behavior
is otherwise unchanged. Adds tests/unit/zim_html.spec.ts (6 tests).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-18 09:03:53 -07:00
Benjamin Smith
5181637926
feat(KnowledgeBase): add document viewer, download, metadata, and sorting (#721) 
Rebuilt on top of dev's RFC #883 state-machine UI rather than the now-defunct
StoredFile shape:

- Extend StoredFileInfo with fileName/size/uploadedAt/isUserUpload
- Populate metadata from on-disk stats in RagService.getStoredFiles
- Add fileSourceSchema validator + getFileContent/downloadFile endpoints
  scoped to the uploads directory only (tighter than the original PR — matches
  docs_service traversal pattern)
- KnowledgeBaseModal: sortable Size and Uploaded columns; View/Download
  buttons on upload-bucket rows; new FileViewerModal for in-browser text
  preview. Bucket grouping preserved — sort applies within each bucket.
- Use formatBytes from ~/lib/util rather than redefining
 2026-06-23 04:47:09 +00:00
jakeaturner
6a2d4c2bf6
feat(content): opt-in automatic updates for installed ZIM & map content 2026-06-23 04:47:01 +00:00
Chris Sherwood
bbd62d8ed1
fix(content): remove superseded curated map/ZIM files when a new version installs 
Only Wikipedia had version cleanup; every other curated map and non-Wikipedia
ZIM left its prior version on disk when a newer one installed, so users silently
accumulated orphaned content (potentially hundreds of GB). (#634)

The install paths already record each resource via InstalledResource
{resource_id, resource_type, version, file_path}, so the authoritative old-file
path for a resource is known. On install of a new version we now capture the
prior row before updateOrCreate repoints it, then delete the old file — gated
behind a pure, fully unit-tested decision function with strict safety rails:

  - tracked-only: requires a prior InstalledResource row for the same
    resource_id, so sideloaded/untracked files are never touched
  - genuine replacement: old and new file paths must differ
  - new-file-verified: the new file must be confirmed on disk first
  - strictly-newer: a re-install or downgrade can't wipe a newer file
  - within-storage-dir: the old path must resolve under the content store

ZIM cleanup deletes the old file directly (NOT via this.delete(), which would
drop the InstalledResource row by resource_id that updateOrCreate just
repointed) and rebuilds the Kiwix library only if a file was actually removed,
so its XML never references a deleted ZIM. Maps need no library step. Wikipedia
keeps its own existing cleanup path. All deletions are best-effort and logged;
a failure never breaks the install.

Closes #634

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-23 04:47:01 +00:00
Chris Sherwood
ca5ec1767f
fix(security): harden assertNotPrivateUrl with ipaddr.js + host normalization 
Replaces the regex blocklist in assertNotPrivateUrl with ipaddr.js range
classification and normalizes the host before checking it. Consolidates two
community proposals (#930 ipaddr.js parsing, #912 trailing-dot normalization)
into one validator so the SSRF-critical path lives in-house with full tests.

- Classify literal IPs by range (loopback / linkLocal / unspecified) via
  ipaddr.js instead of a hand-maintained regex list, which also catches
  alternate IPv4 encodings and avoids over-blocking mapped public IPs (the old
  `::ffff:` regex blocked every mapped address, including public ones). IPv4-
  mapped IPv6 is reduced to its embedded IPv4 before classification.
- Strip a trailing root dot from the host so `localhost.` / `127.0.0.1.` can't
  bypass the checks (they resolve to the same target as the dotless form, #911).
- Strip IPv6 brackets and lowercase for the localhost comparison.
- RFC1918, bare LAN hostnames (e.g. `nomad3`), and external FQDNs remain
  allowed — LAN appliances need them, and DNS rebinding is a fetch-time concern
  outside this guard's scope.

Adds a consolidated unit spec covering loopback/link-local/unspecified literals,
alternate encodings, IPv4-mapped v6, mixed-case + trailing-dot localhost, and
the allowed LAN/FQDN/mapped-public cases.

Resolves #922. Supersedes #930 and #912 (thanks @Gujiassh and @luyua9).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-23 04:46:55 +00:00
Chris Sherwood
3977c723c2
fix(KB): stop partial_stall warning firing on atypical ZIMs (link-out/PDF-heavy) 
The Stored Files "partial stall" warning compares chunks in Qdrant against an
expected count from the ratio registry. The registry has an empty-pattern
catch-all (100 chunks/MB) that matches any filename, so a ZIM that matches no
specific pattern still gets a size-based estimate. For archives that are mostly
PDFs, images, or link-out stubs (e.g. irp.fas.org military-medicine), byte size
wildly over-predicts embeddable text: a 75 MB ZIM estimates ~7,236 chunks but
produces ~1, tripping a false "ingestion may have stalled" warning that re-embed
can't clear.

The catch-all is fine for rough aggregate disk-cost estimates, but it should not
drive a per-file stall signal. Add an `ignoreCatchAll` option to the ratio
lookup that excludes the empty-pattern row (returning null when only the
fallback would match), and use it in the warnings path so partial_stall only
fires when the registry has a *specific* expectation for the file. Files that
match a real pattern (wikipedia_, devdocs_, ifixit_, ...) are unaffected;
disk-cost/batch estimates keep using the fallback.

Closes #913

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-23 04:46:54 +00:00
jakeaturner
cf8db6218d
feat(supply-depot): opt-in automatic updates for installed apps 2026-06-23 04:46:52 +00:00
Jake Turner
be434d755a
feat: supply depot 2026-06-23 04:46:45 +00:00
jakeaturner
736c9bd672 fix(security): canonicalize hostnames to block IPv4-mapped IPv6 IMDS bypass 
Replace literal string matching with ipaddr.js parsing
so equivalent encodings of 169.254.169.254
(::ffff:169.254.169.254, ::ffff:a9fe:a9fe,fully-expanded forms)
and fd00:ec2::254 are all rejected.
 2026-05-20 10:16:00 -07:00
Chris Sherwood
d850cb9588 feat(KB): per-file ingest action + state indicator on Stored Files (RFC #883 §5) 
Closes the Manual-mode UX dead-end: after toggling 'Auto-index new content
for AI?' to Manual, a freshly-downloaded ZIM (or any pending_decision file)
had no UI path to opt in for embedding short of the global Sync Storage /
Re-embed All bulk actions. Per RFC #883 §5, each Stored Files row now
carries a state pill and an adaptive single-button action.

State pill (left of any existing warning chips):
  - 'Indexed'    — green; row had chunks in Qdrant or state row is 'indexed'
  - 'Not Indexed' — neutral; state is pending_decision or browse_only
  - 'Failed'     — red
  - 'Stalled'    — amber
  - admin_docs collapsed row has no pill ('Managed by NOMAD' carries it)

Adaptive action button (paired with the existing Delete button per row):
  - pending_decision         → 'Index' (force=false)
  - browse_only              → 'Index' (force=true)
  - failed / stalled         → 'Retry' (force=true)
  - indexed + warning chip   → 'Re-embed' (force=true; confirm modal first)
  - indexed healthy / null   → no action button (bulk Re-embed All covers it)

Backend: GET /api/rag/files now returns
  { files: Array<{ source, state, chunksEmbedded }> }
instead of a flat string[]. State + chunk-count come from a single
KbIngestState query unioned into the existing Qdrant-derived source list
(no new round trips). New POST /api/rag/files/embed validates the source is
known, refuses if any inflight job already targets the same filePath
(prevents double-click duplicate-chunk hazard), pre-deletes Qdrant points
when force=true, then dispatches via the existing _dispatchEmbedJobsFor
helper used by reembedAll.

Per-file Re-embed (force=true on an already-indexed file) routes through a
StyledModal confirmation since it deletes existing vectors before queueing
a fresh job — same destructive-action weight as Delete's inline confirm but
heavier since it affects search until the rebuild finishes.

Folds in PR #907's blank-screen fix because my new render needs the same
generic restored: `<StyledTable<KbFileGroup>>` and `record.displayName`
(instead of the unresolved `sourceToDisplayName(record.source)` that ships
in rc.5 and ReferenceErrors on modal open). PR #907 also adds title
tooltips on the three bulk-action buttons; those tooltips are NOT included
here — let PR #907 land first or independently for that part.

Multi-select bulk-opt-in deferred per discussion: most Manual-mode users
ingest 1-2 files at a time, the existing global toggle covers the bulk
case, and checkboxes would expand scope past what rc.6 should hold. Will
file a follow-up issue for an 'Index N pending files' single-click button
once this lands.

Tests-in-PR scope was limited to keeping `kb_file_grouping.spec.ts` green
after the StoredFileInfo[] signature change (added asInfos() wrapper).
Dedicated unit tests for embedSingleFile (unknown source / inflight refused
/ force=true delete-then-dispatch) and the new state-pill rendering will
land in a follow-up PR alongside Playwright coverage of the row actions.

Verification path: NOMAD3 currently runs project-nomad-admin:integration-
rc6-preview (PRs #907 + #908 atop rc.5). After this branch is built into a
new integration tag, I'll re-run targeted Playwright UAT on the KB modal
covering: state pill rendering per state, Index click on pending_decision
opts in cleanly, Retry on failed re-dispatches successfully, Re-embed
confirmation modal copy + delete-then-dispatch on the military-medicine
partial-stall row, and Delete flow untouched.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 10:16:00 -07:00
Chris Sherwood
cf3a924b9f feat(KB): guardrail modal at 50GB / 10%-free thresholds (RFC #883 §7) 
One-time confirmation step gating bulk indexing actions that would
consume a substantial amount of disk for embedding storage. Fires only
when the user has policy=Always (i.e., the system would auto-index)
AND the estimate trips either:

  - GUARDRAIL_ABSOLUTE_BYTES = 50 GB embedding cost, OR
  - GUARDRAIL_FREE_DISK_RATIO = 10% of current free disk space

Under policy=Manual the guardrail is silent because the user has
already opted out of automatic ingestion — the files would just queue
as pending_decision either way.

Pieces
- inertia/lib/kb_guardrail.ts: pure decision helper with two constants
  and an evaluateGuardrail() that returns a verdict + reasons. No I/O
  on the helper itself so the logic is trivially testable
- inertia/components/KbGuardrailModal.tsx: confirmation dialog. Headless
  UI Transition + Dialog, amber 'large operation' header, plain-English
  estimate summary, [Cancel] / [Proceed anyway] footer. z-[60] so it
  layers above the tier modal underneath instead of replacing it
- inertia/components/TierSelectionModal.tsx integration: handleSubmit
  now evaluates the guardrail when policy=Always and embedEstimate is
  available; if it trips, we stash the verdict in state and render the
  guardrail modal as an overlay. Confirm runs finalizeSubmit (which is
  the pre-existing onSelectTier + onClose path); Cancel just closes the
  guardrail and leaves the tier modal as-is so the user can change
  their tier choice or flip the policy

The disk-free signal comes from the existing useSystemInfo hook +
getPrimaryDiskInfo helper. Passing freeBytes=0 (unknown) skips the
relative-disk check, so the modal still works on hosts whose disk
introspection failed — just relies on the absolute 50 GB threshold

Tests
- 9 cases in tests/unit/kb_guardrail.spec.ts: standard small batch (no
  trip), exact absolute threshold trips, over-absolute trips, over 10%
  free trips, both-at-once trips with two reasons, freeBytes=0 skip,
  freeBytes=0 + over-absolute trip, exact-10% boundary trips, just-
  under-both safe. All green.

Stacks on feat/kb-tier-estimate-on-disk (#897) — consumes that PR's
estimate endpoint to compute the verdict input. Auto-rebases to rc
when #897 merges.

Pairs with #894 (policy toggle) and #899 (JIT prompt): together the
three PRs cover the 'how do I avoid surprising the user with auto-
indexing they didn't ask for?' arc.

Out of scope (deferred)
- 6 hr time threshold (RFC §7): needs a per-host chunks-per-second
  metric we don't capture yet; would be a follow-up after Phase 4
  self-calibration (RFC §15) lands
- Wider integration (KbPolicyPromptBanner 'Index now' button, manual
  KB-modal sync): TierSelectionModal is the dominant bulk-decision
  surface and the right place to land this first
 2026-05-20 10:16:00 -07:00
Chris Sherwood
563f86a22b feat(KB): conditional warnings A + B on Stored Files (RFC #883 §6) 
Surfaces two silent failure modes that the prior binary
"any-chunks-in-Qdrant ⇒ embedded" check could not distinguish from
healthy ingestion:

- **Warning A — Zero-chunk file** (file_size > 100 MB, chunks = 0)
  Fires on video-only / image-only ZIMs (`lrnselfreliance_en_all`,
  TED talks, etc.) that the pipeline completes "successfully" with no
  extractable text. AI Assistant literally cannot reference these.

- **Warning B — Partial-embed stall** (chunks < 50% of expected from
  the ratio registry). Surfaces the simple_wiki "266 of 600,000 chunks"
  case observed during NOMAD1 ingestion testing — previously these
  looked identical to fully-completed embeds in the UI.

Both warnings render only when their condition is met (silent by
default; noisy only on real problems).

Base is `feat/kb-ratio-registry` (#891) because Warning B's "expected
chunks" estimate comes from `KbRatioRegistry.estimateChunks()`. GitHub
fast-forwards to `rc` once #891 merges.

- `app/utils/kb_warning_decision.ts` — pure `decideWarnings(inputs)`
  with thresholds (`100 MB`, `0.5×`) as exported constants. 10 unit
  tests cover the healthy case, both warnings, the under/at/over
  boundary, the registry-miss suppression, and the video-only registry
  case (`expectedChunks: 0` correctly skips Warning B).
- `RagService.computeFileWarnings()` — single Qdrant scroll tallies
  chunks per source, filesystem walk fills in zero-chunk files,
  ratio registry estimates the expectation, decision function emits.
- New endpoint `GET /api/rag/file-warnings` returns
  `Record<source, FileWarning[]>` (sources with no warnings are
  omitted, so the frontend can `warnings[source] ?? []` for clean
  defaults).
- KB modal: warnings render inline under the file name as amber-tinted
  pills. Polled every 30s alongside the existing health check.

- Warning C — chunks skipped due to length. PR #890 (#881 fix) prevents
  the silent drop at the embed boundary, so the underlying condition
  shouldn't fire anymore. If we still want to surface "we truncated
  N chunks to fit", that needs separate `skipped_count` tracking in
  EmbedFileJob — a Phase 2 follow-up.
- Suppressing Warning B during active mid-ingestion. The user can cross-
  reference the Processing Queue to know it's in-flight; suppressing
  warnings while a job runs would mask real stalls where the job died
  mid-batch. Will revisit when per-card status is wired through.
- Use of `kb_ingest_state.chunks_embedded` (#888) as the chunk count
  source. This PR uses Qdrant scroll directly so it can land
  independently of #888.

- 10 new unit tests on `decideWarnings`, all pass
- Type-check clean
- Hot-patch + browser smoke test deferred until #891 lands (the ratio
  registry needs to exist in the DB for `estimateChunks()` to return
  non-null estimates — without it, only Warning A fires which is still
  useful but Warning B stays dormant)
 2026-05-20 10:16:00 -07:00
Chris Sherwood
e68c753e39 feat(KB): surface embedding-disk estimate in curated tier-change modal (RFC #883 §1) 
When a user picks a tier in TierSelectionModal, show how much additional
disk space the AI Assistant will need if the new ZIMs are indexed, plus
a policy-aware footer explaining whether they'll auto-index (Always) or
wait for opt-in (Manual). Estimates consume #891's KbRatioRegistry via a
new POST /api/rag/estimate-batch endpoint.

Backend
- New POST /api/rag/estimate-batch route + RagController.estimateBatch
- VineJS schema accepting array of {filename, sizeBytes}, capped at 500
- KbRatioRegistry.estimateBatch aggregates via the existing prefix-match
  lookup, returns {totalChunks, totalBytes, hasUnknown}
- New BYTES_PER_CHUNK_ON_DISK constant (~8 KB: 3 KB vector + ~3 KB chunk
  text + ~2 KB payload/index overhead). Tunable; will be replaced by
  Phase 4 self-calibration once we have real measurements.
- Controller normalizes incoming filenames via path.basename so callers
  that send full paths or URLs still match registry prefixes correctly.

Frontend
- api.estimateEmbeddingBatch() client method
- TierSelectionModal: when localSelectedSlug is set, resolve the tier's
  resources (incl. inherited tiers), POST to /estimate-batch, and render
  a new info block with the +~X GB figure + ingest-policy copy. Also
  fetches rag.defaultIngestPolicy so the same block surfaces whether
  indexing will fire automatically or wait for the user.
- resourceFilename() helper extracts the basename from the resource URL
  so the registry lookup hits the right prefix regardless of mirror.

Tests
- 4 new cases in tests/unit/kb_ratio_lookup.spec.ts covering the
  estimateBatch aggregator: standard sum, unknown-flagging, video-only
  ZIM (0 chunks but known, hasUnknown stays false), empty input.

Stacks on feat/kb-ratio-registry (#891) — consumes the registry table
seeded by that PR. Once #891 merges to rc, this PR auto-rebases.

Out of scope for this PR (deferred to follow-ups):
- Per-batch opt-in checkbox (RFC §1's '☑ Also index these for AI') needs
  a per-batch policy override path and is a separate PR
- Guardrail modal at 50 GB / 10% free / 6 hr thresholds (RFC §7) is also
  separate; this PR is informational, not gating
- Time-to-embed estimate awaits a chunks-per-second metric per host
 2026-05-20 10:16:00 -07:00
chriscrosstalk
8eb8809154 feat(KB): Always/Manual ingest policy toggle (RFC #883 §1/§4) (#894) 
* feat(KB): per-file ingest state machine (Phase 1 of RFC #883)

Adds a persistent state machine for AI knowledge-base ingestion so the
scanner can distinguish "fully indexed", "user opted out", "failed", and
"stalled" from each other — none of which were derivable from the prior
binary "any chunks in Qdrant ⇒ embedded" check.

## What lands

- New table `kb_ingest_state` keyed by `file_path` with enum state column
  (`pending_decision | indexed | browse_only | failed | stalled`).
  Independent of `installed_resources` so it covers both curated downloads
  and manually-uploaded KB files.
- New KV key `rag.defaultIngestPolicy` (string: `Always | Manual`).
  Registered now but not consumed yet — JIT prompt + wizard step land in
  Phase 3 of the RFC.
- `EmbedFileJob.handle` writes state on terminal outcomes:
  - Success (final batch) → `indexed` + chunks count
  - `UnrecoverableError` → `failed` + error message
  - Retryable errors are left to BullMQ's existing retry path
- `scanAndSyncStorage` swaps the binary qdrant check for a state-aware
  decision tree (see `decideScanAction`). Existing installs auto-backfill
  on first scan: files with chunks in Qdrant but no state row become
  `indexed`; new files start as `pending_decision`.
- `deleteFileBySource` drops the state row last, so removed files
  disappear entirely instead of leaving an orphan that the next scan
  would re-dispatch into nothing.

## What does NOT land here

- Ratio registry (separate PR) — needed for partial-stall detection and
  cost estimates, but a separable concern.
- #880 follow-up initial-progress anchor (separate tiny PR).
- Phase 2 UI (status pill, per-card actions, conditional warnings).
- Phase 3 policy surfaces (wizard step, JIT prompt, guardrail modal).
- PR #886's bulk-action hookup — `_deletePointsBySource` / Re-embed All
  / Reset & Rebuild would also want to set state, but #886 isn't merged
  yet; that wiring goes in a follow-up once #886 lands.

## Target

This is forward work for v1.40.0 (RFC #883). Branching off `rc` because
that's the current latest base and post-GA Jake will sync rc→dev; a
retarget at PR-open time is a fast-forward if requested.

## Tests

- 9 new unit tests for `decideScanAction` covering all five states plus
  the no-row / chunks-present / chunks-missing combinations
- Type-check clean
- Smoke-tested end-to-end on NOMAD3 via hot-patch:
  - Backfill: 5 ZIMs + 2 KB uploads with existing chunks in Qdrant all
    came back `indexed` on first scan
  - Pending dispatch: a video-only ZIM with no chunks (`lrnselfreliance`)
    came back `pending_decision` and was correctly re-dispatched (Bull
    deduped to its historical `:completed` jobId — bgauger's #886 fix
    drains that)
  - Delete hook: deleting a KB upload via `DELETE /api/rag/files`
    removed both the disk file and the state row

* feat(KB): Always/Manual ingest policy toggle (RFC #883 §1/§4)

Activates the `rag.defaultIngestPolicy` KV registered in Phase 1
(#888) so users on a fresh install (or anyone who picks Manual mode)
no longer get every new ZIM auto-dispatched to the embed pipeline.

## Stacks on #888

This PR's base is `feat/kb-ingest-state-machine` (#888). The state
machine has to be in place for the decision function to be policy-aware;
GitHub will fast-forward the base to `rc` once #888 merges.

## Backend changes

- `decideScanAction` now takes a `policy: 'Always' | 'Manual'` argument
  (defaults to `Always` for backward compatibility).
- New `ScanAction` kind: `create_pending`. Manual mode records that the
  scanner has seen a new file (so the UI can surface a per-card Index
  affordance later) without dispatching an EmbedFileJob.
- `scanAndSyncStorage` reads the KV and passes it through. The scan-result
  log line now includes the active policy and a `waiting on user` count
  for Manual-mode hits.
- `rag.defaultIngestPolicy` added to `SETTINGS_KEYS` so it's reachable
  through the existing `GET/PATCH /api/system/settings` surface — no new
  endpoint.

## Frontend changes

- New section in the KB panel between "Why upload" and "Processing Queue":
  "Auto-index new content for AI? [Always | Manual]" — segmented radio
  with copy explaining the 5-10× disk multiplier. Default Always.
- `useQuery('ingestPolicy')` reads the current value; clicking the
  inactive option mutates and shows a notification confirming the new
  behavior.

## Tests

- 14 unit tests on `decideScanAction` (was 9) — split into Always-mode
  cases (preserves Phase 1's contract) and Manual-mode cases
  (`create_pending`, `pending_decision → skip`, etc.).
- Type-check clean.
- Hot-patch + browser verification deferred until #888 lands; the state
  machine smoke-tested cleanly on NOMAD3 in #888's PR, and this PR's
  decision-tree changes are exhaustively unit-tested.

## RFC open question §3 — policy-change re-trigger

Switching Manual → Always doesn't auto-dispatch existing `pending_decision`
rows immediately. The next scan re-evaluates and dispatches them under
the new policy. This matches the RFC's "treat the switch as I've-
thought-about-it" instinct for the guardrail; full guardrail
implementation lands in Phase 3 task 14.

---------

Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com>
 2026-05-20 10:16:00 -07:00
Chris Sherwood
43ca584b6c feat(KB): status pill + last-activity timestamp on Processing Queue (RFC #883 §5/§10) 
Each in-flight (or stuck) embedding job gets a colored health pill,
relative-activity timestamp, and chunk counter so users can tell at a
glance whether ingestion is making progress.

## Health states

- **🟢 Active** — last batch < 2 min ago
- **🟡 Slow** — last batch 2-5 min ago (CPU-paced multi-batch ingestion
  lives here naturally; not always a problem)
- **🔴 Stalled** — last batch > 5 min ago (likely real problem)
- **⚪ Waiting** — queued, no batch started yet
- **🔴 Failed** — job recorded failed status

## What lands

- New backend util `kb_job_health.ts` with pure `computeJobHealth(input)`
  decision function. Time-based thresholds (2 min / 5 min) inlined as
  constants. 9 unit tests pin the boundaries.
- `EmbedJobWithProgress` gains `lastBatchAt`, `startedAt`, `chunks` —
  already set by `EmbedFileJob.handle` on every batch transition, just
  not previously surfaced through `listActiveJobs`.
- Frontend `kb_job_health_display.ts` maps each status to a Tailwind
  dot color, label, and aria-label so backend and UI stay in sync.
- `ActiveEmbedJobs.tsx` renders the pill, "last activity Xs ago", and
  chunk counter above each progress bar. Adds a manual Refresh button
  and "Last updated Xs ago" line — the existing 2s/30s auto-poll
  cadence in `useEmbedJobs` is left intact.
- Live tick at 5s keeps the relative timestamps current without
  re-fetching from the API.

## Not in scope

- Per-card Cancel / Retry / Un-index — separate Phase 2 PR
- Conditional warnings A/B/C — separate Phase 2 PR
- Computing throughput rate (chunks/min) — needs ratio registry consumer
  (Phase 2 follow-up); for now the pill answers the "is it stuck?"
  question directly without a rate estimate.
 2026-05-20 10:16:00 -07:00
Chris Sherwood
c64ec97de4 feat(KB): group admin docs into single row in Stored Files (RFC #883 §9) 
Project NOMAD's bundled docs (`/app/docs/*.md` and `README.md`) each
embed as their own KB source — currently rendering as 12+ individual
rows that swamp user-uploaded content in the Stored Files table.
Collapse them into one informational row:

> Project NOMAD documentation · 12 files · Managed by NOMAD

The admin-docs row hides the Delete button (those files would be
re-embedded on the next sync anyway, so deleting is a footgun). User
uploads and ZIMs keep their existing per-row Delete UX.

Also adds deterministic sort: ZIMs → user uploads → admin docs → other,
alphabetical within each bucket. Pure frontend change — `/api/rag/files`
response shape unchanged.

Decision logic extracted to `kb_file_grouping.ts` with 9 unit tests
covering bucket classification, sort order, count noun pluralization,
and empty-input handling.
 2026-05-20 10:16:00 -07:00
Chris Sherwood
159d57b2af feat(KB): ratio registry for disk + time estimates (Phase 1B of RFC #883) 
Foundation for the cost estimates and partial-stall detection that
Phase 2 will surface. No consumers yet — this PR just lays the table,
the seed rows, and the lookup helper so subsequent UI work has
estimates available without a per-ZIM benchmark.

## What lands

- New table `kb_ratio_registry` (pattern, chunks_per_mb, sample_count,
  notes). Migration creates and seeds heuristic defaults from the RFC
  appendix: devdocs (1100/MB), Wikipedia variants (270/MB), iFixit
  (50/MB), Stack Exchange Q&A (200/MB), video-only ZIMs (0), plus a
  catch-all fallback at 100/MB.
- `KbRatioRegistry` model with static `lookup()` and `estimateChunks()`.
- Pure helper `kb_ratio_lookup.ts` doing longest-prefix-match — a
  specific entry (`wikipedia_en_simple_`) overrides a broader one
  (`wikipedia_en_`). 9 unit tests covering the lookup boundary.
- `sample_count` starts at 0 (heuristic seed) and is reserved for
  Phase 4 self-calibration to increment as observed ZIMs update each row.

## Not in scope

- Self-calibration on successful ingestion (Phase 4)
- UI consumers — Warning B (partial-embed stall) and the storage budget
  meter / time estimates land in Phase 2.

## Tested

- Type-check clean
- 9 unit tests pass for `findChunksPerMb` and `estimateChunkCount`
- Migration applied on NOMAD3 via hot-patch; 9 seed rows verified in DB
 2026-05-20 10:16:00 -07:00
chriscrosstalk
743549ca74 feat(KB): per-file ingest state machine (Phase 1 of RFC #883) (#888) 
Adds a persistent state machine for AI knowledge-base ingestion so the
scanner can distinguish "fully indexed", "user opted out", "failed", and
"stalled" from each other — none of which were derivable from the prior
binary "any chunks in Qdrant ⇒ embedded" check.

## What lands

- New table `kb_ingest_state` keyed by `file_path` with enum state column
  (`pending_decision | indexed | browse_only | failed | stalled`).
  Independent of `installed_resources` so it covers both curated downloads
  and manually-uploaded KB files.
- New KV key `rag.defaultIngestPolicy` (string: `Always | Manual`).
  Registered now but not consumed yet — JIT prompt + wizard step land in
  Phase 3 of the RFC.
- `EmbedFileJob.handle` writes state on terminal outcomes:
  - Success (final batch) → `indexed` + chunks count
  - `UnrecoverableError` → `failed` + error message
  - Retryable errors are left to BullMQ's existing retry path
- `scanAndSyncStorage` swaps the binary qdrant check for a state-aware
  decision tree (see `decideScanAction`). Existing installs auto-backfill
  on first scan: files with chunks in Qdrant but no state row become
  `indexed`; new files start as `pending_decision`.
- `deleteFileBySource` drops the state row last, so removed files
  disappear entirely instead of leaving an orphan that the next scan
  would re-dispatch into nothing.

## What does NOT land here

- Ratio registry (separate PR) — needed for partial-stall detection and
  cost estimates, but a separable concern.
- #880 follow-up initial-progress anchor (separate tiny PR).
- Phase 2 UI (status pill, per-card actions, conditional warnings).
- Phase 3 policy surfaces (wizard step, JIT prompt, guardrail modal).
- PR #886's bulk-action hookup — `_deletePointsBySource` / Re-embed All
  / Reset & Rebuild would also want to set state, but #886 isn't merged
  yet; that wiring goes in a follow-up once #886 lands.

## Target

This is forward work for v1.40.0 (RFC #883). Branching off `rc` because
that's the current latest base and post-GA Jake will sync rc→dev; a
retarget at PR-open time is a fast-forward if requested.

## Tests

- 9 new unit tests for `decideScanAction` covering all five states plus
  the no-row / chunks-present / chunks-missing combinations
- Type-check clean
- Smoke-tested end-to-end on NOMAD3 via hot-patch:
  - Backfill: 5 ZIMs + 2 KB uploads with existing chunks in Qdrant all
    came back `indexed` on first scan
  - Pending dispatch: a video-only ZIM with no chunks (`lrnselfreliance`)
    came back `pending_decision` and was correctly re-dispatched (Bull
    deduped to its historical `:completed` jobId — bgauger's #886 fix
    drains that)
  - Delete hook: deleting a KB upload via `DELETE /api/rag/files`
    removed both the disk file and the state row

Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com>
 2026-05-20 10:16:00 -07:00
Chris Sherwood
5e2c599c3e fix(ZIM): preserve co-existing Wikipedia corpora on cleanup (#884) 
onWikipediaDownloadComplete was deleting every file whose name starts
with `wikipedia_en_`, treating distinct corpora (simple, medicine,
wikivoyage, climate_change, etc.) as competing versions of the same
selection slot. Whichever wiki finished second silently wiped the
other from disk.

Match by filename stem instead — strip the trailing `_YYYY-MM(-DD).zim`
date suffix and only delete files with the same stem as the new
download. Different release dates of the same variant still get cleaned
up; distinct variants are preserved.

Extracted the predicate to `app/utils/zim_filename.ts` so the boundary
is covered by unit tests (8 cases incl. the #884 repro scenario).
 2026-05-20 10:16:00 -07:00
Ryanba
5517e826aa fix(UI): improve global map banner display logic (#702) 2026-05-20 10:16:00 -07:00