mirror of
https://github.com/Crosstalk-Solutions/project-nomad.git
synced 2026-10-17 13:19:18 +02:00
dabc869aeb
29 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
chriscrosstalk
|
cdf6c00d4c
|
feat: support gated downloads for self-hosted curated content (#1172)
Adds an optional `auth: 'nomad_app_key'` discriminator to curated manifest
resources so a curated collection tier can carry content we host ourselves,
gated to official release builds. Without this, only Creator Packs could use
the entitlement Worker; curated tier installs always downloaded
unauthenticated.
No behaviour change for any existing manifest entry: absent `auth` means
unauthenticated, exactly as today.
- `auth` declared on both the type and the VineJS validator. It has to be on
the validator or VineJS strips it silently on fetch, and the gated download
would then go out with no header and 401 for everyone. A dedicated spec
guards that regression.
- Gated resources are pinned to their manifest URL (resolveZimDownload skips
the catalog comparison) and excluded from catalog update checks, so a
resource-id collision cannot let a third-party mirror overwrite our content.
Consequence, commented rather than implied: gated content does not
auto-update; new versions ship via the manifest.
- 401/403 on a download now reports that an official build is required instead
of a raw axios status, which is what a fork build will hit.
- The pure `isGatedResource` predicate is deliberately split from the
env-reading header builder: importing `#start/env` into
zim_download_resolution triggers env validation at import time and breaks its
unit tests.
Reuses CREATOR_PACKS_APP_KEY rather than minting a second secret — the question
it answers ("is this an official build?") is identical for both content types.
Verified end to end on a test server: `auth` survives validation into the
cached spec, the Bearer header attaches to only the gated resource, the file
lands byte-exact with an installed_resources row and a Kiwix library entry, and
an entry with a gated URL but no `auth` field fails with the intended message.
No catalog entry is included here. Manifests are fetched live from `main`, so a
gated entry must not merge until this ships and is adopted — pre-`auth` builds
strip the field and 401.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
chriscrosstalk
|
eda6560e00
|
feat(benchmark): official multi-arch sysbench, resolved digest, platform metadata (#1158)
Three changes that together let ARM hardware appear on the leaderboard honestly. Shipping them separately would leave ARM half-supported either way: without the image a Pi cannot submit at all, and without the architecture field it submits but is indistinguishable from x86. 1. PIN THE OFFICIAL MULTI-ARCH SYSBENCH IMAGE severalnines/sysbench publishes amd64 only, so ARM hosts could not run the System Benchmark at all — not a graceful failure, the container simply cannot execute. Apple Silicon could only run it under Rosetta emulation, which distorts the measurement it is taking, and that is what drove a community macOS fork to substitute a different benchmark and submit incomparable numbers. Swaps to ghcr.io/crosstalk-solutions/nomad-sysbench (Debian 12 + sysbench 1.0.20+ds-5, built for linux/amd64 + linux/arm64). One digest covers both architectures; verified that pulling the pinned manifest-list digest resolves to arm64 on a Raspberry Pi 5 and amd64 on x86, and that RepoDigests reports the same manifest-list digest on both — so a single allowlist entry serves both. No rescoring: 1.0.17 -> 1.0.20 measured 1.25% apart on identical hardware with identical flags (7170.18 vs 7259.56 events/sec), inside run-to-run noise and ~0.3% on a composite. Both digests are allowlisted server-side, so the fleet can cross over gradually. 2. REPORT THE DIGEST ACTUALLY RESOLVED The submission previously sent SYSBENCH_DIGEST, the constant the client was compiled with. The leaderboard validates that field, but a constant attests to how a client was BUILT rather than what it RAN, so any build inherits a valid value simply by carrying the same source. Now reads it back from the image. Uses RepoDigests (the manifest digest we pulled by), never Id — Id is the config digest, differs per architecture, and would never match the allowlist. Falls back to the constant if inspection yields nothing usable, so a benchmark never fails over provenance metadata. Still forgeable, and always will be with an open-source client. It moves the bar from "no effort" to "deliberate", which is the distinction that matters when judging whether a submission is a mistake or a choice. 3. RECORD CPU ARCHITECTURE AND OS The leaderboard is a single board across instruction sets by design, with disclosure as the fairness mechanism. Without an architecture field an ARM result sits unlabelled beside x86 — exactly what the disclosure exists to prevent. All three fields come from the Docker daemon, reusing the docker.info() call _detectRunEnvironment already makes. That is deliberate: inside the admin container os.arch() and si.osInfo() describe the CONTAINER, not the host being benchmarked. cpu_architecture Architecture x86_64 -> amd64, aarch64 -> arm64 os_version OSVersion '24.04' (already structured, no parsing) os_name OperatingSystem 'Ubuntu 24.04.4 LTS' minus the version run_environment is kept rather than replaced: "which distro" and "is this virtualised" are different questions, and WSL2 is a real performance factor. String handling lives in app/utils/platform_metadata.ts with unit tests, matching the amd_hsa_override convention, so it is testable without a Docker daemon. Unknown architectures pass through verbatim rather than being guessed at, and os_name falls back to the full description whenever the version is missing or absent from it — an over-long name is harmless, a wrong one is not. Columns are nullable and the submission fields optional, so results recorded before this shipped remain submittable. Closes #1156 Refs #1151 |
||
|
Jake Turner
|
90489e06da
|
fix(amd): coerce gfx1103 (780M) to HSA_OVERRIDE 11.0.0 so it stays on GPU (#1134)
PR #1076 stopped forcing HSA_OVERRIDE_GFX_VERSION=11.0.0 on "natively-supported" AMD iGPUs. That was correct for gfx1150/gfx1151 (Strix 890M / Strix Halo, which are in the bundled rocblas allowlist) but wrong for gfx1103 (Phoenix/Hawk Point 780M/760M), which is NOT in that list. Without the override, ollama drops the 780M with "no rocblas support for gfx target" and falls back to CPU on a fresh AI provision. Extract the gfx→HSA mapping into a pure, unit-tested util and map gfx1103 → 11.0.0 (gfx1100 kernels), the value that worked on v1.33.0 and that restores full GPU offload in the field. gfx1150/1151 stay native. Also harden the installer's 780M detection (Hawk Point / "Radeon 780M/ 760M" strings) so the gfx marker isn't silently deleted, and upgrade the no-marker fallback log from info to warn since it can mask CPU fallback. |
||
|
caweis
|
6c0271cd6f
|
refactor(drug-reference): make collections JSON the single source for curated data (#1130)
Generate app/data/{conditions,natural_remedies,home_remedies}.ts from the
repo-root collections/*.json via `npm run gen:curated-data`, so the JSON is the
only file edited by hand. The generated modules keep the data compiled into the
image (no runtime file read, no path fragility, which is why the data was a TS
constant), and curated_data_sync.standalone.ts fails CI if a generated module
ever drifts from its JSON.
This removes the burden of hand-keeping the .json mirror and the .ts constant in
sync. Follow-up to the review discussion on #1040.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Jake Turner
|
04b5c1dfd9
|
feat: auto-generating OpenAPI docs with Scalar UI (#1128) | ||
|
caweis
|
762a4a12ad
|
Add offline FDA drug reference (labels, interaction view, conditions, remedies) (#1040)
* feat(drug-reference): offline FDA drug labels, conditions, and remedies
Adds an offline medical-reference feature with three coupled layers:
- Drug Reference: full-text search over openFDA drug-label indications,
a detail page per label, and a side-by-side single-drug comparison view.
A two-phase background pipeline downloads the openFDA label parts to the
storage volume (resumable) and ingests them into the search table.
- Conditions ("When to use what"): a curated spine of first-aid situations
that maps each situation to matching OTC drugs, each linking back to its
Drug Reference detail page.
- Curated remedies: hand-authored natural and home-remedy entries drawn from
US-government public-domain sources (NCCIH, CDC, MedlinePlus, FDA), shown
with their source links and the same safety disclaimers as the rest of the
feature.
The drug-reference and conditions layers are intentionally coupled: the drug
detail page shows the situations a drug treats, and the conditions controller
reads the same drug_labels table.
Safety surfaces ship as written. The amber SafetyBanner ("informational only,
not medical advice, not an FDA endorsement, not a drug-interaction checker, in
an emergency call emergency services") renders on the condition pages and the
search page; the detail and comparison pages carry their own "not a cross-drug
interaction checker" callout; and every page carries the openFDA CC0 source
citation and no-FDA-affiliation footer.
Wiring on this branch:
- start/routes.ts: the /drug-reference and /conditions page GETs plus their
/api/* groups.
- commands/queue/work.ts: the drug-download and drug-ingest queues, both at
concurrency 1. The two drug queues get a per-queue stall override
(lockDuration 1_800_000, maxStalledCount 3) because each part is one long
stream; every other queue keeps the existing 300000 default.
- inertia/pages/home.tsx: Drug Reference and "When to use what" tiles. The
icon and display_order are a starting point, open to change.
- types/kv_store.ts: the two drugReference.* keys the pipeline reads and writes.
- package.json: yauzl and stream-json (plus their @types), used by the ingest
job to stream the label JSON out of the downloaded zips.
The app reads the conditions and remedy data from the compiled TS constants in
app/data/; the repo-root collections/*.json files are the browseable mirrors.
The natural-remedies standalone test reads collections/natural_remedies.json to
assert the two stay in sync, so that file is also a test fixture.
The four standalone tests pass (drug_interactions, drug_ingest_status,
conditions, natural_remedies). tsc reports no errors in the feature code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(drug-reference): drop a fork-internal comment reference
* fix(drug-reference): address review — defer FULLTEXT, fallback remedies prop, strip fork refs
- migration: wrap the FULLTEXT ALTER in this.defer so it runs after the
deferred createTable (was silently swallowed, index never created)
- controller: add remedies:[] to the index() error fallback (required prop)
- strip fork-internal issue/spec references from ported comments
- tsconfig: exclude tests/standalone (node --experimental-strip-types only)
- correct the varchar(768) byte-math comment; extend remedy-spine test
* fix(drug-reference): make the interaction comparison readable at five drugs
The comparison view laid its columns out on an equal-fraction CSS grid
(repeat(N, minmax(0, 1fr))), so each added drug shrank every column; at the
five-drug maximum the FDA interaction text was squeezed into unreadable slivers.
Lay the columns out with flex instead: full-width and stacked on phones, then
fixed-width columns that scroll sideways from the sm: breakpoint up, so they
never shrink below a readable width. Theme the columns with the same palette as
the rest of the page (they were on stock gray), and give the headers a fixed
min-height so columns line up when drug names wrap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(collections): route a 'dataset' tier resource to the drug pipeline
Add an optional `type` discriminator to SpecResource ('zim' | 'dataset',
absent == 'zim'), so the tier installer can carry a DB-ingested resource
alongside ZIM files. ZimService.downloadCategoryTier branches on it: a
'dataset' resource dispatches the existing FDA download+ingest pipeline
instead of RunDownloadJob, guarded against duplicate dispatch by the drug
ingest status. Every existing manifest entry has no `type` and keeps the
exact ZIM path.
Widen InstalledResource.resource_type to include 'dataset' and exclude
dataset rows from the ZIM/map catalog-update scan (datasets aren't
filename-versioned; their freshness path is separate). No dataset rows are
written yet: the InstalledResource 'dataset' row on ingest-ready, the
manifest entry, install-gating, and the downloads-aggregator integration
are follow-up commits.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drug-reference): rework into an opt-in medicine-standard tier dataset
Reshapes the offline FDA drug reference from a built-in feature into curated
content installed by selecting the Medicine / Standard tier, per maintainer
direction.
- install-state: the ingest writes an installed_resources 'dataset' row on
ready (version = the openFDA export_date), threaded installer to download to
ingest; the tier-status math and the home-tile gate read it. Manual ingests
write no row, so install-state stays tied to the curated path.
- manifest: declare the dataset in the medicine-standard tier (runtime fetches
the remote manifest, so this also needs to land upstream).
- install-gating: the drug-reference home tiles render only when installed.
- uninstall: DrugReferenceService.uninstall() stops the two drug queues, deletes
the on-disk parts, truncates drug_labels (schema kept), clears the KV markers,
and drops the install row. Best-effort, logged, scoped to drug data only.
- downloads: the download phase reports the canonical {percent, downloadedBytes,
totalBytes} shape as one drug-data card in the Active Downloads aggregator with
cancel/remove; the heavy ingest stays in the IngestStatus surface with an
Indexing handoff on the card.
- auto-update: a daily DrugAutoUpdateJob compares the manifest export_date and
re-downloads when newer, gated on installed + no active job.
typecheck clean; the drug standalone suites pass. Three points are flagged in
code for the maintainer: the InstalledResource 'dataset' approach, the tier
home, and the export_date string format.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(downloads): keep the drug download card live across all parts
The Active Downloads card filtered to the deterministic jobId, but the
download's continuations run under auto-generated jobIds (only part 0 uses the
deterministic one). So the card tracked part 0 and then vanished while parts
2..N kept downloading. The queue is concurrency 1, so collapse to whichever
single part is in flight and report the deterministic jobId: one card tracks
aggregate progress through the whole download and cancel/remove still routes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drug-reference): co-locate a persistent safety note with affirmative remedy guidance
Add RemedySafetyNote at the head of every natural-remedy section — the two on
Drug Reference and the one on "When to use what" — so the "informational only,
not medical advice, seek real medical care in an emergency" framing appears with
the guidance itself, not only in the page-top banner. Replaces the terse
per-section caveat with the same amber alert language as SafetyBanner.
Addresses the upstream #1040 review request that the disclaimer be unmistakable
and present wherever affirmative self-care guidance appears, not a one-time banner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(drug-reference): fold dataset freshness into the content-auto-update path
The drug dataset auto-updated on its own daily cron that ignored the
contentAutoUpdate.* master switch, so it would refresh even with content
auto-update turned off, and it didn't ride the content-update path the way
ZIMs and maps do.
Move the export_date freshness/apply orchestration onto
DrugReferenceService.attemptAutoUpdate(), add
ContentAutoUpdateService.attemptDrugDataset() gated on the same enabled +
window config, and have the hourly ContentAutoUpdateJob drive both. Retire
the standalone DrugAutoUpdateJob. The ZIM/map attempt() path is unchanged.
Addresses the upstream #1040 request to wire the openFDA export_date check
into the content updater so the dataset updates alongside ZIMs and maps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drug-reference): strengthen the remedy note to consult a clinician before combining with meds
Widen the affirmative-remedy safety note from "talk to a clinician before use"
to explicitly cover using a remedy AND combining one with a medication the user
already takes — the interaction case is the higher-risk path for an off-grid
user self-treating.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drug-reference): gate affirmative remedy content behind an off-by-default flag
Add drugReference.remediesEnabled (default off), independent of the tier
install. When off, the server emits no remedy data at any boundary — the
drug-reference page prop, conditions show, and the /api/conditions/drugs
situation search — and the "Natural" filter is hidden, so installing the
medicine-standard tier lights up the verbatim FDA label search and the
condition-to-OTC matching but not the hand-authored self-care and herbal
sections. No user-facing toggle: it is flipped on after a clinician content-pass.
Implements the upstream #1040 split-by-risk request: the regulated label content
ships with the tier; the authored remedy guidance stays gated until sign-off.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com>
|
||
|
NgoQuocViet2001
|
42e1d584ef
|
fix(content): resolve current ZIM URL before download (#1091)
* fix(content): resolve current ZIM URL before download * fix: compare ZIM catalog versions numerically |
||
|
just-jbc
|
ac141a33b4
|
feat(rag): add subject/collection organization to knowledge base (#1063)
* feat(rag): add subject/collection organization to knowledge base - Add nullable collection field to KbIngestState, propagated through the embed job, RAG service, and Qdrant point payloads (indexed for filtering) - Add upload-time category selection and per-file collection reassignment in the Knowledge Base modal, with a filterable Stored Files table - Add a 'Search in' collection filter to the chat interface, threaded through to searchSimilarDocuments as an optional Qdrant filter - Fix .docx extraction: previously routed through raw-text extraction (garbage output for a ZIP-based XML format); adds a proper mammoth-based extractor and a dedicated 'docx' file-type case * feat(rag): support dynamic KB collection creation, rename, and removal Extends collection organization with a Manage Collections UI: collections are created on the fly when a file is assigned to a new name, can be renamed (bulk-updates every tagged file and Qdrant point), and can be removed (reassigns tagged files back to Uncategorized rather than deleting anything). * fix(rag): use dynamic collections query in chat search filter chat/index.tsx still imported the static KB_COLLECTIONS constant for its 'Search in' dropdown, inconsistent with KnowledgeBaseModal.tsx which already uses the live getKnowledgeCollections() query. Renamed/added collections via the new Manage Collections UI weren't reflected in the chat filter. * feat(rag): broaden preset tags and add creatable collection combobox Replaces the survival-specific preset list with general-purpose starter tags (recipes, diy, health, technology, finance, travel, hobbies, reference, survival, energy) so the Knowledge Base reads well for home-lab/reference use, not just prepping. Adds sanitizeCollectionName() (trim, lowercase, length cap) applied on every write path server-side, and a dependency-free CollectionCombobox component replacing the plain <select> + window.prompt pattern for tagging — autocompletes against presets + tags already in use, with a '+ Create' option for anything new. * chore(rag): remove .docx fix from this branch, split into #1100 Per review feedback, the .docx extraction fix is unrelated to the collections feature and can merge independently. Moved to a standalone PR (Crosstalk-Solutions/project-nomad#1100) off dev. * chore: remove unrelated diff noise (lockfile, comments, indentation) --------- Co-authored-by: John Cortright <jcortright@zscaler.com> |
||
|
chriscrosstalk
|
ab87e29dd5
|
chore(KB): filter non-content sections + render tables in ZIM extraction (#1044)
Closes #902. Two gaps in the structured ZIM extraction path: 1. NON_CONTENT_HEADING_PATTERNS was only used by the structure heuristic to count meaningful sections, never at section-emit time. Sections under "See also" / "References" / "External links" / etc. were still chunked and embedded. They're now flagged when the heading opens and dropped. 2. <table> elements were run through cheerio's `.text()`, concatenating every cell with no separators ("AgeDoseAdult500mg") into unsearchable word salad. New tableToText() joins cells with " | " and rows with newlines so row/column structure survives into the chunk. Refactor: moved extractStructuredContent out of ZIMExtractionService into a pure, cheerio-only util (app/utils/zim_html.ts) so it can be unit-tested without the native @openzim/libzim binding. Service delegates to it; behavior is otherwise unchanged. Adds tests/unit/zim_html.spec.ts (6 tests). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Benjamin Smith
|
5181637926
|
feat(KnowledgeBase): add document viewer, download, metadata, and sorting (#721)
Rebuilt on top of dev's RFC #883 state-machine UI rather than the now-defunct StoredFile shape: - Extend StoredFileInfo with fileName/size/uploadedAt/isUserUpload - Populate metadata from on-disk stats in RagService.getStoredFiles - Add fileSourceSchema validator + getFileContent/downloadFile endpoints scoped to the uploads directory only (tighter than the original PR — matches docs_service traversal pattern) - KnowledgeBaseModal: sortable Size and Uploaded columns; View/Download buttons on upload-bucket rows; new FileViewerModal for in-browser text preview. Bucket grouping preserved — sort applies within each bucket. - Use formatBytes from ~/lib/util rather than redefining |
||
|
jakeaturner
|
6a2d4c2bf6
|
feat(content): opt-in automatic updates for installed ZIM & map content | ||
|
Chris Sherwood
|
bbd62d8ed1
|
fix(content): remove superseded curated map/ZIM files when a new version installs
Only Wikipedia had version cleanup; every other curated map and non-Wikipedia ZIM left its prior version on disk when a newer one installed, so users silently accumulated orphaned content (potentially hundreds of GB). (#634) The install paths already record each resource via InstalledResource {resource_id, resource_type, version, file_path}, so the authoritative old-file path for a resource is known. On install of a new version we now capture the prior row before updateOrCreate repoints it, then delete the old file — gated behind a pure, fully unit-tested decision function with strict safety rails: - tracked-only: requires a prior InstalledResource row for the same resource_id, so sideloaded/untracked files are never touched - genuine replacement: old and new file paths must differ - new-file-verified: the new file must be confirmed on disk first - strictly-newer: a re-install or downgrade can't wipe a newer file - within-storage-dir: the old path must resolve under the content store ZIM cleanup deletes the old file directly (NOT via this.delete(), which would drop the InstalledResource row by resource_id that updateOrCreate just repointed) and rebuilds the Kiwix library only if a file was actually removed, so its XML never references a deleted ZIM. Maps need no library step. Wikipedia keeps its own existing cleanup path. All deletions are best-effort and logged; a failure never breaks the install. Closes #634 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris Sherwood
|
ca5ec1767f
|
fix(security): harden assertNotPrivateUrl with ipaddr.js + host normalization
Replaces the regex blocklist in assertNotPrivateUrl with ipaddr.js range classification and normalizes the host before checking it. Consolidates two community proposals (#930 ipaddr.js parsing, #912 trailing-dot normalization) into one validator so the SSRF-critical path lives in-house with full tests. - Classify literal IPs by range (loopback / linkLocal / unspecified) via ipaddr.js instead of a hand-maintained regex list, which also catches alternate IPv4 encodings and avoids over-blocking mapped public IPs (the old `::ffff:` regex blocked every mapped address, including public ones). IPv4- mapped IPv6 is reduced to its embedded IPv4 before classification. - Strip a trailing root dot from the host so `localhost.` / `127.0.0.1.` can't bypass the checks (they resolve to the same target as the dotless form, #911). - Strip IPv6 brackets and lowercase for the localhost comparison. - RFC1918, bare LAN hostnames (e.g. `nomad3`), and external FQDNs remain allowed — LAN appliances need them, and DNS rebinding is a fetch-time concern outside this guard's scope. Adds a consolidated unit spec covering loopback/link-local/unspecified literals, alternate encodings, IPv4-mapped v6, mixed-case + trailing-dot localhost, and the allowed LAN/FQDN/mapped-public cases. Resolves #922. Supersedes #930 and #912 (thanks @Gujiassh and @luyua9). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Chris Sherwood
|
3977c723c2
|
fix(KB): stop partial_stall warning firing on atypical ZIMs (link-out/PDF-heavy)
The Stored Files "partial stall" warning compares chunks in Qdrant against an expected count from the ratio registry. The registry has an empty-pattern catch-all (100 chunks/MB) that matches any filename, so a ZIM that matches no specific pattern still gets a size-based estimate. For archives that are mostly PDFs, images, or link-out stubs (e.g. irp.fas.org military-medicine), byte size wildly over-predicts embeddable text: a 75 MB ZIM estimates ~7,236 chunks but produces ~1, tripping a false "ingestion may have stalled" warning that re-embed can't clear. The catch-all is fine for rough aggregate disk-cost estimates, but it should not drive a per-file stall signal. Add an `ignoreCatchAll` option to the ratio lookup that excludes the empty-pattern row (returning null when only the fallback would match), and use it in the warnings path so partial_stall only fires when the registry has a *specific* expectation for the file. Files that match a real pattern (wikipedia_, devdocs_, ifixit_, ...) are unaffected; disk-cost/batch estimates keep using the fallback. Closes #913 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
jakeaturner
|
cf8db6218d
|
feat(supply-depot): opt-in automatic updates for installed apps | ||
|
Jake Turner
|
be434d755a
|
feat: supply depot | ||
|
jakeaturner
|
736c9bd672 |
fix(security): canonicalize hostnames to block IPv4-mapped IPv6 IMDS bypass
Replace literal string matching with ipaddr.js parsing so equivalent encodings of 169.254.169.254 (::ffff:169.254.169.254, ::ffff:a9fe:a9fe,fully-expanded forms) and fd00:ec2::254 are all rejected. |
||
|
Chris Sherwood
|
d850cb9588 |
feat(KB): per-file ingest action + state indicator on Stored Files (RFC #883 §5)
Closes the Manual-mode UX dead-end: after toggling 'Auto-index new content for AI?' to Manual, a freshly-downloaded ZIM (or any pending_decision file) had no UI path to opt in for embedding short of the global Sync Storage / Re-embed All bulk actions. Per RFC #883 §5, each Stored Files row now carries a state pill and an adaptive single-button action. State pill (left of any existing warning chips): - 'Indexed' — green; row had chunks in Qdrant or state row is 'indexed' - 'Not Indexed' — neutral; state is pending_decision or browse_only - 'Failed' — red - 'Stalled' — amber - admin_docs collapsed row has no pill ('Managed by NOMAD' carries it) Adaptive action button (paired with the existing Delete button per row): - pending_decision → 'Index' (force=false) - browse_only → 'Index' (force=true) - failed / stalled → 'Retry' (force=true) - indexed + warning chip → 'Re-embed' (force=true; confirm modal first) - indexed healthy / null → no action button (bulk Re-embed All covers it) Backend: GET /api/rag/files now returns { files: Array<{ source, state, chunksEmbedded }> } instead of a flat string[]. State + chunk-count come from a single KbIngestState query unioned into the existing Qdrant-derived source list (no new round trips). New POST /api/rag/files/embed validates the source is known, refuses if any inflight job already targets the same filePath (prevents double-click duplicate-chunk hazard), pre-deletes Qdrant points when force=true, then dispatches via the existing _dispatchEmbedJobsFor helper used by reembedAll. Per-file Re-embed (force=true on an already-indexed file) routes through a StyledModal confirmation since it deletes existing vectors before queueing a fresh job — same destructive-action weight as Delete's inline confirm but heavier since it affects search until the rebuild finishes. Folds in PR #907's blank-screen fix because my new render needs the same generic restored: `<StyledTable<KbFileGroup>>` and `record.displayName` (instead of the unresolved `sourceToDisplayName(record.source)` that ships in rc.5 and ReferenceErrors on modal open). PR #907 also adds title tooltips on the three bulk-action buttons; those tooltips are NOT included here — let PR #907 land first or independently for that part. Multi-select bulk-opt-in deferred per discussion: most Manual-mode users ingest 1-2 files at a time, the existing global toggle covers the bulk case, and checkboxes would expand scope past what rc.6 should hold. Will file a follow-up issue for an 'Index N pending files' single-click button once this lands. Tests-in-PR scope was limited to keeping `kb_file_grouping.spec.ts` green after the StoredFileInfo[] signature change (added asInfos() wrapper). Dedicated unit tests for embedSingleFile (unknown source / inflight refused / force=true delete-then-dispatch) and the new state-pill rendering will land in a follow-up PR alongside Playwright coverage of the row actions. Verification path: NOMAD3 currently runs project-nomad-admin:integration- rc6-preview (PRs #907 + #908 atop rc.5). After this branch is built into a new integration tag, I'll re-run targeted Playwright UAT on the KB modal covering: state pill rendering per state, Index click on pending_decision opts in cleanly, Retry on failed re-dispatches successfully, Re-embed confirmation modal copy + delete-then-dispatch on the military-medicine partial-stall row, and Delete flow untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris Sherwood
|
cf3a924b9f |
feat(KB): guardrail modal at 50GB / 10%-free thresholds (RFC #883 §7)
One-time confirmation step gating bulk indexing actions that would consume a substantial amount of disk for embedding storage. Fires only when the user has policy=Always (i.e., the system would auto-index) AND the estimate trips either: - GUARDRAIL_ABSOLUTE_BYTES = 50 GB embedding cost, OR - GUARDRAIL_FREE_DISK_RATIO = 10% of current free disk space Under policy=Manual the guardrail is silent because the user has already opted out of automatic ingestion — the files would just queue as pending_decision either way. Pieces - inertia/lib/kb_guardrail.ts: pure decision helper with two constants and an evaluateGuardrail() that returns a verdict + reasons. No I/O on the helper itself so the logic is trivially testable - inertia/components/KbGuardrailModal.tsx: confirmation dialog. Headless UI Transition + Dialog, amber 'large operation' header, plain-English estimate summary, [Cancel] / [Proceed anyway] footer. z-[60] so it layers above the tier modal underneath instead of replacing it - inertia/components/TierSelectionModal.tsx integration: handleSubmit now evaluates the guardrail when policy=Always and embedEstimate is available; if it trips, we stash the verdict in state and render the guardrail modal as an overlay. Confirm runs finalizeSubmit (which is the pre-existing onSelectTier + onClose path); Cancel just closes the guardrail and leaves the tier modal as-is so the user can change their tier choice or flip the policy The disk-free signal comes from the existing useSystemInfo hook + getPrimaryDiskInfo helper. Passing freeBytes=0 (unknown) skips the relative-disk check, so the modal still works on hosts whose disk introspection failed — just relies on the absolute 50 GB threshold Tests - 9 cases in tests/unit/kb_guardrail.spec.ts: standard small batch (no trip), exact absolute threshold trips, over-absolute trips, over 10% free trips, both-at-once trips with two reasons, freeBytes=0 skip, freeBytes=0 + over-absolute trip, exact-10% boundary trips, just- under-both safe. All green. Stacks on feat/kb-tier-estimate-on-disk (#897) — consumes that PR's estimate endpoint to compute the verdict input. Auto-rebases to rc when #897 merges. Pairs with #894 (policy toggle) and #899 (JIT prompt): together the three PRs cover the 'how do I avoid surprising the user with auto- indexing they didn't ask for?' arc. Out of scope (deferred) - 6 hr time threshold (RFC §7): needs a per-host chunks-per-second metric we don't capture yet; would be a follow-up after Phase 4 self-calibration (RFC §15) lands - Wider integration (KbPolicyPromptBanner 'Index now' button, manual KB-modal sync): TierSelectionModal is the dominant bulk-decision surface and the right place to land this first |
||
|
Chris Sherwood
|
563f86a22b |
feat(KB): conditional warnings A + B on Stored Files (RFC #883 §6)
Surfaces two silent failure modes that the prior binary "any-chunks-in-Qdrant ⇒ embedded" check could not distinguish from healthy ingestion: - **Warning A — Zero-chunk file** (file_size > 100 MB, chunks = 0) Fires on video-only / image-only ZIMs (`lrnselfreliance_en_all`, TED talks, etc.) that the pipeline completes "successfully" with no extractable text. AI Assistant literally cannot reference these. - **Warning B — Partial-embed stall** (chunks < 50% of expected from the ratio registry). Surfaces the simple_wiki "266 of 600,000 chunks" case observed during NOMAD1 ingestion testing — previously these looked identical to fully-completed embeds in the UI. Both warnings render only when their condition is met (silent by default; noisy only on real problems). Base is `feat/kb-ratio-registry` (#891) because Warning B's "expected chunks" estimate comes from `KbRatioRegistry.estimateChunks()`. GitHub fast-forwards to `rc` once #891 merges. - `app/utils/kb_warning_decision.ts` — pure `decideWarnings(inputs)` with thresholds (`100 MB`, `0.5×`) as exported constants. 10 unit tests cover the healthy case, both warnings, the under/at/over boundary, the registry-miss suppression, and the video-only registry case (`expectedChunks: 0` correctly skips Warning B). - `RagService.computeFileWarnings()` — single Qdrant scroll tallies chunks per source, filesystem walk fills in zero-chunk files, ratio registry estimates the expectation, decision function emits. - New endpoint `GET /api/rag/file-warnings` returns `Record<source, FileWarning[]>` (sources with no warnings are omitted, so the frontend can `warnings[source] ?? []` for clean defaults). - KB modal: warnings render inline under the file name as amber-tinted pills. Polled every 30s alongside the existing health check. - Warning C — chunks skipped due to length. PR #890 (#881 fix) prevents the silent drop at the embed boundary, so the underlying condition shouldn't fire anymore. If we still want to surface "we truncated N chunks to fit", that needs separate `skipped_count` tracking in EmbedFileJob — a Phase 2 follow-up. - Suppressing Warning B during active mid-ingestion. The user can cross- reference the Processing Queue to know it's in-flight; suppressing warnings while a job runs would mask real stalls where the job died mid-batch. Will revisit when per-card status is wired through. - Use of `kb_ingest_state.chunks_embedded` (#888) as the chunk count source. This PR uses Qdrant scroll directly so it can land independently of #888. - 10 new unit tests on `decideWarnings`, all pass - Type-check clean - Hot-patch + browser smoke test deferred until #891 lands (the ratio registry needs to exist in the DB for `estimateChunks()` to return non-null estimates — without it, only Warning A fires which is still useful but Warning B stays dormant) |
||
|
Chris Sherwood
|
e68c753e39 |
feat(KB): surface embedding-disk estimate in curated tier-change modal (RFC #883 §1)
When a user picks a tier in TierSelectionModal, show how much additional disk space the AI Assistant will need if the new ZIMs are indexed, plus a policy-aware footer explaining whether they'll auto-index (Always) or wait for opt-in (Manual). Estimates consume #891's KbRatioRegistry via a new POST /api/rag/estimate-batch endpoint. Backend - New POST /api/rag/estimate-batch route + RagController.estimateBatch - VineJS schema accepting array of {filename, sizeBytes}, capped at 500 - KbRatioRegistry.estimateBatch aggregates via the existing prefix-match lookup, returns {totalChunks, totalBytes, hasUnknown} - New BYTES_PER_CHUNK_ON_DISK constant (~8 KB: 3 KB vector + ~3 KB chunk text + ~2 KB payload/index overhead). Tunable; will be replaced by Phase 4 self-calibration once we have real measurements. - Controller normalizes incoming filenames via path.basename so callers that send full paths or URLs still match registry prefixes correctly. Frontend - api.estimateEmbeddingBatch() client method - TierSelectionModal: when localSelectedSlug is set, resolve the tier's resources (incl. inherited tiers), POST to /estimate-batch, and render a new info block with the +~X GB figure + ingest-policy copy. Also fetches rag.defaultIngestPolicy so the same block surfaces whether indexing will fire automatically or wait for the user. - resourceFilename() helper extracts the basename from the resource URL so the registry lookup hits the right prefix regardless of mirror. Tests - 4 new cases in tests/unit/kb_ratio_lookup.spec.ts covering the estimateBatch aggregator: standard sum, unknown-flagging, video-only ZIM (0 chunks but known, hasUnknown stays false), empty input. Stacks on feat/kb-ratio-registry (#891) — consumes the registry table seeded by that PR. Once #891 merges to rc, this PR auto-rebases. Out of scope for this PR (deferred to follow-ups): - Per-batch opt-in checkbox (RFC §1's '☑ Also index these for AI') needs a per-batch policy override path and is a separate PR - Guardrail modal at 50 GB / 10% free / 6 hr thresholds (RFC §7) is also separate; this PR is informational, not gating - Time-to-embed estimate awaits a chunks-per-second metric per host |
||
|
chriscrosstalk
|
8eb8809154 |
feat(KB): Always/Manual ingest policy toggle (RFC #883 §1/§4) (#894)
* feat(KB): per-file ingest state machine (Phase 1 of RFC #883) Adds a persistent state machine for AI knowledge-base ingestion so the scanner can distinguish "fully indexed", "user opted out", "failed", and "stalled" from each other — none of which were derivable from the prior binary "any chunks in Qdrant ⇒ embedded" check. ## What lands - New table `kb_ingest_state` keyed by `file_path` with enum state column (`pending_decision | indexed | browse_only | failed | stalled`). Independent of `installed_resources` so it covers both curated downloads and manually-uploaded KB files. - New KV key `rag.defaultIngestPolicy` (string: `Always | Manual`). Registered now but not consumed yet — JIT prompt + wizard step land in Phase 3 of the RFC. - `EmbedFileJob.handle` writes state on terminal outcomes: - Success (final batch) → `indexed` + chunks count - `UnrecoverableError` → `failed` + error message - Retryable errors are left to BullMQ's existing retry path - `scanAndSyncStorage` swaps the binary qdrant check for a state-aware decision tree (see `decideScanAction`). Existing installs auto-backfill on first scan: files with chunks in Qdrant but no state row become `indexed`; new files start as `pending_decision`. - `deleteFileBySource` drops the state row last, so removed files disappear entirely instead of leaving an orphan that the next scan would re-dispatch into nothing. ## What does NOT land here - Ratio registry (separate PR) — needed for partial-stall detection and cost estimates, but a separable concern. - #880 follow-up initial-progress anchor (separate tiny PR). - Phase 2 UI (status pill, per-card actions, conditional warnings). - Phase 3 policy surfaces (wizard step, JIT prompt, guardrail modal). - PR #886's bulk-action hookup — `_deletePointsBySource` / Re-embed All / Reset & Rebuild would also want to set state, but #886 isn't merged yet; that wiring goes in a follow-up once #886 lands. ## Target This is forward work for v1.40.0 (RFC #883). Branching off `rc` because that's the current latest base and post-GA Jake will sync rc→dev; a retarget at PR-open time is a fast-forward if requested. ## Tests - 9 new unit tests for `decideScanAction` covering all five states plus the no-row / chunks-present / chunks-missing combinations - Type-check clean - Smoke-tested end-to-end on NOMAD3 via hot-patch: - Backfill: 5 ZIMs + 2 KB uploads with existing chunks in Qdrant all came back `indexed` on first scan - Pending dispatch: a video-only ZIM with no chunks (`lrnselfreliance`) came back `pending_decision` and was correctly re-dispatched (Bull deduped to its historical `:completed` jobId — bgauger's #886 fix drains that) - Delete hook: deleting a KB upload via `DELETE /api/rag/files` removed both the disk file and the state row * feat(KB): Always/Manual ingest policy toggle (RFC #883 §1/§4) Activates the `rag.defaultIngestPolicy` KV registered in Phase 1 (#888) so users on a fresh install (or anyone who picks Manual mode) no longer get every new ZIM auto-dispatched to the embed pipeline. ## Stacks on #888 This PR's base is `feat/kb-ingest-state-machine` (#888). The state machine has to be in place for the decision function to be policy-aware; GitHub will fast-forward the base to `rc` once #888 merges. ## Backend changes - `decideScanAction` now takes a `policy: 'Always' | 'Manual'` argument (defaults to `Always` for backward compatibility). - New `ScanAction` kind: `create_pending`. Manual mode records that the scanner has seen a new file (so the UI can surface a per-card Index affordance later) without dispatching an EmbedFileJob. - `scanAndSyncStorage` reads the KV and passes it through. The scan-result log line now includes the active policy and a `waiting on user` count for Manual-mode hits. - `rag.defaultIngestPolicy` added to `SETTINGS_KEYS` so it's reachable through the existing `GET/PATCH /api/system/settings` surface — no new endpoint. ## Frontend changes - New section in the KB panel between "Why upload" and "Processing Queue": "Auto-index new content for AI? [Always | Manual]" — segmented radio with copy explaining the 5-10× disk multiplier. Default Always. - `useQuery('ingestPolicy')` reads the current value; clicking the inactive option mutates and shows a notification confirming the new behavior. ## Tests - 14 unit tests on `decideScanAction` (was 9) — split into Always-mode cases (preserves Phase 1's contract) and Manual-mode cases (`create_pending`, `pending_decision → skip`, etc.). - Type-check clean. - Hot-patch + browser verification deferred until #888 lands; the state machine smoke-tested cleanly on NOMAD3 in #888's PR, and this PR's decision-tree changes are exhaustively unit-tested. ## RFC open question §3 — policy-change re-trigger Switching Manual → Always doesn't auto-dispatch existing `pending_decision` rows immediately. The next scan re-evaluates and dispatches them under the new policy. This matches the RFC's "treat the switch as I've- thought-about-it" instinct for the guardrail; full guardrail implementation lands in Phase 3 task 14. --------- Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com> |
||
|
Chris Sherwood
|
43ca584b6c |
feat(KB): status pill + last-activity timestamp on Processing Queue (RFC #883 §5/§10)
Each in-flight (or stuck) embedding job gets a colored health pill, relative-activity timestamp, and chunk counter so users can tell at a glance whether ingestion is making progress. ## Health states - **🟢 Active** — last batch < 2 min ago - **🟡 Slow** — last batch 2-5 min ago (CPU-paced multi-batch ingestion lives here naturally; not always a problem) - **🔴 Stalled** — last batch > 5 min ago (likely real problem) - **⚪ Waiting** — queued, no batch started yet - **🔴 Failed** — job recorded failed status ## What lands - New backend util `kb_job_health.ts` with pure `computeJobHealth(input)` decision function. Time-based thresholds (2 min / 5 min) inlined as constants. 9 unit tests pin the boundaries. - `EmbedJobWithProgress` gains `lastBatchAt`, `startedAt`, `chunks` — already set by `EmbedFileJob.handle` on every batch transition, just not previously surfaced through `listActiveJobs`. - Frontend `kb_job_health_display.ts` maps each status to a Tailwind dot color, label, and aria-label so backend and UI stay in sync. - `ActiveEmbedJobs.tsx` renders the pill, "last activity Xs ago", and chunk counter above each progress bar. Adds a manual Refresh button and "Last updated Xs ago" line — the existing 2s/30s auto-poll cadence in `useEmbedJobs` is left intact. - Live tick at 5s keeps the relative timestamps current without re-fetching from the API. ## Not in scope - Per-card Cancel / Retry / Un-index — separate Phase 2 PR - Conditional warnings A/B/C — separate Phase 2 PR - Computing throughput rate (chunks/min) — needs ratio registry consumer (Phase 2 follow-up); for now the pill answers the "is it stuck?" question directly without a rate estimate. |
||
|
Chris Sherwood
|
c64ec97de4 |
feat(KB): group admin docs into single row in Stored Files (RFC #883 §9)
Project NOMAD's bundled docs (`/app/docs/*.md` and `README.md`) each embed as their own KB source — currently rendering as 12+ individual rows that swamp user-uploaded content in the Stored Files table. Collapse them into one informational row: > Project NOMAD documentation · 12 files · Managed by NOMAD The admin-docs row hides the Delete button (those files would be re-embedded on the next sync anyway, so deleting is a footgun). User uploads and ZIMs keep their existing per-row Delete UX. Also adds deterministic sort: ZIMs → user uploads → admin docs → other, alphabetical within each bucket. Pure frontend change — `/api/rag/files` response shape unchanged. Decision logic extracted to `kb_file_grouping.ts` with 9 unit tests covering bucket classification, sort order, count noun pluralization, and empty-input handling. |
||
|
Chris Sherwood
|
159d57b2af |
feat(KB): ratio registry for disk + time estimates (Phase 1B of RFC #883)
Foundation for the cost estimates and partial-stall detection that Phase 2 will surface. No consumers yet — this PR just lays the table, the seed rows, and the lookup helper so subsequent UI work has estimates available without a per-ZIM benchmark. ## What lands - New table `kb_ratio_registry` (pattern, chunks_per_mb, sample_count, notes). Migration creates and seeds heuristic defaults from the RFC appendix: devdocs (1100/MB), Wikipedia variants (270/MB), iFixit (50/MB), Stack Exchange Q&A (200/MB), video-only ZIMs (0), plus a catch-all fallback at 100/MB. - `KbRatioRegistry` model with static `lookup()` and `estimateChunks()`. - Pure helper `kb_ratio_lookup.ts` doing longest-prefix-match — a specific entry (`wikipedia_en_simple_`) overrides a broader one (`wikipedia_en_`). 9 unit tests covering the lookup boundary. - `sample_count` starts at 0 (heuristic seed) and is reserved for Phase 4 self-calibration to increment as observed ZIMs update each row. ## Not in scope - Self-calibration on successful ingestion (Phase 4) - UI consumers — Warning B (partial-embed stall) and the storage budget meter / time estimates land in Phase 2. ## Tested - Type-check clean - 9 unit tests pass for `findChunksPerMb` and `estimateChunkCount` - Migration applied on NOMAD3 via hot-patch; 9 seed rows verified in DB |
||
|
chriscrosstalk
|
743549ca74 |
feat(KB): per-file ingest state machine (Phase 1 of RFC #883) (#888)
Adds a persistent state machine for AI knowledge-base ingestion so the scanner can distinguish "fully indexed", "user opted out", "failed", and "stalled" from each other — none of which were derivable from the prior binary "any chunks in Qdrant ⇒ embedded" check. ## What lands - New table `kb_ingest_state` keyed by `file_path` with enum state column (`pending_decision | indexed | browse_only | failed | stalled`). Independent of `installed_resources` so it covers both curated downloads and manually-uploaded KB files. - New KV key `rag.defaultIngestPolicy` (string: `Always | Manual`). Registered now but not consumed yet — JIT prompt + wizard step land in Phase 3 of the RFC. - `EmbedFileJob.handle` writes state on terminal outcomes: - Success (final batch) → `indexed` + chunks count - `UnrecoverableError` → `failed` + error message - Retryable errors are left to BullMQ's existing retry path - `scanAndSyncStorage` swaps the binary qdrant check for a state-aware decision tree (see `decideScanAction`). Existing installs auto-backfill on first scan: files with chunks in Qdrant but no state row become `indexed`; new files start as `pending_decision`. - `deleteFileBySource` drops the state row last, so removed files disappear entirely instead of leaving an orphan that the next scan would re-dispatch into nothing. ## What does NOT land here - Ratio registry (separate PR) — needed for partial-stall detection and cost estimates, but a separable concern. - #880 follow-up initial-progress anchor (separate tiny PR). - Phase 2 UI (status pill, per-card actions, conditional warnings). - Phase 3 policy surfaces (wizard step, JIT prompt, guardrail modal). - PR #886's bulk-action hookup — `_deletePointsBySource` / Re-embed All / Reset & Rebuild would also want to set state, but #886 isn't merged yet; that wiring goes in a follow-up once #886 lands. ## Target This is forward work for v1.40.0 (RFC #883). Branching off `rc` because that's the current latest base and post-GA Jake will sync rc→dev; a retarget at PR-open time is a fast-forward if requested. ## Tests - 9 new unit tests for `decideScanAction` covering all five states plus the no-row / chunks-present / chunks-missing combinations - Type-check clean - Smoke-tested end-to-end on NOMAD3 via hot-patch: - Backfill: 5 ZIMs + 2 KB uploads with existing chunks in Qdrant all came back `indexed` on first scan - Pending dispatch: a video-only ZIM with no chunks (`lrnselfreliance`) came back `pending_decision` and was correctly re-dispatched (Bull deduped to its historical `:completed` jobId — bgauger's #886 fix drains that) - Delete hook: deleting a KB upload via `DELETE /api/rag/files` removed both the disk file and the state row Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com> |
||
|
Chris Sherwood
|
5e2c599c3e |
fix(ZIM): preserve co-existing Wikipedia corpora on cleanup (#884)
onWikipediaDownloadComplete was deleting every file whose name starts with `wikipedia_en_`, treating distinct corpora (simple, medicine, wikivoyage, climate_change, etc.) as competing versions of the same selection slot. Whichever wiki finished second silently wiped the other from disk. Match by filename stem instead — strip the trailing `_YYYY-MM(-DD).zim` date suffix and only delete files with the same stem as the new download. Different release dates of the same variant still get cleaned up; distinct variants are preserved. Extracted the predicate to `app/utils/zim_filename.ts` so the boundary is covered by unit tests (8 cases incl. the #884 repro scenario). |
||
|
Ryanba
|
5517e826aa | fix(UI): improve global map banner display logic (#702) | ||
|
Jake Turner
|
b33a1b3e37 | feat: initial commit |