project-nomad/admin/database/migrations
chriscrosstalk 880c80f52f feat(kb): per-file active toggle + collection grouping/bulk toggle (#1286)
* feat(kb): add per-file active/inactive toggle for search inclusion

Implements the per-file toggle agreed on in #1119, in place of disabling
ZIM auto-discovery by default. Any indexed file (ZIM, upload, or other)
can now be excluded from RAG search results instantly, without deleting
or re-embedding its vectors.

- Vectors stay in Qdrant permanently; each point's payload carries an
  active boolean. Toggling flips this payload in place, both directions,
  with no re-embedding cost.
- Search filter is a denylist (must_not: active == false), not an
  allowlist, so a point missing the field entirely is still found by
  default -- search correctness never depends on backfill timing.
- embedAndStoreText() looks up a file's existing active state before
  writing new points, so retries, force re-embeds, and replaced-content
  reindexing never silently reactivate a file the user explicitly
  toggled off.
- Applies uniformly to all non-admin_docs buckets (ZIM, uploads, other),
  matching how the rest of this pipeline already treats buckets, rather
  than special-casing ZIM.

New POST /api/rag/files/active endpoint and KB panel toggle switch,
following the existing updateFileCollection pattern.

Real vector deletion for genuinely deleted/replaced files remains
#1170's separate concern.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* feat(kb): group personal KB uploads by collection with bulk toggle

Groups personal KB uploads into collapsible per-collection header
rows (with an "Uncategorized" bucket for files with no collection) to
reduce clutter in the file table. Each collection header also exposes
a bulk active/inactive toggle to flip every file in that collection at
once.

Also fixes three pre-existing bugs found in the collection-assignment
dropdown while building this: the popover was clipped by table cell
truncation, clipped again by the table wrapper's overflow, and the
combobox query was stale-filtering results on open.

Refs #1193

* fix(kb): widen the modal for the Active column and coerce active to a boolean

Both found by browser-verifying on NOMAD3.

The Active column adds ~57px, which put the table back over the modal width and
reintroduced the exact defect #1198 fixed: the Delete button falls off the right
edge at every viewport. Measured on the box at 1018px of table against 961px of
usable width at max-w-5xl. Widened to 6xl.

MySQL returns tinyint(1) as 0/1 rather than a boolean, and the raw 1 reached the
DOM as aria-checked="1". That is not a valid ARIA value, so a screen reader is
told nothing useful about whether the toggle is on. Coerced in the service so
every consumer sees a real boolean, with a !! guard in Switch as well.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV

* fix(kb): validate the bulk collection toggle's input before sanitising

sanitizeCollectionName ran before the type check, so a form-encoded request
sending collection[] as an array reached .trim() on a non-string and produced
an uncaught TypeError, surfacing as a 500. The vision controller already
guards this the same way.

Also documents why requiring `active` to be a real boolean is load-bearing
rather than mere input hygiene: a cross-site HTML form can only send the string
"true", so that check is what keeps a bulk Qdrant write across every point in
the index out of reach of a form POST from another origin. Worth a comment,
because coercing it would look like a harmless usability fix.

Found in a security review of the release PRs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV

---------

Co-authored-by: John Cortright <jcortright@zscaler.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-29 15:54:41 -07:00
..
1751086751801_create_services_table.ts feat: openwebui+ollama and zim management 2025-07-09 09:08:21 -07:00
1763499145832_update_services_table.ts feat(Services): friendly names and descriptions 2025-11-18 14:02:22 -08:00
1764912210741_create_curated_collections_table.ts feat: curated zim collections 2025-12-05 15:47:22 -08:00
1764912270123_create_curated_collection_resources_table.ts feat: curated zim collections 2025-12-05 15:47:22 -08:00
1768170944482_update_services_add_installation_statuses_table.ts fix(admin): improve duplicate install request handling 2026-01-13 06:58:05 -08:00
1768453747522_update_services_add_icon.ts feat: improved app cards and custom icons 2026-01-15 15:54:59 -08:00
1769097600001_create_benchmark_results_table.ts feat: Add system benchmark feature with NOMAD Score 2026-01-22 21:48:12 -08:00
1769097600002_create_benchmark_settings_table.ts feat: Add system benchmark feature with NOMAD Score 2026-01-22 21:48:12 -08:00
1769300000001_add_powered_by_and_display_order_to_services.ts feat: Use friendly app names on Dashboard with open source attribution 2026-01-22 16:43:32 -08:00
1769300000002_update_services_friendly_names.ts fix: service name defs and ollama ui location 2026-02-01 05:46:23 +00:00
1769324448000_add_builder_tag_to_benchmark_results.ts fix(migrations): Fix timestamp ordering for builder_tag migration 2026-01-28 13:43:21 -08:00
1769400000001_create_installed_tiers_table.ts feat: Improve curated collections UX with persistent tier selection 2026-01-24 15:33:50 -08:00
1769400000002_create_kv_store_table.ts feat(RAG): [wip] RAG capabilities 2026-01-31 20:39:49 -08:00
1769500000001_create_wikipedia_selection_table.ts feat: Add dedicated Wikipedia Selector with smart package management 2026-01-31 21:00:51 -08:00
1769646771604_create_create_chat_sessions_table.ts feat: [wip] native AI chat interface 2026-01-31 20:39:49 -08:00
1769646798266_create_create_chat_messages_table.ts feat: [wip] native AI chat interface 2026-01-31 20:39:49 -08:00
1769700000001_create_zim_file_metadata_table.ts feat(ContentManager): Display friendly names instead of filenames 2026-02-03 23:14:28 -08:00
1770269324176_add_unique_constraint_to_curated_collection_resources_table.ts fix(Curated Collections): ensure resources are not duplicated on fetch-latest 2026-02-04 21:35:18 -08:00
1770273423670_drop_installed_tiers_table.ts fix: rework content tier system to dynamically determine install status 2026-02-04 22:58:21 -08:00
1770849108030_create_create_collection_manifests_table.ts feat: curated content system overhaul 2026-02-11 15:44:46 -08:00
1770849119787_create_create_installed_resources_table.ts feat: curated content system overhaul 2026-02-11 15:44:46 -08:00
1770850092871_create_drop_legacy_curated_tables_table.ts feat: curated content system overhaul 2026-02-11 15:44:46 -08:00
1771000000001_add_update_fields_to_services.ts feat: support for updating services 2026-03-11 14:08:09 -07:00
1771000000002_pin_latest_service_images.ts feat: support for updating services 2026-03-11 14:08:09 -07:00
1771100000001_migrate_kiwix_to_library_mode.ts feat(Kiwix): migrate to Kiwix library mode for improved stability (#622) 2026-04-03 14:26:50 -07:00
1771200000001_create_map_markers_table.ts feat(maps): map marker management 2026-09-29 15:54:41 -07:00
1771200000002_add_visibility_and_icon_fields_to_map_markers.ts feat(maps): map marker management 2026-09-29 15:54:41 -07:00
1772000000001_add_supply_depot_fields_to_services.ts feat: supply depot 2026-06-23 04:46:45 +00:00
1772000000002_add_user_modified_to_services.ts feat: edit curated apps + fix dropdown clip + stale _old rollback 2026-06-23 04:46:46 +00:00
1772000000003_add_app_auto_update_fields_to_services.ts feat(supply-depot): opt-in automatic updates for installed apps 2026-06-23 04:46:52 +00:00
1775100000001_create_custom_library_sources_table.ts feat(Content): custom ZIM library sources with pre-seeded mirrors (#593) 2026-05-20 10:16:00 -07:00
1776000000001_create_kb_ingest_state_table.ts feat(KB): per-file ingest state machine (Phase 1 of RFC #883) (#888) 2026-05-20 10:16:00 -07:00
1776100000001_create_kb_ratio_registry_table.ts fix(KB): align chunks_per_mb column type with TS contract 2026-05-20 10:16:00 -07:00
1776200000001_add_content_auto_update_fields_to_installed_resources.ts feat(content): opt-in automatic updates for installed ZIM & map content 2026-06-23 04:47:01 +00:00
1776200000001_add_custom_url_to_services.ts feat(supply-depot): add custom launch URLs for apps 2026-06-23 04:46:56 +00:00
1776300000001_deprecate_legacy_kolibri.ts feat: replace legacy Kolibri image default with latest v19 image (#1019) 2026-06-23 04:47:07 +00:00
1776400000001_add_benchmark_harness_metadata.ts feat(benchmark): harness hardening — fail loudly + pin sysbench + record provenance (#1089) 2026-08-04 03:37:41 +00:00
1776400000001_add_collection_to_kb_ingest_state.ts feat(rag): add subject/collection organization to knowledge base (#1063) 2026-08-04 03:37:42 +00:00
1776400000001_sunset_meshtastic_daemon.ts chore(catalog): sunset orphaned Meshtastic Daemon card (#1049) 2026-08-04 03:37:31 +00:00
1776400000002_add_benchmark_score_v2.ts feat(benchmark): Score v2 app client — raws, uncapped score, v2 payload + UI 2026-08-04 03:38:30 +00:00
1776400000003_add_benchmark_platform_metadata.ts feat(benchmark): official multi-arch sysbench, resolved digest, platform metadata (#1158) 2026-08-04 03:39:53 +00:00
1778600000004_create_drug_labels_table.ts Add offline FDA drug reference (labels, interaction view, conditions, remedies) (#1040) 2026-08-04 03:38:33 +00:00
1778600000006_add_indications_fulltext_index.ts Add offline FDA drug reference (labels, interaction view, conditions, remedies) (#1040) 2026-08-04 03:38:33 +00:00
1778700000006_add_ingested_at_default_drug_labels.ts Add offline FDA drug reference (labels, interaction view, conditions, remedies) (#1040) 2026-08-04 03:38:33 +00:00
1785468975052_create_add_sources_to_chat_messages_table.ts feat: show source document and date as citations under RAG chat answers 2026-09-29 15:54:41 -07:00
1785712100000_add_active_to_kb_ingest_state.ts feat(kb): per-file active toggle + collection grouping/bulk toggle (#1286) 2026-09-29 15:54:41 -07:00
1786000000001_add_is_link_tile_to_services.ts feat(dashboard): user-added link tiles for services NOMAD does not manage 2026-09-29 15:54:41 -07:00
1786000000002_add_link_color_to_services.ts feat(dashboard): let a link tile carry a brand colour 2026-09-29 15:54:41 -07:00