mirror of
https://github.com/Crosstalk-Solutions/project-nomad.git
synced 2026-10-17 05:14:19 +02:00
8eb8809154
* feat(KB): per-file ingest state machine (Phase 1 of RFC #883) Adds a persistent state machine for AI knowledge-base ingestion so the scanner can distinguish "fully indexed", "user opted out", "failed", and "stalled" from each other — none of which were derivable from the prior binary "any chunks in Qdrant ⇒ embedded" check. ## What lands - New table `kb_ingest_state` keyed by `file_path` with enum state column (`pending_decision | indexed | browse_only | failed | stalled`). Independent of `installed_resources` so it covers both curated downloads and manually-uploaded KB files. - New KV key `rag.defaultIngestPolicy` (string: `Always | Manual`). Registered now but not consumed yet — JIT prompt + wizard step land in Phase 3 of the RFC. - `EmbedFileJob.handle` writes state on terminal outcomes: - Success (final batch) → `indexed` + chunks count - `UnrecoverableError` → `failed` + error message - Retryable errors are left to BullMQ's existing retry path - `scanAndSyncStorage` swaps the binary qdrant check for a state-aware decision tree (see `decideScanAction`). Existing installs auto-backfill on first scan: files with chunks in Qdrant but no state row become `indexed`; new files start as `pending_decision`. - `deleteFileBySource` drops the state row last, so removed files disappear entirely instead of leaving an orphan that the next scan would re-dispatch into nothing. ## What does NOT land here - Ratio registry (separate PR) — needed for partial-stall detection and cost estimates, but a separable concern. - #880 follow-up initial-progress anchor (separate tiny PR). - Phase 2 UI (status pill, per-card actions, conditional warnings). - Phase 3 policy surfaces (wizard step, JIT prompt, guardrail modal). - PR #886's bulk-action hookup — `_deletePointsBySource` / Re-embed All / Reset & Rebuild would also want to set state, but #886 isn't merged yet; that wiring goes in a follow-up once #886 lands. ## Target This is forward work for v1.40.0 (RFC #883). Branching off `rc` because that's the current latest base and post-GA Jake will sync rc→dev; a retarget at PR-open time is a fast-forward if requested. ## Tests - 9 new unit tests for `decideScanAction` covering all five states plus the no-row / chunks-present / chunks-missing combinations - Type-check clean - Smoke-tested end-to-end on NOMAD3 via hot-patch: - Backfill: 5 ZIMs + 2 KB uploads with existing chunks in Qdrant all came back `indexed` on first scan - Pending dispatch: a video-only ZIM with no chunks (`lrnselfreliance`) came back `pending_decision` and was correctly re-dispatched (Bull deduped to its historical `:completed` jobId — bgauger's #886 fix drains that) - Delete hook: deleting a KB upload via `DELETE /api/rag/files` removed both the disk file and the state row * feat(KB): Always/Manual ingest policy toggle (RFC #883 §1/§4) Activates the `rag.defaultIngestPolicy` KV registered in Phase 1 (#888) so users on a fresh install (or anyone who picks Manual mode) no longer get every new ZIM auto-dispatched to the embed pipeline. ## Stacks on #888 This PR's base is `feat/kb-ingest-state-machine` (#888). The state machine has to be in place for the decision function to be policy-aware; GitHub will fast-forward the base to `rc` once #888 merges. ## Backend changes - `decideScanAction` now takes a `policy: 'Always' | 'Manual'` argument (defaults to `Always` for backward compatibility). - New `ScanAction` kind: `create_pending`. Manual mode records that the scanner has seen a new file (so the UI can surface a per-card Index affordance later) without dispatching an EmbedFileJob. - `scanAndSyncStorage` reads the KV and passes it through. The scan-result log line now includes the active policy and a `waiting on user` count for Manual-mode hits. - `rag.defaultIngestPolicy` added to `SETTINGS_KEYS` so it's reachable through the existing `GET/PATCH /api/system/settings` surface — no new endpoint. ## Frontend changes - New section in the KB panel between "Why upload" and "Processing Queue": "Auto-index new content for AI? [Always | Manual]" — segmented radio with copy explaining the 5-10× disk multiplier. Default Always. - `useQuery('ingestPolicy')` reads the current value; clicking the inactive option mutates and shows a notification confirming the new behavior. ## Tests - 14 unit tests on `decideScanAction` (was 9) — split into Always-mode cases (preserves Phase 1's contract) and Manual-mode cases (`create_pending`, `pending_decision → skip`, etc.). - Type-check clean. - Hot-patch + browser verification deferred until #888 lands; the state machine smoke-tested cleanly on NOMAD3 in #888's PR, and this PR's decision-tree changes are exhaustively unit-tested. ## RFC open question §3 — policy-change re-trigger Switching Manual → Always doesn't auto-dispatch existing `pending_decision` rows immediately. The next scan re-evaluates and dispatches them under the new policy. This matches the RFC's "treat the switch as I've- thought-about-it" instinct for the guardrail; full guardrail implementation lands in Phase 3 task 14. --------- Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com>
83 lines
3.1 KiB
TypeScript
83 lines
3.1 KiB
TypeScript
import * as assert from 'node:assert/strict'
|
|
import { test } from 'node:test'
|
|
|
|
import { decideScanAction } from '../../app/utils/kb_ingest_decision.js'
|
|
|
|
// ---------- Always-policy cases (default behavior; preserves pre-policy install) ----------
|
|
|
|
test('Always: no state row, no chunks → dispatch and create row (new file)', () => {
|
|
assert.deepEqual(decideScanAction(null, false, 'Always'), {
|
|
kind: 'dispatch',
|
|
createStateRow: true,
|
|
})
|
|
})
|
|
|
|
test('Always: no state row, chunks present → backfill_indexed (pre-RFC install, existing Qdrant volume)', () => {
|
|
assert.deepEqual(decideScanAction(null, true, 'Always'), { kind: 'backfill_indexed' })
|
|
})
|
|
|
|
test('Always: indexed + chunks present → skip', () => {
|
|
assert.deepEqual(decideScanAction({ state: 'indexed' }, true, 'Always'), { kind: 'skip' })
|
|
})
|
|
|
|
test('Always: indexed + chunks missing → re-dispatch (Qdrant collection rebuilt or chunks deleted)', () => {
|
|
assert.deepEqual(decideScanAction({ state: 'indexed' }, false, 'Always'), {
|
|
kind: 'dispatch',
|
|
createStateRow: false,
|
|
})
|
|
})
|
|
|
|
test('Always: pending_decision → dispatch', () => {
|
|
assert.deepEqual(decideScanAction({ state: 'pending_decision' }, false, 'Always'), {
|
|
kind: 'dispatch',
|
|
createStateRow: false,
|
|
})
|
|
})
|
|
|
|
test('Always: browse_only → skip (user opted out of indexing)', () => {
|
|
assert.deepEqual(decideScanAction({ state: 'browse_only' }, false, 'Always'), { kind: 'skip' })
|
|
})
|
|
|
|
test('Always: failed → skip (manual retry needed, do not auto-redispatch)', () => {
|
|
assert.deepEqual(decideScanAction({ state: 'failed' }, false, 'Always'), { kind: 'skip' })
|
|
})
|
|
|
|
test('Always: stalled → skip (manual retry needed)', () => {
|
|
assert.deepEqual(decideScanAction({ state: 'stalled' }, false, 'Always'), { kind: 'skip' })
|
|
})
|
|
|
|
// ---------- Manual-policy cases ----------
|
|
|
|
test('Manual: no state row, no chunks → create_pending (do not auto-dispatch new content)', () => {
|
|
assert.deepEqual(decideScanAction(null, false, 'Manual'), { kind: 'create_pending' })
|
|
})
|
|
|
|
test('Manual: no state row, chunks present → backfill_indexed (same as Always — Qdrant is authoritative)', () => {
|
|
assert.deepEqual(decideScanAction(null, true, 'Manual'), { kind: 'backfill_indexed' })
|
|
})
|
|
|
|
test('Manual: pending_decision → skip (waiting for user to opt in via Index button)', () => {
|
|
assert.deepEqual(decideScanAction({ state: 'pending_decision' }, false, 'Manual'), {
|
|
kind: 'skip',
|
|
})
|
|
})
|
|
|
|
test('Manual: indexed + chunks missing → re-dispatch (user has already opted in for this file)', () => {
|
|
// Policy switch from Always→Manual must not break in-flight or partially-deleted indexes
|
|
// for files the user previously chose to index.
|
|
assert.deepEqual(decideScanAction({ state: 'indexed' }, false, 'Manual'), {
|
|
kind: 'dispatch',
|
|
createStateRow: false,
|
|
})
|
|
})
|
|
|
|
test('Manual: browse_only → skip (same as Always)', () => {
|
|
assert.deepEqual(decideScanAction({ state: 'browse_only' }, false, 'Manual'), { kind: 'skip' })
|
|
})
|
|
|
|
// ---------- Policy default ----------
|
|
|
|
test('omitted policy defaults to Always (unset KV preserves legacy behavior)', () => {
|
|
assert.deepEqual(decideScanAction(null, false), { kind: 'dispatch', createStateRow: true })
|
|
})
|