mirror of
https://github.com/Crosstalk-Solutions/project-nomad.git
synced 2026-10-17 05:14:19 +02:00
feat(kb): per-file active toggle + collection grouping/bulk toggle (#1286)
* feat(kb): add per-file active/inactive toggle for search inclusion Implements the per-file toggle agreed on in #1119, in place of disabling ZIM auto-discovery by default. Any indexed file (ZIM, upload, or other) can now be excluded from RAG search results instantly, without deleting or re-embedding its vectors. - Vectors stay in Qdrant permanently; each point's payload carries an active boolean. Toggling flips this payload in place, both directions, with no re-embedding cost. - Search filter is a denylist (must_not: active == false), not an allowlist, so a point missing the field entirely is still found by default -- search correctness never depends on backfill timing. - embedAndStoreText() looks up a file's existing active state before writing new points, so retries, force re-embeds, and replaced-content reindexing never silently reactivate a file the user explicitly toggled off. - Applies uniformly to all non-admin_docs buckets (ZIM, uploads, other), matching how the rest of this pipeline already treats buckets, rather than special-casing ZIM. New POST /api/rag/files/active endpoint and KB panel toggle switch, following the existing updateFileCollection pattern. Real vector deletion for genuinely deleted/replaced files remains #1170's separate concern. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(kb): group personal KB uploads by collection with bulk toggle Groups personal KB uploads into collapsible per-collection header rows (with an "Uncategorized" bucket for files with no collection) to reduce clutter in the file table. Each collection header also exposes a bulk active/inactive toggle to flip every file in that collection at once. Also fixes three pre-existing bugs found in the collection-assignment dropdown while building this: the popover was clipped by table cell truncation, clipped again by the table wrapper's overflow, and the combobox query was stale-filtering results on open. Refs #1193 * fix(kb): widen the modal for the Active column and coerce active to a boolean Both found by browser-verifying on NOMAD3. The Active column adds ~57px, which put the table back over the modal width and reintroduced the exact defect #1198 fixed: the Delete button falls off the right edge at every viewport. Measured on the box at 1018px of table against 961px of usable width at max-w-5xl. Widened to 6xl. MySQL returns tinyint(1) as 0/1 rather than a boolean, and the raw 1 reached the DOM as aria-checked="1". That is not a valid ARIA value, so a screen reader is told nothing useful about whether the toggle is on. Coerced in the service so every consumer sees a real boolean, with a !! guard in Switch as well. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV * fix(kb): validate the bulk collection toggle's input before sanitising sanitizeCollectionName ran before the type check, so a form-encoded request sending collection[] as an array reached .trim() on a non-string and produced an uncaught TypeError, surfacing as a 500. The vision controller already guards this the same way. Also documents why requiring `active` to be a real boolean is load-bearing rather than mere input hygiene: a cross-site HTML form can only send the string "true", so that check is what keeps a bulk Qdrant write across every point in the index out of reach of a form POST from another origin. Worth a comment, because coercing it would look like a harmless usability fix. Found in a security review of the release PRs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV --------- Co-authored-by: John Cortright <jcortright@zscaler.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
chriscrosstalk
Jake Turner
parent
1b5d73f64e
commit
880c80f52f
|
|
@ -94,6 +94,57 @@ export default class RagController {
|
|||
return response.status(200).json({ message: result.message })
|
||||
}
|
||||
|
||||
public async setFileActive({ request, response }: HttpContext) {
|
||||
const source: string | null = request.input('source', null)
|
||||
const active: boolean | null = request.input('active', null)
|
||||
|
||||
if (!source || typeof active !== 'boolean') {
|
||||
return response.status(400).json({ error: 'source and active are required.' })
|
||||
}
|
||||
|
||||
const result = await this.ragService.setFileActive(source, active)
|
||||
if (!result.success) {
|
||||
return response.status(500).json({ error: result.message })
|
||||
}
|
||||
return response.status(200).json({ message: result.message })
|
||||
}
|
||||
|
||||
/** Bulk active/inactive toggle for every file tagged with a KB collection
|
||||
* (the personal-upload `collection` field) -- distinct from the unrelated
|
||||
* curated ZIM-pack "collections" feature. `collection: null` targets the
|
||||
* "Uncategorized" bucket. */
|
||||
public async setKnowledgeCollectionActive({ request, response }: HttpContext) {
|
||||
const rawCollection = request.input('collection', undefined)
|
||||
const active: boolean | null = request.input('active', null)
|
||||
|
||||
// Validate before sanitising. Form-encoded input can yield an array
|
||||
// (`collection[]=a&collection[]=b`), and sanitizeCollectionName would then
|
||||
// throw on `.trim()` and surface as an uncaught 500.
|
||||
//
|
||||
// Requiring `active` to be a real boolean is also load-bearing beyond input
|
||||
// hygiene: a cross-site HTML form can only send the string "true", so this
|
||||
// check is what keeps a bulk Qdrant write across every point in the index
|
||||
// out of reach of a form POST from another origin. Do not "helpfully"
|
||||
// coerce it.
|
||||
if (rawCollection === undefined || typeof active !== 'boolean') {
|
||||
return response.status(400).json({ error: 'collection and active are required.' })
|
||||
}
|
||||
|
||||
if (rawCollection !== null && typeof rawCollection !== 'string') {
|
||||
return response.status(400).json({ error: 'collection must be a string or null.' })
|
||||
}
|
||||
|
||||
const collection = rawCollection === null ? null : sanitizeCollectionName(rawCollection)
|
||||
|
||||
const result = await this.ragService.setKnowledgeCollectionActive(collection, active)
|
||||
if (!result.success) {
|
||||
return response.status(500).json({ error: result.message })
|
||||
}
|
||||
return response
|
||||
.status(200)
|
||||
.json({ message: result.message, affectedCount: result.affectedCount })
|
||||
}
|
||||
|
||||
public async renameKnowledgeCollection({ request, response }: HttpContext) {
|
||||
const oldName = sanitizeCollectionName(request.input('oldName', null))
|
||||
const newName = sanitizeCollectionName(request.input('newName', null))
|
||||
|
|
|
|||
|
|
@ -31,6 +31,9 @@ export default class KbIngestState extends BaseModel {
|
|||
@column()
|
||||
declare collection: string | null
|
||||
|
||||
@column()
|
||||
declare active: boolean
|
||||
|
||||
@column()
|
||||
declare last_error: string | null
|
||||
|
||||
|
|
@ -43,7 +46,7 @@ export default class KbIngestState extends BaseModel {
|
|||
static async getOrCreate(filePath: string, collection?: string): Promise<KbIngestState> {
|
||||
return this.firstOrCreate(
|
||||
{ file_path: filePath },
|
||||
{ file_path: filePath, state: 'pending_decision', chunks_embedded: 0, collection: collection ?? null }
|
||||
{ file_path: filePath, state: 'pending_decision', chunks_embedded: 0, collection: collection ?? null, active: true }
|
||||
)
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -4,7 +4,14 @@ import { inject } from '@adonisjs/core'
|
|||
import logger from '@adonisjs/core/services/logger'
|
||||
import { TokenChunker } from '@chonkiejs/core'
|
||||
import sharp from 'sharp'
|
||||
import { deleteFileIfExists, determineFileType, getFile, getFileStatsIfExists, listDirectoryContentsRecursive, ZIM_STORAGE_PATH } from '../utils/fs.js'
|
||||
import {
|
||||
deleteFileIfExists,
|
||||
determineFileType,
|
||||
getFile,
|
||||
getFileStatsIfExists,
|
||||
listDirectoryContentsRecursive,
|
||||
ZIM_STORAGE_PATH,
|
||||
} from '../utils/fs.js'
|
||||
import { PDFParse } from 'pdf-parse'
|
||||
import { createWorker } from 'tesseract.js'
|
||||
import { fromBuffer } from 'pdf2pic'
|
||||
|
|
@ -40,7 +47,12 @@ import { ZIMExtractionService } from './zim_extraction_service.js'
|
|||
import { ZIM_BATCH_SIZE } from '../../constants/zim_extraction.js'
|
||||
import { hasMoreArticleBatches } from '../utils/zim_batch_decision.js'
|
||||
import { EMBEDDING_MODEL_NAME } from '../../constants/ollama.js'
|
||||
import { ProcessAndEmbedFileResponse, ProcessZIMFileResponse, RAGResult, RerankedRAGResult } from '../../types/rag.js'
|
||||
import {
|
||||
ProcessAndEmbedFileResponse,
|
||||
ProcessZIMFileResponse,
|
||||
RAGResult,
|
||||
RerankedRAGResult,
|
||||
} from '../../types/rag.js'
|
||||
|
||||
export type EmbedSingleFileFailureCode =
|
||||
| 'not_found'
|
||||
|
|
@ -78,8 +90,8 @@ export class RagService {
|
|||
public static CHUNK_OVERLAP_TOKENS = 150
|
||||
public static PREFIX_TOKEN_BUDGET = 10 // Reserve ~10 tokens for prefixes
|
||||
public static CHAR_TO_TOKEN_RATIO = 2 // Conservative chars-per-token estimate; technical docs
|
||||
// (numbers, symbols, abbreviations) tokenize denser
|
||||
// than plain prose (~3), so 2 avoids context overflows
|
||||
// (numbers, symbols, abbreviations) tokenize denser
|
||||
// than plain prose (~3), so 2 avoids context overflows
|
||||
// Nomic Embed Text v1.5 uses task-specific prefixes for optimal performance
|
||||
public static SEARCH_DOCUMENT_PREFIX = 'search_document: '
|
||||
public static SEARCH_QUERY_PREFIX = 'search_query: '
|
||||
|
|
@ -88,14 +100,16 @@ export class RagService {
|
|||
constructor(
|
||||
private dockerService: DockerService,
|
||||
private ollamaService: OllamaService
|
||||
) { }
|
||||
) {}
|
||||
|
||||
private async _initializeQdrantClient() {
|
||||
if (!this.qdrantInitPromise) {
|
||||
this.qdrantInitPromise = (async () => {
|
||||
const qdrantUrl = await this.dockerService.getServiceURL(SERVICE_NAMES.QDRANT)
|
||||
if (!qdrantUrl) {
|
||||
throw new Error('Qdrant vector database is offline. Restart the AI Assistant service in Settings to restore the Knowledge Base.')
|
||||
throw new Error(
|
||||
'Qdrant vector database is offline. Restart the AI Assistant service in Settings to restore the Knowledge Base.'
|
||||
)
|
||||
}
|
||||
this.qdrant = new QdrantClient({ url: qdrantUrl })
|
||||
})().catch((err) => {
|
||||
|
|
@ -119,7 +133,8 @@ export class RagService {
|
|||
this.ensuredCollections.clear()
|
||||
return {
|
||||
online: false,
|
||||
message: 'Qdrant vector database is offline. Restart the AI Assistant service in Settings to restore the Knowledge Base.',
|
||||
message:
|
||||
'Qdrant vector database is offline. Restart the AI Assistant service in Settings to restore the Knowledge Base.',
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -166,6 +181,21 @@ export class RagService {
|
|||
field_name: 'collection',
|
||||
field_schema: 'keyword',
|
||||
})
|
||||
await this.qdrant!.createPayloadIndex(collectionName, {
|
||||
field_name: 'active',
|
||||
field_schema: 'bool',
|
||||
})
|
||||
|
||||
// Backfill: stamp `active: true` on any point that predates this field.
|
||||
// `is_empty` (not a match-all filter) means this only ever touches points
|
||||
// that have never been explicitly set, so re-running this on every boot
|
||||
// can never clobber a user's explicit toggle-off. This is data hygiene
|
||||
// only — search correctness never depends on it having run (see the
|
||||
// must_not filter in searchSimilarDocuments).
|
||||
await this.qdrant!.setPayload(collectionName, {
|
||||
payload: { active: true },
|
||||
filter: { must: [{ is_empty: { key: 'active' } }] },
|
||||
})
|
||||
|
||||
// Only memoize after every step succeeded, so a partial failure is retried
|
||||
this.ensuredCollections.add(collectionName)
|
||||
|
|
@ -183,15 +213,17 @@ export class RagService {
|
|||
* - Control characters (except newlines, tabs, and carriage returns)
|
||||
*/
|
||||
private sanitizeText(text: string): string {
|
||||
return text
|
||||
// Null bytes
|
||||
.replace(/\x00/g, '')
|
||||
// Problematic control characters (keep \n, \r, \t)
|
||||
.replace(/[\x01-\x08\x0B-\x0C\x0E-\x1F\x7F]/g, '')
|
||||
// Invalid Unicode surrogates
|
||||
.replace(/[\uD800-\uDFFF]/g, '')
|
||||
// Trim extra whitespace
|
||||
.trim()
|
||||
return (
|
||||
text
|
||||
// Null bytes
|
||||
.replace(/\x00/g, '')
|
||||
// Problematic control characters (keep \n, \r, \t)
|
||||
.replace(/[\x01-\x08\x0B-\x0C\x0E-\x1F\x7F]/g, '')
|
||||
// Invalid Unicode surrogates
|
||||
.replace(/[\uD800-\uDFFF]/g, '')
|
||||
// Trim extra whitespace
|
||||
.trim()
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -245,33 +277,33 @@ export class RagService {
|
|||
* TODO: We could probably move this to a separate QueryPreprocessor class if it grows more complex, but for now it's manageable here.
|
||||
*/
|
||||
private static QUERY_EXPANSION_DICTIONARY: Record<string, string> = {
|
||||
'bob': 'bug out bag',
|
||||
'bov': 'bug out vehicle',
|
||||
'bol': 'bug out location',
|
||||
'edc': 'every day carry',
|
||||
'mre': 'meal ready to eat',
|
||||
'shtf': 'shit hits the fan',
|
||||
'teotwawki': 'the end of the world as we know it',
|
||||
'opsec': 'operational security',
|
||||
'ifak': 'individual first aid kit',
|
||||
'ghb': 'get home bag',
|
||||
'ghi': 'get home in',
|
||||
'wrol': 'without rule of law',
|
||||
'emp': 'electromagnetic pulse',
|
||||
'ham': 'ham amateur radio',
|
||||
'nbr': 'nuclear biological radiological',
|
||||
'cbrn': 'chemical biological radiological nuclear',
|
||||
'sar': 'search and rescue',
|
||||
'comms': 'communications radio',
|
||||
'fifo': 'first in first out',
|
||||
'mylar': 'mylar bag food storage',
|
||||
'paracord': 'paracord 550 cord',
|
||||
'ferro': 'ferro rod fire starter',
|
||||
'bivvy': 'bivvy bivy emergency shelter',
|
||||
'bdu': 'battle dress uniform',
|
||||
'gmrs': 'general mobile radio service',
|
||||
'frs': 'family radio service',
|
||||
'nbc': 'nuclear biological chemical',
|
||||
bob: 'bug out bag',
|
||||
bov: 'bug out vehicle',
|
||||
bol: 'bug out location',
|
||||
edc: 'every day carry',
|
||||
mre: 'meal ready to eat',
|
||||
shtf: 'shit hits the fan',
|
||||
teotwawki: 'the end of the world as we know it',
|
||||
opsec: 'operational security',
|
||||
ifak: 'individual first aid kit',
|
||||
ghb: 'get home bag',
|
||||
ghi: 'get home in',
|
||||
wrol: 'without rule of law',
|
||||
emp: 'electromagnetic pulse',
|
||||
ham: 'ham amateur radio',
|
||||
nbr: 'nuclear biological radiological',
|
||||
cbrn: 'chemical biological radiological nuclear',
|
||||
sar: 'search and rescue',
|
||||
comms: 'communications radio',
|
||||
fifo: 'first in first out',
|
||||
mylar: 'mylar bag food storage',
|
||||
paracord: 'paracord 550 cord',
|
||||
ferro: 'ferro rod fire starter',
|
||||
bivvy: 'bivvy bivy emergency shelter',
|
||||
bdu: 'battle dress uniform',
|
||||
gmrs: 'general mobile radio service',
|
||||
frs: 'family radio service',
|
||||
nbc: 'nuclear biological chemical',
|
||||
}
|
||||
|
||||
private preprocessQuery(query: string): string {
|
||||
|
|
@ -400,9 +432,14 @@ export class RagService {
|
|||
const batchStart = batchIdx * batchSize
|
||||
const batch = prefixedChunks.slice(batchStart, batchStart + batchSize)
|
||||
|
||||
logger.debug(`[RAG] Embedding batch ${batchIdx + 1}/${totalBatches} (${batch.length} chunks)`)
|
||||
logger.debug(
|
||||
`[RAG] Embedding batch ${batchIdx + 1}/${totalBatches} (${batch.length} chunks)`
|
||||
)
|
||||
|
||||
const response = await this.ollamaService.embed(this.resolvedEmbeddingModel ?? EMBEDDING_MODEL_NAME, batch)
|
||||
const response = await this.ollamaService.embed(
|
||||
this.resolvedEmbeddingModel ?? EMBEDDING_MODEL_NAME,
|
||||
batch
|
||||
)
|
||||
|
||||
embeddings.push(...response.embeddings)
|
||||
|
||||
|
|
@ -413,6 +450,21 @@ export class RagService {
|
|||
}
|
||||
|
||||
const timestamp = Date.now()
|
||||
|
||||
// Sanitize source metadata once up front — every chunk in this call shares
|
||||
// the same source.
|
||||
const sanitizedSource =
|
||||
typeof metadata.source === 'string' ? this.sanitizeText(metadata.source) : 'unknown'
|
||||
|
||||
// Preserve an existing file's active/inactive toggle across re-ingestion
|
||||
// (retries, force re-embeds, replaced-content reindexing, etc.) instead of
|
||||
// resetting it to active on every write. A genuinely new file (no row yet)
|
||||
// still defaults to active.
|
||||
const existingIngestState = await KbIngestState.query()
|
||||
.where('file_path', sanitizedSource)
|
||||
.first()
|
||||
const active = existingIngestState ? existingIngestState.active : true
|
||||
|
||||
const points = chunks.map((chunkText, index) => {
|
||||
// Sanitize text to prevent JSON encoding errors
|
||||
const sanitizedText = this.sanitizeText(chunkText)
|
||||
|
|
@ -433,14 +485,11 @@ export class RagService {
|
|||
|
||||
logger.debug(`[RAG] Extracted keywords for chunk ${index}: [${allKeywords.join(', ')}]`)
|
||||
if (structuralKeywords.length > 0) {
|
||||
logger.debug(`[RAG] - Structural: [${structuralKeywords.join(', ')}], Content: [${contentKeywords.join(', ')}]`)
|
||||
logger.debug(
|
||||
`[RAG] - Structural: [${structuralKeywords.join(', ')}], Content: [${contentKeywords.join(', ')}]`
|
||||
)
|
||||
}
|
||||
|
||||
// Sanitize source metadata as well
|
||||
const sanitizedSource = typeof metadata.source === 'string'
|
||||
? this.sanitizeText(metadata.source)
|
||||
: 'unknown'
|
||||
|
||||
return {
|
||||
id: randomUUID(), // qdrant requires either uuid or unsigned int
|
||||
vector: embeddings[index],
|
||||
|
|
@ -452,7 +501,8 @@ export class RagService {
|
|||
keywords: allKeywords.join(' '), // store as space-separated string for text search
|
||||
char_count: sanitizedText.length,
|
||||
created_at: timestamp,
|
||||
source: sanitizedSource
|
||||
source: sanitizedSource,
|
||||
active,
|
||||
},
|
||||
}
|
||||
})
|
||||
|
|
@ -728,9 +778,7 @@ export class RagService {
|
|||
})
|
||||
|
||||
// If no spine found, fall back to all manifest items
|
||||
const contentFiles = spineOrder.length > 0
|
||||
? spineOrder
|
||||
: Array.from(manifestItems.values())
|
||||
const contentFiles = spineOrder.length > 0 ? spineOrder : Array.from(manifestItems.values())
|
||||
|
||||
// Extract text from each content file in order
|
||||
const textParts: string[] = []
|
||||
|
|
@ -749,7 +797,9 @@ export class RagService {
|
|||
}
|
||||
|
||||
const fullText = textParts.join('\n\n')
|
||||
logger.debug(`[RAG] EPUB extracted ${textParts.length} chapters, ${fullText.length} characters total`)
|
||||
logger.debug(
|
||||
`[RAG] EPUB extracted ${textParts.length} chapters, ${fullText.length} characters total`
|
||||
)
|
||||
return fullText
|
||||
}
|
||||
|
||||
|
|
@ -761,13 +811,20 @@ export class RagService {
|
|||
collection?: string
|
||||
): Promise<{ success: boolean; message: string; chunks?: number }> {
|
||||
if (!extractedText || extractedText.trim().length === 0) {
|
||||
return { success: false, message: 'Process completed succesfully, but no text was found to embed.' }
|
||||
return {
|
||||
success: false,
|
||||
message: 'Process completed succesfully, but no text was found to embed.',
|
||||
}
|
||||
}
|
||||
|
||||
const embedResult = await this.embedAndStoreText(extractedText, {
|
||||
source: filepath,
|
||||
...(collection ? { collection } : {})
|
||||
}, onProgress)
|
||||
const embedResult = await this.embedAndStoreText(
|
||||
extractedText,
|
||||
{
|
||||
source: filepath,
|
||||
...(collection ? { collection } : {}),
|
||||
},
|
||||
onProgress
|
||||
)
|
||||
|
||||
if (!embedResult) {
|
||||
return { success: false, message: 'Failed to embed and store the extracted text.' }
|
||||
|
|
@ -788,7 +845,7 @@ export class RagService {
|
|||
/**
|
||||
* Main pipeline to process and embed an uploaded file into the RAG knowledge base.
|
||||
* This includes text extraction, chunking, embedding, and storing in Qdrant.
|
||||
*
|
||||
*
|
||||
* Orchestrates file type detection and delegates to specialized processors.
|
||||
* For ZIM files, supports batch processing via batchOffset parameter.
|
||||
*/
|
||||
|
|
@ -844,12 +901,16 @@ export class RagService {
|
|||
|
||||
// Extraction done — scale remaining embedding progress from 15% to 100%
|
||||
if (onProgress) await onProgress(15)
|
||||
const scaledProgress = onProgress
|
||||
? (p: number) => onProgress(15 + p * 0.85)
|
||||
: undefined
|
||||
const scaledProgress = onProgress ? (p: number) => onProgress(15 + p * 0.85) : undefined
|
||||
|
||||
// Embed extracted text and cleanup
|
||||
return await this.embedTextAndCleanup(extractedText, filepath, deleteAfterEmbedding, scaledProgress, collection)
|
||||
return await this.embedTextAndCleanup(
|
||||
extractedText,
|
||||
filepath,
|
||||
deleteAfterEmbedding,
|
||||
scaledProgress,
|
||||
collection
|
||||
)
|
||||
} catch (error) {
|
||||
logger.error('[RAG] Error processing and embedding file:', error)
|
||||
return { success: false, message: 'Error processing and embedding file.' }
|
||||
|
|
@ -913,9 +974,7 @@ export class RagService {
|
|||
allModels.find((model) => model.name.toLowerCase().includes('nomic-embed-text'))
|
||||
|
||||
if (!embeddingModel) {
|
||||
logger.warn(
|
||||
`[RAG] ${EMBEDDING_MODEL_NAME} not found. Cannot perform similarity search.`
|
||||
)
|
||||
logger.warn(`[RAG] ${EMBEDDING_MODEL_NAME} not found. Cannot perform similarity search.`)
|
||||
this.embeddingModelVerified = false
|
||||
return []
|
||||
}
|
||||
|
|
@ -946,7 +1005,10 @@ export class RagService {
|
|||
return []
|
||||
}
|
||||
|
||||
const response = await this.ollamaService.embed(this.resolvedEmbeddingModel ?? EMBEDDING_MODEL_NAME, [prefixedQuery])
|
||||
const response = await this.ollamaService.embed(
|
||||
this.resolvedEmbeddingModel ?? EMBEDDING_MODEL_NAME,
|
||||
[prefixedQuery]
|
||||
)
|
||||
|
||||
// Perform semantic search with a higher limit to enable reranking
|
||||
const searchLimit = limit * 3 // Get more results for reranking
|
||||
|
|
@ -959,7 +1021,13 @@ export class RagService {
|
|||
limit: searchLimit,
|
||||
score_threshold: scoreThreshold,
|
||||
with_payload: true,
|
||||
...(collection ? { filter: { must: [{ key: 'collection', match: { value: collection } }] } } : {}),
|
||||
filter: {
|
||||
// Denylist, not allowlist: a point with no `active` field at all
|
||||
// (not yet backfilled) must still be found by default. Search
|
||||
// correctness must never depend on the backfill's timing.
|
||||
must_not: [{ key: 'active', match: { value: false } }],
|
||||
...(collection ? { must: [{ key: 'collection', match: { value: collection } }] } : {}),
|
||||
},
|
||||
})
|
||||
|
||||
logger.debug(`[RAG] Found ${searchResults.length} results above threshold ${scoreThreshold}`)
|
||||
|
|
@ -1196,9 +1264,7 @@ export class RagService {
|
|||
* Uses greedy selection: for each result, apply 0.85^n penalty where n is the
|
||||
* number of results already selected from the same source.
|
||||
*/
|
||||
private applySourceDiversity(
|
||||
results: Array<RerankedRAGResult>
|
||||
) {
|
||||
private applySourceDiversity(results: Array<RerankedRAGResult>) {
|
||||
const sourceCounts = new Map<string, number>()
|
||||
const DIVERSITY_PENALTY = 0.85
|
||||
|
||||
|
|
@ -1228,7 +1294,10 @@ export class RagService {
|
|||
*/
|
||||
public async hasDocuments(): Promise<boolean> {
|
||||
try {
|
||||
await this._ensureCollection(RagService.CONTENT_COLLECTION_NAME, RagService.EMBEDDING_DIMENSION)
|
||||
await this._ensureCollection(
|
||||
RagService.CONTENT_COLLECTION_NAME,
|
||||
RagService.EMBEDDING_DIMENSION
|
||||
)
|
||||
const collectionInfo = await this.qdrant!.getCollection(RagService.CONTENT_COLLECTION_NAME)
|
||||
return (collectionInfo.points_count ?? 0) > 0
|
||||
} catch {
|
||||
|
|
@ -1271,15 +1340,34 @@ export class RagService {
|
|||
// in particular) have no row to attach to. The state machine is the
|
||||
// authoritative "what's on disk?" view; Qdrant is "what made it into
|
||||
// the vector store?". Both are needed to render the KB UI honestly.
|
||||
const stateByPath = new Map<string, { state: KbIngestStateValue; chunks_embedded: number; collection: string | null }>()
|
||||
const stateByPath = new Map<
|
||||
string,
|
||||
{
|
||||
state: KbIngestStateValue
|
||||
chunks_embedded: number
|
||||
collection: string | null
|
||||
active: boolean
|
||||
}
|
||||
>()
|
||||
try {
|
||||
const stateRows = await KbIngestState.query().select('file_path', 'state', 'chunks_embedded', 'collection')
|
||||
const stateRows = await KbIngestState.query().select(
|
||||
'file_path',
|
||||
'state',
|
||||
'chunks_embedded',
|
||||
'collection',
|
||||
'active'
|
||||
)
|
||||
for (const row of stateRows) {
|
||||
sources.add(row.file_path)
|
||||
stateByPath.set(row.file_path, {
|
||||
state: row.state,
|
||||
chunks_embedded: row.chunks_embedded,
|
||||
collection: row.collection,
|
||||
// MySQL hands back tinyint(1) as 0/1, not a boolean. Coerce here so
|
||||
// every consumer sees a real boolean -- the raw 1 reached the DOM as
|
||||
// aria-checked="1", which is not a valid ARIA value and leaves screen
|
||||
// readers with no idea whether the toggle is on.
|
||||
active: Boolean(row.active),
|
||||
})
|
||||
}
|
||||
} catch (error) {
|
||||
|
|
@ -1307,6 +1395,7 @@ export class RagService {
|
|||
uploadedAt: stats?.modifiedTime.toISOString() ?? null,
|
||||
isUserUpload,
|
||||
collection: row?.collection ?? null,
|
||||
active: row ? Boolean(row.active) : true,
|
||||
}
|
||||
})
|
||||
)
|
||||
|
|
@ -1349,7 +1438,10 @@ export class RagService {
|
|||
collection: string | null
|
||||
): Promise<{ success: boolean; message: string }> {
|
||||
try {
|
||||
await this._ensureCollection(RagService.CONTENT_COLLECTION_NAME, RagService.EMBEDDING_DIMENSION)
|
||||
await this._ensureCollection(
|
||||
RagService.CONTENT_COLLECTION_NAME,
|
||||
RagService.EMBEDDING_DIMENSION
|
||||
)
|
||||
|
||||
await this.qdrant!.setPayload(RagService.CONTENT_COLLECTION_NAME, {
|
||||
payload: { collection },
|
||||
|
|
@ -1364,13 +1456,112 @@ export class RagService {
|
|||
row.collection = collection
|
||||
await row.save()
|
||||
|
||||
return { success: true, message: collection ? `Moved to "${collection}".` : 'Moved to Uncategorized.' }
|
||||
return {
|
||||
success: true,
|
||||
message: collection ? `Moved to "${collection}".` : 'Moved to Uncategorized.',
|
||||
}
|
||||
} catch (error) {
|
||||
logger.error('[RAG] Error updating file collection:', error)
|
||||
return { success: false, message: 'Error updating file collection.' }
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Toggle a file's active (searchable) state. Updates the `active` payload
|
||||
* field on every existing Qdrant point for this source in place — no
|
||||
* deletion or re-embedding, so this is instant in either direction — then
|
||||
* mirrors the change onto the KbIngestState row so getStoredFiles() reflects
|
||||
* it immediately. Vectors stay in Qdrant permanently either way; only
|
||||
* `searchSimilarDocuments()`'s query-time filter is affected. See #1119.
|
||||
*/
|
||||
public async setFileActive(
|
||||
source: string,
|
||||
active: boolean
|
||||
): Promise<{ success: boolean; message: string }> {
|
||||
try {
|
||||
await this._ensureCollection(
|
||||
RagService.CONTENT_COLLECTION_NAME,
|
||||
RagService.EMBEDDING_DIMENSION
|
||||
)
|
||||
|
||||
await this.qdrant!.setPayload(RagService.CONTENT_COLLECTION_NAME, {
|
||||
payload: { active },
|
||||
filter: { must: [{ key: 'source', match: { value: source } }] },
|
||||
})
|
||||
|
||||
const row = await KbIngestState.query().where('file_path', source).first()
|
||||
if (row) {
|
||||
row.active = active
|
||||
await row.save()
|
||||
}
|
||||
|
||||
return { success: true, message: active ? 'File is now active.' : 'File is now inactive.' }
|
||||
} catch (error) {
|
||||
logger.error('[RAG] Error updating file active state:', error)
|
||||
return { success: false, message: 'Error updating file active state.' }
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Toggle every file tagged with a KB collection (the personal-upload
|
||||
* `collection` field -- not the unrelated curated ZIM-pack "collections"
|
||||
* feature) active/inactive in one bulk operation, mirroring the
|
||||
* collection-scoped bulk writes in renameKnowledgeCollection/
|
||||
* deleteKnowledgeCollection rather than looping setFileActive() per file.
|
||||
* `collection: null` targets the "Uncategorized" bucket.
|
||||
*
|
||||
* Returns the count of files whose active value actually changed (not the
|
||||
* collection's total membership), so the caller can report accurate scope
|
||||
* for what is otherwise an opaque bulk action -- e.g. "Turned off 9 files",
|
||||
* not "Turned off 12" when 3 were already inactive.
|
||||
*/
|
||||
public async setKnowledgeCollectionActive(
|
||||
collection: string | null,
|
||||
active: boolean
|
||||
): Promise<{ success: boolean; message: string; affectedCount: number }> {
|
||||
try {
|
||||
await this._ensureCollection(
|
||||
RagService.CONTENT_COLLECTION_NAME,
|
||||
RagService.EMBEDDING_DIMENSION
|
||||
)
|
||||
|
||||
const collectionQuery = () =>
|
||||
collection === null
|
||||
? KbIngestState.query().whereNull('collection')
|
||||
: KbIngestState.query().where('collection', collection)
|
||||
|
||||
const countRow = await collectionQuery().where('active', !active).count('* as total').first()
|
||||
const affectedCount = Number((countRow as any)?.$extras?.total ?? 0)
|
||||
|
||||
const collectionFilterClause =
|
||||
collection === null
|
||||
? { is_empty: { key: 'collection' } }
|
||||
: { key: 'collection', match: { value: collection } }
|
||||
|
||||
await this.qdrant!.setPayload(RagService.CONTENT_COLLECTION_NAME, {
|
||||
payload: { active },
|
||||
filter: { must: [collectionFilterClause] },
|
||||
})
|
||||
|
||||
await collectionQuery().update({ active })
|
||||
|
||||
const label = collection ?? 'Uncategorized'
|
||||
const verb = active ? 'Turned on' : 'Turned off'
|
||||
return {
|
||||
success: true,
|
||||
message: `${verb} ${affectedCount} file${affectedCount === 1 ? '' : 's'} in "${label}".`,
|
||||
affectedCount,
|
||||
}
|
||||
} catch (error) {
|
||||
logger.error('[RAG] Error updating KB collection active state:', error)
|
||||
return {
|
||||
success: false,
|
||||
message: 'Error updating collection active state.',
|
||||
affectedCount: 0,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Rename a knowledge-base collection everywhere it's referenced: updates every
|
||||
* Qdrant point tagged with the old name in place, and mirrors the change onto
|
||||
|
|
@ -1384,7 +1575,10 @@ export class RagService {
|
|||
if (!oldName || !newName || oldName === newName) {
|
||||
return { success: false, message: 'Invalid collection names.' }
|
||||
}
|
||||
await this._ensureCollection(RagService.CONTENT_COLLECTION_NAME, RagService.EMBEDDING_DIMENSION)
|
||||
await this._ensureCollection(
|
||||
RagService.CONTENT_COLLECTION_NAME,
|
||||
RagService.EMBEDDING_DIMENSION
|
||||
)
|
||||
|
||||
await this.qdrant!.setPayload(RagService.CONTENT_COLLECTION_NAME, {
|
||||
payload: { collection: newName },
|
||||
|
|
@ -1413,7 +1607,10 @@ export class RagService {
|
|||
if (!name) {
|
||||
return { success: false, message: 'Invalid collection name.' }
|
||||
}
|
||||
await this._ensureCollection(RagService.CONTENT_COLLECTION_NAME, RagService.EMBEDDING_DIMENSION)
|
||||
await this._ensureCollection(
|
||||
RagService.CONTENT_COLLECTION_NAME,
|
||||
RagService.EMBEDDING_DIMENSION
|
||||
)
|
||||
|
||||
await this.qdrant!.setPayload(RagService.CONTENT_COLLECTION_NAME, {
|
||||
payload: { collection: null },
|
||||
|
|
@ -1483,7 +1680,15 @@ export class RagService {
|
|||
}
|
||||
|
||||
private static readonly VIEWABLE_TEXT_EXTENSIONS: ReadonlySet<string> = new Set([
|
||||
'md', 'txt', 'csv', 'json', 'yaml', 'yml', 'toml', 'xml', 'html',
|
||||
'md',
|
||||
'txt',
|
||||
'csv',
|
||||
'json',
|
||||
'yaml',
|
||||
'yml',
|
||||
'toml',
|
||||
'xml',
|
||||
'html',
|
||||
])
|
||||
|
||||
/**
|
||||
|
|
@ -1625,7 +1830,9 @@ export class RagService {
|
|||
// false "ingestion stalled" warnings. Suppress Warning B in that case. (#913)
|
||||
const expectedChunks =
|
||||
fileSizeBytes > 0
|
||||
? await KbRatioRegistry.estimateChunks(fileName, fileSizeBytes, { ignoreCatchAll: true })
|
||||
? await KbRatioRegistry.estimateChunks(fileName, fileSizeBytes, {
|
||||
ignoreCatchAll: true,
|
||||
})
|
||||
: null
|
||||
|
||||
const warnings = decideWarnings({ fileSizeBytes, chunksInQdrant, expectedChunks })
|
||||
|
|
@ -1660,17 +1867,19 @@ export class RagService {
|
|||
logger.info(`[RAG] Deleted all points for source: ${source}`)
|
||||
|
||||
/** Delete the physical file only if it lives inside the uploads directory.
|
||||
* resolve() normalises path traversal sequences (e.g. "/../..") before the
|
||||
* check to prevent path traversal vulns
|
||||
* The trailing sep is to ensure a prefix like "kb_uploads_{something_incorrect}" can't slip through.
|
||||
*/
|
||||
* resolve() normalises path traversal sequences (e.g. "/../..") before the
|
||||
* check to prevent path traversal vulns
|
||||
* The trailing sep is to ensure a prefix like "kb_uploads_{something_incorrect}" can't slip through.
|
||||
*/
|
||||
const uploadsAbsPath = join(process.cwd(), RagService.UPLOADS_STORAGE_PATH)
|
||||
const resolvedSource = resolve(source)
|
||||
if (resolvedSource.startsWith(uploadsAbsPath + sep)) {
|
||||
await deleteFileIfExists(resolvedSource)
|
||||
logger.info(`[RAG] Deleted uploaded file from disk: ${resolvedSource}`)
|
||||
} else {
|
||||
logger.warn(`[RAG] File was removed from knowledge base but doesn't live in Nomad's uploads directory, so it can't be safely removed. Skipping deletion of physical file...`)
|
||||
logger.warn(
|
||||
`[RAG] File was removed from knowledge base but doesn't live in Nomad's uploads directory, so it can't be safely removed. Skipping deletion of physical file...`
|
||||
)
|
||||
}
|
||||
|
||||
// Drop the ingest state row last so the file disappears entirely. Without
|
||||
|
|
@ -1785,7 +1994,10 @@ export class RagService {
|
|||
const alreadyEmbeddedRaw = await KVStore.getValue('rag.docsEmbedded')
|
||||
if (alreadyEmbeddedRaw && !force) {
|
||||
logger.info('[RAG] Nomad docs have already been discovered and queued. Skipping.')
|
||||
return { success: true, message: 'Nomad docs have already been discovered and queued. Skipping.' }
|
||||
return {
|
||||
success: true,
|
||||
message: 'Nomad docs have already been discovered and queued. Skipping.',
|
||||
}
|
||||
}
|
||||
|
||||
const filesToEmbed: Array<{ path: string; source: string }> = []
|
||||
|
|
@ -1817,17 +2029,17 @@ export class RagService {
|
|||
})
|
||||
logger.info(`[RAG] Successfully dispatched job for ${fileInfo.source}`)
|
||||
} catch (fileError) {
|
||||
logger.error(
|
||||
`[RAG] Error dispatching job for file ${fileInfo.source}:`,
|
||||
fileError
|
||||
)
|
||||
logger.error(`[RAG] Error dispatching job for file ${fileInfo.source}:`, fileError)
|
||||
}
|
||||
}
|
||||
|
||||
// Update KV store to mark docs as discovered so we don't redo this unnecessarily
|
||||
await KVStore.setValue('rag.docsEmbedded', true)
|
||||
|
||||
return { success: true, message: `Nomad docs discovery completed. Dispatched ${filesToEmbed.length} embedding jobs.` }
|
||||
return {
|
||||
success: true,
|
||||
message: `Nomad docs discovery completed. Dispatched ${filesToEmbed.length} embedding jobs.`,
|
||||
}
|
||||
} catch (error) {
|
||||
logger.error('Error discovering Nomad docs:', error)
|
||||
return { success: false, message: 'Error discovering Nomad docs.' }
|
||||
|
|
@ -1946,7 +2158,8 @@ export class RagService {
|
|||
return {
|
||||
success: false,
|
||||
code: 'inflight',
|
||||
message: 'A job for this file is already in progress. Wait for it to finish before re-queuing.',
|
||||
message:
|
||||
'A job for this file is already in progress. Wait for it to finish before re-queuing.',
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1983,10 +2196,7 @@ export class RagService {
|
|||
* by reembedAll() where the file must remain so it can be re-ingested.
|
||||
*/
|
||||
private async _deletePointsBySource(source: string): Promise<void> {
|
||||
await this._ensureCollection(
|
||||
RagService.CONTENT_COLLECTION_NAME,
|
||||
RagService.EMBEDDING_DIMENSION
|
||||
)
|
||||
await this._ensureCollection(RagService.CONTENT_COLLECTION_NAME, RagService.EMBEDDING_DIMENSION)
|
||||
await this.qdrant!.delete(RagService.CONTENT_COLLECTION_NAME, {
|
||||
filter: { must: [{ key: 'source', match: { value: source } }] },
|
||||
})
|
||||
|
|
@ -2003,7 +2213,10 @@ export class RagService {
|
|||
const { QueueService } = await import('#services/queue_service')
|
||||
const queue = QueueService.getInstance().getQueue(EmbedFileJob.queue)
|
||||
const counts = await queue.getJobCounts('waiting', 'active', 'delayed', 'paused')
|
||||
return (counts.waiting || 0) + (counts.active || 0) + (counts.delayed || 0) + (counts.paused || 0) > 0
|
||||
return (
|
||||
(counts.waiting || 0) + (counts.active || 0) + (counts.delayed || 0) + (counts.paused || 0) >
|
||||
0
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -2164,7 +2377,8 @@ export class RagService {
|
|||
if (await this._hasInflightEmbedJobs()) {
|
||||
return {
|
||||
success: false,
|
||||
message: 'Embed jobs are already in progress. Wait for the queue to drain (or clean up failed jobs) before triggering a bulk re-embed.',
|
||||
message:
|
||||
'Embed jobs are already in progress. Wait for the queue to drain (or clean up failed jobs) before triggering a bulk re-embed.',
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -2197,7 +2411,10 @@ export class RagService {
|
|||
try {
|
||||
await this._deletePointsBySource(filePath)
|
||||
} catch (err) {
|
||||
logger.error(`[RAG] Failed to delete prior points for ${filePath}; skipping dispatch:`, err)
|
||||
logger.error(
|
||||
`[RAG] Failed to delete prior points for ${filePath}; skipping dispatch:`,
|
||||
err
|
||||
)
|
||||
failedPaths.push(filePath)
|
||||
continue
|
||||
}
|
||||
|
|
@ -2212,7 +2429,10 @@ export class RagService {
|
|||
} catch (fileError) {
|
||||
// Old points already deleted but the new job never made it onto the
|
||||
// queue. Logged + surfaced so an operator can rerun a sync.
|
||||
logger.error(`[RAG] Re-embed dispatch failed for ${filePath} after delete; file is now unindexed until next sync:`, fileError)
|
||||
logger.error(
|
||||
`[RAG] Re-embed dispatch failed for ${filePath} after delete; file is now unindexed until next sync:`,
|
||||
fileError
|
||||
)
|
||||
failedPaths.push(filePath)
|
||||
}
|
||||
}
|
||||
|
|
@ -2261,7 +2481,8 @@ export class RagService {
|
|||
if (await this._hasInflightEmbedJobs()) {
|
||||
return {
|
||||
success: false,
|
||||
message: 'Embed jobs are already in progress. Wait for the queue to drain (or clean up failed jobs) before triggering a reset.',
|
||||
message:
|
||||
'Embed jobs are already in progress. Wait for the queue to drain (or clean up failed jobs) before triggering a reset.',
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,17 @@
|
|||
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||||
|
||||
export default class extends BaseSchema {
|
||||
protected tableName = 'kb_ingest_state'
|
||||
|
||||
async up() {
|
||||
this.schema.alterTable(this.tableName, (table) => {
|
||||
table.boolean('active').notNullable().defaultTo(true)
|
||||
})
|
||||
}
|
||||
|
||||
async down() {
|
||||
this.schema.alterTable(this.tableName, (table) => {
|
||||
table.dropColumn('active')
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -17,6 +17,11 @@ export type StyledTableProps<T extends { [key: string]: any }> = {
|
|||
title?: React.ReactNode
|
||||
render?: (record: T, index: number) => React.ReactNode
|
||||
className?: string
|
||||
/** Opt out of the default single-line truncation/`overflow: hidden` on
|
||||
* this column's cells. Needed for columns whose render() opens a
|
||||
* floating popover (e.g. a combobox) -- otherwise the cell clips the
|
||||
* popover to its own row height instead of letting it float. */
|
||||
noTruncate?: boolean
|
||||
}[]
|
||||
className?: string
|
||||
rowLines?: boolean
|
||||
|
|
@ -160,7 +165,8 @@ function StyledTable<T extends { [key: string]: any }>({
|
|||
<td
|
||||
key={index}
|
||||
className={classNames(
|
||||
'relative text-sm whitespace-nowrap max-w-72 truncate break-words text-left',
|
||||
'relative text-sm max-w-72 break-words text-left',
|
||||
column.noTruncate ? '' : 'whitespace-nowrap truncate',
|
||||
column.className || '',
|
||||
compact ? `${leftPadding} py-2` : `${leftPadding} py-4 pr-3`
|
||||
)}
|
||||
|
|
|
|||
|
|
@ -20,6 +20,12 @@ interface CollectionComboboxProps {
|
|||
* match something. Actual normalization (trim/lowercase/length cap) happens
|
||||
* server-side via sanitizeCollectionName; this just lowercases for the
|
||||
* match-check and display so duplicates-by-case don't look like real options.
|
||||
*
|
||||
* The option list is portaled to `document.body` and positioned via
|
||||
* `getBoundingClientRect()` rather than living in normal flow. This is used
|
||||
* inside table cells (e.g. the Stored Files panel), where any ancestor with
|
||||
* non-`visible` overflow -- a `<td>`, or `StyledTable`'s own scroll wrapper --
|
||||
* would otherwise clip the popover to that ancestor's box.
|
||||
*/
|
||||
export default function CollectionCombobox({
|
||||
value,
|
||||
|
|
@ -32,6 +38,10 @@ export default function CollectionCombobox({
|
|||
}: CollectionComboboxProps) {
|
||||
const [query, setQuery] = useState(value)
|
||||
const [isOpen, setIsOpen] = useState(false)
|
||||
// Whether the user has typed since the popover opened. The input is
|
||||
// pre-filled with `value` on open, but that shouldn't filter the option
|
||||
// list down to just the current value -- only actual typing should.
|
||||
const [userTyped, setUserTyped] = useState(false)
|
||||
const containerRef = useRef<HTMLDivElement>(null)
|
||||
const dropdownRef = useRef<HTMLDivElement>(null)
|
||||
const [position, setPosition] = useState<{ left: number; top: number; width: number } | null>(null)
|
||||
|
|
@ -50,6 +60,7 @@ export default function CollectionCombobox({
|
|||
if (!insideInput && !insideList) {
|
||||
setIsOpen(false)
|
||||
setQuery(value)
|
||||
setUserTyped(false)
|
||||
}
|
||||
}
|
||||
document.addEventListener('mousedown', handleClickOutside)
|
||||
|
|
@ -95,16 +106,21 @@ export default function CollectionCombobox({
|
|||
}, [isOpen, reposition])
|
||||
|
||||
const normalizedQuery = query.trim().toLowerCase()
|
||||
const filtered = normalizedQuery
|
||||
? options.filter((o) => o.toLowerCase().includes(normalizedQuery))
|
||||
// Filtering (and the "+ Create" row) only kick in once the user has typed
|
||||
// -- otherwise the pre-filled current value would filter the list down to
|
||||
// just itself the instant the popover opens.
|
||||
const filterQuery = userTyped ? normalizedQuery : ''
|
||||
const filtered = filterQuery
|
||||
? options.filter((o) => o.toLowerCase().includes(filterQuery))
|
||||
: options
|
||||
const exactMatch = options.find((o) => o.toLowerCase() === normalizedQuery)
|
||||
const showCreateOption = normalizedQuery.length > 0 && !exactMatch
|
||||
const showCreateOption = userTyped && normalizedQuery.length > 0 && !exactMatch
|
||||
|
||||
const commit = (val: string) => {
|
||||
onChange(val)
|
||||
setQuery(val)
|
||||
setIsOpen(false)
|
||||
setUserTyped(false)
|
||||
}
|
||||
|
||||
return (
|
||||
|
|
@ -116,6 +132,7 @@ export default function CollectionCombobox({
|
|||
onChange={(e) => {
|
||||
setQuery(e.target.value)
|
||||
setIsOpen(true)
|
||||
setUserTyped(true)
|
||||
}}
|
||||
onFocus={() => setIsOpen(true)}
|
||||
onKeyDown={(e) => {
|
||||
|
|
@ -130,6 +147,7 @@ export default function CollectionCombobox({
|
|||
} else if (e.key === 'Escape') {
|
||||
setIsOpen(false)
|
||||
setQuery(value)
|
||||
setUserTyped(false)
|
||||
}
|
||||
}}
|
||||
placeholder={placeholder}
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ import { useNotifications } from '~/context/NotificationContext'
|
|||
import api from '~/lib/api'
|
||||
import {
|
||||
groupAndSortKbFiles,
|
||||
UNCATEGORIZED_COLLECTION_KEY,
|
||||
type KbFileGroup,
|
||||
type KbFileSort,
|
||||
type KbFileSortKey,
|
||||
|
|
@ -17,6 +18,8 @@ import type { KbIngestStateValue } from '../../../types/kb_ingest_state'
|
|||
import { formatBytes } from '~/lib/util'
|
||||
import {
|
||||
IconArrowsSort,
|
||||
IconChevronDown,
|
||||
IconChevronRight,
|
||||
IconDownload,
|
||||
IconEye,
|
||||
IconSortAscending,
|
||||
|
|
@ -30,6 +33,7 @@ import { SERVICE_NAMES } from '../../../constants/service_names'
|
|||
import CollectionsManager from './CollectionsManager'
|
||||
import { KB_COLLECTIONS } from '../../../constants/kb_collections'
|
||||
import CollectionCombobox from './CollectionCombobox'
|
||||
import Switch from '~/components/inputs/Switch'
|
||||
|
||||
interface KnowledgeBaseModalProps {
|
||||
aiAssistantName?: string
|
||||
|
|
@ -39,7 +43,17 @@ interface KnowledgeBaseModalProps {
|
|||
// File extensions the in-browser viewer can render. Must stay in sync with
|
||||
// `RagService.VIEWABLE_TEXT_EXTENSIONS` -- anything outside this set falls back
|
||||
// to Download.
|
||||
const VIEWABLE_EXTENSIONS = new Set(['md', 'txt', 'csv', 'json', 'yaml', 'yml', 'toml', 'xml', 'html'])
|
||||
const VIEWABLE_EXTENSIONS = new Set([
|
||||
'md',
|
||||
'txt',
|
||||
'csv',
|
||||
'json',
|
||||
'yaml',
|
||||
'yml',
|
||||
'toml',
|
||||
'xml',
|
||||
'html',
|
||||
])
|
||||
|
||||
function isViewableExtension(filename: string): boolean {
|
||||
const ext = filename.split('.').at(-1)?.toLowerCase() ?? ''
|
||||
|
|
@ -53,7 +67,11 @@ function renderSortHeader(
|
|||
setSort: (s: KbFileSort) => void
|
||||
): React.ReactNode {
|
||||
const active = sort.key === key
|
||||
const Icon = !active ? IconArrowsSort : sort.direction === 'asc' ? IconSortAscending : IconSortDescending
|
||||
const Icon = !active
|
||||
? IconArrowsSort
|
||||
: sort.direction === 'asc'
|
||||
? IconSortAscending
|
||||
: IconSortDescending
|
||||
return (
|
||||
<button
|
||||
type="button"
|
||||
|
|
@ -67,7 +85,11 @@ function renderSortHeader(
|
|||
}}
|
||||
>
|
||||
<span>{label}</span>
|
||||
<Icon size={14} className={active ? 'text-text-primary' : 'text-text-muted'} aria-hidden="true" />
|
||||
<Icon
|
||||
size={14}
|
||||
className={active ? 'text-text-primary' : 'text-text-muted'}
|
||||
aria-hidden="true"
|
||||
/>
|
||||
</button>
|
||||
)
|
||||
}
|
||||
|
|
@ -87,7 +109,9 @@ function renderStatePill(record: KbFileGroup): React.ReactNode {
|
|||
switch (effective) {
|
||||
case 'indexed':
|
||||
return (
|
||||
<span className={`${base} text-green-700 bg-green-50 border-green-200 dark:text-green-300 dark:bg-green-950/40 dark:border-green-800`}>
|
||||
<span
|
||||
className={`${base} text-green-700 bg-green-50 border-green-200 dark:text-green-300 dark:bg-green-950/40 dark:border-green-800`}
|
||||
>
|
||||
Indexed
|
||||
</span>
|
||||
)
|
||||
|
|
@ -100,13 +124,17 @@ function renderStatePill(record: KbFileGroup): React.ReactNode {
|
|||
)
|
||||
case 'failed':
|
||||
return (
|
||||
<span className={`${base} text-red-700 bg-red-50 border-red-200 dark:text-red-300 dark:bg-red-950/40 dark:border-red-800`}>
|
||||
<span
|
||||
className={`${base} text-red-700 bg-red-50 border-red-200 dark:text-red-300 dark:bg-red-950/40 dark:border-red-800`}
|
||||
>
|
||||
Failed
|
||||
</span>
|
||||
)
|
||||
case 'stalled':
|
||||
return (
|
||||
<span className={`${base} text-amber-700 bg-amber-50 border-amber-200 dark:text-amber-300 dark:bg-amber-950/40 dark:border-amber-800`}>
|
||||
<span
|
||||
className={`${base} text-amber-700 bg-amber-50 border-amber-200 dark:text-amber-300 dark:bg-amber-950/40 dark:border-amber-800`}
|
||||
>
|
||||
Stalled
|
||||
</span>
|
||||
)
|
||||
|
|
@ -130,19 +158,40 @@ function pickRowAction(record: KbFileGroup, hasWarnings: boolean): RowAction | n
|
|||
switch (effective) {
|
||||
case 'indexed':
|
||||
return hasWarnings
|
||||
? { kind: 'reembed', label: 'Re-embed', force: true, variant: 'secondary', icon: 'IconRefreshAlert' }
|
||||
? {
|
||||
kind: 'reembed',
|
||||
label: 'Re-embed',
|
||||
force: true,
|
||||
variant: 'secondary',
|
||||
icon: 'IconRefreshAlert',
|
||||
}
|
||||
: null
|
||||
case 'pending_decision':
|
||||
return { kind: 'index', label: 'Index', force: false, variant: 'primary', icon: 'IconDownload' }
|
||||
return {
|
||||
kind: 'index',
|
||||
label: 'Index',
|
||||
force: false,
|
||||
variant: 'primary',
|
||||
icon: 'IconDownload',
|
||||
}
|
||||
case 'browse_only':
|
||||
return { kind: 'index', label: 'Index', force: true, variant: 'primary', icon: 'IconDownload' }
|
||||
return {
|
||||
kind: 'index',
|
||||
label: 'Index',
|
||||
force: true,
|
||||
variant: 'primary',
|
||||
icon: 'IconDownload',
|
||||
}
|
||||
case 'failed':
|
||||
case 'stalled':
|
||||
return { kind: 'index', label: 'Retry', force: true, variant: 'primary', icon: 'IconRefresh' }
|
||||
}
|
||||
}
|
||||
|
||||
export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", onClose }: KnowledgeBaseModalProps) {
|
||||
export default function KnowledgeBaseModal({
|
||||
aiAssistantName = 'AI Assistant',
|
||||
onClose,
|
||||
}: KnowledgeBaseModalProps) {
|
||||
const { addNotification } = useNotifications()
|
||||
const [files, setFiles] = useState<File[]>([])
|
||||
const [isUploading, setIsUploading] = useState(false)
|
||||
|
|
@ -150,10 +199,17 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
const [collectionFilter, setCollectionFilter] = useState<string>('All')
|
||||
const [manageCollectionsOpen, setManageCollectionsOpen] = useState(false)
|
||||
const [confirmDeleteSource, setConfirmDeleteSource] = useState<string | null>(null)
|
||||
const [confirmReembed, setConfirmReembed] = useState<{ source: string; displayName: string } | null>(null)
|
||||
const [confirmReembed, setConfirmReembed] = useState<{
|
||||
source: string
|
||||
displayName: string
|
||||
} | null>(null)
|
||||
const [bulkMode, setBulkMode] = useState<null | 'reembed' | 'reset'>(null)
|
||||
const [resetTyped, setResetTyped] = useState('')
|
||||
const [sort, setSort] = useState<KbFileSort>({ key: 'name', direction: 'asc' })
|
||||
// KB-collection group keys (see UNCATEGORIZED_COLLECTION_KEY) whose member
|
||||
// files are currently expanded into individual rows. Collapsed by default
|
||||
// so grouping actually declutters the panel.
|
||||
const [expandedCollections, setExpandedCollections] = useState<Set<string>>(new Set())
|
||||
const [viewerSource, setViewerSource] = useState<string | null>(null)
|
||||
const fileUploaderRef = useRef<React.ComponentRef<typeof FileUploader>>(null)
|
||||
const { openModal, closeModal } = useModals()
|
||||
|
|
@ -248,6 +304,39 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
},
|
||||
})
|
||||
|
||||
const toggleActiveMutation = useMutation({
|
||||
mutationFn: ({ source, active }: { source: string; active: boolean }) =>
|
||||
api.setFileActive(source, active),
|
||||
onSuccess: (data) => {
|
||||
addNotification({ type: 'success', message: data?.message || 'Active state updated.' })
|
||||
queryClient.invalidateQueries({ queryKey: ['storedFiles'] })
|
||||
},
|
||||
onError: (error: any) => {
|
||||
addNotification({
|
||||
type: 'error',
|
||||
message: error?.message || 'Failed to update active state.',
|
||||
})
|
||||
},
|
||||
})
|
||||
|
||||
const toggleCollectionActiveMutation = useMutation({
|
||||
mutationFn: ({ collection, active }: { collection: string | null; active: boolean }) =>
|
||||
api.setKnowledgeCollectionActive(collection, active),
|
||||
onSuccess: (data) => {
|
||||
addNotification({
|
||||
type: 'success',
|
||||
message: data?.message || 'Collection active state updated.',
|
||||
})
|
||||
queryClient.invalidateQueries({ queryKey: ['storedFiles'] })
|
||||
},
|
||||
onError: (error: any) => {
|
||||
addNotification({
|
||||
type: 'error',
|
||||
message: error?.message || 'Failed to update collection active state.',
|
||||
})
|
||||
},
|
||||
})
|
||||
|
||||
const deleteMutation = useMutation({
|
||||
mutationFn: (source: string) => api.deleteRAGFile(source),
|
||||
onSuccess: () => {
|
||||
|
|
@ -294,7 +383,10 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
const cancelAllMutation = useMutation({
|
||||
mutationFn: () => api.cancelAllEmbedJobs(),
|
||||
onSuccess: (data) => {
|
||||
addNotification({ type: 'success', message: data?.message || 'All embedding jobs cancelled.' })
|
||||
addNotification({
|
||||
type: 'success',
|
||||
message: data?.message || 'All embedding jobs cancelled.',
|
||||
})
|
||||
queryClient.invalidateQueries({ queryKey: ['embed-jobs'] })
|
||||
queryClient.invalidateQueries({ queryKey: ['failedEmbedJobs'] })
|
||||
queryClient.invalidateQueries({ queryKey: ['storedFiles'] })
|
||||
|
|
@ -321,7 +413,9 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
onSuccess: (data) => {
|
||||
addNotification({
|
||||
type: 'success',
|
||||
message: data?.message || 'Storage synced successfully. If new files were found, they have been queued for processing.',
|
||||
message:
|
||||
data?.message ||
|
||||
'Storage synced successfully. If new files were found, they have been queued for processing.',
|
||||
})
|
||||
},
|
||||
onError: (error: any) => {
|
||||
|
|
@ -401,21 +495,33 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
}
|
||||
}
|
||||
|
||||
const toggleCollectionExpanded = (key: string) => {
|
||||
setExpandedCollections((prev) => {
|
||||
const next = new Set(prev)
|
||||
if (next.has(key)) {
|
||||
next.delete(key)
|
||||
} else {
|
||||
next.add(key)
|
||||
}
|
||||
return next
|
||||
})
|
||||
}
|
||||
|
||||
const handleConfirmCancelAll = () => {
|
||||
openModal(
|
||||
<StyledModal
|
||||
title='Cancel All Embedding Jobs?'
|
||||
title="Cancel All Embedding Jobs?"
|
||||
onConfirm={() => {
|
||||
cancelAllMutation.mutate()
|
||||
closeModal('confirm-cancel-all-modal')
|
||||
}}
|
||||
onCancel={() => closeModal('confirm-cancel-all-modal')}
|
||||
open={true}
|
||||
confirmText='Cancel All Jobs'
|
||||
cancelText='Keep Jobs'
|
||||
confirmVariant='danger'
|
||||
confirmText="Cancel All Jobs"
|
||||
cancelText="Keep Jobs"
|
||||
confirmVariant="danger"
|
||||
>
|
||||
<p className='text-text-primary'>
|
||||
<p className="text-text-primary">
|
||||
This stops <strong>every</strong> embedding job — including ones still in progress or
|
||||
stuck — and clears the processing queue. The uploaded source files for those jobs are
|
||||
deleted, so you'll need to re-upload anything you still want indexed. Stored files that
|
||||
|
|
@ -429,31 +535,35 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
const handleConfirmSync = () => {
|
||||
openModal(
|
||||
<StyledModal
|
||||
title='Confirm Sync?'
|
||||
title="Confirm Sync?"
|
||||
onConfirm={() => {
|
||||
syncMutation.mutate()
|
||||
closeModal(
|
||||
"confirm-sync-modal"
|
||||
)
|
||||
closeModal('confirm-sync-modal')
|
||||
}}
|
||||
onCancel={() => closeModal("confirm-sync-modal")}
|
||||
onCancel={() => closeModal('confirm-sync-modal')}
|
||||
open={true}
|
||||
confirmText='Confirm Sync'
|
||||
cancelText='Cancel'
|
||||
confirmVariant='primary'
|
||||
confirmText="Confirm Sync"
|
||||
cancelText="Cancel"
|
||||
confirmVariant="primary"
|
||||
>
|
||||
<p className='text-text-primary'>
|
||||
This will scan the NOMAD's storage directories for any new files and queue them for processing. This is useful if you've manually added files to the storage or want to ensure everything is up to date.
|
||||
This may cause a temporary increase in resource usage if new files are found and being processed. Are you sure you want to proceed?
|
||||
<p className="text-text-primary">
|
||||
This will scan the NOMAD's storage directories for any new files and queue them for
|
||||
processing. This is useful if you've manually added files to the storage or want to ensure
|
||||
everything is up to date. This may cause a temporary increase in resource usage if new
|
||||
files are found and being processed. Are you sure you want to proceed?
|
||||
</p>
|
||||
</StyledModal>,
|
||||
"confirm-sync-modal"
|
||||
'confirm-sync-modal'
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="fixed inset-0 z-50 flex items-center justify-center p-4 bg-black/30 backdrop-blur-sm transition-opacity">
|
||||
<div className="bg-surface-primary rounded-lg shadow-xl max-w-5xl w-full max-h-[90vh] overflow-hidden flex flex-col">
|
||||
{/* 6xl, not 5xl: the Active column added ~57px and put the table back over the
|
||||
modal width, which is the exact defect #1198 fixed (the Delete button falls
|
||||
off the right edge at every viewport). Measured on NOMAD3: table 1018px
|
||||
against 961px of usable width at 5xl. */}
|
||||
<div className="bg-surface-primary rounded-lg shadow-xl max-w-6xl w-full max-h-[90vh] overflow-hidden flex flex-col">
|
||||
<div className="flex items-center justify-between p-6 border-b border-border-subtle shrink-0">
|
||||
<h2 className="text-2xl font-semibold text-text-primary">Knowledge Base</h2>
|
||||
<button
|
||||
|
|
@ -560,7 +670,8 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
</p>
|
||||
<p className="text-sm text-desert-stone">
|
||||
NOMAD will automatically discover and extract any content you save to your
|
||||
Information Library (if installed), making it instantly available to {aiAssistantName} without any extra steps.
|
||||
Information Library (if installed), making it instantly available to{' '}
|
||||
{aiAssistantName} without any extra steps.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -574,8 +685,8 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
Auto-index new content for AI?
|
||||
</p>
|
||||
<p className="text-xs text-text-muted mt-1">
|
||||
Indexed content typically uses 5–10× the original file size on disk.
|
||||
Changes apply to new content added after this setting changes.
|
||||
Indexed content typically uses 5–10× the original file size on disk. Changes apply
|
||||
to new content added after this setting changes.
|
||||
</p>
|
||||
</div>
|
||||
<div
|
||||
|
|
@ -591,9 +702,7 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
type="button"
|
||||
role="radio"
|
||||
aria-checked={isActive}
|
||||
onClick={() =>
|
||||
!isActive && updateIngestPolicyMutation.mutate(option)
|
||||
}
|
||||
onClick={() => !isActive && updateIngestPolicyMutation.mutate(option)}
|
||||
disabled={updateIngestPolicyMutation.isPending}
|
||||
className={`px-4 py-2 text-sm font-medium transition-colors ${
|
||||
isActive
|
||||
|
|
@ -640,8 +749,8 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
</div>
|
||||
|
||||
<div className="my-12">
|
||||
<div className='flex items-center justify-between mb-6 gap-2 flex-wrap'>
|
||||
<StyledSectionHeader title="Stored Knowledge Base Files" className='!mb-0' />
|
||||
<div className="flex items-center justify-between mb-6 gap-2 flex-wrap">
|
||||
<StyledSectionHeader title="Stored Knowledge Base Files" className="!mb-0" />
|
||||
<div className="flex items-center gap-2 flex-wrap">
|
||||
<label className="flex items-center gap-2 text-sm text-text-secondary">
|
||||
Search in:
|
||||
|
|
@ -652,8 +761,11 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
>
|
||||
<option value="All">All</option>
|
||||
{knownCollections.map((c) => (
|
||||
<option key={c} value={c}>{c}</option>
|
||||
<option key={c} value={c}>
|
||||
{c}
|
||||
</option>
|
||||
))}
|
||||
<option value={UNCATEGORIZED_COLLECTION_KEY}>Uncategorized</option>
|
||||
</select>
|
||||
</label>
|
||||
<StyledButton
|
||||
|
|
@ -667,8 +779,11 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
<StyledButton
|
||||
variant="danger"
|
||||
size="md"
|
||||
icon='IconAlertTriangle'
|
||||
onClick={() => { setResetTyped(''); setBulkMode('reset') }}
|
||||
icon="IconAlertTriangle"
|
||||
onClick={() => {
|
||||
setResetTyped('')
|
||||
setBulkMode('reset')
|
||||
}}
|
||||
disabled={isUploading || qdrantOffline || bulkBusy}
|
||||
loading={resetMutation.isPending}
|
||||
title="Drop the entire embeddings collection and re-embed everything from scratch. Permanently removes vectors for files no longer on disk. Destructive: requires typing RESET to confirm."
|
||||
|
|
@ -678,7 +793,7 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
<StyledButton
|
||||
variant="secondary"
|
||||
size="md"
|
||||
icon='IconRefreshAlert'
|
||||
icon="IconRefreshAlert"
|
||||
onClick={() => setBulkMode('reembed')}
|
||||
disabled={isUploading || qdrantOffline || bulkBusy || storedFiles.length === 0}
|
||||
loading={reembedMutation.isPending}
|
||||
|
|
@ -689,7 +804,7 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
<StyledButton
|
||||
variant="secondary"
|
||||
size="md"
|
||||
icon='IconRefresh'
|
||||
icon="IconRefresh"
|
||||
onClick={handleConfirmSync}
|
||||
disabled={syncMutation.isPending || isUploading || qdrantOffline || bulkBusy}
|
||||
loading={syncMutation.isPending || isUploading}
|
||||
|
|
@ -697,15 +812,12 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
>
|
||||
Sync Storage
|
||||
</StyledButton>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
{warningsUnavailable && (
|
||||
<div className="mb-4 inline-flex items-center gap-2 text-xs text-amber-700 dark:text-amber-300 bg-amber-50 dark:bg-amber-950/40 border border-amber-200 dark:border-amber-800 rounded px-3 py-2">
|
||||
<span aria-hidden="true">⚠</span>
|
||||
<span>
|
||||
File warnings unavailable — couldn't read storage state. Retrying…
|
||||
</span>
|
||||
<span>File warnings unavailable — couldn't read storage state. Retrying…</span>
|
||||
</div>
|
||||
)}
|
||||
<StyledTable<KbFileGroup>
|
||||
|
|
@ -716,13 +828,38 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
accessor: 'source',
|
||||
title: renderSortHeader('File Name', 'name', sort, setSort),
|
||||
render(record) {
|
||||
if (record.isCollectionHeader) {
|
||||
const key = record.collection ?? UNCATEGORIZED_COLLECTION_KEY
|
||||
const expanded = expandedCollections.has(key)
|
||||
return (
|
||||
<button
|
||||
type="button"
|
||||
className="inline-flex items-center gap-1.5 font-semibold text-text-primary hover:text-desert-green transition-colors"
|
||||
onClick={() => toggleCollectionExpanded(key)}
|
||||
aria-expanded={expanded}
|
||||
>
|
||||
{expanded ? (
|
||||
<IconChevronDown
|
||||
size={16}
|
||||
className="text-text-muted"
|
||||
aria-hidden="true"
|
||||
/>
|
||||
) : (
|
||||
<IconChevronRight
|
||||
size={16}
|
||||
className="text-text-muted"
|
||||
aria-hidden="true"
|
||||
/>
|
||||
)}
|
||||
{record.displayName}
|
||||
</button>
|
||||
)
|
||||
}
|
||||
const warnings = fileWarnings[record.source] ?? []
|
||||
const pill = renderStatePill(record)
|
||||
return (
|
||||
<div className="flex flex-col gap-1.5">
|
||||
<span className="text-text-primary">
|
||||
{record.displayName}
|
||||
</span>
|
||||
<span className="text-text-primary">{record.displayName}</span>
|
||||
{(pill || warnings.length > 0) && (
|
||||
<div className="flex flex-wrap items-center gap-1.5">
|
||||
{pill}
|
||||
|
|
@ -734,15 +871,15 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
<span aria-hidden="true">⚠</span>
|
||||
{w.kind === 'zero_chunks' && (
|
||||
<span>
|
||||
Embedded 0 chunks — this file has no text content.
|
||||
AI Assistant cannot reference it.
|
||||
Embedded 0 chunks — this file has no text content. AI Assistant
|
||||
cannot reference it.
|
||||
</span>
|
||||
)}
|
||||
{w.kind === 'partial_stall' && (
|
||||
<span>
|
||||
Only {w.chunksEmbedded.toLocaleString()} of est.{' '}
|
||||
{w.chunksExpected.toLocaleString()} chunks embedded —
|
||||
ingestion may have stalled.
|
||||
{w.chunksExpected.toLocaleString()} chunks embedded — ingestion
|
||||
may have stalled.
|
||||
</span>
|
||||
)}
|
||||
</span>
|
||||
|
|
@ -784,8 +921,9 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
accessor: 'collection',
|
||||
title: 'Collection',
|
||||
className: 'whitespace-nowrap',
|
||||
noTruncate: true,
|
||||
render(record) {
|
||||
if (record.bucket === 'admin_docs') {
|
||||
if (record.bucket === 'admin_docs' || record.isCollectionHeader) {
|
||||
return <span className="text-text-muted">—</span>
|
||||
}
|
||||
const isSaving =
|
||||
|
|
@ -794,7 +932,12 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
return (
|
||||
<CollectionCombobox
|
||||
value={record.collection ?? ''}
|
||||
onChange={(val) => updateCollectionMutation.mutate({ source: record.source, collection: val })}
|
||||
onChange={(val) =>
|
||||
updateCollectionMutation.mutate({
|
||||
source: record.source,
|
||||
collection: val,
|
||||
})
|
||||
}
|
||||
options={comboboxOptions}
|
||||
disabled={isSaving}
|
||||
className="w-40"
|
||||
|
|
@ -802,6 +945,45 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
)
|
||||
},
|
||||
},
|
||||
{
|
||||
accessor: 'active',
|
||||
title: 'Active',
|
||||
render(record) {
|
||||
if (record.bucket === 'admin_docs') {
|
||||
return <span className="text-text-muted">—</span>
|
||||
}
|
||||
if (record.isCollectionHeader) {
|
||||
const isSaving =
|
||||
toggleCollectionActiveMutation.isPending &&
|
||||
toggleCollectionActiveMutation.variables?.collection === record.collection
|
||||
return (
|
||||
<Switch
|
||||
checked={record.collectionActiveState === 'all-active'}
|
||||
indeterminate={record.collectionActiveState === 'mixed'}
|
||||
onChange={(val) =>
|
||||
toggleCollectionActiveMutation.mutate({
|
||||
collection: record.collection,
|
||||
active: val,
|
||||
})
|
||||
}
|
||||
disabled={isSaving}
|
||||
/>
|
||||
)
|
||||
}
|
||||
const isSaving =
|
||||
toggleActiveMutation.isPending &&
|
||||
toggleActiveMutation.variables?.source === record.source
|
||||
return (
|
||||
<Switch
|
||||
checked={record.active}
|
||||
onChange={(val) =>
|
||||
toggleActiveMutation.mutate({ source: record.source, active: val })
|
||||
}
|
||||
disabled={isSaving}
|
||||
/>
|
||||
)
|
||||
},
|
||||
},
|
||||
{
|
||||
accessor: 'source',
|
||||
title: '',
|
||||
|
|
@ -809,30 +991,34 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
if (record.bucket === 'admin_docs') {
|
||||
return (
|
||||
<div className="flex justify-end">
|
||||
<span className="text-sm text-text-muted italic">
|
||||
Managed by NOMAD
|
||||
</span>
|
||||
<span className="text-sm text-text-muted italic">Managed by NOMAD</span>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
if (record.isCollectionHeader) {
|
||||
return null
|
||||
}
|
||||
|
||||
const isConfirming = confirmDeleteSource === record.source
|
||||
const isDeleting = deleteMutation.isPending && confirmDeleteSource === record.source
|
||||
const isDeleting =
|
||||
deleteMutation.isPending && confirmDeleteSource === record.source
|
||||
if (isConfirming) {
|
||||
return (
|
||||
<div className="flex items-center gap-2 justify-end">
|
||||
<span className="text-sm text-text-secondary">Remove from knowledge base?</span>
|
||||
<span className="text-sm text-text-secondary">
|
||||
Remove from knowledge base?
|
||||
</span>
|
||||
<StyledButton
|
||||
variant='danger'
|
||||
size='sm'
|
||||
variant="danger"
|
||||
size="sm"
|
||||
onClick={() => deleteMutation.mutate(record.source)}
|
||||
disabled={isDeleting}
|
||||
>
|
||||
{isDeleting ? 'Deleting…' : 'Confirm'}
|
||||
</StyledButton>
|
||||
<StyledButton
|
||||
variant='ghost'
|
||||
size='sm'
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
onClick={() => setConfirmDeleteSource(null)}
|
||||
disabled={isDeleting}
|
||||
>
|
||||
|
|
@ -847,7 +1033,10 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
const actionPendingForThisRow =
|
||||
embedMutation.isPending && embedMutation.variables?.source === record.source
|
||||
|
||||
const canView = record.isUserUpload && isViewableExtension(record.displayName) && record.size !== null
|
||||
const canView =
|
||||
record.isUserUpload &&
|
||||
isViewableExtension(record.displayName) &&
|
||||
record.size !== null
|
||||
const canDownload = record.isUserUpload && record.size !== null
|
||||
|
||||
return (
|
||||
|
|
@ -859,12 +1048,17 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
icon={action.icon}
|
||||
onClick={() => {
|
||||
if (action.kind === 'reembed') {
|
||||
setConfirmReembed({ source: record.source, displayName: record.displayName })
|
||||
setConfirmReembed({
|
||||
source: record.source,
|
||||
displayName: record.displayName,
|
||||
})
|
||||
} else {
|
||||
embedMutation.mutate({ source: record.source, force: action.force })
|
||||
}
|
||||
}}
|
||||
disabled={qdrantOffline || deleteMutation.isPending || embedMutation.isPending}
|
||||
disabled={
|
||||
qdrantOffline || deleteMutation.isPending || embedMutation.isPending
|
||||
}
|
||||
loading={actionPendingForThisRow}
|
||||
>
|
||||
{action.label}
|
||||
|
|
@ -876,7 +1070,9 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
size="sm"
|
||||
icon="IconEye"
|
||||
onClick={() => setViewerSource(record.source)}
|
||||
>View</StyledButton>
|
||||
>
|
||||
View
|
||||
</StyledButton>
|
||||
)}
|
||||
{canDownload && (
|
||||
<StyledButton
|
||||
|
|
@ -886,7 +1082,9 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
onClick={() => {
|
||||
window.location.href = `/api/rag/files/download?source=${encodeURIComponent(record.source)}`
|
||||
}}
|
||||
>Download</StyledButton>
|
||||
>
|
||||
Download
|
||||
</StyledButton>
|
||||
)}
|
||||
<StyledButton
|
||||
variant="danger"
|
||||
|
|
@ -894,8 +1092,12 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
icon="IconTrash"
|
||||
onClick={() => setConfirmDeleteSource(record.source)}
|
||||
disabled={deleteMutation.isPending || embedMutation.isPending}
|
||||
loading={deleteMutation.isPending && confirmDeleteSource === record.source}
|
||||
>Delete</StyledButton>
|
||||
loading={
|
||||
deleteMutation.isPending && confirmDeleteSource === record.source
|
||||
}
|
||||
>
|
||||
Delete
|
||||
</StyledButton>
|
||||
</div>
|
||||
)
|
||||
},
|
||||
|
|
@ -904,8 +1106,13 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
data={groupAndSortKbFiles(
|
||||
collectionFilter === 'All'
|
||||
? storedFiles
|
||||
: storedFiles.filter((f) => f.collection === collectionFilter),
|
||||
sort
|
||||
: storedFiles.filter((f) =>
|
||||
collectionFilter === UNCATEGORIZED_COLLECTION_KEY
|
||||
? f.collection === null
|
||||
: f.collection === collectionFilter
|
||||
),
|
||||
sort,
|
||||
expandedCollections
|
||||
)}
|
||||
loading={isLoadingFiles}
|
||||
/>
|
||||
|
|
@ -915,38 +1122,53 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
|
||||
{bulkMode === 'reembed' && (
|
||||
<StyledModal
|
||||
title='Re-embed All Documents?'
|
||||
title="Re-embed All Documents?"
|
||||
open={true}
|
||||
confirmText={reembedMutation.isPending ? 'Re-embedding…' : 'Re-embed All'}
|
||||
cancelText='Cancel'
|
||||
confirmVariant='primary'
|
||||
cancelText="Cancel"
|
||||
confirmVariant="primary"
|
||||
confirmLoading={reembedMutation.isPending}
|
||||
onConfirm={() => reembedMutation.mutate()}
|
||||
onCancel={() => setBulkMode(null)}
|
||||
>
|
||||
<div className='text-text-primary text-sm space-y-3 text-left'>
|
||||
<div className="text-text-primary text-sm space-y-3 text-left">
|
||||
<p>
|
||||
This will re-process every document currently in your knowledge base — about
|
||||
<strong> {storedFiles.length} file{storedFiles.length === 1 ? '' : 's'}</strong>.
|
||||
For each file, NOMAD will delete the existing embeddings from Qdrant and queue a fresh
|
||||
embedding job using the current chunking and embedding model.
|
||||
<strong>
|
||||
{' '}
|
||||
{storedFiles.length} file{storedFiles.length === 1 ? '' : 's'}
|
||||
</strong>
|
||||
. For each file, NOMAD will delete the existing embeddings from Qdrant and queue a
|
||||
fresh embedding job using the current chunking and embedding model.
|
||||
</p>
|
||||
<div className='rounded border border-border-subtle bg-surface-secondary p-3'>
|
||||
<p className='font-semibold mb-1'>What this is for</p>
|
||||
<p className='text-text-secondary'>
|
||||
<div className="rounded border border-border-subtle bg-surface-secondary p-3">
|
||||
<p className="font-semibold mb-1">What this is for</p>
|
||||
<p className="text-text-secondary">
|
||||
Use this when the embedding model or chunking logic has changed, or when you suspect
|
||||
stored vectors are stale. Files on disk are <em>not</em> deleted, and any orphan
|
||||
points whose source file is no longer present will be preserved untouched (see
|
||||
<em> Reset & Rebuild </em>if you want a fully clean slate).
|
||||
</p>
|
||||
</div>
|
||||
<div className='rounded border border-amber-300 bg-amber-50 dark:bg-amber-950 dark:border-amber-800 p-3 text-amber-900 dark:text-amber-200'>
|
||||
<p className='font-semibold mb-1'>Heads up</p>
|
||||
<ul className='list-disc pl-5 space-y-1'>
|
||||
<li>Embedding {storedFiles.length} file{storedFiles.length === 1 ? '' : 's'} may take a long time, especially for large PDFs or ZIM archives.</li>
|
||||
<li>On systems without GPU acceleration, expect sustained high CPU usage for the duration.</li>
|
||||
<li>Knowledge Base search results may be incomplete until every file finishes re-embedding.</li>
|
||||
<li>If embed jobs are already in progress, this action will be refused — wait for the queue to drain first.</li>
|
||||
<div className="rounded border border-amber-300 bg-amber-50 dark:bg-amber-950 dark:border-amber-800 p-3 text-amber-900 dark:text-amber-200">
|
||||
<p className="font-semibold mb-1">Heads up</p>
|
||||
<ul className="list-disc pl-5 space-y-1">
|
||||
<li>
|
||||
Embedding {storedFiles.length} file{storedFiles.length === 1 ? '' : 's'} may take
|
||||
a long time, especially for large PDFs or ZIM archives.
|
||||
</li>
|
||||
<li>
|
||||
On systems without GPU acceleration, expect sustained high CPU usage for the
|
||||
duration.
|
||||
</li>
|
||||
<li>
|
||||
Knowledge Base search results may be incomplete until every file finishes
|
||||
re-embedding.
|
||||
</li>
|
||||
<li>
|
||||
If embed jobs are already in progress, this action will be refused — wait for the
|
||||
queue to drain first.
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -955,53 +1177,79 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
|
||||
{bulkMode === 'reset' && (
|
||||
<StyledModal
|
||||
title='Reset & Rebuild Knowledge Base?'
|
||||
title="Reset & Rebuild Knowledge Base?"
|
||||
open={true}
|
||||
confirmText={resetMutation.isPending ? 'Resetting…' : 'Wipe & Rebuild'}
|
||||
cancelText='Cancel'
|
||||
confirmVariant='danger'
|
||||
cancelText="Cancel"
|
||||
confirmVariant="danger"
|
||||
confirmLoading={resetMutation.isPending}
|
||||
onConfirm={() => {
|
||||
if (resetTyped === 'RESET') resetMutation.mutate()
|
||||
}}
|
||||
onCancel={() => { setBulkMode(null); setResetTyped('') }}
|
||||
onCancel={() => {
|
||||
setBulkMode(null)
|
||||
setResetTyped('')
|
||||
}}
|
||||
>
|
||||
<div className='text-text-primary text-sm space-y-3 text-left'>
|
||||
<div className="text-text-primary text-sm space-y-3 text-left">
|
||||
<p>
|
||||
This will <strong>permanently delete every point</strong> in the
|
||||
<code> nomad_knowledge_base </code>Qdrant collection and rebuild from the
|
||||
<strong> {storedFiles.length} file{storedFiles.length === 1 ? '' : 's'}</strong> currently
|
||||
on disk. The collection is dropped, recreated, and every file is re-queued for embedding.
|
||||
<strong>
|
||||
{' '}
|
||||
{storedFiles.length} file{storedFiles.length === 1 ? '' : 's'}
|
||||
</strong>{' '}
|
||||
currently on disk. The collection is dropped, recreated, and every file is re-queued
|
||||
for embedding.
|
||||
</p>
|
||||
<div className='rounded border border-border-subtle bg-surface-secondary p-3'>
|
||||
<p className='font-semibold mb-1'>How this differs from Re-embed All</p>
|
||||
<ul className='list-disc pl-5 space-y-1 text-text-secondary'>
|
||||
<li><strong>Re-embed All</strong> replaces vectors file-by-file. Any orphan points (vectors whose source file was deleted from disk at some point) are preserved.</li>
|
||||
<li><strong>Reset & Rebuild</strong> drops the entire collection. Orphan points are <strong>gone forever</strong>. Only files currently on disk will exist in Qdrant afterwards.</li>
|
||||
<div className="rounded border border-border-subtle bg-surface-secondary p-3">
|
||||
<p className="font-semibold mb-1">How this differs from Re-embed All</p>
|
||||
<ul className="list-disc pl-5 space-y-1 text-text-secondary">
|
||||
<li>
|
||||
<strong>Re-embed All</strong> replaces vectors file-by-file. Any orphan points
|
||||
(vectors whose source file was deleted from disk at some point) are preserved.
|
||||
</li>
|
||||
<li>
|
||||
<strong>Reset & Rebuild</strong> drops the entire collection. Orphan points
|
||||
are <strong>gone forever</strong>. Only files currently on disk will exist in
|
||||
Qdrant afterwards.
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div className='rounded border border-red-300 bg-red-50 dark:bg-red-950 dark:border-red-800 p-3 text-red-900 dark:text-red-200'>
|
||||
<p className='font-semibold mb-1'>This action is destructive and cannot be undone</p>
|
||||
<ul className='list-disc pl-5 space-y-1'>
|
||||
<li>Knowledge Base search will be empty until embedding finishes (potentially hours on CPU-only systems).</li>
|
||||
<li>For a few seconds during the reset, the Qdrant collection does not exist — any chat-with-RAG queries in that window may return a "collection not found" error. Avoid using chat until the rebuild has begun.</li>
|
||||
<li>If embed jobs are already in progress, this action will be refused — wait for the queue to drain first.</li>
|
||||
<div className="rounded border border-red-300 bg-red-50 dark:bg-red-950 dark:border-red-800 p-3 text-red-900 dark:text-red-200">
|
||||
<p className="font-semibold mb-1">This action is destructive and cannot be undone</p>
|
||||
<ul className="list-disc pl-5 space-y-1">
|
||||
<li>
|
||||
Knowledge Base search will be empty until embedding finishes (potentially hours on
|
||||
CPU-only systems).
|
||||
</li>
|
||||
<li>
|
||||
For a few seconds during the reset, the Qdrant collection does not exist — any
|
||||
chat-with-RAG queries in that window may return a "collection not found" error.
|
||||
Avoid using chat until the rebuild has begun.
|
||||
</li>
|
||||
<li>
|
||||
If embed jobs are already in progress, this action will be refused — wait for the
|
||||
queue to drain first.
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div>
|
||||
<label className='block text-sm font-semibold mb-1'>
|
||||
<label className="block text-sm font-semibold mb-1">
|
||||
Type <code>RESET</code> to confirm:
|
||||
</label>
|
||||
<input
|
||||
type='text'
|
||||
type="text"
|
||||
value={resetTyped}
|
||||
onChange={(e) => setResetTyped(e.target.value)}
|
||||
placeholder='RESET'
|
||||
placeholder="RESET"
|
||||
autoFocus
|
||||
className='w-full rounded border border-border-subtle bg-surface-primary px-3 py-2 text-text-primary focus:outline-none focus:ring-2 focus:ring-red-500'
|
||||
className="w-full rounded border border-border-subtle bg-surface-primary px-3 py-2 text-text-primary focus:outline-none focus:ring-2 focus:ring-red-500"
|
||||
/>
|
||||
{resetTyped.length > 0 && resetTyped !== 'RESET' && (
|
||||
<p className='text-xs text-red-600 mt-1'>Type RESET exactly (uppercase, no spaces) to enable the confirm button.</p>
|
||||
<p className="text-xs text-red-600 mt-1">
|
||||
Type RESET exactly (uppercase, no spaces) to enable the confirm button.
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -1010,29 +1258,35 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
|
||||
{confirmReembed && (
|
||||
<StyledModal
|
||||
title='Re-embed this file?'
|
||||
title="Re-embed this file?"
|
||||
open={true}
|
||||
confirmText={embedMutation.isPending ? 'Queuing…' : 'Re-embed'}
|
||||
cancelText='Cancel'
|
||||
confirmVariant='primary'
|
||||
cancelText="Cancel"
|
||||
confirmVariant="primary"
|
||||
confirmLoading={embedMutation.isPending}
|
||||
onConfirm={() =>
|
||||
embedMutation.mutate({ source: confirmReembed.source, force: true })
|
||||
}
|
||||
onConfirm={() => embedMutation.mutate({ source: confirmReembed.source, force: true })}
|
||||
onCancel={() => setConfirmReembed(null)}
|
||||
>
|
||||
<div className='text-text-primary text-sm space-y-3 text-left'>
|
||||
<div className="text-text-primary text-sm space-y-3 text-left">
|
||||
<p>
|
||||
This will delete the existing embeddings for{' '}
|
||||
<strong>{confirmReembed.displayName}</strong> and queue
|
||||
a fresh embedding job. The file on disk is not touched.
|
||||
<strong>{confirmReembed.displayName}</strong> and queue a fresh embedding job. The
|
||||
file on disk is not touched.
|
||||
</p>
|
||||
<div className='rounded border border-amber-300 bg-amber-50 dark:bg-amber-950 dark:border-amber-800 p-3 text-amber-900 dark:text-amber-200'>
|
||||
<p className='font-semibold mb-1'>Heads up</p>
|
||||
<ul className='list-disc pl-5 space-y-1'>
|
||||
<li>For large ZIM archives this can take a long time, especially on CPU-only systems.</li>
|
||||
<li>Search results that referenced this file will be incomplete until the new embedding finishes.</li>
|
||||
<li>If a job for this file is already running, the re-embed will be refused — wait for it to finish first.</li>
|
||||
<div className="rounded border border-amber-300 bg-amber-50 dark:bg-amber-950 dark:border-amber-800 p-3 text-amber-900 dark:text-amber-200">
|
||||
<p className="font-semibold mb-1">Heads up</p>
|
||||
<ul className="list-disc pl-5 space-y-1">
|
||||
<li>
|
||||
For large ZIM archives this can take a long time, especially on CPU-only systems.
|
||||
</li>
|
||||
<li>
|
||||
Search results that referenced this file will be incomplete until the new
|
||||
embedding finishes.
|
||||
</li>
|
||||
<li>
|
||||
If a job for this file is already running, the re-embed will be refused — wait for
|
||||
it to finish first.
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -1040,10 +1294,7 @@ export default function KnowledgeBaseModal({ aiAssistantName = "AI Assistant", o
|
|||
)}
|
||||
|
||||
{viewerSource && (
|
||||
<FileViewerModal
|
||||
source={viewerSource}
|
||||
onClose={() => setViewerSource(null)}
|
||||
/>
|
||||
<FileViewerModal source={viewerSource} onClose={() => setViewerSource(null)} />
|
||||
)}
|
||||
|
||||
{manageCollectionsOpen && (
|
||||
|
|
@ -1079,9 +1330,7 @@ function FileViewerModal({ source, onClose }: { source: string; onClose: () => v
|
|||
large
|
||||
>
|
||||
<div className="text-left text-sm">
|
||||
{isLoading && (
|
||||
<div className="text-text-secondary">Loading…</div>
|
||||
)}
|
||||
{isLoading && <div className="text-text-secondary">Loading…</div>}
|
||||
{showError && (
|
||||
<div className="text-amber-700 dark:text-amber-300">
|
||||
Couldn't load file. It may have been moved or its type isn't viewable.
|
||||
|
|
|
|||
|
|
@ -7,6 +7,12 @@ interface SwitchProps {
|
|||
description?: string
|
||||
disabled?: boolean
|
||||
id?: string
|
||||
/** Renders a third, visually distinct "mixed" state (e.g. a collection
|
||||
* toggle whose member files aren't uniformly active/inactive) instead of
|
||||
* the normal on/off look. Clicking a mixed switch always resolves to the
|
||||
* less-destructive direction -- turning everything on -- regardless of
|
||||
* `checked`. */
|
||||
indeterminate?: boolean
|
||||
}
|
||||
|
||||
export default function Switch({
|
||||
|
|
@ -16,6 +22,7 @@ export default function Switch({
|
|||
description,
|
||||
disabled = false,
|
||||
id,
|
||||
indeterminate = false,
|
||||
}: SwitchProps) {
|
||||
const switchId = id || `switch-${label?.replace(/\s+/g, '-').toLowerCase()}`
|
||||
|
||||
|
|
@ -39,13 +46,13 @@ export default function Switch({
|
|||
id={switchId}
|
||||
type="button"
|
||||
role="switch"
|
||||
aria-checked={checked}
|
||||
aria-checked={indeterminate ? 'mixed' : !!checked}
|
||||
disabled={disabled}
|
||||
onClick={() => !disabled && onChange(!checked)}
|
||||
onClick={() => !disabled && onChange(indeterminate ? true : !checked)}
|
||||
className={clsx(
|
||||
'relative inline-flex h-6 w-11 flex-shrink-0 cursor-pointer rounded-full border-2 border-transparent',
|
||||
'transition-colors duration-200 ease-in-out focus:outline-none focus:ring-2 focus:ring-desert-green focus:ring-offset-2',
|
||||
checked ? 'bg-desert-green' : 'bg-border-default',
|
||||
indeterminate ? 'bg-amber-400' : checked ? 'bg-desert-green' : 'bg-border-default',
|
||||
disabled ? 'opacity-50 cursor-not-allowed' : ''
|
||||
)}
|
||||
>
|
||||
|
|
@ -53,7 +60,7 @@ export default function Switch({
|
|||
className={clsx(
|
||||
'pointer-events-none inline-block h-5 w-5 transform rounded-full bg-white shadow ring-0',
|
||||
'transition duration-200 ease-in-out',
|
||||
checked ? 'translate-x-5' : 'translate-x-0'
|
||||
indeterminate ? 'translate-x-2.5' : checked ? 'translate-x-5' : 'translate-x-0'
|
||||
)}
|
||||
/>
|
||||
</button>
|
||||
|
|
|
|||
|
|
@ -1112,6 +1112,16 @@ class API {
|
|||
})()
|
||||
}
|
||||
|
||||
async setFileActive(source: string, active: boolean) {
|
||||
return catchInternal(async () => {
|
||||
const response = await this.client.post<{ message: string }>('/rag/files/active', {
|
||||
source,
|
||||
active,
|
||||
})
|
||||
return response.data
|
||||
})()
|
||||
}
|
||||
|
||||
async renameCollection(oldName: string, newName: string) {
|
||||
return catchInternal(async () => {
|
||||
const response = await this.client.post<{ message: string }>('/rag/rename-collection', {
|
||||
|
|
@ -1131,6 +1141,16 @@ class API {
|
|||
})()
|
||||
}
|
||||
|
||||
async setKnowledgeCollectionActive(collection: string | null, active: boolean) {
|
||||
return catchInternal(async () => {
|
||||
const response = await this.client.post<{ message: string; affectedCount: number }>(
|
||||
'/rag/collection-active',
|
||||
{ collection, active }
|
||||
)
|
||||
return response.data
|
||||
})()
|
||||
}
|
||||
|
||||
async createLinkTile(data: {
|
||||
friendly_name: string
|
||||
url: string
|
||||
|
|
|
|||
|
|
@ -19,11 +19,7 @@ const ZIM_PREFIX = '/app/storage/zim/'
|
|||
const UPLOADS_PREFIX = '/app/storage/kb_uploads/'
|
||||
|
||||
export function classifyKbFile(source: string): KbFileBucket {
|
||||
if (
|
||||
ADMIN_DOCS_PREFIXES.some((p) =>
|
||||
p.endsWith('/') ? source.startsWith(p) : source === p
|
||||
)
|
||||
) {
|
||||
if (ADMIN_DOCS_PREFIXES.some((p) => (p.endsWith('/') ? source.startsWith(p) : source === p))) {
|
||||
return 'admin_docs'
|
||||
}
|
||||
if (source.startsWith(ZIM_PREFIX)) return 'zim'
|
||||
|
|
@ -39,7 +35,8 @@ export function sourceToDisplayName(source: string): string {
|
|||
export interface KbFileGroup {
|
||||
bucket: KbFileBucket
|
||||
/** Source path used as the row's stable React key. For collapsed admin docs
|
||||
* this is a synthetic marker; individual file paths live in `members`. */
|
||||
* or a KB-collection header this is a synthetic marker; individual file
|
||||
* paths live in `members`. */
|
||||
source: string
|
||||
displayName: string
|
||||
/** Number of underlying files this row represents (1 for non-collapsed). */
|
||||
|
|
@ -61,11 +58,29 @@ export interface KbFileGroup {
|
|||
/** True when the row corresponds to a user upload — drives whether the
|
||||
* view/download buttons render. False for the collapsed admin_docs group. */
|
||||
isUserUpload: boolean
|
||||
/** Subject/category tag, or null if uncategorized. Always null for the
|
||||
* collapsed admin_docs group. */
|
||||
/** Subject/category (KB collection) tag, or null if uncategorized. Always
|
||||
* null for the collapsed admin_docs group. Not to be confused with the
|
||||
* unrelated curated ZIM-pack "collections" feature elsewhere in the app. */
|
||||
collection: string | null
|
||||
/** Whether this file's chunks are included in RAG search results. Always
|
||||
* true (non-toggleable) for the collapsed admin_docs group. */
|
||||
active: boolean
|
||||
/** True for a collapsible per-KB-collection header row that clusters
|
||||
* upload-bucket files sharing a `collection` tag (or the "Uncategorized"
|
||||
* bucket for `collection: null` files). Undefined for ordinary file rows
|
||||
* and the admin_docs group. */
|
||||
isCollectionHeader?: boolean
|
||||
/** Aggregate active state across a header row's members. Only set when
|
||||
* `isCollectionHeader` is true. */
|
||||
collectionActiveState?: 'all-active' | 'all-inactive' | 'mixed'
|
||||
}
|
||||
|
||||
/** Sentinel key (not a valid `collection` value — `sanitizeCollectionName`
|
||||
* never produces an empty string) used to group and expand/collapse
|
||||
* upload-bucket files with `collection: null` under an "Uncategorized"
|
||||
* header, and as the header row's `collection` lookup key from the UI. */
|
||||
export const UNCATEGORIZED_COLLECTION_KEY = ''
|
||||
|
||||
const BUCKET_SORT_ORDER: KbFileBucket[] = ['zim', 'upload', 'admin_docs', 'other']
|
||||
|
||||
export type KbFileSortKey = 'name' | 'size' | 'uploadedAt'
|
||||
|
|
@ -80,8 +95,10 @@ const DEFAULT_SORT: KbFileSort = { key: 'name', direction: 'asc' }
|
|||
function compareForSort(a: StoredFileInfo, b: StoredFileInfo, sort: KbFileSort): number {
|
||||
// Files the scanner couldn't stat sort to the end regardless of direction so
|
||||
// they don't pollute the top of size/uploaded-at views.
|
||||
const aMissing = sort.key !== 'name' && (sort.key === 'size' ? a.size === null : a.uploadedAt === null)
|
||||
const bMissing = sort.key !== 'name' && (sort.key === 'size' ? b.size === null : b.uploadedAt === null)
|
||||
const aMissing =
|
||||
sort.key !== 'name' && (sort.key === 'size' ? a.size === null : a.uploadedAt === null)
|
||||
const bMissing =
|
||||
sort.key !== 'name' && (sort.key === 'size' ? b.size === null : b.uploadedAt === null)
|
||||
if (aMissing && !bMissing) return 1
|
||||
if (!aMissing && bMissing) return -1
|
||||
|
||||
|
|
@ -98,18 +115,32 @@ function compareForSort(a: StoredFileInfo, b: StoredFileInfo, sort: KbFileSort):
|
|||
return sort.direction === 'desc' ? -cmp : cmp
|
||||
}
|
||||
|
||||
/** Build the synthetic React key for a KB-collection header row. */
|
||||
function collectionHeaderKey(collectionKey: string): string {
|
||||
return `__collection_header__:${collectionKey}`
|
||||
}
|
||||
|
||||
/**
|
||||
* Group stored-file rows into table rows for the Stored Files panel.
|
||||
*
|
||||
* - Admin docs (`/app/docs/*`, README) collapse into a single
|
||||
* "Project NOMAD documentation · N files" row.
|
||||
* - ZIMs, uploads, and others stay as individual rows, sorted within their
|
||||
* bucket by the active sort key. Bucket order itself is fixed — sorting
|
||||
* never flattens or reorders the groups themselves.
|
||||
* - Upload-bucket files (personal uploads) are clustered under a collapsible
|
||||
* header row per KB-collection tag -- including an "Uncategorized" header
|
||||
* for files with no tag -- so a user with many uploads doesn't see one flat
|
||||
* row per file. A header's members only appear as individual rows when its
|
||||
* key is present in `expandedCollections`; collapsed is the default so the
|
||||
* grouping actually declutters the panel rather than just adding a label.
|
||||
* - ZIMs and others stay as individual rows, sorted within their bucket by
|
||||
* the active sort key. Bucket order itself is fixed, and collection headers
|
||||
* within the upload bucket are always ordered alphabetically (Uncategorized
|
||||
* last) regardless of sort -- sorting only ever reorders files *within* a
|
||||
* group, never the groups themselves.
|
||||
*/
|
||||
export function groupAndSortKbFiles(
|
||||
files: StoredFileInfo[],
|
||||
sort: KbFileSort = DEFAULT_SORT
|
||||
sort: KbFileSort = DEFAULT_SORT,
|
||||
expandedCollections: ReadonlySet<string> = new Set()
|
||||
): KbFileGroup[] {
|
||||
const buckets: Record<KbFileBucket, StoredFileInfo[]> = {
|
||||
zim: [],
|
||||
|
|
@ -140,10 +171,80 @@ export function groupAndSortKbFiles(
|
|||
uploadedAt: null,
|
||||
isUserUpload: false,
|
||||
collection: null,
|
||||
active: true,
|
||||
})
|
||||
continue
|
||||
}
|
||||
|
||||
if (bucket === 'upload') {
|
||||
const byCollection = new Map<string, StoredFileInfo[]>()
|
||||
for (const file of members) {
|
||||
const key = file.collection ?? UNCATEGORIZED_COLLECTION_KEY
|
||||
const bucketMembers = byCollection.get(key)
|
||||
if (bucketMembers) {
|
||||
bucketMembers.push(file)
|
||||
} else {
|
||||
byCollection.set(key, [file])
|
||||
}
|
||||
}
|
||||
|
||||
const collectionKeys = Array.from(byCollection.keys()).sort((a, b) => {
|
||||
if (a === UNCATEGORIZED_COLLECTION_KEY) return 1
|
||||
if (b === UNCATEGORIZED_COLLECTION_KEY) return -1
|
||||
return a.localeCompare(b)
|
||||
})
|
||||
|
||||
for (const key of collectionKeys) {
|
||||
const groupMembers = byCollection.get(key)!
|
||||
const collectionName = key === UNCATEGORIZED_COLLECTION_KEY ? null : key
|
||||
const label = collectionName ?? 'Uncategorized'
|
||||
const activeCount = groupMembers.filter((m) => m.active).length
|
||||
const collectionActiveState: KbFileGroup['collectionActiveState'] =
|
||||
activeCount === 0
|
||||
? 'all-inactive'
|
||||
: activeCount === groupMembers.length
|
||||
? 'all-active'
|
||||
: 'mixed'
|
||||
|
||||
groups.push({
|
||||
bucket,
|
||||
source: collectionHeaderKey(key),
|
||||
displayName: `${label} · ${groupMembers.length} file${groupMembers.length === 1 ? '' : 's'}`,
|
||||
count: groupMembers.length,
|
||||
members: groupMembers.map((m) => m.source),
|
||||
state: null,
|
||||
chunksEmbedded: 0,
|
||||
size: null,
|
||||
uploadedAt: null,
|
||||
isUserUpload: false,
|
||||
collection: collectionName,
|
||||
active: collectionActiveState !== 'all-inactive',
|
||||
isCollectionHeader: true,
|
||||
collectionActiveState,
|
||||
})
|
||||
|
||||
if (!expandedCollections.has(key)) continue
|
||||
|
||||
for (const file of groupMembers.sort((a, b) => compareForSort(a, b, sort))) {
|
||||
groups.push({
|
||||
bucket,
|
||||
source: file.source,
|
||||
displayName: sourceToDisplayName(file.source),
|
||||
count: 1,
|
||||
members: [],
|
||||
state: file.state,
|
||||
chunksEmbedded: file.chunksEmbedded,
|
||||
size: file.size,
|
||||
uploadedAt: file.uploadedAt,
|
||||
isUserUpload: file.isUserUpload,
|
||||
collection: file.collection,
|
||||
active: file.active,
|
||||
})
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
for (const file of members.sort((a, b) => compareForSort(a, b, sort))) {
|
||||
groups.push({
|
||||
bucket,
|
||||
|
|
@ -157,6 +258,7 @@ export function groupAndSortKbFiles(
|
|||
uploadedAt: file.uploadedAt,
|
||||
isUserUpload: file.isUserUpload,
|
||||
collection: file.collection,
|
||||
active: file.active,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -504,6 +504,10 @@ router
|
|||
summary: "Update a file's knowledge collection",
|
||||
tags: ['rag'],
|
||||
})
|
||||
documented(router.post('/files/active', [RagController, 'setFileActive']), {
|
||||
summary: "Toggle a file's active (searchable) state",
|
||||
tags: ['rag'],
|
||||
})
|
||||
documented(router.post('/rename-collection', [RagController, 'renameKnowledgeCollection']), {
|
||||
summary: 'Rename a knowledge collection',
|
||||
tags: ['rag'],
|
||||
|
|
@ -512,6 +516,10 @@ router
|
|||
summary: 'Delete a knowledge collection',
|
||||
tags: ['rag'],
|
||||
})
|
||||
documented(router.post('/collection-active', [RagController, 'setKnowledgeCollectionActive']), {
|
||||
summary: "Bulk-toggle every file in a knowledge collection's active (searchable) state",
|
||||
tags: ['rag'],
|
||||
})
|
||||
})
|
||||
.prefix('/api/rag')
|
||||
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import {
|
|||
classifyKbFile,
|
||||
groupAndSortKbFiles,
|
||||
sourceToDisplayName,
|
||||
UNCATEGORIZED_COLLECTION_KEY,
|
||||
} from '../../inertia/lib/kb_file_grouping.js'
|
||||
import type { StoredFileInfo } from '../../types/rag.js'
|
||||
|
||||
|
|
@ -24,17 +25,12 @@ const asInfos = (sources: string[]): StoredFileInfo[] =>
|
|||
uploadedAt: null,
|
||||
isUserUpload: classifyKbFile(source) === 'upload',
|
||||
collection: null,
|
||||
active: true,
|
||||
}))
|
||||
|
||||
test('classifyKbFile distinguishes ZIM, upload, admin_docs, and other', () => {
|
||||
assert.equal(
|
||||
classifyKbFile('/app/storage/zim/devdocs_en_python_2026-02.zim'),
|
||||
'zim'
|
||||
)
|
||||
assert.equal(
|
||||
classifyKbFile('/app/storage/kb_uploads/federalist.txt-8cc4ec95aa8f.txt'),
|
||||
'upload'
|
||||
)
|
||||
assert.equal(classifyKbFile('/app/storage/zim/devdocs_en_python_2026-02.zim'), 'zim')
|
||||
assert.equal(classifyKbFile('/app/storage/kb_uploads/federalist.txt-8cc4ec95aa8f.txt'), 'upload')
|
||||
assert.equal(classifyKbFile('/app/docs/release-notes.md'), 'admin_docs')
|
||||
assert.equal(classifyKbFile('/app/README.md'), 'admin_docs')
|
||||
assert.equal(classifyKbFile('/unexpected/path/file.txt'), 'other')
|
||||
|
|
@ -53,12 +49,14 @@ test('sourceToDisplayName returns the basename', () => {
|
|||
})
|
||||
|
||||
test('groupAndSortKbFiles collapses all admin docs into a single row', () => {
|
||||
const groups = groupAndSortKbFiles(asInfos([
|
||||
'/app/docs/release-notes.md',
|
||||
'/app/docs/getting-started.md',
|
||||
'/app/docs/maps.md',
|
||||
'/app/README.md',
|
||||
]))
|
||||
const groups = groupAndSortKbFiles(
|
||||
asInfos([
|
||||
'/app/docs/release-notes.md',
|
||||
'/app/docs/getting-started.md',
|
||||
'/app/docs/maps.md',
|
||||
'/app/README.md',
|
||||
])
|
||||
)
|
||||
|
||||
assert.equal(groups.length, 1)
|
||||
assert.equal(groups[0].bucket, 'admin_docs')
|
||||
|
|
@ -73,12 +71,14 @@ test('groupAndSortKbFiles collapses all admin docs into a single row', () => {
|
|||
})
|
||||
|
||||
test('groupAndSortKbFiles orders buckets ZIM → upload → admin_docs → other', () => {
|
||||
const groups = groupAndSortKbFiles(asInfos([
|
||||
'/app/docs/release-notes.md',
|
||||
'/unexpected/foo.txt',
|
||||
'/app/storage/kb_uploads/upload.pdf',
|
||||
'/app/storage/zim/devdocs.zim',
|
||||
]))
|
||||
const groups = groupAndSortKbFiles(
|
||||
asInfos([
|
||||
'/app/docs/release-notes.md',
|
||||
'/unexpected/foo.txt',
|
||||
'/app/storage/kb_uploads/upload.pdf',
|
||||
'/app/storage/zim/devdocs.zim',
|
||||
])
|
||||
)
|
||||
|
||||
assert.deepEqual(
|
||||
groups.map((g) => g.bucket),
|
||||
|
|
@ -87,11 +87,13 @@ test('groupAndSortKbFiles orders buckets ZIM → upload → admin_docs → other
|
|||
})
|
||||
|
||||
test('groupAndSortKbFiles alphabetizes within a bucket', () => {
|
||||
const groups = groupAndSortKbFiles(asInfos([
|
||||
'/app/storage/zim/wikipedia.zim',
|
||||
'/app/storage/zim/devdocs.zim',
|
||||
'/app/storage/zim/ifixit.zim',
|
||||
]))
|
||||
const groups = groupAndSortKbFiles(
|
||||
asInfos([
|
||||
'/app/storage/zim/wikipedia.zim',
|
||||
'/app/storage/zim/devdocs.zim',
|
||||
'/app/storage/zim/ifixit.zim',
|
||||
])
|
||||
)
|
||||
|
||||
assert.deepEqual(
|
||||
groups.map((g) => g.displayName),
|
||||
|
|
@ -109,10 +111,7 @@ test('groupAndSortKbFiles handles empty input', () => {
|
|||
})
|
||||
|
||||
test('groupAndSortKbFiles preserves a stable synthetic key for the admin docs group', () => {
|
||||
const groups = groupAndSortKbFiles(asInfos([
|
||||
'/app/docs/release-notes.md',
|
||||
'/app/docs/maps.md',
|
||||
]))
|
||||
const groups = groupAndSortKbFiles(asInfos(['/app/docs/release-notes.md', '/app/docs/maps.md']))
|
||||
// The admin-docs row uses a synthetic source key (not a real path) so it
|
||||
// can be used as a React key without colliding with any real file row.
|
||||
assert.equal(groups[0].source, '__admin_docs_group__')
|
||||
|
|
@ -122,49 +121,154 @@ test('groupAndSortKbFiles preserves a stable synthetic key for the admin docs gr
|
|||
* three sort keys (name, size, uploadedAt) produce visibly different orders —
|
||||
* if a test passed for name it had better fail for size. */
|
||||
const sized: StoredFileInfo[] = [
|
||||
{ source: '/app/storage/kb_uploads/charlie.txt', state: null, chunksEmbedded: 0, fileName: 'charlie.txt', size: 100, uploadedAt: '2026-01-01T00:00:00Z', isUserUpload: true, collection: null },
|
||||
{ source: '/app/storage/kb_uploads/alpha.txt', state: null, chunksEmbedded: 0, fileName: 'alpha.txt', size: 300, uploadedAt: '2026-03-01T00:00:00Z', isUserUpload: true, collection: null },
|
||||
{ source: '/app/storage/kb_uploads/bravo.txt', state: null, chunksEmbedded: 0, fileName: 'bravo.txt', size: 200, uploadedAt: '2026-02-01T00:00:00Z', isUserUpload: true, collection: null },
|
||||
{
|
||||
source: '/app/storage/kb_uploads/charlie.txt',
|
||||
state: null,
|
||||
chunksEmbedded: 0,
|
||||
fileName: 'charlie.txt',
|
||||
size: 100,
|
||||
uploadedAt: '2026-01-01T00:00:00Z',
|
||||
isUserUpload: true,
|
||||
collection: null,
|
||||
active: true,
|
||||
},
|
||||
{
|
||||
source: '/app/storage/kb_uploads/alpha.txt',
|
||||
state: null,
|
||||
chunksEmbedded: 0,
|
||||
fileName: 'alpha.txt',
|
||||
size: 300,
|
||||
uploadedAt: '2026-03-01T00:00:00Z',
|
||||
isUserUpload: true,
|
||||
collection: null,
|
||||
active: true,
|
||||
},
|
||||
{
|
||||
source: '/app/storage/kb_uploads/bravo.txt',
|
||||
state: null,
|
||||
chunksEmbedded: 0,
|
||||
fileName: 'bravo.txt',
|
||||
size: 200,
|
||||
uploadedAt: '2026-02-01T00:00:00Z',
|
||||
isUserUpload: true,
|
||||
collection: null,
|
||||
active: true,
|
||||
},
|
||||
]
|
||||
|
||||
// These fixtures are all `collection: null`, so they land in a single
|
||||
// "Uncategorized" header row that must be expanded (via
|
||||
// UNCATEGORIZED_COLLECTION_KEY) to see the sorted member rows.
|
||||
const expandUncategorized = new Set([UNCATEGORIZED_COLLECTION_KEY])
|
||||
const fileRowsOf = (groups: ReturnType<typeof groupAndSortKbFiles>) =>
|
||||
groups.filter((g) => !g.isCollectionHeader)
|
||||
|
||||
test('groupAndSortKbFiles sorts by size ascending', () => {
|
||||
const groups = groupAndSortKbFiles(sized, { key: 'size', direction: 'asc' })
|
||||
assert.deepEqual(groups.map((g) => g.displayName), ['charlie.txt', 'bravo.txt', 'alpha.txt'])
|
||||
const groups = groupAndSortKbFiles(sized, { key: 'size', direction: 'asc' }, expandUncategorized)
|
||||
assert.deepEqual(
|
||||
fileRowsOf(groups).map((g) => g.displayName),
|
||||
['charlie.txt', 'bravo.txt', 'alpha.txt']
|
||||
)
|
||||
})
|
||||
|
||||
test('groupAndSortKbFiles sorts by size descending', () => {
|
||||
const groups = groupAndSortKbFiles(sized, { key: 'size', direction: 'desc' })
|
||||
assert.deepEqual(groups.map((g) => g.displayName), ['alpha.txt', 'bravo.txt', 'charlie.txt'])
|
||||
const groups = groupAndSortKbFiles(sized, { key: 'size', direction: 'desc' }, expandUncategorized)
|
||||
assert.deepEqual(
|
||||
fileRowsOf(groups).map((g) => g.displayName),
|
||||
['alpha.txt', 'bravo.txt', 'charlie.txt']
|
||||
)
|
||||
})
|
||||
|
||||
test('groupAndSortKbFiles sorts by uploadedAt ascending', () => {
|
||||
const groups = groupAndSortKbFiles(sized, { key: 'uploadedAt', direction: 'asc' })
|
||||
assert.deepEqual(groups.map((g) => g.displayName), ['charlie.txt', 'bravo.txt', 'alpha.txt'])
|
||||
const groups = groupAndSortKbFiles(
|
||||
sized,
|
||||
{ key: 'uploadedAt', direction: 'asc' },
|
||||
expandUncategorized
|
||||
)
|
||||
assert.deepEqual(
|
||||
fileRowsOf(groups).map((g) => g.displayName),
|
||||
['charlie.txt', 'bravo.txt', 'alpha.txt']
|
||||
)
|
||||
})
|
||||
|
||||
test('groupAndSortKbFiles sorts by uploadedAt descending', () => {
|
||||
const groups = groupAndSortKbFiles(sized, { key: 'uploadedAt', direction: 'desc' })
|
||||
assert.deepEqual(groups.map((g) => g.displayName), ['alpha.txt', 'bravo.txt', 'charlie.txt'])
|
||||
const groups = groupAndSortKbFiles(
|
||||
sized,
|
||||
{ key: 'uploadedAt', direction: 'desc' },
|
||||
expandUncategorized
|
||||
)
|
||||
assert.deepEqual(
|
||||
fileRowsOf(groups).map((g) => g.displayName),
|
||||
['alpha.txt', 'bravo.txt', 'charlie.txt']
|
||||
)
|
||||
})
|
||||
|
||||
test('groupAndSortKbFiles parks files with null size at the end of size sort', () => {
|
||||
const withMissing: StoredFileInfo[] = [
|
||||
...sized,
|
||||
{ source: '/app/storage/kb_uploads/zzz_missing.txt', state: null, chunksEmbedded: 0, fileName: 'zzz_missing.txt', size: null, uploadedAt: null, isUserUpload: true, collection: null },
|
||||
{
|
||||
source: '/app/storage/kb_uploads/zzz_missing.txt',
|
||||
state: null,
|
||||
chunksEmbedded: 0,
|
||||
fileName: 'zzz_missing.txt',
|
||||
size: null,
|
||||
uploadedAt: null,
|
||||
isUserUpload: true,
|
||||
collection: null,
|
||||
active: true,
|
||||
},
|
||||
]
|
||||
// Missing-size files sort last regardless of direction so the "real" data
|
||||
// owns the top of the view either way.
|
||||
const asc = groupAndSortKbFiles(withMissing, { key: 'size', direction: 'asc' })
|
||||
assert.equal(asc.at(-1)?.displayName, 'zzz_missing.txt')
|
||||
const desc = groupAndSortKbFiles(withMissing, { key: 'size', direction: 'desc' })
|
||||
assert.equal(desc.at(-1)?.displayName, 'zzz_missing.txt')
|
||||
const asc = groupAndSortKbFiles(
|
||||
withMissing,
|
||||
{ key: 'size', direction: 'asc' },
|
||||
expandUncategorized
|
||||
)
|
||||
assert.equal(fileRowsOf(asc).at(-1)?.displayName, 'zzz_missing.txt')
|
||||
const desc = groupAndSortKbFiles(
|
||||
withMissing,
|
||||
{ key: 'size', direction: 'desc' },
|
||||
expandUncategorized
|
||||
)
|
||||
assert.equal(fileRowsOf(desc).at(-1)?.displayName, 'zzz_missing.txt')
|
||||
})
|
||||
|
||||
test('groupAndSortKbFiles preserves bucket order across all sort modes', () => {
|
||||
const mixed: StoredFileInfo[] = [
|
||||
{ source: '/app/storage/zim/big.zim', state: null, chunksEmbedded: 0, fileName: 'big.zim', size: 999, uploadedAt: '2026-01-01T00:00:00Z', isUserUpload: false, collection: null },
|
||||
{ source: '/app/storage/kb_uploads/small.txt', state: null, chunksEmbedded: 0, fileName: 'small.txt', size: 1, uploadedAt: '2026-09-01T00:00:00Z', isUserUpload: true, collection: null },
|
||||
{ source: '/app/docs/release-notes.md', state: null, chunksEmbedded: 0, fileName: 'release-notes.md', size: 50, uploadedAt: '2026-05-01T00:00:00Z', isUserUpload: false, collection: null },
|
||||
{
|
||||
source: '/app/storage/zim/big.zim',
|
||||
state: null,
|
||||
chunksEmbedded: 0,
|
||||
fileName: 'big.zim',
|
||||
size: 999,
|
||||
uploadedAt: '2026-01-01T00:00:00Z',
|
||||
isUserUpload: false,
|
||||
collection: null,
|
||||
active: true,
|
||||
},
|
||||
{
|
||||
source: '/app/storage/kb_uploads/small.txt',
|
||||
state: null,
|
||||
chunksEmbedded: 0,
|
||||
fileName: 'small.txt',
|
||||
size: 1,
|
||||
uploadedAt: '2026-09-01T00:00:00Z',
|
||||
isUserUpload: true,
|
||||
collection: null,
|
||||
active: true,
|
||||
},
|
||||
{
|
||||
source: '/app/docs/release-notes.md',
|
||||
state: null,
|
||||
chunksEmbedded: 0,
|
||||
fileName: 'release-notes.md',
|
||||
size: 50,
|
||||
uploadedAt: '2026-05-01T00:00:00Z',
|
||||
isUserUpload: false,
|
||||
collection: null,
|
||||
active: true,
|
||||
},
|
||||
]
|
||||
// Even if size-desc would put zim first naturally, sort runs *within* a
|
||||
// bucket — buckets themselves stay in the canonical zim → upload → admin_docs
|
||||
|
|
@ -183,12 +287,115 @@ test('groupAndSortKbFiles preserves bucket order across all sort modes', () => {
|
|||
})
|
||||
|
||||
test('groupAndSortKbFiles emits null metadata + isUserUpload=false for the admin_docs group', () => {
|
||||
const groups = groupAndSortKbFiles(asInfos([
|
||||
'/app/docs/release-notes.md',
|
||||
'/app/README.md',
|
||||
]))
|
||||
const groups = groupAndSortKbFiles(asInfos(['/app/docs/release-notes.md', '/app/README.md']))
|
||||
assert.equal(groups[0].bucket, 'admin_docs')
|
||||
assert.equal(groups[0].size, null)
|
||||
assert.equal(groups[0].uploadedAt, null)
|
||||
assert.equal(groups[0].isUserUpload, false)
|
||||
})
|
||||
|
||||
/** Fixtures for the KB-collection grouping tests: two named collections
|
||||
* ("diy" with one file, "recipes" with a mixed active/inactive pair) plus an
|
||||
* untagged file that should fall into the "Uncategorized" header. */
|
||||
const collectioned: StoredFileInfo[] = [
|
||||
{
|
||||
source: '/app/storage/kb_uploads/recipe_active.txt',
|
||||
state: null,
|
||||
chunksEmbedded: 0,
|
||||
fileName: 'recipe_active.txt',
|
||||
size: 10,
|
||||
uploadedAt: null,
|
||||
isUserUpload: true,
|
||||
collection: 'recipes',
|
||||
active: true,
|
||||
},
|
||||
{
|
||||
source: '/app/storage/kb_uploads/recipe_off.txt',
|
||||
state: null,
|
||||
chunksEmbedded: 0,
|
||||
fileName: 'recipe_off.txt',
|
||||
size: 20,
|
||||
uploadedAt: null,
|
||||
isUserUpload: true,
|
||||
collection: 'recipes',
|
||||
active: false,
|
||||
},
|
||||
{
|
||||
source: '/app/storage/kb_uploads/diy_guide.txt',
|
||||
state: null,
|
||||
chunksEmbedded: 0,
|
||||
fileName: 'diy_guide.txt',
|
||||
size: 30,
|
||||
uploadedAt: null,
|
||||
isUserUpload: true,
|
||||
collection: 'diy',
|
||||
active: true,
|
||||
},
|
||||
{
|
||||
source: '/app/storage/kb_uploads/loose_note.txt',
|
||||
state: null,
|
||||
chunksEmbedded: 0,
|
||||
fileName: 'loose_note.txt',
|
||||
size: 40,
|
||||
uploadedAt: null,
|
||||
isUserUpload: true,
|
||||
collection: null,
|
||||
active: true,
|
||||
},
|
||||
]
|
||||
|
||||
test('groupAndSortKbFiles collapses upload files into per-collection header rows by default', () => {
|
||||
const groups = groupAndSortKbFiles(collectioned)
|
||||
// Nothing expanded -> just the three headers, no individual file rows.
|
||||
assert.equal(groups.length, 3)
|
||||
assert.ok(groups.every((g) => g.isCollectionHeader))
|
||||
})
|
||||
|
||||
test('groupAndSortKbFiles orders KB-collection headers alphabetically with Uncategorized last', () => {
|
||||
const groups = groupAndSortKbFiles(collectioned)
|
||||
assert.deepEqual(
|
||||
groups.map((g) => g.collection),
|
||||
['diy', 'recipes', null]
|
||||
)
|
||||
})
|
||||
|
||||
test('groupAndSortKbFiles labels the null-collection header "Uncategorized" with a file count', () => {
|
||||
const groups = groupAndSortKbFiles(collectioned)
|
||||
const header = groups.find((g) => g.collection === null)
|
||||
assert.equal(header?.displayName, 'Uncategorized · 1 file')
|
||||
})
|
||||
|
||||
test('groupAndSortKbFiles computes all-active, all-inactive, and mixed aggregate state per header', () => {
|
||||
const groups = groupAndSortKbFiles(collectioned)
|
||||
const diy = groups.find((g) => g.collection === 'diy')
|
||||
const recipes = groups.find((g) => g.collection === 'recipes')
|
||||
assert.equal(diy?.collectionActiveState, 'all-active')
|
||||
assert.equal(recipes?.collectionActiveState, 'mixed')
|
||||
})
|
||||
|
||||
test('groupAndSortKbFiles expands only the requested collection into member rows', () => {
|
||||
const groups = groupAndSortKbFiles(
|
||||
collectioned,
|
||||
{ key: 'name', direction: 'asc' },
|
||||
new Set(['recipes'])
|
||||
)
|
||||
const recipesIdx = groups.findIndex((g) => g.collection === 'recipes' && g.isCollectionHeader)
|
||||
assert.equal(groups[recipesIdx + 1]?.displayName, 'recipe_active.txt')
|
||||
assert.equal(groups[recipesIdx + 2]?.displayName, 'recipe_off.txt')
|
||||
// diy and Uncategorized stay collapsed -- their next entry is another header.
|
||||
const diyIdx = groups.findIndex((g) => g.collection === 'diy')
|
||||
assert.equal(groups[diyIdx + 1]?.isCollectionHeader, true)
|
||||
})
|
||||
|
||||
test('groupAndSortKbFiles uses UNCATEGORIZED_COLLECTION_KEY to expand null-collection files', () => {
|
||||
const groups = groupAndSortKbFiles(
|
||||
collectioned,
|
||||
{ key: 'name', direction: 'asc' },
|
||||
new Set([UNCATEGORIZED_COLLECTION_KEY])
|
||||
)
|
||||
const fileRows = groups.filter((g) => !g.isCollectionHeader)
|
||||
assert.deepEqual(
|
||||
fileRows.map((g) => g.displayName),
|
||||
['loose_note.txt']
|
||||
)
|
||||
})
|
||||
|
|
|
|||
|
|
@ -183,6 +183,9 @@ export type StoredFileInfo = {
|
|||
isUserUpload: boolean
|
||||
/** Subject/category tag, or null if uncategorized. */
|
||||
collection: string | null
|
||||
/** Whether this file's chunks are included in RAG search results. Toggling
|
||||
* this never deletes or re-embeds vectors — see #1119. */
|
||||
active: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user