mirror of
https://github.com/Crosstalk-Solutions/project-nomad.git
synced 2026-10-17 21:29:20 +02:00
97c65cca67
The Stored Files modal and per-file warnings panel both enumerate distinct 'source' values in the qdrant content collection. The existing implementation scrolls every point in the collection, 100 per page, just to learn the unique sources. On a fully-ingested NOMAD this is brutally slow: NOMAD3's 3.1M-point collection takes 50+ seconds to return its ~40 distinct sources, and the same scan runs twice per modal open (getStoredFiles + computeFileWarnings each pay it independently). Qdrant 1.10+ supports a facet aggregation that returns the distinct values of a payload field with their counts in a single call. @qdrant/js-client-rest@1.16.2 (already pinned) exposes this as `client.facet()`. Swap the three scroll loops: - RagService.getStoredFiles - RagService.computeFileWarnings - RagService.\_scanAndSync (the source-set used for skip-already-embedded decisions) As a bonus, computeFileWarnings's per-source chunk count comes back inline from the facet, so the secondary scroll that was incrementing a counter is also gone. `exact: true` everywhere because the counts feed Warning A's zero_chunks decision and could mis-fire near thresholds otherwise. New RagService.FACET_SOURCE_LIMIT = 10000 caps the facet response; real NOMADs run ~100 sources max, this is comfortable headroom. Expected impact on NOMAD3 (3M points, 40 sources): ~50s -> ~100ms per endpoint. On an empty/fresh KB: no measurable change either way. |
||
|---|---|---|
| .. | ||
| controllers | ||
| exceptions | ||
| jobs | ||
| middleware | ||
| models | ||
| services | ||
| utils | ||
| validators | ||