mirror of
https://github.com/Crosstalk-Solutions/project-nomad.git
synced 2026-10-17 05:14:19 +02:00
a3811ad49f
The batch continuation gated on articles that produced chunks rather than articles the extractor consumed. Articles whose text is empty after HTML cleaning — redirect stubs, category listings, media and PDF wrappers — return no chunks and so no documentIds, which made a normal sparse batch look like the end of the archive. Ingestion stopped after the first batch and the rest of the file was silently skipped. Medicine LibreTexts embedded 16 chunks out of 23,171 articles: its first 50 entries are navigation pages, only 10 of which had text, so 10 >= 50 was false and no continuation was ever dispatched. Measured against the same install, WikiMed reached 28% of its articles, Libre Pathology 31%, MedlinePlus 84%, and CDC Travelers' Health 2.7%. The same value was also returned as `articlesProcessed`, which EmbedFileJob uses to advance the next batch offset. Advancing by the content-bearing count re-read the overlap on every sparse batch, and a batch yielding no text at all would have advanced by zero and re-run the same window indefinitely. Both now use the extractor's consumed-article count, which it already computed and logged but discarded. The end-of-archive signal is the iterator running dry (articlesProcessed < batchSize); archive.articleCount cannot serve as a bound because iterByPath() legitimately yields more article entries than that figure for some archives. The log line now reports both counts (content-bearing/consumed) so a sparse batch is visible without re-deriving it.
42 lines
1.9 KiB
TypeScript
42 lines
1.9 KiB
TypeScript
/**
|
|
* Decision for whether a batched ZIM ingestion should dispatch a continuation.
|
|
*
|
|
* This is the pure, I/O-free core of the batch loop in
|
|
* `RagService.processZIMFile` (mirrors `decideScanAction` in
|
|
* `kb_ingest_decision.ts`).
|
|
*
|
|
* The signal MUST be the number of articles the extractor *consumed*, never the
|
|
* number that produced chunks. An article yields zero chunks whenever its text
|
|
* is empty after HTML cleaning — redirect stubs, category listings, image/video
|
|
* wrappers, PDF containers. Those are perfectly normal ZIM entries, but they are
|
|
* invisible to any count derived from the returned chunks.
|
|
*
|
|
* Gating on chunk-derived counts silently truncates ingestion the first time a
|
|
* window of `batchSize` articles happens to contain mostly empty ones: the
|
|
* continuation is never dispatched and every remaining article in the archive is
|
|
* skipped. That is not a rare edge case — it is the norm for scraped-site and
|
|
* media-heavy archives, where the opening entries are usually navigation pages.
|
|
*
|
|
* `articlesProcessed < batchSize` means the extractor's iterator ran dry, which
|
|
* is the only reliable end-of-archive signal available: `archive.articleCount`
|
|
* cannot serve as an upper bound, because `iterByPath()` legitimately yields
|
|
* more article entries than that figure for some archives.
|
|
*
|
|
* A full final batch costs one extra dispatch that extracts nothing and stops.
|
|
* That is deliberate: over-running by a single empty batch is cheap, while
|
|
* stopping one batch early loses the remainder of the archive.
|
|
*/
|
|
export interface ZimBatchProgress {
|
|
/** Articles the extractor consumed in this batch (not articles that produced chunks). */
|
|
articlesProcessed: number
|
|
/** The article ceiling requested for this batch. */
|
|
batchSize: number
|
|
}
|
|
|
|
export function hasMoreArticleBatches({
|
|
articlesProcessed,
|
|
batchSize,
|
|
}: ZimBatchProgress): boolean {
|
|
return articlesProcessed >= batchSize
|
|
}
|