mirror of
https://github.com/Crosstalk-Solutions/project-nomad.git
synced 2026-10-17 05:14:19 +02:00
a3811ad49f
The batch continuation gated on articles that produced chunks rather than articles the extractor consumed. Articles whose text is empty after HTML cleaning — redirect stubs, category listings, media and PDF wrappers — return no chunks and so no documentIds, which made a normal sparse batch look like the end of the archive. Ingestion stopped after the first batch and the rest of the file was silently skipped. Medicine LibreTexts embedded 16 chunks out of 23,171 articles: its first 50 entries are navigation pages, only 10 of which had text, so 10 >= 50 was false and no continuation was ever dispatched. Measured against the same install, WikiMed reached 28% of its articles, Libre Pathology 31%, MedlinePlus 84%, and CDC Travelers' Health 2.7%. The same value was also returned as `articlesProcessed`, which EmbedFileJob uses to advance the next batch offset. Advancing by the content-bearing count re-read the overlap on every sparse batch, and a batch yielding no text at all would have advanced by zero and re-run the same window indefinitely. Both now use the extractor's consumed-article count, which it already computed and logged but discarded. The end-of-archive signal is the iterator running dry (articlesProcessed < batchSize); archive.articleCount cannot serve as a bound because iterByPath() legitimately yields more article entries than that figure for some archives. The log line now reports both counts (content-bearing/consumed) so a sparse batch is visible without re-deriving it. |
||
|---|---|---|
| .. | ||
| eval | ||
| affirmative_remedies.ts | ||
| amd_hsa_override.ts | ||
| content_auto_update_backoff.ts | ||
| content_reindex_decision.ts | ||
| context_budget.ts | ||
| context_window.ts | ||
| downloads.ts | ||
| fs.ts | ||
| hosted_content_auth.ts | ||
| hosted_content.ts | ||
| image_disk_preflight.ts | ||
| kb_ingest_decision.ts | ||
| kb_job_health.ts | ||
| kb_ratio_lookup.ts | ||
| kb_warning_decision.ts | ||
| misc.ts | ||
| platform_metadata.ts | ||
| rag_prompt.ts | ||
| rag_relevance.ts | ||
| structured_output.ts | ||
| superseded_resource.ts | ||
| tasks_model.ts | ||
| think_stream.ts | ||
| token_estimate.ts | ||
| update_window.ts | ||
| version.ts | ||
| zim_batch_decision.ts | ||
| zim_download_resolution.ts | ||
| zim_filename.ts | ||
| zim_html.ts | ||