project-nomad/admin/commands
Chris Sherwood baf94bbd87 fix(queue): stop stalled-job recovery forking the embedding chain
Fixes #1269. Fixes #1203.

`file-embeddings` and `downloads` both routinely outlive the default
5 minute lock, so BullMQ calls healthy jobs stalled on both. They need
opposite recovery policies, which is why neither was covered by the
existing drug-queue override.

Embedding is a self-continuing chain, so re-queueing a stalled job while
the original is alive runs two chains over the same file. Both write to
Qdrant with freshly minted point ids, so the second cannot overwrite the
first, and the queue's concurrency of 2 means the fork never contends
for a slot. maxStalledCount 0 fails on the first stall instead.

A download is resumable, so recovery is safe there; the default of 1 was
failing jobs outright and bypassing their own attempts: 10 policy.

The decision moves to a pure, unit-tested util alongside the other
decision helpers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LtGWCVMPYn6qPWVYiycK74
2026-09-29 15:54:41 -07:00
..
app_auto_update feat(supply-depot): opt-in automatic updates for installed apps 2026-06-23 04:46:52 +00:00
auto_update feat(system): add opt-in automatic updates for the core NOMAD app 2026-06-23 04:46:51 +00:00
benchmark feat(Docker): container URL resolution util and networking improvs 2026-01-24 15:27:56 -08:00
content_auto_update feat(content): opt-in automatic updates for installed ZIM & map content 2026-06-23 04:47:01 +00:00
eval feat(AI): let retrieval decline instead of making the prompt paper over it 2026-09-29 15:54:41 -07:00
queue fix(queue): stop stalled-job recovery forking the embedding chain 2026-09-29 15:54:41 -07:00