github-mirror/project-nomad
1
0
Fork 0
mirror of https://github.com/Crosstalk-Solutions/project-nomad.git synced 2026-10-17 05:14:19 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

fix(zim): use the OPDS catalog host, not Kiwix's gated browse host

Browse Source 
browse.library.kiwix.org grew an anti-crawler interstitial on 2026-08-29
that answers HTTP 200 with an HTML confirmation page. Content Explorer's
remote listing 500s on it and ZIM auto-update silently reports "no
updates". opds.library.kiwix.org serves the same OPDS API ungated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV
This commit is contained in:
Chris Sherwood 2026-08-31 14:03:41 -07:00 committed by Jake Turner
parent 95db80d156
commit 9ed3ac6a46
2 changed files with 12 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
admin/app/services/kiwix_catalog_service.ts
View File

@ -24,7 +24,14 @@ import { isRawListRemoteZimFilesResponse } from '../../util/zim.js'
* defensive — a malformed entry is skipped, never thrown.
*/
const KIWIX_CATALOG_URL = 'https://browse.library.kiwix.org/catalog/v2/entries'
// `browse.library.kiwix.org` is Kiwix's *human* browsing host. On 2026-08-29 it grew an
// anti-crawler interstitial that answers HTTP 200 with an HTML "Please confirm..." page
// requiring JS + a cookie, so every catalog request here returned unparseable HTML.
// `parseZimEntries` swallows that as an empty feed, which reads as "no updates available"
// -- a silent, permanent stall of ZIM auto-update. `opds.library.kiwix.org` is the
// machine-facing host for the same API and is not gated; `library.kiwix.org` now 301s to
// it. Keep catalog traffic off the browse host.
const KIWIX_CATALOG_URL = 'https://opds.library.kiwix.org/catalog/v2/entries'
const GITHUB_PMTILES_URL =
'https://api.github.com/repos/Crosstalk-Solutions/project-nomad-maps/contents/pmtiles'

5
admin/app/services/zim_service.ts
View File

@ -81,7 +81,10 @@ export class ZimService {
count: number
query?: string
}): Promise<ListRemoteZimFilesResponse> {
const LIBRARY_BASE_URL = 'https://browse.library.kiwix.org/catalog/v2/entries'
// The machine-facing OPDS host. NOT `browse.library.kiwix.org`, which since 2026-08-29
// serves an anti-crawler confirmation page (HTTP 200 + HTML) that fails
// `isRawListRemoteZimFilesResponse` and 500s this endpoint. See kiwix_catalog_service.
const LIBRARY_BASE_URL = 'https://opds.library.kiwix.org/catalog/v2/entries'
// Kiwix returns pages of content unaware of what the user has installed locally. When
// the installed set is large, a single 12-item Kiwix page can come back with everything
// already installed → 0 post-filter items → frontend deadlock (#731). Accumulate across