mirror of
https://github.com/Crosstalk-Solutions/project-nomad.git
synced 2026-10-17 05:14:19 +02:00
dev
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Chris Sherwood
|
ee203df485 |
fix(translate): download every released language, and list them in the docs
fetch_models.py sorted versions with float(), and Firefox Remote Settings
now publishes Nightly pre-releases ("1.0a1") alongside each release. The
first one crashed the whole fetch, so adding Swedish, or 21 of the 34 named
languages, silently did nothing and the button never appeared.
Only take records desktop Firefox would use on the release channel, pick
the newest complete version per direction instead of mixing files across
versions, support pairs that publish a separate source and target vocab
(Japanese), and keep one failing language from stopping the rest.
That opens 50 languages, up from the ~40 the docs claimed, including
Arabic, Japanese, Korean and Thai, which the docs said were unavailable.
All 50 downloaded, loaded and translated on the bergamot 0.4.5 engine in
the shipped image (NOMAD3). Chinese stays out: it is published as
zh-Hans/zh-Hant, which the proxy's two-letter codes cannot carry yet.
Translated pages now send Cache-Control: no-cache and put the installed
language set in the ETag. Kiwix's max-age=3600 kept the old language bar
for up to an hour after a language was added, and a normal refresh does
not reach the article frame.
Docs: step-by-step instructions and the full code table in the Supply
Depot section; sizes corrected to 45-140 MB per language.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Chris Sherwood
|
eef5b6cd37 |
fix(translate): forward upstream redirects instead of following them
urlopen() follows redirects silently. Kiwix answers a book root such as /content/<book> with a 302 to the ZIM's main page, so the proxy served that page's HTML at the original, shallower URL. Every relative link and stylesheet then resolved one directory too high: the page lost its styling and every article link 404'd (e.g. atenolol/ -> /content/atenolol/). Kiwix's /random is a 302 too and broke the same way. Use an opener that returns 3xx to the browser like any other non-200 response, and keep an absolute Location on this origin rather than the upstream container name. Bumps the catalog image to 0.1.1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
chriscrosstalk
|
1b67a87254 |
feat(translate): offline translation for the Information Library (#1292)
* feat(translate): offline translation for the Information Library Implements #1272. Ships as a Supply Depot app so users who do not need translation never see it. A path-preserving reverse proxy in front of kiwix-serve. /viewer, /skin/*, /search, /random and /catalog/* are forwarded byte for byte; only text/html under /content/ is rewritten. Preserving the path is the whole design: viewer.js keeps the content iframe in sync with location.hash by comparing contentWindow.location.pathname, so moving the path starts the ping-pong its own comment warns about and forces us to patch Kiwix's JavaScript on every Kiwix bump. Because the path never moves, no Kiwix JS is modified, and Kiwix's header, search, library and random keep working because they are Kiwix. Language is a cookie, so links inside the ZIM need no rewriting either. Bergamot rather than the AI Assistant: measured on the same box and the same passage, 12,818 words/s on CPU against llama3.1:8b at 8 words/s on an RTX 5060. It is also more careful with names, where the LLM translated the product name and a smaller model hallucinated a clause. No GPU, so it works on CPU-only boxes where the LLM path is unusable. Models come from Firefox Remote Settings, the endpoint Firefox itself uses, not the mozilla/firefox-translations-models repo, which was archived on 2026-08-21. MPL-2.0, about 37 MB per direction. Failure handling is deliberate throughout: a failed model fetch is non-fatal, a translation error returns the original article with a comment rather than an error page, and a box with no models still starts and still forwards the library. Translation being unavailable must never make the Information Library unreachable. amd64 only, and pinned as such in the build workflow. Bergamot's intgemm backend is x86 specific and there is no aarch64 wheel, so a multi-arch build would produce an ARM image that pulls cleanly and dies at start. Also adds .gitattributes pinning *.sh to LF. The repo stores LF but Windows checks out CRLF, and a CRLF after a shebang makes the interpreter unresolvable inside the image. CI builds on Linux so this never bit, but it breaks a local image build and would have bitten the existing sidecar script too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV * fix(translate): vary the cache on the language cookie Found by clicking the language buttons in a browser rather than trusting curl. Kiwix sends ETag and Cache-Control: max-age=3600, and the proxy forwarded both unchanged on a response whose content now varies by cookie. The browser cached the English article for an hour and served it straight back when the reader picked a language, so the feature looked like it simply did not work. A shared cache would have been worse, serving one reader's language to another. Translatable responses now send Vary: Cookie, and their ETag is namespaced by language so revalidation still works per language rather than being turned off. Forwarded paths are untouched, so /viewer, /skin/* and /catalog/* stay byte for byte identical to Kiwix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV * fix(translate): move off port 8480, which Vaultwarden already uses Caught by seeing the two cards side by side in Supply Depot with the same port. My port survey grepped ui_location for a bare number, and Vaultwarden's is 'https:8480' because it needs an https Open link, so it did not match and 8480 looked free. Moved to 8460, which fits the existing spacing between Meshtastic Web on 8450 and Homebox on 8470. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV * fix(translate): pin the base image to python 3.10 The build fails outright on 3.12: bergamot publishes no wheel for it and pip resolves no further than 0.1.2. The spike used 3.10 for this reason and I had modernised it without checking. Caught by building the image rather than reading it. Moving off this pin means moving off the 2022 PyPI wheel to a build from mozilla/translations, which is the follow-up already noted in the Dockerfile. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV * fix(translate): translate text that lives in a bare div Sotoki-generated StackExchange ZIMs put every question excerpt in a <div class="...excerpt">, so on a questions list the titles translated and the descriptions underneath them did not. Reported against the ham radio ZIM. div could not simply be added to BLOCK_TAGS: the outermost-block rule would claim a top-level layout wrapper and send the whole page as one job. It is now claimed speculatively and abandoned the moment anything block-level opens inside it, so a wrapper yields to the blocks it contains and only a genuine text leaf is translated. Also moves the test harness's reporting block back to the end of the file. New checks had been appended after it, so they executed but their failures were never reported: verified that removing div from the set now fails 3 checks, where before it reported success. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV * fix(translate): decode numeric character references before translating Bergamot's HTML mode decodes named entities (`&` arrives as "&") but not numeric ones. It treats `'` as literal text and escapes the ampersand on the way out, so "If I'm in the wilderness" came back as "Si I&#x27;m dans la nature" and rendered the escape sequence visibly to the user. Confirmed by round-tripping fragments through the model directly. Numeric refs are now decoded before the round trip. `&`, `<` and `>` are deliberately left encoded: decoding those would inject real markup into the fragment and fail the balance check that gates HTML mode, dropping the block to plain text and losing its inline links. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F8Mgg1vd2eKSs8StZiuyMV --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |