docs(release): finalize v1.31.0 release notes [skip ci]

Defaults to metric for global audience. Persists choice in localStorage.
Segmented button styled to match MapLibre controls.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

.dockerignore

@@ -3,3 +3,6 @@
 .git
 node_modules
 *.log
+admin/storage
+admin/node_modules
+admin/build

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,193 @@
+name: Bug Report
+description: Report a bug or issue with Project N.O.M.A.D.
+title: "[Bug]: "
+labels: ["bug", "needs-triage"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to report a bug! Please fill out the information below to help us diagnose and fix the issue.
+        
+        **Before submitting:**
+        - Search existing issues to avoid duplicates
+        - Ensure you're running the latest version of N.O.M.A.D.
+        - Redact any personal or sensitive information from logs/configs
+        - Please don't submit issues related to running N.O.M.A.D. on Unraid or another NAS - we don't have plans to support these kinds of platforms at this time
+
+  - type: dropdown
+    id: issue-category
+    attributes:
+      label: Issue Category
+      description: What area is this issue related to?
+      options:
+        - Installation/Setup
+        - AI Assistant (Ollama)
+        - Knowledge Base/RAG (Document Upload)
+        - Docker/Container Issues
+        - GPU Configuration
+        - Content Downloads (ZIM, Maps, Collections)
+        - Service Management (Start/Stop/Update)
+        - System Performance/Resources
+        - UI/Frontend Issue
+        - Other
+    validations:
+      required: true
+
+  - type: textarea
+    id: description
+    attributes:
+      label: Bug Description
+      description: Provide a clear and concise description of what the bug is
+      placeholder: What happened? What did you expect to happen?
+    validations:
+      required: true
+
+  - type: textarea
+    id: reproduction
+    attributes:
+      label: Steps to Reproduce
+      description: How can we reproduce this issue?
+      placeholder: |
+        1. Go to '...'
+        2. Click on '...'
+        3. See error
+    validations:
+      required: true
+
+  - type: textarea
+    id: expected-behavior
+    attributes:
+      label: Expected Behavior
+      description: What did you expect to happen?
+      placeholder: Describe the expected outcome
+    validations:
+      required: true
+
+  - type: textarea
+    id: actual-behavior
+    attributes:
+      label: Actual Behavior
+      description: What actually happened?
+      placeholder: Describe what actually occurred, including any error messages
+    validations:
+      required: true
+
+  - type: input
+    id: nomad-version
+    attributes:
+      label: N.O.M.A.D. Version
+      description: What version of N.O.M.A.D. are you running? (Check Settings > Update or run `docker ps` and check nomad_admin image tag)
+      placeholder: "e.g., 1.29.0"
+    validations:
+      required: true
+
+  - type: dropdown
+    id: os
+    attributes:
+      label: Operating System
+      description: What OS are you running N.O.M.A.D. on?
+      options:
+        - Ubuntu 24.04
+        - Ubuntu 22.04
+        - Ubuntu 20.04
+        - Debian 13 (Trixie)
+        - Debian 12 (Bookworm)
+        - Debian 11 (Bullseye)
+        - Other Debian-based
+        - Other (not yet officially supported)
+    validations:
+      required: true
+
+  - type: input
+    id: docker-version
+    attributes:
+      label: Docker Version
+      description: What version of Docker are you running? (`docker --version`)
+      placeholder: "e.g., Docker version 24.0.7"
+
+  - type: dropdown
+    id: gpu-present
+    attributes:
+      label: Do you have a dedicated GPU?
+      options:
+        - "Yes"
+        - "No"
+        - "Not sure"
+    validations:
+      required: true
+
+  - type: input
+    id: gpu-model
+    attributes:
+      label: GPU Model (if applicable)
+      description: What GPU model do you have? (Check Settings > System or run `nvidia-smi` if NVIDIA GPU)
+      placeholder: "e.g., NVIDIA GeForce RTX 3060"
+
+  - type: textarea
+    id: system-specs
+    attributes:
+      label: System Specifications
+      description: Provide relevant system specs (CPU, RAM, available disk space)
+      placeholder: |
+        CPU: 
+        RAM: 
+        Available Disk Space: 
+        GPU (if any): 
+
+  - type: textarea
+    id: service-status
+    attributes:
+      label: Service Status (if relevant)
+      description: If this is a service-related issue, what's the status of relevant services? (Check Settings > Apps or run `docker ps`)
+      placeholder: |
+        Paste output from `docker ps` or describe service states from the UI
+
+  - type: textarea
+    id: logs
+    attributes:
+      label: Relevant Logs
+      description: |
+        Include any relevant logs or error messages. **Please redact any personal/sensitive information.**
+        
+        Useful commands for collecting logs:
+        - N.O.M.A.D. management app: `docker logs nomad_admin`
+        - Ollama: `docker logs nomad_ollama`
+        - Qdrant: `docker logs nomad_qdrant`
+        - Specific service: `docker logs nomad_<service-name>`
+      placeholder: Paste relevant log output here
+      render: shell
+
+  - type: textarea
+    id: browser-console
+    attributes:
+      label: Browser Console Errors (if UI issue)
+      description: If this is a UI issue, include any errors from your browser's developer console (F12)
+      placeholder: Paste browser console errors here
+      render: javascript
+
+  - type: textarea
+    id: screenshots
+    attributes:
+      label: Screenshots
+      description: If applicable, add screenshots to help explain your problem (drag and drop images here)
+
+  - type: textarea
+    id: additional-context
+    attributes:
+      label: Additional Context
+      description: Add any other context about the problem here (network setup, custom configurations, recent changes, etc.)
+
+  - type: checkboxes
+    id: terms
+    attributes:
+      label: Pre-submission Checklist
+      description: Please confirm the following before submitting
+      options:
+        - label: I have searched for existing issues that might be related to this bug
+          required: true
+        - label: I am running the latest version of Project N.O.M.A.D. (or have noted my version above)
+          required: true
+        - label: I have redacted any personal or sensitive information from logs and screenshots
+          required: true
+        - label: This issue is NOT related to running N.O.M.A.D. on an unsupported/non-Debian-based OS
+          required: false

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,17 @@
+blank_issues_enabled: false
+contact_links:
+  - name: 💬 Discord Community
+    url: https://discord.com/invite/crosstalksolutions
+    about: Join our Discord community for general questions, support, and discussions
+  - name: 📖 Documentation
+    url: https://projectnomad.us
+    about: Check the official documentation and guides
+  - name: 🏆 Community Leaderboard
+    url: https://benchmark.projectnomad.us
+    about: View the N.O.M.A.D. benchmark leaderboard
+  - name: 🤝 Contributing Guide
+    url: https://github.com/Crosstalk-Solutions/project-nomad/blob/main/CONTRIBUTING.md
+    about: Learn how to contribute to Project N.O.M.A.D.
+  - name: 📅 Roadmap
+    url: https://roadmap.projectnomad.us
+    about: See our public roadmap, vote on features, and suggest new ones

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,150 @@
+name: Feature Request
+description: Suggest a new feature or enhancement for Project N.O.M.A.D.
+title: "[Feature]: "
+labels: ["enhancement", "needs-discussion"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for your interest in improving Project N.O.M.A.D.! Before you submit a feature request, consider checking our [roadmap](https://roadmap.projectnomad.us) to see if it's already planned or in progress. You're welcome to suggest new ideas there if you don't plan on opening PRs yourself.
+
+        
+        **Please note:** Feature requests are not guaranteed to be implemented. All requests are evaluated based on alignment with the project's goals, feasibility, and community demand.
+        
+        **Before submitting:**
+        - Search existing feature requests and our [roadmap](https://roadmap.projectnomad.us) to avoid duplicates
+        - Consider if this aligns with N.O.M.A.D.'s mission: offline-first knowledge and education
+        - Consider the technical feasibility of the feature: N.O.M.A.D. is designed to be containerized and run on a wide range of hardware, so features that require heavy resources (aside from GPU-intensive tasks) or complex host configurations may be less likely to be implemented
+        - Consider the scope of the feature: Small, focused enhancements that can be implemented incrementally are more likely to be implemented than large, broad features that would require significant development effort or have an unclear path forward
+        - If you're able to contribute code, testing, or documentation, that significantly increases the chances of your feature being implemented
+
+  - type: dropdown
+    id: feature-category
+    attributes:
+      label: Feature Category
+      description: What area does this feature relate to?
+      options:
+        - New Service/Tool Integration
+        - AI Assistant Enhancement
+        - Knowledge Base/RAG Improvement
+        - Content Management (ZIM, Maps, Collections)
+        - UI/UX Improvement
+        - System Management
+        - Performance Optimization
+        - Documentation
+        - Security
+        - Other
+    validations:
+      required: true
+
+  - type: textarea
+    id: problem
+    attributes:
+      label: Problem Statement
+      description: What problem does this feature solve? Is your feature request related to a pain point?
+      placeholder: I find it frustrating when... / It would be helpful if... / Users struggle with...
+    validations:
+      required: true
+
+  - type: textarea
+    id: solution
+    attributes:
+      label: Proposed Solution
+      description: Describe the feature or enhancement you'd like to see
+      placeholder: Add a feature that... / Change the behavior to... / Integrate with...
+    validations:
+      required: true
+
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: Alternative Solutions
+      description: Have you considered any alternative solutions or workarounds?
+      placeholder: I've tried... / Another approach could be... / A workaround is...
+
+  - type: textarea
+    id: use-case
+    attributes:
+      label: Use Case
+      description: Describe a specific scenario where this feature would be valuable
+      placeholder: |
+        As a [type of user], when I [do something], I want to [accomplish something] so that [benefit].
+        
+        Example: Because I have a dedicated GPU, I want to be able to see in the UI if GPU support is enabled so that I can optimize performance and troubleshoot issues more easily.
+
+  - type: dropdown
+    id: user-type
+    attributes:
+      label: Who would benefit from this feature?
+      description: What type of users would find this most valuable?
+      multiple: true
+      options:
+        - Individual/Home Users
+        - Families
+        - Teachers/Educators
+        - Students
+        - Survivalists/Preppers
+        - Developers/Contributors
+        - Organizations
+        - All Users
+    validations:
+      required: true
+
+  - type: dropdown
+    id: priority
+    attributes:
+      label: How important is this feature to you?
+      options:
+        - Critical - Blocking my use of N.O.M.A.D.
+        - High - Would significantly improve my experience
+        - Medium - Would be nice to have
+        - Low - Minor convenience
+    validations:
+      required: true
+
+  - type: textarea
+    id: implementation-ideas
+    attributes:
+      label: Implementation Ideas (Optional)
+      description: If you have technical suggestions for how this could be implemented, share them here
+      placeholder: This could potentially use... / It might integrate with... / A possible approach is...
+
+  - type: textarea
+    id: examples
+    attributes:
+      label: Examples or References
+      description: Are there similar features in other applications? Include links, screenshots, or descriptions
+      placeholder: Similar to how [app name] does... / See this example at [URL]
+
+  - type: dropdown
+    id: willing-to-contribute
+    attributes:
+      label: Would you be willing to help implement this?
+      description: Contributing increases the likelihood of implementation
+      options:
+        - "Yes - I can write the code"
+        - "Yes - I can help test"
+        - "Yes - I can help with documentation"
+        - "Maybe - with guidance"
+        - "No - I don't have the skills/time"
+    validations:
+      required: true
+
+  - type: textarea
+    id: additional-context
+    attributes:
+      label: Additional Context
+      description: Add any other context, mockups, diagrams, or information about the feature request
+
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Pre-submission Checklist
+      description: Please confirm the following before submitting
+      options:
+        - label: I have searched for existing feature requests that might be similar
+          required: true
+        - label: This feature aligns with N.O.M.A.D.'s mission of offline-first knowledge and education
+          required: true
+        - label: I understand that feature requests are not guaranteed to be implemented
+          required: true

.github/dependabot.yaml

@@ -0,0 +1,7 @@
+version: 2
+updates:
+  - package-ecosystem: "npm"
+    directory: "/admin"
+    schedule:
+      interval: "weekly"
+    target-branch: "rc"

.github/scripts/finalize-release-notes.sh

@@ -0,0 +1,133 @@
+#!/usr/bin/env bash
+#
+# finalize-release-notes.sh
+#
+# Stamps the "## Unreleased" section in a release-notes file with a version
+# and date, and extracts the section content for use in GitHub releases / email.
+# Also includes all commits since the last release for complete transparency.
+#
+# Usage:  finalize-release-notes.sh <version> <file-path>
+#
+# Exit codes:
+#   0 - Success: section stamped and extracted
+#   1 - No "## Unreleased" section found (skip gracefully)
+#   2 - Unreleased section exists but is empty (skip gracefully)
+
+set -euo pipefail
+
+VERSION="${1:?Usage: finalize-release-notes.sh <version> <file-path>}"
+FILE="${2:?Usage: finalize-release-notes.sh <version> <file-path>}"
+
+if [[ ! -f "$FILE" ]]; then
+  echo "Error: File not found: $FILE" >&2
+  exit 1
+fi
+
+# Find the line number of the ## Unreleased header (case-insensitive)
+HEADER_LINE=$(grep -inm1 '^## unreleased' "$FILE" | cut -d: -f1)
+
+if [[ -z "$HEADER_LINE" ]]; then
+  echo "No '## Unreleased' section found. Skipping."
+  exit 1
+fi
+
+TOTAL_LINES=$(wc -l < "$FILE")
+
+# Find the next section header (## Version ...) or --- separator after the Unreleased header
+NEXT_SECTION_LINE=""
+if [[ $HEADER_LINE -lt $TOTAL_LINES ]]; then
+  NEXT_SECTION_LINE=$(tail -n +"$((HEADER_LINE + 1))" "$FILE" \
+    | grep -nm1 '^## \|^---$' \
+    | cut -d: -f1)
+fi
+
+if [[ -n "$NEXT_SECTION_LINE" ]]; then
+  # NEXT_SECTION_LINE is relative to HEADER_LINE+1, convert to absolute
+  END_LINE=$((HEADER_LINE + NEXT_SECTION_LINE - 1))
+else
+  # Section runs to end of file
+  END_LINE=$TOTAL_LINES
+fi
+
+# Extract content between header and next section (exclusive of both boundaries)
+CONTENT_START=$((HEADER_LINE + 1))
+CONTENT_END=$END_LINE
+
+# Extract the section body (between header line and the next boundary)
+SECTION_BODY=$(sed -n "${CONTENT_START},${CONTENT_END}p" "$FILE" | sed '/^$/N;/^\n$/d')
+
+# Check for actual content: strip blank lines and lines that are only markdown headers (###...)
+TRIMMED=$(echo "$SECTION_BODY" | sed '/^[[:space:]]*$/d')
+HAS_CONTENT=$(echo "$SECTION_BODY" | sed '/^[[:space:]]*$/d' | grep -v '^###' || true)
+
+if [[ -z "$TRIMMED" || -z "$HAS_CONTENT" ]]; then
+  echo "Unreleased section is empty. Skipping."
+  exit 2
+fi
+
+# Format the date as "Month Day, Year"
+DATE_STAMP=$(date +'%B %-d, %Y')
+NEW_HEADER="## Version ${VERSION} - ${DATE_STAMP}"
+
+# Build the replacement: swap the header line, keep everything else intact
+{
+  # Lines before the Unreleased header
+  if [[ $HEADER_LINE -gt 1 ]]; then
+    head -n "$((HEADER_LINE - 1))" "$FILE"
+  fi
+  # New versioned header
+  echo "$NEW_HEADER"
+  # Content between header and next section
+  sed -n "${CONTENT_START},${CONTENT_END}p" "$FILE"
+  # Rest of the file after the section
+  if [[ $END_LINE -lt $TOTAL_LINES ]]; then
+    tail -n +"$((END_LINE + 1))" "$FILE"
+  fi
+} > "${FILE}.tmp"
+
+mv "${FILE}.tmp" "$FILE"
+
+# Get commits since the last release
+LAST_TAG=$(git describe --tags --abbrev=0 HEAD^ 2>/dev/null || echo "")
+COMMIT_LIST=""
+
+if [[ -n "$LAST_TAG" ]]; then
+  echo "Fetching commits since ${LAST_TAG}..."
+  # Get commits between last tag and HEAD, excluding merge commits and skip ci commits
+  COMMIT_LIST=$(git log "${LAST_TAG}..HEAD" \
+    --no-merges \
+    --pretty=format:"- %s ([%h](https://github.com/${GITHUB_REPOSITORY}/commit/%H))" \
+    --grep="\[skip ci\]" --invert-grep \
+    || echo "")
+else
+  echo "No previous tag found, fetching all commits..."
+  COMMIT_LIST=$(git log \
+    --no-merges \
+    --pretty=format:"- %s ([%h](https://github.com/${GITHUB_REPOSITORY}/commit/%H))" \
+    --grep="\[skip ci\]" --invert-grep \
+    || echo "")
+fi
+
+# Write the extracted section content (for GitHub release body / future email)
+{
+  echo "$NEW_HEADER"
+  echo ""
+  if [[ -n "$TRIMMED" ]]; then
+    echo "$TRIMMED"
+    echo ""
+  fi
+  
+  # Add commit history if available
+  if [[ -n "$COMMIT_LIST" ]]; then
+    echo "---"
+    echo ""
+    echo "### 📝 All Changes"
+    echo ""
+    echo "$COMMIT_LIST"
+  fi
+} > "${FILE}.section"
+
+echo "Finalized release notes for v${VERSION}"
+echo "  Updated: ${FILE}"
+echo "  Extracted: ${FILE}.section"
+exit 0

.github/workflows/build-admin-on-pr.yml

@@ -0,0 +1,25 @@
+name: Build Admin
+
+on: pull_request
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v6
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v6
+        with:
+          node-version: '24'
+          cache: 'npm'
+
+      - name: Install dependencies
+        run: npm ci
+        working-directory: ./admin
+
+      - name: Run build
+        run: npm run build
+        working-directory: ./admin

.github/workflows/build-disk-collector.yml

@@ -0,0 +1,51 @@
+name: Build Disk Collector Image
+
+on:
+  workflow_dispatch:
+    inputs:
+      version:
+        description: 'Semantic version to label the Docker image under (no "v" prefix, e.g. "1.2.3")'
+        required: true
+        type: string
+      tag_latest:
+        description: 'Also tag this image as :latest?'
+        required: false
+        type: boolean
+        default: false
+
+jobs:
+  check_authorization:
+    name: Check authorization to publish new Docker image
+    runs-on: ubuntu-latest
+    outputs:
+      isAuthorized: ${{ steps.check-auth.outputs.is_authorized }}
+    steps:
+      - name: check-auth
+        id: check-auth
+        run: echo "is_authorized=${{ contains(secrets.DEPLOYMENT_AUTHORIZED_USERS, github.triggering_actor) }}" >> $GITHUB_OUTPUT
+  build:
+    name: Build disk-collector image
+    needs: check_authorization
+    if: needs.check_authorization.outputs.isAuthorized == 'true'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v6
+      - name: Log in to GitHub Container Registry
+        uses: docker/login-action@v4
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Build and push
+        uses: docker/build-push-action@v7
+        with:
+          context: install/sidecar-disk-collector
+          push: true
+          tags: |
+            ghcr.io/crosstalk-solutions/project-nomad-disk-collector:${{ inputs.version }}
+            ghcr.io/crosstalk-solutions/project-nomad-disk-collector:v${{ inputs.version }}
+            ${{ inputs.tag_latest && 'ghcr.io/crosstalk-solutions/project-nomad-disk-collector:latest' || '' }}

.github/workflows/build-primary-image.yml

@@ -0,0 +1,54 @@
+name: Build Primary Docker Image
+
+on:
+  workflow_dispatch:
+    inputs:
+      version:
+        description: 'Semantic version to label the Docker image under (no "v" prefix, e.g. "1.2.3")'
+        required: true
+        type: string
+      tag_latest:
+        description: 'Also tag this image as :latest? (Keep false for RC and beta releases)'
+        required: false
+        type: boolean
+        default: false
+
+jobs:
+  check_authorization:
+    name: Check authorization to publish new Docker image
+    runs-on: ubuntu-latest
+    outputs:
+      isAuthorized: ${{ steps.check-auth.outputs.is_authorized }}
+    steps:
+      - name: check-auth
+        id: check-auth
+        run: echo "is_authorized=${{ contains(secrets.DEPLOYMENT_AUTHORIZED_USERS, github.triggering_actor) }}" >> $GITHUB_OUTPUT      
+  build:
+    name: Build Docker image
+    needs: check_authorization
+    if: needs.check_authorization.outputs.isAuthorized == 'true'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v6
+      - name: Log in to GitHub Container Registry
+        uses: docker/login-action@v4
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Build and push
+        uses: docker/build-push-action@v7
+        with:
+          push: true
+          tags: |
+            ghcr.io/crosstalk-solutions/project-nomad:${{ inputs.version }}
+            ghcr.io/crosstalk-solutions/project-nomad:v${{ inputs.version }}
+            ${{ inputs.tag_latest && 'ghcr.io/crosstalk-solutions/project-nomad:latest' || '' }}
+          build-args: |
+            VERSION=${{ inputs.version }}
+            BUILD_DATE=${{ github.event.workflow_run.created_at }}
+            VCS_REF=${{ github.sha }}

.github/workflows/build-sidecar-updater.yml

@@ -1,12 +1,17 @@
-name: Build Docker Image
+name: Build Sidecar Updater Image
 
 on:
   workflow_dispatch:
     inputs:
       version:
-        description: 'Semantic version to label the Docker image under'
+        description: 'Semantic version to label the Docker image under (no "v" prefix, e.g. "1.2.3")'
         required: true
         type: string
+      tag_latest:
+        description: 'Also tag this image as :latest?'
+        required: false
+        type: boolean
+        default: false
 
 jobs:
   check_authorization:
@@ -19,7 +24,7 @@ jobs:
         id: check-auth
         run: echo "is_authorized=${{ contains(secrets.DEPLOYMENT_AUTHORIZED_USERS, github.triggering_actor) }}" >> $GITHUB_OUTPUT
   build:
-    name: Build Docker image
+    name: Build sidecar-updater image
     needs: check_authorization
     if: needs.check_authorization.outputs.isAuthorized == 'true'
     runs-on: ubuntu-latest
@@ -28,7 +33,7 @@ jobs:
       packages: write
     steps:
       - name: Checkout code
-        uses: actions/checkout@v4
+        uses: actions/checkout@v6
       - name: Log in to GitHub Container Registry
         uses: docker/login-action@v2
         with:
@@ -38,7 +43,9 @@ jobs:
       - name: Build and push
         uses: docker/build-push-action@v5
         with:
+          context: install/sidecar-updater
           push: true
           tags: |
-            ghcr.io/crosstalk-solutions/project-nomad:${{ inputs.version }}
-            ghcr.io/crosstalk-solutions/project-nomad:latest
+            ghcr.io/crosstalk-solutions/project-nomad-sidecar-updater:${{ inputs.version }}
+            ghcr.io/crosstalk-solutions/project-nomad-sidecar-updater:v${{ inputs.version }}
+            ${{ inputs.tag_latest && 'ghcr.io/crosstalk-solutions/project-nomad-sidecar-updater:latest' || '' }}

.github/workflows/release.yml

@@ -22,12 +22,12 @@ jobs:
       newVersion: ${{ steps.semver.outputs.new_release_version }}
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@v6
         with:
           fetch-depth: 0
           persist-credentials: false
       - name: semantic-release
-        uses: cycjimmy/semantic-release-action@v3
+        uses: cycjimmy/semantic-release-action@v6
         id: semver
         env:
           GITHUB_TOKEN: ${{ secrets.COSMISTACKBOT_ACCESS_TOKEN }}
@@ -35,3 +35,57 @@ jobs:
           GIT_AUTHOR_EMAIL: dev@cosmistack.com
           GIT_COMMITTER_NAME: cosmistack-bot
           GIT_COMMITTER_EMAIL: dev@cosmistack.com
+
+      - name: Finalize release notes
+        # Skip for pre-releases (versions containing a hyphen, e.g. 1.27.0-rc.1)
+        if: |
+          steps.semver.outputs.new_release_published == 'true' &&
+          !contains(steps.semver.outputs.new_release_version, '-')
+        id: finalize-notes
+        env:
+          GITHUB_REPOSITORY: ${{ github.repository }}
+        run: |
+          git pull origin main
+          chmod +x .github/scripts/finalize-release-notes.sh
+          EXIT_CODE=0
+          .github/scripts/finalize-release-notes.sh \
+            "${{ steps.semver.outputs.new_release_version }}" \
+            admin/docs/release-notes.md || EXIT_CODE=$?
+          if [[ "$EXIT_CODE" -eq 0 ]]; then
+            echo "has_notes=true" >> $GITHUB_OUTPUT
+          else
+            echo "has_notes=false" >> $GITHUB_OUTPUT
+          fi
+
+      - name: Commit finalized release notes
+        if: |
+          steps.semver.outputs.new_release_published == 'true' &&
+          steps.finalize-notes.outputs.has_notes == 'true' &&
+          !contains(steps.semver.outputs.new_release_version, '-')
+        run: |
+          git config user.name "cosmistack-bot"
+          git config user.email "dev@cosmistack.com"
+          git remote set-url origin https://x-access-token:${{ secrets.COSMISTACKBOT_ACCESS_TOKEN }}@github.com/${{ github.repository }}.git
+          git add admin/docs/release-notes.md
+          git commit -m "docs(release): finalize v${{ steps.semver.outputs.new_release_version }} release notes [skip ci]"
+          git push origin main
+
+      - name: Update GitHub release body
+        if: |
+          steps.semver.outputs.new_release_published == 'true' &&
+          steps.finalize-notes.outputs.has_notes == 'true' &&
+          !contains(steps.semver.outputs.new_release_version, '-')
+        env:
+          GH_TOKEN: ${{ secrets.COSMISTACKBOT_ACCESS_TOKEN }}
+        run: |
+          gh release edit "v${{ steps.semver.outputs.new_release_version }}" \
+            --notes-file admin/docs/release-notes.md.section
+
+      # Future: Send release notes email
+      # - name: Send release notes email
+      #   if: steps.semver.outputs.new_release_published == 'true' && steps.finalize-notes.outputs.has_notes == 'true'
+      #   run: |
+      #     curl -X POST "https://api.projectnomad.us/api/v1/newsletter/release" \
+      #       -H "Authorization: Bearer ${{ secrets.NOMAD_API_KEY }}" \
+      #       -H "Content-Type: application/json" \
+      #       -d "{\"version\": \"${{ steps.semver.outputs.new_release_version }}\", \"body\": $(cat admin/docs/release-notes.md.section | jq -Rs .)}"

.github/workflows/validate-collection-urls.yml

@@ -0,0 +1,58 @@
+name: Validate Collection URLs
+
+on:
+  push:
+    paths:
+      - 'collections/**.json'
+  pull_request:
+    paths:
+      - 'collections/**.json'
+
+jobs:
+  validate-urls:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Extract and validate URLs
+        run: |
+          FAILED=0
+          CHECKED=0
+          FAILED_URLS=""
+
+          # Recursively extract all non-null string URLs from every JSON file in collections/
+          URLS=$(jq -r '.. | .url? | select(type == "string")' collections/*.json | sort -u)
+
+          while IFS= read -r url; do
+            [ -z "$url" ] && continue
+            CHECKED=$((CHECKED + 1))
+            printf "Checking: %s ... " "$url"
+
+            # Use Range: bytes=0-0 to avoid downloading the full file.
+            # --max-filesize 1 aborts early if the server ignores the Range header
+            # and returns 200 with the full body. The HTTP status is still captured.
+            HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
+              --range 0-0 \
+              --max-filesize 1 \
+              --max-time 30 \
+              --location \
+              "$url")
+
+            if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "206" ]; then
+              echo "OK ($HTTP_CODE)"
+            else
+              echo "FAILED ($HTTP_CODE)"
+              FAILED=$((FAILED + 1))
+              FAILED_URLS="$FAILED_URLS\n  - $url (HTTP $HTTP_CODE)"
+            fi
+          done <<< "$URLS"
+
+          echo ""
+          echo "Checked $CHECKED URLs, $FAILED failed."
+
+          if [ "$FAILED" -gt 0 ]; then
+            echo ""
+            echo "Broken URLs:"
+            printf "%b\n" "$FAILED_URLS"
+            exit 1
+          fi

.releaserc.json

@@ -1,5 +1,8 @@
 {
-  "branches": ["master"],
+  "branches": [
+    "main",
+    { "name": "rc", "prerelease": "rc" }
+  ],
   "plugins": [
     "@semantic-release/commit-analyzer",
     "@semantic-release/release-notes-generator",

CODE_OF_CONDUCT.md

@@ -0,0 +1,128 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity
+and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the
+  overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or
+  advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email
+  address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official e-mail address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+.
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series
+of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or
+permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior,  harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within
+the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.0, available at
+https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct
+enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see the FAQ at
+https://www.contributor-covenant.org/faq. Translations are available at
+https://www.contributor-covenant.org/translations.

CONTRIBUTING.md

@@ -0,0 +1,168 @@
+# Contributing to Project N.O.M.A.D.
+
+Thank you for your interest in contributing to Project N.O.M.A.D.! Community contributions are what keep this project growing and improving. Please read this guide fully before getting started — it will save you (and the maintainers) a lot of time.
+
+> **Note:** Acceptance of contributions is not guaranteed. All pull requests are evaluated based on quality, relevance, and alignment with the project's goals. The maintainers of Project N.O.M.A.D. ("Nomad") reserve the right accept, deny, or modify any pull request at their sole discretion.
+
+---
+
+## Table of Contents
+
+- [Code of Conduct](#code-of-conduct)
+- [Before You Start](#before-you-start)
+- [Getting Started](#getting-started)
+- [Development Workflow](#development-workflow)
+- [Commit Messages](#commit-messages)
+- [Release Notes](#release-notes)
+- [Versioning](#versioning)
+- [Submitting a Pull Request](#submitting-a-pull-request)
+- [Feedback & Community](#feedback--community)
+
+---
+
+## Code of Conduct
+
+Please read and review our full [Code of Conduct](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/CODE_OF_CONDUCT.md) before contributing. In short: please be respectful and considerate in all interactions with maintainers and other contributors.
+
+We are committed to providing a welcoming environment for everyone. Disrespectful or abusive behavior will not be tolerated. 
+
+---
+
+## Before You Start
+
+**Open an issue first.** Before writing any code, please [open an issue](../../issues/new) to discuss your proposed change. This helps avoid duplicate work and ensures your contribution aligns with the project's direction.
+
+When opening an issue:
+- Use a clear, descriptive title
+- Describe the problem you're solving or the feature you want to add
+- If it's a bug, include steps to reproduce it and as much detail about your environment as possible
+- Ensure you redact any personal or sensitive information in any logs, configs, etc.
+
+---
+
+## Getting Started with Contributing
+**Please note**: this is the Getting Started guide for developing and contributing to Nomad, NOT [installing Nomad](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/README.md) for regular use! 
+
+### Prerequisites
+
+- A Debian-based OS (Ubuntu recommended)
+- `sudo`/root privileges
+- Docker installed and running
+- A stable internet connection (required for dependency downloads)
+- Node.js (for frontend/admin work)
+
+### Fork & Clone
+
+1. Click **Fork** at the top right of this repository
+2. Clone your fork locally:
+   ```bash
+   git clone https://github.com/YOUR_USERNAME/project-nomad.git
+   cd project-nomad
+   ```
+3. Add the upstream remote so you can stay in sync:
+   ```bash
+   git remote add upstream https://github.com/Crosstalk-Solutions/project-nomad.git
+   ```
+
+### Avoid Installing a Release Version Locally
+Because Nomad relies heavily on Docker, we actually recommend against installing a release version of the project on the same local machine where you are developing. This can lead to conflicts with ports, volumes, and other resources. Instead, you can run your development version in a separate Docker environment while keeping your local machine clean. It certainly __can__ be done, but it adds complexity to your setup and workflow. If you choose to install a release version locally, please ensure you have a clear strategy for managing potential conflicts and resource usage.
+
+---
+
+## Development Workflow
+
+1. **Sync with upstream** before starting any new work. We prefer rebasing over merge commits to keep a clean, linear git history as much as possible (this also makes it easier for maintainers to review and merge your changes). To sync with upstream:
+   ```bash
+   git fetch upstream
+   git checkout dev
+   git rebase upstream/dev
+   ```
+
+2. **Create a feature branch** off `dev` with a descriptive name:
+   ```bash
+   git checkout -b fix/issue-123
+   # or
+   git checkout -b feature/add-new-tool
+   ```
+
+3. **Make your changes.** Follow existing code style and conventions. Test your changes locally against a running N.O.M.A.D. instance before submitting.
+
+4. **Add release notes** (see [Release Notes](#release-notes) below).
+
+5. **Commit your changes** using [Conventional Commits](#commit-messages).
+
+6. **Push your branch** and open a pull request.
+
+---
+
+## Commit Messages
+
+This project uses [Conventional Commits](https://www.conventionalcommits.org/). All commit messages must follow this format:
+
+```
+<type>(<scope>): <description>
+```
+
+**Common types:**
+
+| Type | When to use |
+|------|-------------|
+| `feat` | A new user-facing feature |
+| `fix` | A bug fix |
+| `docs` | Documentation changes only |
+| `refactor` | Code change that isn't a fix or feature and does not affect functionality |
+| `chore` | Build process, dependency updates, tooling |
+| `test` | Adding or updating tests |
+
+**Scope** is optional but encouraged — use it to indicate the area of the codebase affected (e.g., `api`, `ui`, `maps`).
+
+**Examples:**
+```
+feat(ui): add dark mode toggle to Command Center
+fix(api): resolve container status not updating after restart
+docs: update hardware requirements in README
+chore(deps): bump docker-compose to v2.24
+```
+
+---
+
+## Release Notes
+
+Human-readable release notes live in [`admin/docs/release-notes.md`](admin/docs/release-notes.md) and are displayed directly in the Command Center UI.
+
+If your PR is merged in, the maintainers will update the release notes with a summary of your contribution and credit you as the author. You do not need to add this yourself in the PR (please don't, as it may cause merge conflicts), but you can include a suggested note in the PR description if you like.
+
+---
+
+## Versioning
+
+This project uses [Semantic Versioning](https://semver.org/). Versions are managed in the root `package.json` and updated automatically by `semantic-release`. The `project-nomad` Docker image uses this version. The `admin/package.json` version stays at `0.0.0` and should not be changed manually.
+
+---
+
+## Submitting a Pull Request
+
+1. Push your branch to your fork:
+   ```bash
+   git push origin your-branch-name
+   ```
+2. Open a pull request against the `dev` branch of this repository
+3. In the PR description:
+   - Summarize what your changes do and why
+   - Reference the related issue (e.g., `Closes #123`)
+   - Note any relevant testing steps or environment details
+4. Be responsive to feedback — maintainers may request changes. Pull requests with no activity for an extended period may be closed.
+
+---
+
+## Feedback & Community
+
+Have questions or want to discuss ideas before opening an issue? Join the community:
+
+- **Discord:** [Join the Crosstalk Solutions server](https://discord.com/invite/crosstalksolutions) — the best place to get help, share your builds, and talk with other N.O.M.A.D. users
+- **Website:** [www.projectnomad.us](https://www.projectnomad.us)
+- **Benchmark Leaderboard:** [benchmark.projectnomad.us](https://benchmark.projectnomad.us)
+
+---
+
+*Project N.O.M.A.D. is licensed under the [Apache License 2.0](LICENSE).*

Dockerfile

@@ -1,7 +1,7 @@
-FROM node:22.16.0-alpine3.22 AS base
+FROM node:22-slim AS base
 
 # Install bash & curl for entrypoint script compatibility, graphicsmagick for pdf2pic, and vips-dev & build-base for sharp 
-RUN apk add --no-cache bash curl graphicsmagick vips-dev build-base
+RUN apt-get update && apt-get install -y bash curl graphicsmagick libvips-dev build-essential
 
 # All deps stage
 FROM base AS deps
@@ -24,13 +24,35 @@ RUN node ace build
 
 # Production stage
 FROM base
+ARG VERSION=dev
+ARG BUILD_DATE
+ARG VCS_REF
+
+# Labels
+LABEL org.opencontainers.image.title="Project N.O.M.A.D" \
+      org.opencontainers.image.description="The Project N.O.M.A.D Official Docker image" \
+      org.opencontainers.image.version="${VERSION}" \
+      org.opencontainers.image.created="${BUILD_DATE}" \
+      org.opencontainers.image.revision="${VCS_REF}" \
+      org.opencontainers.image.vendor="Crosstalk Solutions, LLC" \
+      org.opencontainers.image.documentation="https://github.com/CrosstalkSolutions/project-nomad/blob/main/README.md" \
+      org.opencontainers.image.source="https://github.com/CrosstalkSolutions/project-nomad" \
+      org.opencontainers.image.licenses="Apache-2.0"
+
 ENV NODE_ENV=production
 WORKDIR /app
 COPY --from=production-deps /app/node_modules /app/node_modules
 COPY --from=build /app/build /app
 # Copy root package.json for version info
 COPY package.json /app/version.json
+
+# Copy docs and README for access within the container
 COPY admin/docs /app/docs
 COPY README.md /app/README.md
+
+# Copy entrypoint script and ensure it's executable
+COPY install/entrypoint.sh /usr/local/bin/entrypoint.sh
+RUN chmod +x /usr/local/bin/entrypoint.sh
+
 EXPOSE 8080
-CMD ["node", "./bin/server.js"]
+ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

FAQ.md

@@ -0,0 +1,106 @@
+# Frequently Asked Questions (FAQ)
+
+Find answers to some of the most common questions about Project N.O.M.A.D.
+
+## Can I customize the port(s) that NOMAD uses?
+
+Yes, you can customize the ports that NOMAD's core services (Command Center, MySQL, Redis) use. Please refer to the [Advanced Installation](README.md#advanced-installation) section of the README for more details on how to do this.
+
+Note: As of 3/24/2026, only the core services defined in the `docker-compose.yml` file currently support port customization - the installable applications (e.g. Ollama, Kiwix, etc.) do not yet support this, but we have multiple PR's in the works to add this feature for all installable applications in a future release.
+
+## Can I customize the storage location for NOMAD's data?
+
+Yes, you can customize the storage location for NOMAD's content by modifying the `docker-compose.yml` file to adjust the appropriate bind mounts to point to your desired storage location on your host machine. Please refer to the [Advanced Installation](README.md#advanced-installation) section of the README for more details on how to do this.
+
+## Can I store NOMAD's data on an external drive or network storage?
+
+Short answer: yes, but we can't do it for you (and we recommend a local drive for best performance).
+
+Long answer: Custom storage paths, mount points, and external drives (like iSCSI or SMB/NFS volumes) **are possible**, but this will be up to your individual configuration on the host before NOMAD starts, and then passed in via the compose.yml as this is a *host-level concern*, not a NOMAD-level concern (see above for details). NOMAD itself can't configure this for you, nor could we support all possible configurations in the install script.
+
+## Can I run NOMAD on MAC, WSL2, or a non-Debian-based Distro?
+
+See [Why does NOMAD require a Debian-based OS?](#why-does-nomad-require-a-debian-based-os)
+
+## Why does NOMAD require a Debian-based OS?
+
+Project N.O.M.A.D. is currently designed to run on Debian-based Linux distributions (with Ubuntu being the recommended distro) because our installation scripts and Docker configurations are optimized for this environment. While it's technically possible to run the Docker containers on other operating systems that support Docker, we have not tested or optimized the installation process for non-Debian-based systems, so we cannot guarantee a smooth experience on those platforms at this time.
+
+Support for other operating systems will come in the future, but because our development resources are limited as a free and open-source project, we needed to prioritize our efforts and focus on a narrower set of supported platforms for the initial release. We chose Debian-based Linux as our starting point because it's widely used, easy to spin up, and provides a stable environment for running Docker containers.
+
+Community members have provided guides for running N.O.M.A.D. on other platforms (e.g. WSL2, Mac, etc.) in our Discord community and [Github Discussions](https://github.com/Crosstalk-Solutions/project-nomad/discussions), so if you're interested in running N.O.M.A.D. on a non-Debian-based system, we recommend checking there for any available resources or guides. However, keep in mind that if you choose to run N.O.M.A.D. on a non-Debian-based system, you may encounter issues that we won't be able to provide support for, and you may need to have a higher level of technical expertise to troubleshoot and resolve any problems that arise.
+
+## Can I run NOMAD on a Raspberry Pi or other ARM-based device?
+Project N.O.M.A.D. is currently designed to run on x86-64 architecture, and we have not yet tested or optimized it for ARM-based devices like the Raspberry Pi (and have not published any official images for ARM architecture).
+
+Support for ARM-based devices is on our roadmap, but our initial focus was on x86-64 hardware due to its widespread use and compatibility with a wide range of applications.
+
+Community members have forked and published their own ARM-compatible images and installation guides for running N.O.M.A.D. on Raspberry Pi and other ARM-based devices in our Discord community and [Github Discussions](https://github.com/Crosstalk-Solutions/project-nomad/discussions), but these are not officially supported by the core development team, and we cannot guarantee their functionality or provide support for any issues that arise when using these community-created resources.
+
+## What are the hardware requirements for running NOMAD?
+
+Project N.O.M.A.D. itself is quite lightweight and can run on even modest x86-64 hardware, but the tools and resources you choose to install with N.O.M.A.D. will determine the specs required for your unique deployment. Please see the [Hardware Guide](https://www.projectnomad.us/hardware) for detailed build recommendations at various price points.
+
+## Does NOMAD support languages other than English?
+
+As of March 2026, Project N.O.M.A.D.'s UI is only available in English, and the majority of the tools and resources available through N.O.M.A.D. are also primarily in English. However, we have multi-language support on our roadmap for a future release, and we are actively working on adding support for additional languages both in the UI and in the available tools/resources. If you're interested in contributing to this effort, please check out our [CONTRIBUTING.md](CONTRIBUTING.md) file for guidelines on how to get involved.
+
+## What technologies is NOMAD built with?
+
+Project N.O.M.A.D. is built using a combination of technologies, including:
+- **Docker:** for containerization of the Command Center and its dependencies
+- **Node.js & TypeScript:** for the backend of the Command Center, particularly the [AdonisJS](https://adonisjs.com/) framework
+- **React:** for the frontend of the Command Center, utilizing [Vite](https://vitejs.dev/) and [Inertia.js](https://inertiajs.com/) under the hood
+- **MySQL:** for the Command Center's database
+- **Redis:** for various caching, background jobs, "cron" tasks, and other internal processes within the Command Center
+
+NOMAD makes use of the Docker-outside-of-Docker ("DooD") pattern, which allows the Command Center to manage and orchestrate other Docker containers on the host machine without needing to run Docker itself inside a container. This approach provides better performance and compatibility with a wider range of host environments while still allowing for powerful container management capabilities through the Command Center's UI.
+
+## Can I run NOMAD if I have existing Docker containers on my machine?
+Yes, you can safely run Project N.O.M.A.D. on a machine that already has existing Docker containers. NOMAD is designed to coexist with other Docker containers and will not interfere with them as long as there are no port conflicts or resource constraints.
+
+All of NOMAD's containers are prefixed with `nomad_` in their names, so they can be easily identified and managed separately from any other containers you may have running. Just make sure to review the ports that NOMAD's core services (Command Center, MySQL, Redis) use during installation and adjust them if necessary to avoid conflicts with your existing containers.
+
+## Why does NOMAD require access to the Docker socket?
+
+See [What technologies is NOMAD built with?](#what-technologies-is-nomad-built-with)
+
+## Can I use any AI models?
+NOMAD by default uses Ollama inside of a docker container to run LLM Models for the AI Assistant. So if you find a model on HuggingFace for example, you won't be able to use that model in NOMAD. The list of available models in the AI Assistant settings (/settings/models) may not show all of the models you are looking for. If you found a model from https://ollama.com/search that you'd like to try and its not in the settings page, you can use a curl command to download the model.  
+`curl -X POST -H "Content-Type: application/json" -d '{"model":"MODEL_NAME_HERE"}' http://localhost:8080/api/ollama/models` replacing MODEL_NAME_HERE with the model name from whats in the ollama website.
+
+## Do I have to install the AI features in NOMAD?
+
+No, the AI features in NOMAD (Ollama, Qdrant, custom RAG pipeline, etc.) are all optional and not required to use the core functionality of NOMAD.
+
+## Is NOMAD actually free? Are there any hidden costs?
+Yes, Project N.O.M.A.D. is completely free and open-source software licensed under the Apache License 2.0. There are no hidden costs or fees associated with using NOMAD itself, and we don't have any plans to introduce "premium" features or paid tiers.
+
+Aside from the cost of the hardware you choose to run it on, there are no costs associated with using NOMAD.
+
+## Do you sell hardware or pre-built devices with NOMAD pre-installed?
+
+No, we do not sell hardware or pre-built devices with NOMAD pre-installed at this time. Project N.O.M.A.D. is a free and open-source software project, and we provide detailed installation instructions and hardware recommendations for users to set up their own NOMAD instances on compatible hardware of their choice. The tradeoff to this DIY approach is some additional setup time and technical know-how required on the user's end, but it also allows for greater flexibility and customization in terms of hardware selection and configuration to best suit each user's unique needs, budget, and preferences.
+
+## How quickly are issues resolved when reported?
+
+We strive to address and resolve issues as quickly as possible, but please keep in mind that Project N.O.M.A.D. is a free and open-source project maintained by a small team of volunteers. We prioritize issues based on their severity, impact on users, and the resources required to resolve them. Critical issues that affect a large number of users are typically addressed more quickly, while less severe issues may take longer to resolve. Aside from the development efforts needed to address the issue, we do our best to conduct thorough testing and validation to ensure that any fix we implement doesn't introduce new issues or regressions, which also adds to the time it takes to resolve an issue.
+
+We also encourage community involvement in troubleshooting and resolving issues, so if you encounter a problem, please consider checking our Discord community and Github Discussions for potential solutions or workarounds while we work on an official fix.
+
+## How often are new features added or updates released?
+
+We aim to release updates and new features on a regular basis, but the exact timing can vary based on the complexity of the features being developed, the resources available to our volunteer development team, and the feedback and needs of our community. We typically release smaller "patch" versions more frequently to address bugs and make minor improvements, while larger feature releases may take more time to develop and test before they're ready for release.
+
+## I opened a PR to contribute a new feature or fix a bug. How long does it usually take for PRs to be reviewed and merged?
+We appreciate all contributions to the project and strive to review and merge pull requests (PRs) as quickly as possible. The time it takes for a PR to be reviewed and merged can vary based on several factors, including the complexity of the changes, the current workload of our maintainers, and the need for any additional testing or revisions.
+
+Because NOMAD is still a young project, some PRs (particularly those for new features) may take longer to review and merge as we prioritize building out the core functionality and ensuring stability before adding new features. However, we do our best to provide timely feedback on all PRs and keep contributors informed about the status of their contributions.
+
+## I have a question that isn't answered here. Where can I ask for help?
+
+If you have a question that isn't answered in this FAQ, please feel free to ask for help in our Discord community (https://discord.com/invite/crosstalksolutions) or on our Github Discussions page (https://github.com/Crosstalk-Solutions/project-nomad/discussions).
+
+## I have a suggestion for a new feature or improvement. How can I share it?
+
+We welcome and encourage suggestions for new features and improvements! We highly encourage sharing your ideas (or upvoting existing suggestions) on our public roadmap at https://roadmap.projectnomad.us, where we track new feature requests. This is the best way to ensure that your suggestion is seen by the development team and the community, and it also allows other community members to upvote and show support for your idea, which can help prioritize it for future development.

LICENSE

@@ -0,0 +1,190 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to the Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by the Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding any notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   Copyright 2024-2026 Crosstalk Solutions LLC
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

README.md

@@ -1,7 +1,5 @@
-# *NOTE: Project N.O.M.A.D. is still in active development and should not be considered stable!*
-
 <div align="center">
-<img src="https://raw.githubusercontent.com/Crosstalk-Solutions/project-nomad/refs/heads/master/admin/public/project_nomad_logo.png" width="200" height="200"/>
+<img src="admin/public/project_nomad_logo.webp" width="200" height="200"/>
 
 # Project N.O.M.A.D.
 ### Node for Offline Media, Archives, and Data
@@ -23,27 +21,48 @@ Project N.O.M.A.D. can be installed on any Debian-based operating system (we rec
 
 *Note: sudo/root privileges are required to run the install script*
 
-#### One-liner (recommended for fresh Ubuntu installs)
-```bash
-sudo apt-get update && sudo apt-get install -y curl && curl -fsSL https://raw.githubusercontent.com/Crosstalk-Solutions/project-nomad/refs/heads/master/install/install_nomad.sh | sudo bash
-```
-
-#### Two-step install (if you already have curl)
-```bash
-curl -fsSL https://raw.githubusercontent.com/Crosstalk-Solutions/project-nomad/refs/heads/master/install/install_nomad.sh -o install_nomad.sh
-```
-
+### Quick Install (Debian-based OS Only)
 ```bash
+sudo apt-get update && \
+sudo apt-get install -y curl && \
+curl -fsSL https://raw.githubusercontent.com/Crosstalk-Solutions/project-nomad/refs/heads/main/install/install_nomad.sh \
+  -o install_nomad.sh && \
 sudo bash install_nomad.sh
 ```
 
 Project N.O.M.A.D. is now installed on your device! Open a browser and navigate to `http://localhost:8080` (or `http://DEVICE_IP:8080`) to start exploring!
 
-## How It Works
-From a technical standpoint, N.O.M.A.D. is primarily a management UI ("Command Center") and API that orchestrates a goodie basket of containerized offline archive tools and resources such as 
-[Kiwix](https://kiwix.org/), [ProtoMaps](https://protomaps.com), [Ollama](https://ollama.com/), and more.
+For a complete step-by-step walkthrough (including Ubuntu installation), see the [Installation Guide](https://www.projectnomad.us/install).
 
-By abstracting the installation of each of these awesome tools, N.O.M.A.D. makes getting your offline survival computer up and running a breeze! N.O.M.A.D. also includes some additional built-in handy tools, such as a ZIM library managment interface, calculators, and more.
+### Advanced Installation
+For more control over the installation process, copy and paste the [Docker Compose template](https://raw.githubusercontent.com/Crosstalk-Solutions/project-nomad/refs/heads/main/install/management_compose.yaml) into a `docker-compose.yml` file and customize it to your liking (be sure to replace any placeholders with your actual values). Then, run `docker compose up -d` to start the Command Center and its dependencies. Note: this method is recommended for advanced users only, as it requires familiarity with Docker and manual configuration before starting.
+
+## How It Works
+N.O.M.A.D. is a management UI ("Command Center") and API that orchestrates a collection of containerized tools and resources via [Docker](https://www.docker.com/). It handles installation, configuration, and updates for everything — so you don't have to.
+
+**Built-in capabilities include:**
+- **AI Chat with Knowledge Base** — local AI chat powered by [Ollama](https://ollama.com/) or you can use OpenAI API compatible software such as LM Studio or llama.cpp, with document upload and semantic search (RAG via [Qdrant](https://qdrant.tech/))
+- **Information Library** — offline Wikipedia, medical references, ebooks, and more via [Kiwix](https://kiwix.org/)
+- **Education Platform** — Khan Academy courses with progress tracking via [Kolibri](https://learningequality.org/kolibri/)
+- **Offline Maps** — downloadable regional maps via [ProtoMaps](https://protomaps.com)
+- **Data Tools** — encryption, encoding, and analysis via [CyberChef](https://gchq.github.io/CyberChef/)
+- **Notes** — local note-taking via [FlatNotes](https://github.com/dullage/flatnotes)
+- **System Benchmark** — hardware scoring with a [community leaderboard](https://benchmark.projectnomad.us)
+- **Easy Setup Wizard** — guided first-time configuration with curated content collections
+
+N.O.M.A.D. also includes built-in tools like a Wikipedia content selector, ZIM library manager, and content explorer.
+
+## What's Included
+
+| Capability | Powered By | What You Get |
+|-----------|-----------|-------------|
+| Information Library | Kiwix | Offline Wikipedia, medical references, survival guides, ebooks |
+| AI Assistant | Ollama + Qdrant | Built-in chat with document upload and semantic search |
+| Education Platform | Kolibri | Khan Academy courses, progress tracking, multi-user support |
+| Offline Maps | ProtoMaps | Downloadable regional maps with search and navigation |
+| Data Tools | CyberChef | Encryption, encoding, hashing, and data analysis |
+| Notes | FlatNotes | Local note-taking with markdown support |
+| System Benchmark | Built-in | Hardware scoring, Builder Tags, and community leaderboard |
 
 ## Device Requirements
 While many similar offline survival computers are designed to be run on bare-minimum, lightweight hardware, Project N.O.M.A.D. is quite the opposite. To install and run the
@@ -65,13 +84,24 @@ To run LLM's and other included AI tools:
 #### Optimal Specs
 - Processor: AMD Ryzen 7 or Intel Core i7 or better
 - RAM: 32 GB system memory
-- Graphics: NVIDIA RTX 3060 or better (more VRAM = run larger models)
+- Graphics: NVIDIA RTX 3060 or AMD equivalent or better (more VRAM = run larger models)
 - Storage: At least 250 GB free disk space (preferably on SSD)
 - OS: Debian-based (Ubuntu recommended)
 - Stable internet connection (required during install only)
 
+**For detailed build recommendations at three price points ($150–$1,000+), see the [Hardware Guide](https://www.projectnomad.us/hardware).**
+
 Again, Project N.O.M.A.D. itself is quite lightweight - it's the tools and resources you choose to install with N.O.M.A.D. that will determine the specs required for your unique deployment
 
+#### Running AI models on a different host
+By default, N.O.M.A.D.'s installer will attempt to setup Ollama on the host when the AI Assistant is installed. However, if you would like to run the AI model on a different host, you can go to the settings of of the AI assistant and input a URL for either an ollama or OpenAI-compatible API server (such as LM Studio).  
+Note that if you use Ollama on a different host, you must start the server with this option `OLLAMA_HOST=0.0.0.0`.  
+Ollama is the preferred way to use the AI assistant as it has features such as model download that OpenAI API does not support. So when using LM Studio for example, you will have to use LM Studio to download models.
+You are responsible for the setup of Ollama/OpenAI server on the other host.
+
+## Frequently Asked Questions (FAQ)
+For answers to common questions about Project N.O.M.A.D., please see our [FAQ](FAQ.md) page.
+
 ## About Internet Usage & Privacy
 Project N.O.M.A.D. is designed for offline usage. An internet connection is only required during the initial installation (to download dependencies) and if you (the user) decide to download additional tools and resources at a later time. Otherwise, N.O.M.A.D. does not require an internet connection and has ZERO built-in telemetry.
 
@@ -80,16 +110,24 @@ To test internet connectivity, N.O.M.A.D. attempts to make a request to Cloudfla
 ## About Security
 By design, Project N.O.M.A.D. is intended to be open and available without hurdles - it includes no authentication. If you decide to connect your device to a local network after install (e.g. for allowing other devices to access it's resources), you can block/open ports to control which services are exposed.
 
-## Versioning
-This project uses semantic versioning. The version is managed in the root `package.json` 
-and automatically updated by semantic-release. For simplicity's sake, the "project-nomad" container
-uses the same version defined there instead of the version in `admin/package.json` (stays at 0.0.0), as it's the only container derived from the code.
+**Will authentication be added in the future?** Maybe. It's not currently a priority, but if there's enough demand for it, we may consider building in an optional authentication layer in a future release to support uses cases where multiple users need access to the same instance but with different permission levels (e.g. family use with parental controls, classroom use with teacher/admin accounts, etc.). We have a suggestion for this on our public roadmap, so if this is something you'd like to see, please upvote it here: https://roadmap.projectnomad.us/posts/1/user-authentication-please-build-in-user-auth-with-admin-user-roles
+
+For now, we recommend using network-level controls to manage access if you're planning to expose your N.O.M.A.D. instance to other devices on a local network. N.O.M.A.D. is not designed to be exposed directly to the internet, and we strongly advise against doing so unless you really know what you're doing, have taken appropriate security measures, and understand the risks involved.
+
+## Contributing
+Contributions are welcome and appreciated! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on how to contribute to the project.
 
 ## Community & Resources
 
 - **Website:** [www.projectnomad.us](https://www.projectnomad.us) - Learn more about the project
 - **Discord:** [Join the Community](https://discord.com/invite/crosstalksolutions) - Get help, share your builds, and connect with other NOMAD users
 - **Benchmark Leaderboard:** [benchmark.projectnomad.us](https://benchmark.projectnomad.us) - See how your hardware stacks up against other NOMAD builds
+- **Troubleshooting Guide:** [TROUBLESHOOTING.md](TROUBLESHOOTING.md) - Find solutions to common issues
+- **FAQ:** [FAQ.md](FAQ.md) - Find answers to frequently asked questions
+
+## License
+
+Project N.O.M.A.D. is licensed under the [Apache License 2.0](LICENSE).
 
 ## Helper Scripts
 Once installed, Project N.O.M.A.D. has a few helper scripts should you ever need to troubleshoot issues or perform maintenance that can't be done through the Command Center. All of these scripts are found in Project N.O.M.A.D.'s install directory, `/opt/project-nomad`
@@ -104,7 +142,7 @@ sudo bash /opt/project-nomad/start_nomad.sh
 
 ###### Stop Script - Stops all installed project containers
 ```bash
-sudo bash /opt/project-nomad/start_nomad.sh
+sudo bash /opt/project-nomad/stop_nomad.sh
 ```
 ###
 
@@ -115,9 +153,5 @@ sudo bash /opt/project-nomad/update_nomad.sh
 
 ###### Uninstall Script - Need to start fresh? Use the uninstall script to make your life easy. Note: this cannot be undone!
 ```bash
-curl -fsSL https://raw.githubusercontent.com/Crosstalk-Solutions/project-nomad/refs/heads/master/install/uninstall_nomad.sh -o uninstall_nomad.sh
-```
-
-```bash
-sudo bash uninstall_nomad.sh
+curl -fsSL https://raw.githubusercontent.com/Crosstalk-Solutions/project-nomad/refs/heads/main/install/uninstall_nomad.sh -o uninstall_nomad.sh && sudo bash uninstall_nomad.sh
 ```

admin/adonisrc.ts

@@ -53,7 +53,8 @@ export default defineConfig({
     () => import('@adonisjs/lucid/database_provider'),
     () => import('@adonisjs/inertia/inertia_provider'),
     () => import('@adonisjs/transmit/transmit_provider'),
-    () => import('#providers/map_static_provider')
+    () => import('#providers/map_static_provider'),
+    () => import('#providers/kiwix_migration_provider'),
   ],
 
   /*

admin/app/controllers/benchmark_controller.ts

@@ -179,7 +179,9 @@ export default class BenchmarkController {
         percentile: submitResult.percentile,
       })
     } catch (error) {
-      return response.status(400).send({
+      // Pass through the status code from the service if available, otherwise default to 400
+      const statusCode = (error as any).statusCode || 400
+      return response.status(statusCode).send({
         success: false,
         error: error.message,
       })

admin/app/controllers/chats_controller.ts

@@ -2,7 +2,6 @@ import { inject } from '@adonisjs/core'
 import type { HttpContext } from '@adonisjs/core/http'
 import { ChatService } from '#services/chat_service'
 import { createSessionSchema, updateSessionSchema, addMessageSchema } from '#validators/chat'
-import { parseBoolean } from '../utils/misc.js'
 import KVStore from '#models/kv_store'
 import { SystemService } from '#services/system_service'
 import { SERVICE_NAMES } from '../../constants/service_names.js'
@@ -20,7 +19,7 @@ export default class ChatsController {
     const chatSuggestionsEnabled = await KVStore.getValue('chat.suggestionsEnabled')
     return inertia.render('chat', {
       settings: {
-        chatSuggestionsEnabled: parseBoolean(chatSuggestionsEnabled),
+        chatSuggestionsEnabled: chatSuggestionsEnabled ?? false,
       },
     })
   }

admin/app/controllers/collection_updates_controller.ts

@@ -0,0 +1,30 @@
+import { CollectionUpdateService } from '#services/collection_update_service'
+import {
+  assertNotPrivateUrl,
+  applyContentUpdateValidator,
+  applyAllContentUpdatesValidator,
+} from '#validators/common'
+import type { HttpContext } from '@adonisjs/core/http'
+
+export default class CollectionUpdatesController {
+  async checkForUpdates({}: HttpContext) {
+    const service = new CollectionUpdateService()
+    return await service.checkForUpdates()
+  }
+
+  async applyUpdate({ request }: HttpContext) {
+    const update = await request.validateUsing(applyContentUpdateValidator)
+    assertNotPrivateUrl(update.download_url)
+    const service = new CollectionUpdateService()
+    return await service.applyUpdate(update)
+  }
+
+  async applyAllUpdates({ request }: HttpContext) {
+    const { updates } = await request.validateUsing(applyAllContentUpdatesValidator)
+    for (const update of updates) {
+      assertNotPrivateUrl(update.download_url)
+    }
+    const service = new CollectionUpdateService()
+    return await service.applyAllUpdates(updates)
+  }
+}

admin/app/controllers/downloads_controller.ts

@@ -15,4 +15,13 @@ export default class DownloadsController {
     const payload = await request.validateUsing(downloadJobsByFiletypeSchema)
     return this.downloadService.listDownloadJobs(payload.params.filetype)
   }
+
+  async removeJob({ params }: HttpContext) {
+    await this.downloadService.removeFailedJob(params.jobId)
+    return { success: true }
+  }
+
+  async cancelJob({ params }: HttpContext) {
+    return this.downloadService.cancelJob(params.jobId)
+  }
 }

admin/app/controllers/easy_setup_controller.ts

@@ -1,5 +1,7 @@
 import { SystemService } from '#services/system_service'
 import { ZimService } from '#services/zim_service'
+import { CollectionManifestService } from '#services/collection_manifest_service'
+import KVStore from '#models/kv_store'
 import { inject } from '@adonisjs/core'
 import type { HttpContext } from '@adonisjs/core/http'
 
@@ -11,10 +13,14 @@ export default class EasySetupController {
   ) {}
 
   async index({ inertia }: HttpContext) {
-    const services = await this.systemService.getServices({ installedOnly: false })
+    const [services, remoteOllamaUrl] = await Promise.all([
+      this.systemService.getServices({ installedOnly: false }),
+      KVStore.getValue('ai.remoteOllamaUrl'),
+    ])
     return inertia.render('easy-setup/index', {
       system: {
         services: services,
+        remoteOllamaUrl: remoteOllamaUrl ?? '',
       },
     })
   }
@@ -26,4 +32,22 @@ export default class EasySetupController {
   async listCuratedCategories({}: HttpContext) {
     return await this.zimService.listCuratedCategories()
   }
+
+  async refreshManifests({}: HttpContext) {
+    const manifestService = new CollectionManifestService()
+    const [zimChanged, mapsChanged, wikiChanged] = await Promise.all([
+      manifestService.fetchAndCacheSpec('zim_categories'),
+      manifestService.fetchAndCacheSpec('maps'),
+      manifestService.fetchAndCacheSpec('wikipedia'),
+    ])
+
+    return {
+      success: true,
+      changed: {
+        zim_categories: zimChanged,
+        maps: mapsChanged,
+        wikipedia: wikiChanged,
+      },
+    }
+  }
 }

admin/app/controllers/maps_controller.ts

@@ -1,5 +1,7 @@
 import { MapService } from '#services/map_service'
+import MapMarker from '#models/map_marker'
 import {
+  assertNotPrivateUrl,
   downloadCollectionValidator,
   filenameParamValidator,
   remoteDownloadValidator,
@@ -7,6 +9,7 @@ import {
 } from '#validators/common'
 import { inject } from '@adonisjs/core'
 import type { HttpContext } from '@adonisjs/core/http'
+import vine from '@vinejs/vine'
 
 @inject()
 export default class MapsController {
@@ -25,12 +28,14 @@ export default class MapsController {
 
   async downloadBaseAssets({ request }: HttpContext) {
     const payload = await request.validateUsing(remoteDownloadValidatorOptional)
+    if (payload.url) assertNotPrivateUrl(payload.url)
     await this.mapService.downloadBaseAssets(payload.url)
     return { success: true }
   }
 
   async downloadRemote({ request }: HttpContext) {
     const payload = await request.validateUsing(remoteDownloadValidator)
+    assertNotPrivateUrl(payload.url)
     const filename = await this.mapService.downloadRemote(payload.url)
     return {
       message: 'Download started successfully',
@@ -52,6 +57,7 @@ export default class MapsController {
   // For providing a "preflight" check in the UI before actually starting a background download
   async downloadRemotePreflight({ request }: HttpContext) {
     const payload = await request.validateUsing(remoteDownloadValidator)
+    assertNotPrivateUrl(payload.url)
     const info = await this.mapService.downloadRemotePreflight(payload.url)
     return info
   }
@@ -69,6 +75,18 @@ export default class MapsController {
     return await this.mapService.listRegions()
   }
 
+  async globalMapInfo({}: HttpContext) {
+    return await this.mapService.getGlobalMapInfo()
+  }
+
+  async downloadGlobalMap({}: HttpContext) {
+    const result = await this.mapService.downloadGlobalMap()
+    return {
+      message: 'Download started successfully',
+      ...result,
+    }
+  }
+
   async styles({ request, response }: HttpContext) {
     // Automatically ensure base assets are present before generating styles
     const baseAssetsExist = await this.mapService.ensureBaseAssets()
@@ -79,7 +97,13 @@ export default class MapsController {
       })
     }
 
-    const styles = await this.mapService.generateStylesJSON(request.host())
+    const forwardedProto = request.headers()['x-forwarded-proto'];
+
+    const protocol: string = forwardedProto
+      ? (typeof forwardedProto === 'string' ? forwardedProto : request.protocol())
+      : request.protocol();
+
+    const styles = await this.mapService.generateStylesJSON(request.host(), protocol)
     return response.json(styles)
   }
 
@@ -101,4 +125,60 @@ export default class MapsController {
       message: 'Map file deleted successfully',
     }
   }
+
+  // --- Map Markers ---
+
+  async listMarkers({}: HttpContext) {
+    return await MapMarker.query().orderBy('created_at', 'asc')
+  }
+
+  async createMarker({ request }: HttpContext) {
+    const payload = await request.validateUsing(
+      vine.compile(
+        vine.object({
+          name: vine.string().trim().minLength(1).maxLength(255),
+          longitude: vine.number(),
+          latitude: vine.number(),
+          color: vine.string().trim().maxLength(20).optional(),
+        })
+      )
+    )
+    const marker = await MapMarker.create({
+      name: payload.name,
+      longitude: payload.longitude,
+      latitude: payload.latitude,
+      color: payload.color ?? 'orange',
+    })
+    return marker
+  }
+
+  async updateMarker({ request, response }: HttpContext) {
+    const { id } = request.params()
+    const marker = await MapMarker.find(id)
+    if (!marker) {
+      return response.status(404).send({ message: 'Marker not found' })
+    }
+    const payload = await request.validateUsing(
+      vine.compile(
+        vine.object({
+          name: vine.string().trim().minLength(1).maxLength(255).optional(),
+          color: vine.string().trim().maxLength(20).optional(),
+        })
+      )
+    )
+    if (payload.name !== undefined) marker.name = payload.name
+    if (payload.color !== undefined) marker.color = payload.color
+    await marker.save()
+    return marker
+  }
+
+  async deleteMarker({ request, response }: HttpContext) {
+    const { id } = request.params()
+    const marker = await MapMarker.find(id)
+    if (!marker) {
+      return response.status(404).send({ message: 'Marker not found' })
+    }
+    await marker.delete()
+    return { message: 'Marker deleted' }
+  }
 }

admin/app/controllers/ollama_controller.ts

@@ -1,56 +1,98 @@
+import { ChatService } from '#services/chat_service'
+import { DockerService } from '#services/docker_service'
 import { OllamaService } from '#services/ollama_service'
 import { RagService } from '#services/rag_service'
+import Service from '#models/service'
+import KVStore from '#models/kv_store'
 import { modelNameSchema } from '#validators/download'
 import { chatSchema, getAvailableModelsSchema } from '#validators/ollama'
 import { inject } from '@adonisjs/core'
 import type { HttpContext } from '@adonisjs/core/http'
-import { SYSTEM_PROMPTS } from '../../constants/ollama.js'
+import { DEFAULT_QUERY_REWRITE_MODEL, RAG_CONTEXT_LIMITS, SYSTEM_PROMPTS } from '../../constants/ollama.js'
+import { SERVICE_NAMES } from '../../constants/service_names.js'
+import logger from '@adonisjs/core/services/logger'
+type Message = { role: 'system' | 'user' | 'assistant'; content: string }
 
 @inject()
 export default class OllamaController {
   constructor(
+    private chatService: ChatService,
+    private dockerService: DockerService,
     private ollamaService: OllamaService,
     private ragService: RagService
-  ) {}
+  ) { }
 
   async availableModels({ request }: HttpContext) {
     const reqData = await request.validateUsing(getAvailableModelsSchema)
     return await this.ollamaService.getAvailableModels({
       sort: reqData.sort,
       recommendedOnly: reqData.recommendedOnly,
+      query: reqData.query || null,
+      limit: reqData.limit || 15,
+      force: reqData.force,
     })
   }
 
-  async chat({ request }: HttpContext) {
+  async chat({ request, response }: HttpContext) {
     const reqData = await request.validateUsing(chatSchema)
 
-    /**If there are no system messages in the chat
-     *(i.e. first message from the user)inject system prompts
-     **/
+    // Flush SSE headers immediately so the client connection is open while
+    // pre-processing (query rewriting, RAG lookup) runs in the background.
+    if (reqData.stream) {
+      response.response.setHeader('Content-Type', 'text/event-stream')
+      response.response.setHeader('Cache-Control', 'no-cache')
+      response.response.setHeader('Connection', 'keep-alive')
+      response.response.flushHeaders()
+    }
+
+    try {
+      // If there are no system messages in the chat inject system prompts
       const hasSystemMessage = reqData.messages.some((msg) => msg.role === 'system')
       if (!hasSystemMessage) {
         const systemPrompt = {
           role: 'system' as const,
           content: SYSTEM_PROMPTS.default,
         }
+        logger.debug('[OllamaController] Injecting system prompt')
         reqData.messages.unshift(systemPrompt)
       }
 
-    // Get the last user message to use for RAG context retrieval
-    const lastUserMessage = [...reqData.messages].reverse().find((msg) => msg.role === 'user')
+      // Query rewriting for better RAG retrieval with manageable context
+      // Will return user's latest message if no rewriting is needed
+      const rewrittenQuery = await this.rewriteQueryWithContext(reqData.messages)
 
-    if (lastUserMessage) {
-      // Search for relevant context in the knowledge base
-      // Using lower threshold (0.3) with improved hybrid search
+      logger.debug(`[OllamaController] Rewritten query for RAG: "${rewrittenQuery}"`)
+      if (rewrittenQuery) {
         const relevantDocs = await this.ragService.searchSimilarDocuments(
-        lastUserMessage.content,
-        5, // Retrieve top 5 most relevant chunks
-        0.3 // Minimum similarity score of 0.3 (lowered from 0.7 for better recall)
+          rewrittenQuery,
+          5, // Top 5 most relevant chunks
+          0.3 // Minimum similarity score of 0.3
         )
 
-      // If relevant context is found, inject as a system message
+        logger.debug(`[RAG] Retrieved ${relevantDocs.length} relevant documents for query: "${rewrittenQuery}"`)
+
+        // If relevant context is found, inject as a system message with adaptive limits
         if (relevantDocs.length > 0) {
-        const contextText = relevantDocs
+          // Determine context budget based on model size
+          const { maxResults, maxTokens } = this.getContextLimitsForModel(reqData.model)
+          let trimmedDocs = relevantDocs.slice(0, maxResults)
+
+          // Apply token cap if set (estimate ~3.5 chars per token)
+          // Always include the first (most relevant) result — the cap only gates subsequent results
+          if (maxTokens > 0) {
+            const charCap = maxTokens * 3.5
+            let totalChars = 0
+            trimmedDocs = trimmedDocs.filter((doc, idx) => {
+              totalChars += doc.text.length
+              return idx === 0 || totalChars <= charCap
+            })
+          }
+
+          logger.debug(
+            `[RAG] Injecting ${trimmedDocs.length}/${relevantDocs.length} results (model: ${reqData.model}, maxResults: ${maxResults}, maxTokens: ${maxTokens || 'unlimited'})`
+          )
+
+          const contextText = trimmedDocs
             .map((doc, idx) => `[Context ${idx + 1}] (Relevance: ${(doc.score * 100).toFixed(1)}%)\n${doc.text}`)
             .join('\n\n')
 
@@ -66,7 +108,166 @@ export default class OllamaController {
         }
       }
 
-    return await this.ollamaService.chat(reqData)
+      // If system messages are large (e.g. due to RAG context), request a context window big
+      // enough to fit them. Ollama respects num_ctx per-request; LM Studio ignores it gracefully.
+      const systemChars = reqData.messages
+        .filter((m) => m.role === 'system')
+        .reduce((sum, m) => sum + m.content.length, 0)
+      const estimatedSystemTokens = Math.ceil(systemChars / 3.5)
+      let numCtx: number | undefined
+      if (estimatedSystemTokens > 3000) {
+        const needed = estimatedSystemTokens + 2048 // leave room for conversation + response
+        numCtx = [8192, 16384, 32768, 65536].find((n) => n >= needed) ?? 65536
+        logger.debug(`[OllamaController] Large system prompt (~${estimatedSystemTokens} tokens), requesting num_ctx: ${numCtx}`)
+      }
+
+      // Check if the model supports "thinking" capability for enhanced response generation
+      // If gpt-oss model, it requires a text param for "think" https://docs.ollama.com/api/chat
+      const thinkingCapability = await this.ollamaService.checkModelHasThinking(reqData.model)
+      const think: boolean | 'medium' = thinkingCapability ? (reqData.model.startsWith('gpt-oss') ? 'medium' : true) : false
+
+      // Separate sessionId from the Ollama request payload — Ollama rejects unknown fields
+      const { sessionId, ...ollamaRequest } = reqData
+
+      // Save user message to DB before streaming if sessionId provided
+      let userContent: string | null = null
+      if (sessionId) {
+        const lastUserMsg = [...reqData.messages].reverse().find((m) => m.role === 'user')
+        if (lastUserMsg) {
+          userContent = lastUserMsg.content
+          await this.chatService.addMessage(sessionId, 'user', userContent)
+        }
+      }
+
+      if (reqData.stream) {
+        logger.debug(`[OllamaController] Initiating streaming response for model: "${reqData.model}" with think: ${think}`)
+        // Headers already flushed above
+        const stream = await this.ollamaService.chatStream({ ...ollamaRequest, think, numCtx })
+        let fullContent = ''
+        for await (const chunk of stream) {
+          if (chunk.message?.content) {
+            fullContent += chunk.message.content
+          }
+          response.response.write(`data: ${JSON.stringify(chunk)}\n\n`)
+        }
+        response.response.end()
+
+        // Save assistant message and optionally generate title
+        if (sessionId && fullContent) {
+          await this.chatService.addMessage(sessionId, 'assistant', fullContent)
+          const messageCount = await this.chatService.getMessageCount(sessionId)
+          if (messageCount <= 2 && userContent) {
+            this.chatService.generateTitle(sessionId, userContent, fullContent).catch((err) => {
+              logger.error(`[OllamaController] Title generation failed: ${err instanceof Error ? err.message : err}`)
+            })
+          }
+        }
+        return
+      }
+
+      // Non-streaming (legacy) path
+      const result = await this.ollamaService.chat({ ...ollamaRequest, think, numCtx })
+
+      if (sessionId && result?.message?.content) {
+        await this.chatService.addMessage(sessionId, 'assistant', result.message.content)
+        const messageCount = await this.chatService.getMessageCount(sessionId)
+        if (messageCount <= 2 && userContent) {
+          this.chatService.generateTitle(sessionId, userContent, result.message.content).catch((err) => {
+            logger.error(`[OllamaController] Title generation failed: ${err instanceof Error ? err.message : err}`)
+          })
+        }
+      }
+
+      return result
+    } catch (error) {
+      if (reqData.stream) {
+        response.response.write(`data: ${JSON.stringify({ error: true })}\n\n`)
+        response.response.end()
+        return
+      }
+      throw error
+    }
+  }
+
+  async remoteStatus() {
+    const remoteUrl = await KVStore.getValue('ai.remoteOllamaUrl')
+    if (!remoteUrl) {
+      return { configured: false, connected: false }
+    }
+    try {
+      const testResponse = await fetch(`${remoteUrl.replace(/\/$/, '')}/v1/models`, {
+        signal: AbortSignal.timeout(3000),
+      })
+      return { configured: true, connected: testResponse.ok }
+    } catch {
+      return { configured: true, connected: false }
+    }
+  }
+
+  async configureRemote({ request, response }: HttpContext) {
+    const remoteUrl: string | null = request.input('remoteUrl', null)
+
+    const ollamaService = await Service.query().where('service_name', SERVICE_NAMES.OLLAMA).first()
+    if (!ollamaService) {
+      return response.status(404).send({ success: false, message: 'Ollama service record not found.' })
+    }
+
+    // Clear path: null or empty URL removes remote config and marks service as not installed
+    if (!remoteUrl || remoteUrl.trim() === '') {
+      await KVStore.clearValue('ai.remoteOllamaUrl')
+      ollamaService.installed = false
+      ollamaService.installation_status = 'idle'
+      await ollamaService.save()
+      return { success: true, message: 'Remote Ollama configuration cleared.' }
+    }
+
+    // Validate URL format
+    if (!remoteUrl.startsWith('http')) {
+      return response.status(400).send({
+        success: false,
+        message: 'Invalid URL. Must start with http:// or https://',
+      })
+    }
+
+    // Test connectivity via OpenAI-compatible /v1/models endpoint (works with Ollama, LM Studio, llama.cpp, etc.)
+    try {
+      const testResponse = await fetch(`${remoteUrl.replace(/\/$/, '')}/v1/models`, {
+        signal: AbortSignal.timeout(5000),
+      })
+      if (!testResponse.ok) {
+        return response.status(400).send({
+          success: false,
+          message: `Could not connect to ${remoteUrl} (HTTP ${testResponse.status}). Make sure the server is running and accessible. For Ollama, start it with OLLAMA_HOST=0.0.0.0.`,
+        })
+      }
+    } catch (error) {
+      return response.status(400).send({
+        success: false,
+        message: `Could not connect to ${remoteUrl}. Make sure the server is running and reachable. For Ollama, start it with OLLAMA_HOST=0.0.0.0.`,
+      })
+    }
+
+    // Save remote URL and mark service as installed
+    await KVStore.setValue('ai.remoteOllamaUrl', remoteUrl.trim())
+    ollamaService.installed = true
+    ollamaService.installation_status = 'idle'
+    await ollamaService.save()
+
+    // Install Qdrant if not already installed (fire-and-forget)
+    const qdrantService = await Service.query().where('service_name', SERVICE_NAMES.QDRANT).first()
+    if (qdrantService && !qdrantService.installed) {
+      this.dockerService.createContainerPreflight(SERVICE_NAMES.QDRANT).catch((error) => {
+        logger.error('[OllamaController] Failed to start Qdrant preflight:', error)
+      })
+    }
+
+    // Mirror post-install side effects: disable suggestions, trigger docs discovery
+    await KVStore.setValue('chat.suggestionsEnabled', false)
+    this.ragService.discoverNomadDocs().catch((error) => {
+      logger.error('[OllamaController] Failed to discover Nomad docs:', error)
+    })
+
+    return { success: true, message: 'Remote Ollama configured.' }
   }
 
   async deleteModel({ request }: HttpContext) {
@@ -87,7 +288,87 @@ export default class OllamaController {
     }
   }
 
-  async installedModels({}: HttpContext) {
+  async installedModels({ }: HttpContext) {
     return await this.ollamaService.getModels()
   }
+
+  /**
+   * Determines RAG context limits based on model size extracted from the model name.
+   * Parses size indicators like "1b", "3b", "8b", "70b" from model names/tags.
+   */
+  private getContextLimitsForModel(modelName: string): { maxResults: number; maxTokens: number } {
+    // Extract parameter count from model name (e.g., "llama3.2:3b", "qwen2.5:1.5b", "gemma:7b")
+    const sizeMatch = modelName.match(/(\d+\.?\d*)[bB]/)
+    const paramBillions = sizeMatch ? parseFloat(sizeMatch[1]) : 8 // default to 8B if unknown
+
+    for (const tier of RAG_CONTEXT_LIMITS) {
+      if (paramBillions <= tier.maxParams) {
+        return { maxResults: tier.maxResults, maxTokens: tier.maxTokens }
+      }
+    }
+
+    // Fallback: no limits
+    return { maxResults: 5, maxTokens: 0 }
+  }
+
+  private async rewriteQueryWithContext(
+    messages: Message[]
+  ): Promise<string | null> {
+    try {
+      // Get recent conversation history (last 6 messages for 3 turns)
+      const recentMessages = messages.slice(-6)
+
+      // Skip rewriting for short conversations. Rewriting adds latency with
+      // little RAG benefit until there is enough context to matter.
+      const userMessages = recentMessages.filter(msg => msg.role === 'user')
+      if (userMessages.length <= 2) {
+        return userMessages[userMessages.length - 1]?.content || null
+      }
+
+      const conversationContext = recentMessages
+        .map(msg => {
+          const role = msg.role === 'user' ? 'User' : 'Assistant'
+          // Truncate assistant messages to first 200 chars to keep context manageable
+          const content = msg.role === 'assistant'
+            ? msg.content.slice(0, 200) + (msg.content.length > 200 ? '...' : '')
+            : msg.content
+          return `${role}: "${content}"`
+        })
+        .join('\n')
+
+      const installedModels = await this.ollamaService.getModels(true)
+      const rewriteModelAvailable = installedModels?.some(model => model.name === DEFAULT_QUERY_REWRITE_MODEL)
+      if (!rewriteModelAvailable) {
+        logger.warn(`[RAG] Query rewrite model "${DEFAULT_QUERY_REWRITE_MODEL}" not available. Skipping query rewriting.`)
+        const lastUserMessage = [...messages].reverse().find(msg => msg.role === 'user')
+        return lastUserMessage?.content || null
+      }
+
+      // FUTURE ENHANCEMENT: allow the user to specify which model to use for rewriting
+      const response = await this.ollamaService.chat({
+        model: DEFAULT_QUERY_REWRITE_MODEL,
+        messages: [
+          {
+            role: 'system',
+            content: SYSTEM_PROMPTS.query_rewrite,
+          },
+          {
+            role: 'user',
+            content: `Conversation:\n${conversationContext}\n\nRewritten Query:`,
+          },
+        ],
+      })
+
+      const rewrittenQuery = response.message.content.trim()
+      logger.info(`[RAG] Query rewritten: "${rewrittenQuery}"`)
+      return rewrittenQuery
+    } catch (error) {
+      logger.error(
+        `[RAG] Query rewriting failed: ${error instanceof Error ? error.message : error}`
+      )
+      // Fallback to last user message if rewriting fails
+      const lastUserMessage = [...messages].reverse().find(msg => msg.role === 'user')
+      return lastUserMessage?.content || null
+    }
+  }
 }

admin/app/controllers/rag_controller.ts

@@ -5,12 +5,11 @@ import type { HttpContext } from '@adonisjs/core/http'
 import app from '@adonisjs/core/services/app'
 import { randomBytes } from 'node:crypto'
 import { sanitizeFilename } from '../utils/fs.js'
-import { stat } from 'node:fs/promises'
-import { getJobStatusSchema } from '#validators/rag'
+import { deleteFileSchema, getJobStatusSchema } from '#validators/rag'
 
 @inject()
 export default class RagController {
-  constructor(private ragService: RagService) {}
+  constructor(private ragService: RagService) { }
 
   public async upload({ request, response }: HttpContext) {
     const uploadedFile = request.file('file')
@@ -28,20 +27,10 @@ export default class RagController {
       name: fileName,
     })
 
-    // Get file size for tracking
-    let fileSize: number | undefined = undefined
-    try {
-      const stats = await stat(fullPath)
-      fileSize = stats.size
-    } catch (error) {
-      // Not critical if we can't get file size, just swallow the error
-    }
-
     // Dispatch background job for embedding
     const result = await EmbedFileJob.dispatch({
       filePath: fullPath,
       fileName,
-      fileSize,
     })
 
     return response.status(202).json({
@@ -53,6 +42,11 @@ export default class RagController {
     })
   }
 
+  public async getActiveJobs({ response }: HttpContext) {
+    const jobs = await EmbedFileJob.listActiveJobs()
+    return response.status(200).json(jobs)
+  }
+
   public async getJobStatus({ request, response }: HttpContext) {
     const reqData = await request.validateUsing(getJobStatusSchema)
 
@@ -70,4 +64,35 @@ export default class RagController {
     const files = await this.ragService.getStoredFiles()
     return response.status(200).json({ files })
   }
+
+  public async deleteFile({ request, response }: HttpContext) {
+    const { source } = await request.validateUsing(deleteFileSchema)
+    const result = await this.ragService.deleteFileBySource(source)
+    if (!result.success) {
+      return response.status(500).json({ error: result.message })
+    }
+    return response.status(200).json({ message: result.message })
+  }
+
+  public async getFailedJobs({ response }: HttpContext) {
+    const jobs = await EmbedFileJob.listFailedJobs()
+    return response.status(200).json(jobs)
+  }
+
+  public async cleanupFailedJobs({ response }: HttpContext) {
+    const result = await EmbedFileJob.cleanupFailedJobs()
+    return response.status(200).json({
+      message: `Cleaned up ${result.cleaned} failed job${result.cleaned !== 1 ? 's' : ''}${result.filesDeleted > 0 ? `, deleted ${result.filesDeleted} file${result.filesDeleted !== 1 ? 's' : ''}` : ''}.`,
+      ...result,
+    })
+  }
+
+  public async scanAndSync({ response }: HttpContext) {
+    try {
+      const syncResult = await this.ragService.scanAndSyncStorage()
+      return response.status(200).json(syncResult)
+    } catch (error) {
+      return response.status(500).json({ error: 'Error scanning and syncing storage', details: error.message })
+    }
+  }
 }

admin/app/controllers/settings_controller.ts

@@ -1,12 +1,11 @@
-import KVStore from '#models/kv_store';
-import { BenchmarkService } from '#services/benchmark_service';
-import { MapService } from '#services/map_service';
-import { OllamaService } from '#services/ollama_service';
-import { SystemService } from '#services/system_service';
-import { updateSettingSchema } from '#validators/settings';
-import { inject } from '@adonisjs/core';
+import KVStore from '#models/kv_store'
+import { BenchmarkService } from '#services/benchmark_service'
+import { MapService } from '#services/map_service'
+import { OllamaService } from '#services/ollama_service'
+import { SystemService } from '#services/system_service'
+import { getSettingSchema, updateSettingSchema } from '#validators/settings'
+import { inject } from '@adonisjs/core'
 import type { HttpContext } from '@adonisjs/core/http'
-import { parseBoolean } from '../utils/misc.js';
 
 @inject()
 export default class SettingsController {
@@ -15,65 +14,80 @@ export default class SettingsController {
     private mapService: MapService,
     private benchmarkService: BenchmarkService,
     private ollamaService: OllamaService
-    ) { }
+  ) {}
 
   async system({ inertia }: HttpContext) {
-        const systemInfo = await this.systemService.getSystemInfo();
+    const systemInfo = await this.systemService.getSystemInfo()
     return inertia.render('settings/system', {
       system: {
-                info: systemInfo
-            }
-        });
+        info: systemInfo,
+      },
+    })
   }
 
   async apps({ inertia }: HttpContext) {
-        const services = await this.systemService.getServices({ installedOnly: false });
+    const services = await this.systemService.getServices({ installedOnly: false })
     return inertia.render('settings/apps', {
       system: {
-                services
-            }
-        });
+        services,
+      },
+    })
   }
 
   async legal({ inertia }: HttpContext) {
-        return inertia.render('settings/legal');
+    return inertia.render('settings/legal')
+  }
+
+  async support({ inertia }: HttpContext) {
+    return inertia.render('settings/support')
   }
 
   async maps({ inertia }: HttpContext) {
-        const baseAssetsCheck = await this.mapService.ensureBaseAssets();
-        const regionFiles = await this.mapService.listRegions();
+    const baseAssetsCheck = await this.mapService.ensureBaseAssets()
+    const regionFiles = await this.mapService.listRegions()
     return inertia.render('settings/maps', {
       maps: {
         baseAssetsExist: baseAssetsCheck,
-                regionFiles: regionFiles.files
-            }
-        });
+        regionFiles: regionFiles.files,
+      },
+    })
   }
 
   async models({ inertia }: HttpContext) {
-        const availableModels = await this.ollamaService.getAvailableModels({ sort: 'pulls', recommendedOnly: false });
-        const installedModels = await this.ollamaService.getModels();
+    const availableModels = await this.ollamaService.getAvailableModels({
+      sort: 'pulls',
+      recommendedOnly: false,
+      query: null,
+      limit: 15,
+    })
+    const installedModels = await this.ollamaService.getModels().catch(() => [])
     const chatSuggestionsEnabled = await KVStore.getValue('chat.suggestionsEnabled')
+    const aiAssistantCustomName = await KVStore.getValue('ai.assistantCustomName')
+    const remoteOllamaUrl = await KVStore.getValue('ai.remoteOllamaUrl')
+    const ollamaFlashAttention = await KVStore.getValue('ai.ollamaFlashAttention')
     return inertia.render('settings/models', {
       models: {
-                availableModels: availableModels || [],
+        availableModels: availableModels?.models || [],
         installedModels: installedModels || [],
         settings: {
-                    chatSuggestionsEnabled: parseBoolean(chatSuggestionsEnabled)
-                }
-            }
-        });
+          chatSuggestionsEnabled: chatSuggestionsEnabled ?? false,
+          aiAssistantCustomName: aiAssistantCustomName ?? '',
+          remoteOllamaUrl: remoteOllamaUrl ?? '',
+          ollamaFlashAttention: ollamaFlashAttention ?? true,
+        },
+      },
+    })
   }
 
   async update({ inertia }: HttpContext) {
-        const updateInfo = await this.systemService.checkLatestVersion();
+    const updateInfo = await this.systemService.checkLatestVersion()
     return inertia.render('settings/update', {
       system: {
         updateAvailable: updateInfo.updateAvailable,
         latestVersion: updateInfo.latestVersion,
-                currentVersion: updateInfo.currentVersion
-            }
-        });
+        currentVersion: updateInfo.currentVersion,
+      },
+    })
   }
 
   async zim({ inertia }: HttpContext) {
@@ -81,30 +95,30 @@ export default class SettingsController {
   }
 
   async zimRemote({ inertia }: HttpContext) {
-        return inertia.render('settings/zim/remote-explorer');
+    return inertia.render('settings/zim/remote-explorer')
   }
 
   async benchmark({ inertia }: HttpContext) {
-        const latestResult = await this.benchmarkService.getLatestResult();
-        const status = this.benchmarkService.getStatus();
+    const latestResult = await this.benchmarkService.getLatestResult()
+    const status = this.benchmarkService.getStatus()
     return inertia.render('settings/benchmark', {
       benchmark: {
         latestResult,
         status: status.status,
-                currentBenchmarkId: status.benchmarkId
-            }
-        });
+        currentBenchmarkId: status.benchmarkId,
+      },
+    })
   }
 
   async getSetting({ request, response }: HttpContext) {
-        const key = request.qs().key;
+    const { key } = await getSettingSchema.validate({ key: request.qs().key });
     const value = await KVStore.getValue(key);
     return response.status(200).send({ key, value });
   }
 
   async updateSetting({ request, response }: HttpContext) {
-        const reqData = await request.validateUsing(updateSettingSchema);
-        await this.systemService.updateSetting(reqData.key, reqData.value);
-        return response.status(200).send({ success: true, message: 'Setting updated successfully' });
+    const reqData = await request.validateUsing(updateSettingSchema)
+    await this.systemService.updateSetting(reqData.key, reqData.value)
+    return response.status(200).send({ success: true, message: 'Setting updated successfully' })
   }
 }

admin/app/controllers/system_controller.ts

@@ -1,7 +1,9 @@
 import { DockerService } from '#services/docker_service';
 import { SystemService } from '#services/system_service'
 import { SystemUpdateService } from '#services/system_update_service'
-import { affectServiceValidator, installServiceValidator, subscribeToReleaseNotesValidator } from '#validators/system';
+import { ContainerRegistryService } from '#services/container_registry_service'
+import { CheckServiceUpdatesJob } from '#jobs/check_service_updates_job'
+import { affectServiceValidator, checkLatestVersionValidator, installServiceValidator, subscribeToReleaseNotesValidator, updateServiceValidator } from '#validators/system';
 import { inject } from '@adonisjs/core'
 import type { HttpContext } from '@adonisjs/core/http'
 
@@ -10,7 +12,8 @@ export default class SystemController {
     constructor(
         private systemService: SystemService,
         private dockerService: DockerService,
-        private systemUpdateService: SystemUpdateService
+        private systemUpdateService: SystemUpdateService,
+        private containerRegistryService: ContainerRegistryService
     ) { }
 
     async getInternetStatus({ }: HttpContext) {
@@ -32,7 +35,7 @@ export default class SystemController {
         if (result.success) {
             response.send({ success: true, message: result.message });
         } else {
-            response.status(400).send({ error: result.message });
+            response.status(400).send({ success: false, message: result.message });
         }
     }
 
@@ -46,8 +49,9 @@ export default class SystemController {
         response.send({ success: result.success, message: result.message });
     }
 
-    async checkLatestVersion({ }: HttpContext) {
-        return await this.systemService.checkLatestVersion();
+    async checkLatestVersion({ request }: HttpContext) {
+        const payload = await request.validateUsing(checkLatestVersionValidator)
+        return await this.systemService.checkLatestVersion(payload.force);
     }
 
     async forceReinstallService({ request, response }: HttpContext) {
@@ -108,4 +112,70 @@ export default class SystemController {
         const reqData = await request.validateUsing(subscribeToReleaseNotesValidator);
         return await this.systemService.subscribeToReleaseNotes(reqData.email);
     }
+
+    async getDebugInfo({}: HttpContext) {
+        const debugInfo = await this.systemService.getDebugInfo()
+        return { debugInfo }
+    }
+
+    async checkServiceUpdates({ response }: HttpContext) {
+        await CheckServiceUpdatesJob.dispatch()
+        response.send({ success: true, message: 'Service update check dispatched' })
+    }
+
+    async getAvailableVersions({ params, response }: HttpContext) {
+        const serviceName = params.name
+        const service = await (await import('#models/service')).default
+            .query()
+            .where('service_name', serviceName)
+            .where('installed', true)
+            .first()
+
+        if (!service) {
+            return response.status(404).send({ error: `Service ${serviceName} not found or not installed` })
+        }
+
+        try {
+            const hostArch = await this.getHostArch()
+            const updates = await this.containerRegistryService.getAvailableUpdates(
+                service.container_image,
+                hostArch,
+                service.source_repo
+            )
+            response.send({ versions: updates })
+        } catch (error) {
+            response.status(500).send({ error: `Failed to fetch versions: ${error.message}` })
+        }
+    }
+
+    async updateService({ request, response }: HttpContext) {
+        const payload = await request.validateUsing(updateServiceValidator)
+        const result = await this.dockerService.updateContainer(
+            payload.service_name,
+            payload.target_version
+        )
+
+        if (result.success) {
+            response.send({ success: true, message: result.message })
+        } else {
+            response.status(400).send({ error: result.message })
+        }
+    }
+
+    private async getHostArch(): Promise<string> {
+        try {
+            const info = await this.dockerService.docker.info()
+            const arch = info.Architecture || ''
+            const archMap: Record<string, string> = {
+                x86_64: 'amd64',
+                aarch64: 'arm64',
+                armv7l: 'arm',
+                amd64: 'amd64',
+                arm64: 'arm64',
+            }
+            return archMap[arch] || arch.toLowerCase()
+        } catch {
+            return 'amd64'
+        }
+    }
 }

admin/app/controllers/zim_controller.ts

@@ -1,9 +1,9 @@
 import { ZimService } from '#services/zim_service'
 import {
-  downloadCollectionValidator,
+  assertNotPrivateUrl,
+  downloadCategoryTierValidator,
   filenameParamValidator,
   remoteDownloadWithMetadataValidator,
-  saveInstalledTierValidator,
   selectWikipediaValidator,
 } from '#validators/common'
 import { listRemoteZimValidator } from '#validators/zim'
@@ -26,6 +26,7 @@ export default class ZimController {
 
   async downloadRemote({ request }: HttpContext) {
     const payload = await request.validateUsing(remoteDownloadWithMetadataValidator)
+    assertNotPrivateUrl(payload.url)
     const { filename, jobId } = await this.zimService.downloadRemote(payload.url, payload.metadata)
 
     return {
@@ -36,32 +37,25 @@ export default class ZimController {
     }
   }
 
-  async downloadCollection({ request }: HttpContext) {
-    const payload = await request.validateUsing(downloadCollectionValidator)
-    const resources = await this.zimService.downloadCollection(payload.slug)
+  async listCuratedCategories({}: HttpContext) {
+    return await this.zimService.listCuratedCategories()
+  }
+
+  async downloadCategoryTier({ request }: HttpContext) {
+    const payload = await request.validateUsing(downloadCategoryTierValidator)
+    const resources = await this.zimService.downloadCategoryTier(
+      payload.categorySlug,
+      payload.tierSlug
+    )
 
     return {
       message: 'Download started successfully',
-      slug: payload.slug,
+      categorySlug: payload.categorySlug,
+      tierSlug: payload.tierSlug,
       resources,
     }
   }
 
-  async listCuratedCollections({}: HttpContext) {
-    return this.zimService.listCuratedCollections()
-  }
-
-  async fetchLatestCollections({}: HttpContext) {
-    const success = await this.zimService.fetchLatestCollections()
-    return { success }
-  }
-
-  async saveInstalledTier({ request }: HttpContext) {
-    const payload = await request.validateUsing(saveInstalledTierValidator)
-    await this.zimService.saveInstalledTier(payload.categorySlug, payload.tierSlug)
-    return { success: true }
-  }
-
   async delete({ request, response }: HttpContext) {
     const payload = await request.validateUsing(filenameParamValidator)
 

admin/app/jobs/check_service_updates_job.ts

@@ -0,0 +1,134 @@
+import { Job } from 'bullmq'
+import { QueueService } from '#services/queue_service'
+import { DockerService } from '#services/docker_service'
+import { ContainerRegistryService } from '#services/container_registry_service'
+import Service from '#models/service'
+import logger from '@adonisjs/core/services/logger'
+import transmit from '@adonisjs/transmit/services/main'
+import { BROADCAST_CHANNELS } from '../../constants/broadcast.js'
+import { DateTime } from 'luxon'
+
+export class CheckServiceUpdatesJob {
+  static get queue() {
+    return 'service-updates'
+  }
+
+  static get key() {
+    return 'check-service-updates'
+  }
+
+  async handle(_job: Job) {
+    logger.info('[CheckServiceUpdatesJob] Checking for service updates...')
+
+    const dockerService = new DockerService()
+    const registryService = new ContainerRegistryService()
+
+    // Determine host architecture
+    const hostArch = await this.getHostArch(dockerService)
+
+    const installedServices = await Service.query().where('installed', true)
+    let updatesFound = 0
+
+    for (const service of installedServices) {
+      try {
+        const updates = await registryService.getAvailableUpdates(
+          service.container_image,
+          hostArch,
+          service.source_repo
+        )
+
+        const latestUpdate = updates.length > 0 ? updates[0].tag : null
+
+        service.available_update_version = latestUpdate
+        service.update_checked_at = DateTime.now()
+        await service.save()
+
+        if (latestUpdate) {
+          updatesFound++
+          logger.info(
+            `[CheckServiceUpdatesJob] Update available for ${service.service_name}: ${service.container_image} → ${latestUpdate}`
+          )
+        }
+      } catch (error) {
+        logger.error(
+          `[CheckServiceUpdatesJob] Failed to check updates for ${service.service_name}: ${error.message}`
+        )
+        // Continue checking other services
+      }
+    }
+
+    logger.info(
+      `[CheckServiceUpdatesJob] Completed. ${updatesFound} update(s) found for ${installedServices.length} service(s).`
+    )
+
+    // Broadcast completion so the frontend can refresh
+    transmit.broadcast(BROADCAST_CHANNELS.SERVICE_UPDATES, {
+      status: 'completed',
+      updatesFound,
+      timestamp: new Date().toISOString(),
+    })
+
+    return { updatesFound }
+  }
+
+  private async getHostArch(dockerService: DockerService): Promise<string> {
+    try {
+      const info = await dockerService.docker.info()
+      const arch = info.Architecture || ''
+
+      // Map Docker architecture names to OCI names
+      const archMap: Record<string, string> = {
+        x86_64: 'amd64',
+        aarch64: 'arm64',
+        armv7l: 'arm',
+        amd64: 'amd64',
+        arm64: 'arm64',
+      }
+
+      return archMap[arch] || arch.toLowerCase()
+    } catch (error) {
+      logger.warn(
+        `[CheckServiceUpdatesJob] Could not detect host architecture: ${error.message}. Defaulting to amd64.`
+      )
+      return 'amd64'
+    }
+  }
+
+  static async scheduleNightly() {
+    const queueService = new QueueService()
+    const queue = queueService.getQueue(this.queue)
+
+    await queue.upsertJobScheduler(
+      'nightly-service-update-check',
+      { pattern: '0 3 * * *' },
+      {
+        name: this.key,
+        opts: {
+          removeOnComplete: { count: 7 },
+          removeOnFail: { count: 5 },
+        },
+      }
+    )
+
+    logger.info('[CheckServiceUpdatesJob] Service update check scheduled with cron: 0 3 * * *')
+  }
+
+  static async dispatch() {
+    const queueService = new QueueService()
+    const queue = queueService.getQueue(this.queue)
+
+    const job = await queue.add(
+      this.key,
+      {},
+      {
+        attempts: 3,
+        backoff: { type: 'exponential', delay: 60000 },
+        removeOnComplete: { count: 7 },
+        removeOnFail: { count: 5 },
+      }
+    )
+
+    logger.info(`[CheckServiceUpdatesJob] Dispatched ad-hoc service update check job ${job.id}`)
+    return job
+  }
+}

admin/app/jobs/check_update_job.ts

@@ -0,0 +1,77 @@
+import { Job } from 'bullmq'
+import { QueueService } from '#services/queue_service'
+import { DockerService } from '#services/docker_service'
+import { SystemService } from '#services/system_service'
+import logger from '@adonisjs/core/services/logger'
+import KVStore from '#models/kv_store'
+
+export class CheckUpdateJob {
+  static get queue() {
+    return 'system'
+  }
+
+  static get key() {
+    return 'check-update'
+  }
+
+  async handle(_job: Job) {
+    logger.info('[CheckUpdateJob] Running update check...')
+
+    const dockerService = new DockerService()
+    const systemService = new SystemService(dockerService)
+
+    try {
+      const result = await systemService.checkLatestVersion()
+
+      if (result.updateAvailable) {
+        logger.info(
+          `[CheckUpdateJob] Update available: ${result.currentVersion} → ${result.latestVersion}`
+        )
+      } else {
+        await KVStore.setValue('system.updateAvailable', false)
+        logger.info(
+          `[CheckUpdateJob] System is up to date (${result.currentVersion})`
+        )
+      }
+
+      return result
+    } catch (error) {
+      logger.error(`[CheckUpdateJob] Update check failed: ${error.message}`)
+      throw error
+    }
+  }
+
+  static async scheduleNightly() {
+    const queueService = new QueueService()
+    const queue = queueService.getQueue(this.queue)
+
+    await queue.upsertJobScheduler(
+      'nightly-update-check',
+      { pattern: '0 2,14 * * *' }, // Every 12 hours at 2am and 2pm
+      {
+        name: this.key,
+        opts: {
+          removeOnComplete: { count: 7 },
+          removeOnFail: { count: 5 },
+        },
+      }
+    )
+
+    logger.info('[CheckUpdateJob] Update check scheduled with cron: 0 2,14 * * *')
+  }
+
+  static async dispatch() {
+    const queueService = new QueueService()
+    const queue = queueService.getQueue(this.queue)
+
+    const job = await queue.add(this.key, {}, {
+      attempts: 3,
+      backoff: { type: 'exponential', delay: 60000 },
+      removeOnComplete: { count: 7 },
+      removeOnFail: { count: 5 },
+    })
+
+    logger.info(`[CheckUpdateJob] Dispatched ad-hoc update check job ${job.id}`)
+    return job
+  }
+}

admin/app/jobs/download_model_job.ts

@@ -1,4 +1,4 @@
-import { Job } from 'bullmq'
+import { Job, UnrecoverableError } from 'bullmq'
 import { QueueService } from '#services/queue_service'
 import { createHash } from 'crypto'
 import logger from '@adonisjs/core/services/logger'
@@ -44,7 +44,9 @@ export class DownloadModelJob {
     // Services are ready, initiate the download with progress tracking
     const result = await ollamaService.downloadModel(modelName, (progressPercent) => {
       if (progressPercent) {
-        job.updateProgress(Math.floor(progressPercent))
+        job.updateProgress(Math.floor(progressPercent)).catch((err) => {
+          if (err?.code !== -1) throw err
+        })
         logger.info(
           `[DownloadModelJob] Model ${modelName}: ${progressPercent}%`
         )
@@ -56,6 +58,8 @@ export class DownloadModelJob {
         status: 'downloading',
         progress: progressPercent,
         progress_timestamp: new Date().toISOString(),
+      }).catch((err) => {
+        if (err?.code !== -1) throw err
       })
     })
 
@@ -63,6 +67,10 @@ export class DownloadModelJob {
       logger.error(
         `[DownloadModelJob] Failed to initiate download for model ${modelName}: ${result.message}`
       )
+      // Don't retry errors that will never succeed (e.g., Ollama version too old)
+      if (result.retryable === false) {
+        throw new UnrecoverableError(result.message)
+      }
       throw new Error(`Failed to initiate download for model: ${result.message}`)
     }
 
@@ -85,6 +93,15 @@ export class DownloadModelJob {
     const queue = queueService.getQueue(this.queue)
     const jobId = this.getJobId(params.modelName)
 
+    // Clear any previous failed job so a fresh attempt can be dispatched
+    const existing = await queue.getJob(jobId)
+    if (existing) {
+      const state = await existing.getState()
+      if (state === 'failed') {
+        await existing.remove()
+      }
+    }
+
     try {
       const job = await queue.add(this.key, params, {
         jobId,
@@ -104,9 +121,9 @@ export class DownloadModelJob {
       }
     } catch (error) {
       if (error.message.includes('job already exists')) {
-        const existing = await queue.getJob(jobId)
+        const active = await queue.getJob(jobId)
         return {
-          job: existing,
+          job: active,
           created: false,
           message: `Job already exists for model ${params.modelName}`,
         }

admin/app/jobs/embed_file_job.ts

@@ -1,15 +1,21 @@
-import { Job } from 'bullmq'
+import { Job, UnrecoverableError } from 'bullmq'
 import { QueueService } from '#services/queue_service'
+import { EmbedJobWithProgress } from '../../types/rag.js'
 import { RagService } from '#services/rag_service'
 import { DockerService } from '#services/docker_service'
 import { OllamaService } from '#services/ollama_service'
 import { createHash } from 'crypto'
 import logger from '@adonisjs/core/services/logger'
+import fs from 'node:fs/promises'
 
 export interface EmbedFileJobParams {
   filePath: string
   fileName: string
   fileSize?: number
+  // Batch processing for large ZIM files
+  batchOffset?: number  // Current batch offset (for ZIM files)
+  totalArticles?: number // Total articles in ZIM (for progress tracking)
+  isFinalBatch?: boolean // Whether this is the last batch (prevents premature deletion)
 }
 
 export class EmbedFileJob {
@@ -25,17 +31,38 @@ export class EmbedFileJob {
     return createHash('sha256').update(filePath).digest('hex').slice(0, 16)
   }
 
-  async handle(job: Job) {
-    const { filePath, fileName } = job.data as EmbedFileJobParams
+  /** Calls job.updateProgress but silently ignores "Missing key" errors (code -1),
+   *  which occur when the job has been removed from Redis (e.g. cancelled externally)
+   *  between the time the await was issued and the Redis write completed. */
+  private async safeUpdateProgress(job: Job, progress: number): Promise<void> {
+    try {
+      await job.updateProgress(progress)
+    } catch (err: any) {
+      if (err?.code !== -1) throw err
+    }
+  }
 
-    logger.info(`[EmbedFileJob] Starting embedding process for: ${fileName}`)
+  async handle(job: Job) {
+    const { filePath, fileName, batchOffset, totalArticles } = job.data as EmbedFileJobParams
+
+    const isZimBatch = batchOffset !== undefined
+    const batchInfo = isZimBatch ? ` (batch offset: ${batchOffset})` : ''
+    logger.info(`[EmbedFileJob] Starting embedding process for: ${fileName}${batchInfo}`)
 
     const dockerService = new DockerService()
     const ollamaService = new OllamaService()
     const ragService = new RagService(dockerService, ollamaService)
 
     try {
-      // Check if Ollama and Qdrant services are ready
+      // Check if Ollama and Qdrant services are installed and ready
+      // Use UnrecoverableError for "not installed" so BullMQ won't retry —
+      // retrying 30x when the service doesn't exist just wastes Redis connections
+      const ollamaUrl = await dockerService.getServiceURL('nomad_ollama')
+      if (!ollamaUrl) {
+        logger.warn('[EmbedFileJob] Ollama is not installed. Skipping embedding for: %s', fileName)
+        throw new UnrecoverableError('Ollama service is not installed. Install AI Assistant to enable file embeddings.')
+      }
+
       const existingModels = await ollamaService.getModels()
       if (!existingModels) {
         logger.warn('[EmbedFileJob] Ollama service not ready yet. Will retry...')
@@ -44,41 +71,95 @@ export class EmbedFileJob {
 
       const qdrantUrl = await dockerService.getServiceURL('nomad_qdrant')
       if (!qdrantUrl) {
-        logger.warn('[EmbedFileJob] Qdrant service not ready yet. Will retry...')
-        throw new Error('Qdrant service not ready yet')
+        logger.warn('[EmbedFileJob] Qdrant is not installed. Skipping embedding for: %s', fileName)
+        throw new UnrecoverableError('Qdrant service is not installed. Install AI Assistant to enable file embeddings.')
       }
 
       logger.info(`[EmbedFileJob] Services ready. Processing file: ${fileName}`)
 
       // Update progress starting
-      await job.updateProgress(0)
+      await this.safeUpdateProgress(job, 5)
       await job.updateData({
         ...job.data,
         status: 'processing',
-        startedAt: Date.now(),
+        startedAt: job.data.startedAt || Date.now(),
       })
 
       logger.info(`[EmbedFileJob] Processing file: ${filePath}`)
 
+      // Progress callback: maps service-reported 0-100% into the 5-95% job range
+      const onProgress = async (percent: number) => {
+        await this.safeUpdateProgress(job, Math.min(95, Math.round(5 + percent * 0.9)))
+      }
+
       // Process and embed the file
-      const result = await ragService.processAndEmbedFile(filePath)
+      // Only allow deletion if explicitly marked as final batch
+      const allowDeletion = job.data.isFinalBatch === true
+      const result = await ragService.processAndEmbedFile(
+        filePath,
+        allowDeletion,
+        batchOffset,
+        onProgress
+      )
 
       if (!result.success) {
         logger.error(`[EmbedFileJob] Failed to process file ${fileName}: ${result.message}`)
         throw new Error(result.message)
       }
 
-      // Update progress complete
-      await job.updateProgress(100)
+      // For ZIM files with batching, check if more batches are needed
+      if (result.hasMoreBatches) {
+        const nextOffset = (batchOffset || 0) + (result.articlesProcessed || 0)
+        logger.info(
+          `[EmbedFileJob] Batch complete. Dispatching next batch at offset ${nextOffset}`
+        )
+
+        // Dispatch next batch (not final yet)
+        await EmbedFileJob.dispatch({
+          filePath,
+          fileName,
+          batchOffset: nextOffset,
+          totalArticles: totalArticles || result.totalArticles,
+          isFinalBatch: false, // Explicitly not final
+        })
+
+        // Calculate progress based on articles processed
+        const progress = totalArticles
+          ? Math.round((nextOffset / totalArticles) * 100)
+          : 50
+
+        await this.safeUpdateProgress(job, progress)
+        await job.updateData({
+          ...job.data,
+          status: 'batch_completed',
+          lastBatchAt: Date.now(),
+          chunks: (job.data.chunks || 0) + (result.chunks || 0),
+        })
+
+        return {
+          success: true,
+          fileName,
+          filePath,
+          chunks: result.chunks,
+          hasMoreBatches: true,
+          nextOffset,
+          message: `Batch embedded ${result.chunks} chunks, next batch queued`,
+        }
+      }
+
+      // Final batch or non-batched file - mark as complete
+      const totalChunks = (job.data.chunks || 0) + (result.chunks || 0)
+      await this.safeUpdateProgress(job, 100)
       await job.updateData({
         ...job.data,
         status: 'completed',
         completedAt: Date.now(),
-        chunks: result.chunks,
+        chunks: totalChunks,
       })
 
+      const batchMsg = isZimBatch ? ` (final batch, total chunks: ${totalChunks})` : ''
       logger.info(
-        `[EmbedFileJob] Successfully embedded ${result.chunks} chunks from file: ${fileName}`
+        `[EmbedFileJob] Successfully embedded ${result.chunks} chunks from file: ${fileName}${batchMsg}`
       )
 
       return {
@@ -102,6 +183,20 @@ export class EmbedFileJob {
     }
   }
 
+  static async listActiveJobs(): Promise<EmbedJobWithProgress[]> {
+    const queueService = new QueueService()
+    const queue = queueService.getQueue(this.queue)
+    const jobs = await queue.getJobs(['waiting', 'active', 'delayed'])
+
+    return jobs.map((job) => ({
+      jobId: job.id!.toString(),
+      fileName: (job.data as EmbedFileJobParams).fileName,
+      filePath: (job.data as EmbedFileJobParams).filePath,
+      progress: typeof job.progress === 'number' ? job.progress : 0,
+      status: ((job.data as any).status as string) ?? 'waiting',
+    }))
+  }
+
   static async getByFilePath(filePath: string): Promise<Job | undefined> {
     const queueService = new QueueService()
     const queue = queueService.getQueue(this.queue)
@@ -149,6 +244,52 @@ export class EmbedFileJob {
     }
   }
 
+  static async listFailedJobs(): Promise<EmbedJobWithProgress[]> {
+    const queueService = new QueueService()
+    const queue = queueService.getQueue(this.queue)
+    // Jobs that have failed at least once are in 'delayed' (retrying) or terminal 'failed' state.
+    // We identify them by job.data.status === 'failed' set in the catch block of handle().
+    const jobs = await queue.getJobs(['waiting', 'delayed', 'failed'])
+
+    return jobs
+      .filter((job) => (job.data as any).status === 'failed')
+      .map((job) => ({
+        jobId: job.id!.toString(),
+        fileName: (job.data as EmbedFileJobParams).fileName,
+        filePath: (job.data as EmbedFileJobParams).filePath,
+        progress: 0,
+        status: 'failed',
+        error: (job.data as any).error,
+      }))
+  }
+
+  static async cleanupFailedJobs(): Promise<{ cleaned: number; filesDeleted: number }> {
+    const queueService = new QueueService()
+    const queue = queueService.getQueue(this.queue)
+    const allJobs = await queue.getJobs(['waiting', 'delayed', 'failed'])
+    const failedJobs = allJobs.filter((job) => (job.data as any).status === 'failed')
+
+    let cleaned = 0
+    let filesDeleted = 0
+
+    for (const job of failedJobs) {
+      const filePath = (job.data as EmbedFileJobParams).filePath
+      if (filePath && filePath.includes(RagService.UPLOADS_STORAGE_PATH)) {
+        try {
+          await fs.unlink(filePath)
+          filesDeleted++
+        } catch {
+          // File may already be deleted — that's fine
+        }
+      }
+      await job.remove()
+      cleaned++
+    }
+
+    logger.info(`[EmbedFileJob] Cleaned up ${cleaned} failed jobs, deleted ${filesDeleted} files`)
+    return { cleaned, filesDeleted }
+  }
+
   static async getStatus(filePath: string): Promise<{
     exists: boolean
     status?: string

admin/app/jobs/run_download_job.ts

@@ -1,11 +1,12 @@
-import { Job } from 'bullmq'
-import { RunDownloadJobParams } from '../../types/downloads.js'
+import { Job, UnrecoverableError } from 'bullmq'
+import { RunDownloadJobParams, DownloadProgressData } from '../../types/downloads.js'
 import { QueueService } from '#services/queue_service'
 import { doResumableDownload } from '../utils/downloads.js'
 import { createHash } from 'crypto'
 import { DockerService } from '#services/docker_service'
 import { ZimService } from '#services/zim_service'
 import { MapService } from '#services/map_service'
+import { EmbedFileJob } from './embed_file_job.js'
 
 export class RunDownloadJob {
   static get queue() {
@@ -16,52 +17,163 @@ export class RunDownloadJob {
     return 'run-download'
   }
 
+  /** In-memory registry of abort controllers for active download jobs */
+  static abortControllers: Map<string, AbortController> = new Map()
+
   static getJobId(url: string): string {
     return createHash('sha256').update(url).digest('hex').slice(0, 16)
   }
 
+  /** Redis key used to signal cancellation across processes */
+  static cancelKey(jobId: string): string {
+    return `nomad:download:cancel:${jobId}`
+  }
+
+  /** Signal cancellation via Redis so the worker process can pick it up */
+  static async signalCancel(jobId: string): Promise<void> {
+    const queueService = new QueueService()
+    const queue = queueService.getQueue(this.queue)
+    const client = await queue.client
+    await client.set(this.cancelKey(jobId), '1', 'EX', 300) // 5 min TTL
+  }
+
   async handle(job: Job) {
-    const { url, filepath, timeout, allowedMimeTypes, forceNew, filetype } =
+    const { url, filepath, timeout, allowedMimeTypes, forceNew, filetype, resourceMetadata } =
       job.data as RunDownloadJobParams
 
-    //    console.log("Simulating delay for job for URL:", url)
-    //  await new Promise((resolve) => setTimeout(resolve, 30000)) // Simulate initial delay
-    //  console.log("Starting download for URL:", url)
+    // Register abort controller for this job
+    const abortController = new AbortController()
+    RunDownloadJob.abortControllers.set(job.id!, abortController)
 
-    // // simulate progress updates for demonstration
-    // for (let progress = 0; progress <= 100; progress += 10) {
-    //   await new Promise((resolve) => setTimeout(resolve, 20000)) // Simulate time taken for each progress step
-    //   job.updateProgress(progress)
-    //   console.log(`Job progress for URL ${url}: ${progress}%`)
-    // }
+    // Get Redis client for checking cancel signals from the API process
+    const queueService = new QueueService()
+    const cancelRedis = await queueService.getQueue(RunDownloadJob.queue).client
 
+    let lastKnownProgress: Pick<DownloadProgressData, 'downloadedBytes' | 'totalBytes'> = {
+      downloadedBytes: 0,
+      totalBytes: 0,
+    }
+
+    // Track whether cancellation was explicitly requested by the user (via Redis signal
+    // or in-process AbortController). BullMQ lock mismatches can also abort the download
+    // stream, but those should be retried — only user-initiated cancels are unrecoverable.
+    let userCancelled = false
+
+    // Poll Redis for cancel signal every 2s — independent of progress events so cancellation
+    // works even when the stream is stalled and no onProgress ticks are firing.
+    let cancelPollInterval: ReturnType<typeof setInterval> | null = setInterval(async () => {
+      try {
+        const val = await cancelRedis.get(RunDownloadJob.cancelKey(job.id!))
+        if (val) {
+          await cancelRedis.del(RunDownloadJob.cancelKey(job.id!))
+          userCancelled = true
+          abortController.abort('user-cancel')
+        }
+      } catch {
+        // Redis errors are non-fatal; in-process AbortController covers same-process cancels
+      }
+    }, 2000)
+
+    try {
       await doResumableDownload({
         url,
         filepath,
         timeout,
         allowedMimeTypes,
         forceNew,
+        signal: abortController.signal,
         onProgress(progress) {
           const progressPercent = (progress.downloadedBytes / (progress.totalBytes || 1)) * 100
-        job.updateProgress(Math.floor(progressPercent))
+          const progressData: DownloadProgressData = {
+            percent: Math.floor(progressPercent),
+            downloadedBytes: progress.downloadedBytes,
+            totalBytes: progress.totalBytes,
+            lastProgressTime: Date.now(),
+          }
+          job.updateProgress(progressData).catch((err) => {
+            // Job was removed from Redis (e.g. cancelled) between the callback firing
+            // and the Redis write completing — this is expected and safe to ignore.
+            if (err?.code !== -1) throw err
+          })
+          lastKnownProgress = { downloadedBytes: progress.downloadedBytes, totalBytes: progress.totalBytes }
         },
         async onComplete(url) {
           try {
+            // Create InstalledResource entry if metadata was provided
+            if (resourceMetadata) {
+              const { default: InstalledResource } = await import('#models/installed_resource')
+              const { DateTime } = await import('luxon')
+              const { getFileStatsIfExists, deleteFileIfExists } = await import('../utils/fs.js')
+              const stats = await getFileStatsIfExists(filepath)
+
+              // Look up the old entry so we can clean up the previous file after updating
+              const oldEntry = await InstalledResource.query()
+                .where('resource_id', resourceMetadata.resource_id)
+                .where('resource_type', filetype as 'zim' | 'map')
+                .first()
+              const oldFilePath = oldEntry?.file_path ?? null
+
+              await InstalledResource.updateOrCreate(
+                { resource_id: resourceMetadata.resource_id, resource_type: filetype as 'zim' | 'map' },
+                {
+                  version: resourceMetadata.version,
+                  collection_ref: resourceMetadata.collection_ref,
+                  url: url,
+                  file_path: filepath,
+                  file_size_bytes: stats ? Number(stats.size) : null,
+                  installed_at: DateTime.now(),
+                }
+              )
+
+              // Delete the old file if it differs from the new one
+              if (oldFilePath && oldFilePath !== filepath) {
+                try {
+                  await deleteFileIfExists(oldFilePath)
+                  console.log(`[RunDownloadJob] Deleted old file: ${oldFilePath}`)
+                } catch (deleteError) {
+                  console.warn(
+                    `[RunDownloadJob] Failed to delete old file ${oldFilePath}:`,
+                    deleteError
+                  )
+                }
+              }
+            }
+
             if (filetype === 'zim') {
               const dockerService = new DockerService()
               const zimService = new ZimService(dockerService)
               await zimService.downloadRemoteSuccessCallback([url], true)
+
+              // Only dispatch embedding job if AI Assistant (Ollama) is installed
+              const ollamaUrl = await dockerService.getServiceURL('nomad_ollama')
+              if (ollamaUrl) {
+                try {
+                  await EmbedFileJob.dispatch({
+                    fileName: url.split('/').pop() || '',
+                    filePath: filepath,
+                  })
+                } catch (error) {
+                  console.error(`[RunDownloadJob] Error dispatching EmbedFileJob for URL ${url}:`, error)
+                }
+              }
             } else if (filetype === 'map') {
               const mapsService = new MapService()
               await mapsService.downloadRemoteSuccessCallback([url], false)
             }
           } catch (error) {
             console.error(
-            `[RunDownloadJob] Error in ZIM download success callback for URL ${url}:`,
+              `[RunDownloadJob] Error in download success callback for URL ${url}:`,
               error
             )
           }
-        job.updateProgress(100)
+          job.updateProgress({
+            percent: 100,
+            downloadedBytes: lastKnownProgress.downloadedBytes,
+            totalBytes: lastKnownProgress.totalBytes,
+            lastProgressTime: Date.now(),
+          } as DownloadProgressData).catch((err) => {
+            if (err?.code !== -1) throw err
+          })
         },
       })
 
@@ -69,6 +181,21 @@ export class RunDownloadJob {
         url,
         filepath,
       }
+    } catch (error: any) {
+      // Only prevent retries for user-initiated cancellations. BullMQ lock mismatches
+      // can also abort the stream, and those should be retried with backoff.
+      // Check both the flag (Redis poll) and abort reason (in-process cancel).
+      if (userCancelled || abortController.signal.reason === 'user-cancel') {
+        throw new UnrecoverableError(`Download cancelled: ${error.message}`)
+      }
+      throw error
+    } finally {
+      if (cancelPollInterval !== null) {
+        clearInterval(cancelPollInterval)
+        cancelPollInterval = null
+      }
+      RunDownloadJob.abortControllers.delete(job.id!)
+    }
   }
 
   static async getByUrl(url: string): Promise<Job | undefined> {
@@ -78,6 +205,29 @@ export class RunDownloadJob {
     return await queue.getJob(jobId)
   }
 
+  /**
+   * Check if a download is actively in progress for the given URL.
+   * Returns the job only if it's in an active state (active, waiting, delayed).
+   * If the job exists in a terminal state (failed, completed), removes it and returns undefined.
+   */
+  static async getActiveByUrl(url: string): Promise<Job | undefined> {
+    const job = await this.getByUrl(url)
+    if (!job) return undefined
+
+    const state = await job.getState()
+    if (state === 'active' || state === 'waiting' || state === 'delayed') {
+      return job
+    }
+
+    // Terminal state -- clean up stale job so it doesn't block re-download
+    try {
+      await job.remove()
+    } catch {
+      // May already be gone
+    }
+    return undefined
+  }
+
   static async dispatch(params: RunDownloadJobParams) {
     const queueService = new QueueService()
     const queue = queueService.getQueue(this.queue)
@@ -86,8 +236,8 @@ export class RunDownloadJob {
     try {
       const job = await queue.add(this.key, params, {
         jobId,
-        attempts: 3,
-        backoff: { type: 'exponential', delay: 2000 },
+        attempts: 10,
+        backoff: { type: 'exponential', delay: 30000 },
         removeOnComplete: true,
       })
 

admin/app/middleware/compression_middleware.ts

@@ -0,0 +1,21 @@
+import env from '#start/env'
+import type { HttpContext } from '@adonisjs/core/http'
+import type { NextFn } from '@adonisjs/core/types/http'
+import compression from 'compression'
+
+const compress = env.get('DISABLE_COMPRESSION') ? null : compression()
+
+export default class CompressionMiddleware {
+  async handle({ request, response }: HttpContext, next: NextFn) {
+    if (!compress) return await next()
+
+    await new Promise<void>((resolve, reject) => {
+      compress(request.request as any, response.response as any, (err?: any) => {
+        if (err) reject(err)
+        else resolve()
+      })
+    })
+
+    await next()
+  }
+}

admin/app/models/collection_manifest.ts

@@ -0,0 +1,22 @@
+import { DateTime } from 'luxon'
+import { BaseModel, column, SnakeCaseNamingStrategy } from '@adonisjs/lucid/orm'
+import type { ManifestType } from '../../types/collections.js'
+
+export default class CollectionManifest extends BaseModel {
+  static namingStrategy = new SnakeCaseNamingStrategy()
+
+  @column({ isPrimary: true })
+  declare type: ManifestType
+
+  @column()
+  declare spec_version: string
+
+  @column({
+    consume: (value: string) => (typeof value === 'string' ? JSON.parse(value) : value),
+    prepare: (value: any) => JSON.stringify(value),
+  })
+  declare spec_data: any
+
+  @column.dateTime()
+  declare fetched_at: DateTime
+}

admin/app/models/curated_collection.ts

@@ -1,39 +0,0 @@
-import { DateTime } from 'luxon'
-import { BaseModel, column, hasMany, SnakeCaseNamingStrategy } from '@adonisjs/lucid/orm'
-import CuratedCollectionResource from './curated_collection_resource.js'
-import type { HasMany } from '@adonisjs/lucid/types/relations'
-import type { CuratedCollectionType } from '../../types/curated_collections.js'
-
-export default class CuratedCollection extends BaseModel {
-  static namingStrategy = new SnakeCaseNamingStrategy()
-
-  @column({ isPrimary: true })
-  declare slug: string
-
-  @column()
-  declare type: CuratedCollectionType
-
-  @column()
-  declare name: string
-
-  @column()
-  declare description: string
-
-  @column()
-  declare icon: string
-
-  @column()
-  declare language: string
-
-  @hasMany(() => CuratedCollectionResource, {
-    foreignKey: 'curated_collection_slug',
-    localKey: 'slug',
-  })
-  declare resources: HasMany<typeof CuratedCollectionResource>
-
-  @column.dateTime({ autoCreate: true })
-  declare created_at: DateTime
-
-  @column.dateTime({ autoCreate: true, autoUpdate: true })
-  declare updated_at: DateTime
-}

admin/app/models/curated_collection_resource.ts

@@ -1,41 +0,0 @@
-import { DateTime } from 'luxon'
-import { BaseModel, belongsTo, column, SnakeCaseNamingStrategy } from '@adonisjs/lucid/orm'
-import CuratedCollection from './curated_collection.js'
-import type { BelongsTo } from '@adonisjs/lucid/types/relations'
-
-export default class CuratedCollectionResource extends BaseModel {
-  static namingStrategy = new SnakeCaseNamingStrategy()
-
-  @column({ isPrimary: true })
-  declare id: number
-
-  @column()
-  declare curated_collection_slug: string
-
-  @belongsTo(() => CuratedCollection, {
-    foreignKey: 'slug',
-    localKey: 'curated_collection_slug',
-  })
-  declare curated_collection: BelongsTo<typeof CuratedCollection>
-
-  @column()
-  declare title: string
-
-  @column()
-  declare url: string
-
-  @column()
-  declare description: string
-
-  @column()
-  declare size_mb: number
-
-  @column()
-  declare downloaded: boolean
-
-  @column.dateTime({ autoCreate: true })
-  declare created_at: DateTime
-
-  @column.dateTime({ autoCreate: true, autoUpdate: true })
-  declare updated_at: DateTime
-}

admin/app/models/installed_resource.ts

@@ -0,0 +1,33 @@
+import { DateTime } from 'luxon'
+import { BaseModel, column, SnakeCaseNamingStrategy } from '@adonisjs/lucid/orm'
+
+export default class InstalledResource extends BaseModel {
+  static namingStrategy = new SnakeCaseNamingStrategy()
+
+  @column({ isPrimary: true })
+  declare id: number
+
+  @column()
+  declare resource_id: string
+
+  @column()
+  declare resource_type: 'zim' | 'map'
+
+  @column()
+  declare collection_ref: string | null
+
+  @column()
+  declare version: string
+
+  @column()
+  declare url: string
+
+  @column()
+  declare file_path: string
+
+  @column()
+  declare file_size_bytes: number | null
+
+  @column.dateTime()
+  declare installed_at: DateTime
+}

admin/app/models/installed_tier.ts

@@ -1,21 +0,0 @@
-import { DateTime } from 'luxon'
-import { BaseModel, column, SnakeCaseNamingStrategy } from '@adonisjs/lucid/orm'
-
-export default class InstalledTier extends BaseModel {
-  static namingStrategy = new SnakeCaseNamingStrategy()
-
-  @column({ isPrimary: true })
-  declare id: number
-
-  @column()
-  declare category_slug: string
-
-  @column()
-  declare tier_slug: string
-
-  @column.dateTime({ autoCreate: true })
-  declare created_at: DateTime
-
-  @column.dateTime({ autoCreate: true, autoUpdate: true })
-  declare updated_at: DateTime
-}

admin/app/models/kv_store.ts

@@ -1,6 +1,7 @@
 import { DateTime } from 'luxon'
 import { BaseModel, column, SnakeCaseNamingStrategy } from '@adonisjs/lucid/orm'
-import type { KVStoreKey, KVStoreValue } from '../../types/kv_store.js'
+import { KV_STORE_SCHEMA, type KVStoreKey, type KVStoreValue } from '../../types/kv_store.js'
+import { parseBoolean } from '../utils/misc.js'
 
 /**
  * Generic key-value store model for storing various settings
@@ -17,7 +18,7 @@ export default class KVStore extends BaseModel {
   declare key: KVStoreKey
 
   @column()
-  declare value: KVStoreValue
+  declare value: string | null
 
   @column.dateTime({ autoCreate: true })
   declare created_at: DateTime
@@ -26,28 +27,38 @@ export default class KVStore extends BaseModel {
   declare updated_at: DateTime
 
   /**
-   * Get a setting value by key
+   * Get a setting value by key, automatically deserializing to the correct type.
    */
-  static async getValue(key: KVStoreKey): Promise<KVStoreValue> {
+  static async getValue<K extends KVStoreKey>(key: K): Promise<KVStoreValue<K> | null> {
     const setting = await this.findBy('key', key)
     if (!setting || setting.value === undefined || setting.value === null) {
       return null
     }
-    if (typeof setting.value === 'string') {
-      return setting.value
-    }
-    return String(setting.value)
+    const raw = String(setting.value)
+    return (KV_STORE_SCHEMA[key] === 'boolean' ? parseBoolean(raw) : raw) as KVStoreValue<K>
   }
 
   /**
-   * Set a setting value by key (creates if not exists)
+   * Set a setting value by key (creates if not exists), automatically serializing to string.
    */
-  static async setValue(key: KVStoreKey, value: KVStoreValue): Promise<KVStore> {
-    const setting = await this.firstOrCreate({ key }, { key, value })
-    if (setting.value !== value) {
-      setting.value = value
+  static async setValue<K extends KVStoreKey>(key: K, value: KVStoreValue<K>): Promise<KVStore> {
+    const serialized = String(value)
+    const setting = await this.firstOrCreate({ key }, { key, value: serialized })
+    if (setting.value !== serialized) {
+      setting.value = serialized
       await setting.save()
     }
     return setting
   }
+
+  /**
+   * Clear a setting value by key, storing null so getValue returns null.
+   */
+  static async clearValue<K extends KVStoreKey>(key: K): Promise<void> {
+    const setting = await this.findBy('key', key)
+    if (setting && setting.value !== null) {
+      setting.value = null
+      await setting.save()
+    }
+  }
 }

admin/app/models/map_marker.ts

@@ -0,0 +1,43 @@
+import { DateTime } from 'luxon'
+import { BaseModel, column, SnakeCaseNamingStrategy } from '@adonisjs/lucid/orm'
+
+export default class MapMarker extends BaseModel {
+  static namingStrategy = new SnakeCaseNamingStrategy()
+
+  @column({ isPrimary: true })
+  declare id: number
+
+  @column()
+  declare name: string
+
+  @column()
+  declare longitude: number
+
+  @column()
+  declare latitude: number
+
+  @column()
+  declare color: string
+
+  // 'pin' for user-placed markers, 'waypoint' for route points (future)
+  @column()
+  declare marker_type: string
+
+  // Groups markers into a route (future)
+  @column()
+  declare route_id: string | null
+
+  // Order within a route (future)
+  @column()
+  declare route_order: number | null
+
+  // Optional user notes for a location
+  @column()
+  declare notes: string | null
+
+  @column.dateTime({ autoCreate: true })
+  declare created_at: DateTime
+
+  @column.dateTime({ autoCreate: true, autoUpdate: true })
+  declare updated_at: DateTime
+}

admin/app/models/service.ts

@@ -62,6 +62,15 @@ export default class Service extends BaseModel {
   @column()
   declare metadata: string | null
 
+  @column()
+  declare source_repo: string | null
+
+  @column()
+  declare available_update_version: string | null
+
+  @column.dateTime()
+  declare update_checked_at: DateTime | null
+
   @column.dateTime({ autoCreate: true })
   declare created_at: DateTime
 

admin/app/models/zim_file_metadata.ts

@@ -1,30 +0,0 @@
-import { DateTime } from 'luxon'
-import { BaseModel, column, SnakeCaseNamingStrategy } from '@adonisjs/lucid/orm'
-
-export default class ZimFileMetadata extends BaseModel {
-  static namingStrategy = new SnakeCaseNamingStrategy()
-
-  @column({ isPrimary: true })
-  declare id: number
-
-  @column()
-  declare filename: string
-
-  @column()
-  declare title: string
-
-  @column()
-  declare summary: string | null
-
-  @column()
-  declare author: string | null
-
-  @column()
-  declare size_bytes: number | null
-
-  @column.dateTime({ autoCreate: true })
-  declare created_at: DateTime
-
-  @column.dateTime({ autoCreate: true, autoUpdate: true })
-  declare updated_at: DateTime
-}

admin/app/services/benchmark_service.ts

@@ -25,12 +25,14 @@ import type {
 import { randomUUID, createHmac } from 'node:crypto'
 import { DockerService } from './docker_service.js'
 import { SERVICE_NAMES } from '../../constants/service_names.js'
+import { BROADCAST_CHANNELS } from '../../constants/broadcast.js'
+import Dockerode from 'dockerode'
 
 // HMAC secret for signing submissions to the benchmark repository
 // This provides basic protection against casual API abuse.
 // Note: Since NOMAD is open source, a determined attacker could extract this.
 // For stronger protection, see challenge-response authentication.
-const BENCHMARK_HMAC_SECRET = 'nomad-benchmark-v1-2026'
+const BENCHMARK_HMAC_SECRET = '778ba65d0bc0e23119e5ffce4b3716648a7d071f0a47ec3f'
 
 // Re-export default weights for use in service
 const SCORE_WEIGHTS = {
@@ -45,7 +47,6 @@ const SCORE_WEIGHTS = {
 // Benchmark configuration constants
 const SYSBENCH_IMAGE = 'severalnines/sysbench:latest'
 const SYSBENCH_CONTAINER_NAME = 'nomad_benchmark_sysbench'
-const BENCHMARK_CHANNEL = 'benchmark-progress'
 
 // Reference model for AI benchmark - small but meaningful
 const AI_BENCHMARK_MODEL = 'llama3.2:1b'
@@ -186,8 +187,14 @@ export class BenchmarkService {
 
       return response.data as RepositorySubmitResponse
     } catch (error) {
-      logger.error(`Failed to submit benchmark to repository: ${error.message}`)
-      throw new Error(`Failed to submit benchmark: ${error.message}`)
+      const detail = error.response?.data?.error || error.message || 'Unknown error'
+      const statusCode = error.response?.status
+      logger.error(`Failed to submit benchmark to repository: ${detail} (Status: ${statusCode})`)
+      
+      // Create an error with the status code attached for proper handling upstream
+      const err: any = new Error(`Failed to submit benchmark: ${detail}`)
+      err.statusCode = statusCode
+      throw err
     }
   }
 
@@ -270,6 +277,27 @@ export class BenchmarkService {
         gpuModel = discreteGpu?.model || graphics.controllers[0]?.model || null
       }
 
+      // Fallback: Check Docker for nvidia runtime and query GPU model via nvidia-smi
+      if (!gpuModel) {
+        try {
+          const dockerInfo = await this.dockerService.docker.info()
+          const runtimes = dockerInfo.Runtimes || {}
+          if ('nvidia' in runtimes) {
+            logger.info('[BenchmarkService] NVIDIA container runtime detected, querying GPU model via nvidia-smi')
+
+            const systemService = new (await import('./system_service.js')).SystemService(this.dockerService)
+            const nvidiaInfo = await systemService.getNvidiaSmiInfo()
+            if (Array.isArray(nvidiaInfo) && nvidiaInfo.length > 0) {
+              gpuModel = nvidiaInfo[0].model
+            } else {
+              logger.warn(`[BenchmarkService] NVIDIA runtime detected but failed to get GPU info: ${typeof nvidiaInfo === 'string' ? nvidiaInfo : JSON.stringify(nvidiaInfo)}`)
+            }
+          }
+        } catch (dockerError) {
+          logger.warn(`[BenchmarkService] Could not query Docker info for GPU detection: ${dockerError.message}`)
+        }
+      }
+
       // Fallback: Extract integrated GPU from CPU model name
       if (!gpuModel) {
         const cpuFullName = `${cpu.manufacturer} ${cpu.brand}`
@@ -543,10 +571,10 @@ export class BenchmarkService {
    */
   private _normalizeScore(value: number, reference: number): number {
     if (value <= 0) return 0
-    // Log scale: score = 50 * (1 + log2(value/reference))
-    // This gives 50 at reference value, scales logarithmically
+    // Log scale with widened range: dividing log2 by 3 prevents scores from
+    // clamping to 0% for below-average hardware. Gives 50% at reference value.
     const ratio = value / reference
-    const score = 50 * (1 + Math.log2(Math.max(0.01, ratio)))
+    const score = 50 * (1 + Math.log2(Math.max(0.01, ratio)) / 3)
     return Math.min(100, Math.max(0, score)) / 100
   }
 
@@ -555,9 +583,9 @@ export class BenchmarkService {
    */
   private _normalizeScoreInverse(value: number, reference: number): number {
     if (value <= 0) return 1
-    // Inverse: lower values = higher scores
+    // Inverse: lower values = higher scores, with widened log range
     const ratio = reference / value
-    const score = 50 * (1 + Math.log2(Math.max(0.01, ratio)))
+    const score = 50 * (1 + Math.log2(Math.max(0.01, ratio)) / 3)
     return Math.min(100, Math.max(0, score)) / 100
   }
 
@@ -591,6 +619,7 @@ export class BenchmarkService {
     const eventsMatch = output.match(/events per second:\s*([\d.]+)/i)
     const totalTimeMatch = output.match(/total time:\s*([\d.]+)s/i)
     const totalEventsMatch = output.match(/total number of events:\s*(\d+)/i)
+    logger.debug(`[BenchmarkService] CPU output parsing - events/s: ${eventsMatch?.[1]}, total_time: ${totalTimeMatch?.[1]}, total_events: ${totalEventsMatch?.[1]}`)
 
     return {
       events_per_second: eventsMatch ? parseFloat(eventsMatch[1]) : 0,
@@ -686,23 +715,26 @@ export class BenchmarkService {
    * Run a sysbench command in a Docker container
    */
   private async _runSysbenchCommand(cmd: string[]): Promise<string> {
+    let container: Dockerode.Container | null = null
     try {
       // Create container with TTY to avoid multiplexed output
-      const container = await this.dockerService.docker.createContainer({
+      container = await this.dockerService.docker.createContainer({
         Image: SYSBENCH_IMAGE,
         Cmd: cmd,
         name: `${SYSBENCH_CONTAINER_NAME}_${Date.now()}`,
         Tty: true, // Important: prevents multiplexed stdout/stderr headers
         HostConfig: {
-          AutoRemove: true,
+          AutoRemove: false, // Don't auto-remove to avoid race condition with fetching logs
         },
       })
 
       // Start container
       await container.start()
 
-      // Wait for completion and get logs
+      // Wait for completion
       await container.wait()
+      
+      // Get logs after container has finished
       const logs = await container.logs({
         stdout: true,
         stderr: true,
@@ -713,8 +745,24 @@ export class BenchmarkService {
         .replace(/[\x00-\x08]/g, '') // Remove control characters
         .trim()
 
+      // Manually remove the container after getting logs
+      try {
+        await container.remove()
+      } catch (removeError) {
+        // Log but don't fail if removal fails (container might already be gone)
+        logger.warn(`Failed to remove sysbench container: ${removeError.message}`)
+      }
+
       return output
     } catch (error) {
+      // Clean up container on error if it exists
+      if (container) {
+        try {
+          await container.remove({ force: true })
+        } catch (removeError) {
+          // Ignore removal errors
+        }
+      }
       logger.error(`Sysbench command failed: ${error.message}`)
       throw new Error(`Sysbench command failed: ${error.message}`)
     }
@@ -734,7 +782,7 @@ export class BenchmarkService {
       timestamp: new Date().toISOString(),
     }
 
-    transmit.broadcast(BENCHMARK_CHANNEL, {
+    transmit.broadcast(BROADCAST_CHANNELS.BENCHMARK_PROGRESS, {
       benchmark_id: this.currentBenchmarkId,
       ...progress,
     })

admin/app/services/chat_service.ts

@@ -4,23 +4,13 @@ import logger from '@adonisjs/core/services/logger'
 import { DateTime } from 'luxon'
 import { inject } from '@adonisjs/core'
 import { OllamaService } from './ollama_service.js'
-import { ChatRequest } from 'ollama'
-import { SYSTEM_PROMPTS } from '../../constants/ollama.js'
+import { DEFAULT_QUERY_REWRITE_MODEL, SYSTEM_PROMPTS } from '../../constants/ollama.js'
 import { toTitleCase } from '../utils/misc.js'
 
 @inject()
 export class ChatService {
   constructor(private ollamaService: OllamaService) {}
 
-  async chat(chatRequest: ChatRequest & { stream?: false }) {
-    try {
-      return await this.ollamaService.chat(chatRequest)
-    } catch (error) {
-      logger.error(`[ChatService] Chat error: ${error instanceof Error ? error.message : error}`)
-      throw new Error('Chat processing failed')
-    }
-  }
-
   async getAllSessions() {
     try {
       const sessions = await ChatSession.query().orderBy('updated_at', 'desc')
@@ -42,7 +32,7 @@ export class ChatService {
   async getChatSuggestions() {
     try {
       const models = await this.ollamaService.getModels()
-      if (!models) {
+      if (!models || models.length === 0) {
         return [] // If no models are available, return empty suggestions
       }
 
@@ -230,9 +220,59 @@ export class ChatService {
     }
   }
 
-  /**
-   * Delete all chat sessions and messages
-   */
+  async getMessageCount(sessionId: number): Promise<number> {
+    try {
+      const count = await ChatMessage.query().where('session_id', sessionId).count('* as total')
+      return Number(count[0].$extras.total)
+    } catch (error) {
+      logger.error(
+        `[ChatService] Failed to get message count for session ${sessionId}: ${error instanceof Error ? error.message : error}`
+      )
+      return 0
+    }
+  }
+
+  async generateTitle(sessionId: number, userMessage: string, assistantMessage: string) {
+    try {
+      const models = await this.ollamaService.getModels()
+      const titleModelAvailable = models?.some((m) => m.name === DEFAULT_QUERY_REWRITE_MODEL)
+
+      let title: string
+
+      if (!titleModelAvailable) {
+        title = userMessage.slice(0, 57) + (userMessage.length > 57 ? '...' : '')
+      } else {
+        const response = await this.ollamaService.chat({
+          model: DEFAULT_QUERY_REWRITE_MODEL,
+          messages: [
+            { role: 'system', content: SYSTEM_PROMPTS.title_generation },
+            { role: 'user', content: userMessage },
+            { role: 'assistant', content: assistantMessage },
+          ],
+        })
+
+        title = response?.message?.content?.trim()
+        if (!title) {
+          title = userMessage.slice(0, 57) + (userMessage.length > 57 ? '...' : '')
+        }
+      }
+
+      await this.updateSession(sessionId, { title })
+      logger.info(`[ChatService] Generated title for session ${sessionId}: "${title}"`)
+    } catch (error) {
+      logger.error(
+        `[ChatService] Failed to generate title for session ${sessionId}: ${error instanceof Error ? error.message : error}`
+      )
+      // Fall back to truncated user message
+      try {
+        const fallbackTitle = userMessage.slice(0, 57) + (userMessage.length > 57 ? '...' : '')
+        await this.updateSession(sessionId, { title: fallbackTitle })
+      } catch {
+        // Silently fail - session keeps "New Chat" title
+      }
+    }
+  }
+
   async deleteAllSessions() {
     try {
       await ChatSession.query().delete()

admin/app/services/collection_manifest_service.ts

@@ -0,0 +1,317 @@
+import axios from 'axios'
+import vine from '@vinejs/vine'
+import logger from '@adonisjs/core/services/logger'
+import { DateTime } from 'luxon'
+import { join } from 'path'
+import CollectionManifest from '#models/collection_manifest'
+import InstalledResource from '#models/installed_resource'
+import { zimCategoriesSpecSchema, mapsSpecSchema, wikipediaSpecSchema } from '#validators/curated_collections'
+import {
+  ensureDirectoryExists,
+  listDirectoryContents,
+  getFileStatsIfExists,
+  ZIM_STORAGE_PATH,
+} from '../utils/fs.js'
+import type {
+  ManifestType,
+  ZimCategoriesSpec,
+  MapsSpec,
+  CategoryWithStatus,
+  CollectionWithStatus,
+  SpecResource,
+  SpecTier,
+} from '../../types/collections.js'
+
+const SPEC_URLS: Record<ManifestType, string> = {
+  zim_categories: 'https://raw.githubusercontent.com/Crosstalk-Solutions/project-nomad/refs/heads/main/collections/kiwix-categories.json',
+  maps: 'https://github.com/Crosstalk-Solutions/project-nomad/raw/refs/heads/main/collections/maps.json',
+  wikipedia: 'https://raw.githubusercontent.com/Crosstalk-Solutions/project-nomad/refs/heads/main/collections/wikipedia.json',
+}
+
+const VALIDATORS: Record<ManifestType, any> = {
+  zim_categories: zimCategoriesSpecSchema,
+  maps: mapsSpecSchema,
+  wikipedia: wikipediaSpecSchema,
+}
+
+export class CollectionManifestService {
+  private readonly mapStoragePath = '/storage/maps'
+
+  // ---- Spec management ----
+
+  async fetchAndCacheSpec(type: ManifestType): Promise<boolean> {
+    try {
+      const response = await axios.get(SPEC_URLS[type], { timeout: 15000 })
+
+      const validated = await vine.validate({
+        schema: VALIDATORS[type],
+        data: response.data,
+      })
+
+      const existing = await CollectionManifest.find(type)
+      const specVersion = validated.spec_version
+
+      if (existing) {
+        const changed = existing.spec_version !== specVersion
+        existing.spec_version = specVersion
+        existing.spec_data = validated
+        existing.fetched_at = DateTime.now()
+        await existing.save()
+        return changed
+      }
+
+      await CollectionManifest.create({
+        type,
+        spec_version: specVersion,
+        spec_data: validated,
+        fetched_at: DateTime.now(),
+      })
+
+      return true
+    } catch (error) {
+      logger.error(`[CollectionManifestService] Failed to fetch spec for ${type}:`, error?.message || error)
+      return false
+    }
+  }
+
+  async getCachedSpec<T>(type: ManifestType): Promise<T | null> {
+    const manifest = await CollectionManifest.find(type)
+    if (!manifest) return null
+    return manifest.spec_data as T
+  }
+
+  async getSpecWithFallback<T>(type: ManifestType): Promise<T | null> {
+    try {
+      await this.fetchAndCacheSpec(type)
+    } catch {
+      // Fetch failed, will fall back to cache
+    }
+    return this.getCachedSpec<T>(type)
+  }
+
+  // ---- Status computation ----
+
+  async getCategoriesWithStatus(): Promise<CategoryWithStatus[]> {
+    const spec = await this.getSpecWithFallback<ZimCategoriesSpec>('zim_categories')
+    if (!spec) return []
+
+    const installedResources = await InstalledResource.query().where('resource_type', 'zim')
+    const installedMap = new Map(installedResources.map((r) => [r.resource_id, r]))
+
+    return spec.categories.map((category) => ({
+      ...category,
+      installedTierSlug: this.getInstalledTierForCategory(category.tiers, installedMap),
+    }))
+  }
+
+  async getMapCollectionsWithStatus(): Promise<CollectionWithStatus[]> {
+    const spec = await this.getSpecWithFallback<MapsSpec>('maps')
+    if (!spec) return []
+
+    const installedResources = await InstalledResource.query().where('resource_type', 'map')
+    const installedIds = new Set(installedResources.map((r) => r.resource_id))
+
+    return spec.collections.map((collection) => {
+      const installedCount = collection.resources.filter((r) => installedIds.has(r.id)).length
+      return {
+        ...collection,
+        all_installed: installedCount === collection.resources.length,
+        installed_count: installedCount,
+        total_count: collection.resources.length,
+      }
+    })
+  }
+
+  // ---- Tier resolution ----
+
+  static resolveTierResources(tier: SpecTier, allTiers: SpecTier[]): SpecResource[] {
+    const visited = new Set<string>()
+    return CollectionManifestService._resolveTierResourcesInner(tier, allTiers, visited)
+  }
+
+  private static _resolveTierResourcesInner(
+    tier: SpecTier,
+    allTiers: SpecTier[],
+    visited: Set<string>
+  ): SpecResource[] {
+    if (visited.has(tier.slug)) return [] // cycle detection
+    visited.add(tier.slug)
+
+    const resources: SpecResource[] = []
+
+    if (tier.includesTier) {
+      const included = allTiers.find((t) => t.slug === tier.includesTier)
+      if (included) {
+        resources.push(...CollectionManifestService._resolveTierResourcesInner(included, allTiers, visited))
+      }
+    }
+
+    resources.push(...tier.resources)
+    return resources
+  }
+
+  getInstalledTierForCategory(
+    tiers: SpecTier[],
+    installedMap: Map<string, InstalledResource>
+  ): string | undefined {
+    // Check from highest tier to lowest (tiers are ordered low to high in spec)
+    const reversedTiers = [...tiers].reverse()
+
+    for (const tier of reversedTiers) {
+      const resolved = CollectionManifestService.resolveTierResources(tier, tiers)
+      if (resolved.length === 0) continue
+
+      const allInstalled = resolved.every((r) => installedMap.has(r.id))
+      if (allInstalled) {
+        return tier.slug
+      }
+    }
+
+    return undefined
+  }
+
+  // ---- Filename parsing ----
+
+  static parseZimFilename(filename: string): { resource_id: string; version: string } | null {
+    const name = filename.replace(/\.zim$/, '')
+    const match = name.match(/^(.+)_(\d{4}-\d{2})$/)
+    if (!match) return null
+    return { resource_id: match[1], version: match[2] }
+  }
+
+  static parseMapFilename(filename: string): { resource_id: string; version: string } | null {
+    const name = filename.replace(/\.pmtiles$/, '')
+    const match = name.match(/^(.+)_(\d{4}-\d{2})$/)
+    if (!match) return null
+    return { resource_id: match[1], version: match[2] }
+  }
+
+  // ---- Filesystem reconciliation ----
+
+  async reconcileFromFilesystem(): Promise<{ zim: number; map: number }> {
+    let zimCount = 0
+    let mapCount = 0
+
+    console.log("RECONCILING FILESYSTEM MANIFESTS...")
+
+    // Reconcile ZIM files
+    try {
+      const zimDir = join(process.cwd(), ZIM_STORAGE_PATH)
+      await ensureDirectoryExists(zimDir)
+      const zimItems = await listDirectoryContents(zimDir)
+      const zimFiles = zimItems.filter((f) => f.name.endsWith('.zim'))
+
+      console.log(`Found ${zimFiles.length} ZIM files on disk. Reconciling with database...`)
+
+      // Get spec for URL lookup
+      const zimSpec = await this.getCachedSpec<ZimCategoriesSpec>('zim_categories')
+      const specResourceMap = new Map<string, SpecResource>()
+      if (zimSpec) {
+        for (const cat of zimSpec.categories) {
+          for (const tier of cat.tiers) {
+            for (const res of tier.resources) {
+              specResourceMap.set(res.id, res)
+            }
+          }
+        }
+      }
+
+      const seenZimIds = new Set<string>()
+
+      for (const file of zimFiles) {
+        console.log(`Processing ZIM file: ${file.name}`)
+        // Skip Wikipedia files (managed by WikipediaSelection model)
+        if (file.name.startsWith('wikipedia_en_')) continue
+
+        const parsed = CollectionManifestService.parseZimFilename(file.name)
+        console.log(`Parsed ZIM filename:`, parsed)
+        if (!parsed) continue
+
+        seenZimIds.add(parsed.resource_id)
+
+        const specRes = specResourceMap.get(parsed.resource_id)
+        const filePath = join(zimDir, file.name)
+        const stats = await getFileStatsIfExists(filePath)
+
+        await InstalledResource.updateOrCreate(
+          { resource_id: parsed.resource_id, resource_type: 'zim' },
+          {
+            version: parsed.version,
+            url: specRes?.url || '',
+            file_path: filePath,
+            file_size_bytes: stats ? Number(stats.size) : null,
+            installed_at: DateTime.now(),
+          }
+        )
+        zimCount++
+      }
+
+      // Remove entries for ZIM files no longer on disk
+      const existingZim = await InstalledResource.query().where('resource_type', 'zim')
+      for (const entry of existingZim) {
+        if (!seenZimIds.has(entry.resource_id)) {
+          await entry.delete()
+        }
+      }
+    } catch (error) {
+      logger.error('[CollectionManifestService] Error reconciling ZIM files:', error)
+    }
+
+    // Reconcile map files
+    try {
+      const mapDir = join(process.cwd(), this.mapStoragePath, 'pmtiles')
+      await ensureDirectoryExists(mapDir)
+      const mapItems = await listDirectoryContents(mapDir)
+      const mapFiles = mapItems.filter((f) => f.name.endsWith('.pmtiles'))
+
+      // Get spec for URL/version lookup
+      const mapSpec = await this.getCachedSpec<MapsSpec>('maps')
+      const mapResourceMap = new Map<string, SpecResource>()
+      if (mapSpec) {
+        for (const col of mapSpec.collections) {
+          for (const res of col.resources) {
+            mapResourceMap.set(res.id, res)
+          }
+        }
+      }
+
+      const seenMapIds = new Set<string>()
+
+      for (const file of mapFiles) {
+        const parsed = CollectionManifestService.parseMapFilename(file.name)
+        if (!parsed) continue
+
+        seenMapIds.add(parsed.resource_id)
+
+        const specRes = mapResourceMap.get(parsed.resource_id)
+        const filePath = join(mapDir, file.name)
+        const stats = await getFileStatsIfExists(filePath)
+
+        await InstalledResource.updateOrCreate(
+          { resource_id: parsed.resource_id, resource_type: 'map' },
+          {
+            version: parsed.version,
+            url: specRes?.url || '',
+            file_path: filePath,
+            file_size_bytes: stats ? Number(stats.size) : null,
+            installed_at: DateTime.now(),
+          }
+        )
+        mapCount++
+      }
+
+      // Remove entries for map files no longer on disk
+      const existingMaps = await InstalledResource.query().where('resource_type', 'map')
+      for (const entry of existingMaps) {
+        if (!seenMapIds.has(entry.resource_id)) {
+          await entry.delete()
+        }
+      }
+    } catch (error) {
+      logger.error('[CollectionManifestService] Error reconciling map files:', error)
+    }
+
+    logger.info(`[CollectionManifestService] Reconciled ${zimCount} ZIM files, ${mapCount} map files`)
+    return { zim: zimCount, map: mapCount }
+  }
+}

admin/app/services/collection_update_service.ts

@@ -0,0 +1,157 @@
+import logger from '@adonisjs/core/services/logger'
+import env from '#start/env'
+import axios from 'axios'
+import InstalledResource from '#models/installed_resource'
+import { RunDownloadJob } from '../jobs/run_download_job.js'
+import { ZIM_STORAGE_PATH } from '../utils/fs.js'
+import { join } from 'path'
+import type {
+  ResourceUpdateCheckRequest,
+  ResourceUpdateInfo,
+  ContentUpdateCheckResult,
+} from '../../types/collections.js'
+import { NOMAD_API_DEFAULT_BASE_URL } from '../../constants/misc.js'
+
+const MAP_STORAGE_PATH = '/storage/maps'
+
+const ZIM_MIME_TYPES = ['application/x-zim', 'application/x-openzim', 'application/octet-stream']
+const PMTILES_MIME_TYPES = ['application/vnd.pmtiles', 'application/octet-stream']
+
+export class CollectionUpdateService {
+  async checkForUpdates(): Promise<ContentUpdateCheckResult> {
+    const nomadAPIURL = env.get('NOMAD_API_URL') || NOMAD_API_DEFAULT_BASE_URL
+    if (!nomadAPIURL) {
+      return {
+        updates: [],
+        checked_at: new Date().toISOString(),
+        error: 'Nomad API is not configured. Set the NOMAD_API_URL environment variable.',
+      }
+    }
+
+    const installed = await InstalledResource.all()
+    if (installed.length === 0) {
+      return {
+        updates: [],
+        checked_at: new Date().toISOString(),
+      }
+    }
+
+    const requestBody: ResourceUpdateCheckRequest = {
+      resources: installed.map((r) => ({
+        resource_id: r.resource_id,
+        resource_type: r.resource_type,
+        installed_version: r.version,
+      })),
+    }
+
+    try {
+      const response = await axios.post<ResourceUpdateInfo[]>(`${nomadAPIURL}/api/v1/resources/check-updates`, requestBody, {
+        timeout: 15000,
+      })
+
+      logger.info(
+        `[CollectionUpdateService] Update check complete: ${response.data.length} update(s) available`
+      )
+
+      return {
+        updates: response.data,
+        checked_at: new Date().toISOString(),
+      }
+    } catch (error) {
+      if (axios.isAxiosError(error) && error.response) {
+        logger.error(
+          `[CollectionUpdateService] Nomad API returned ${error.response.status}: ${JSON.stringify(error.response.data)}`
+        )
+        return {
+          updates: [],
+          checked_at: new Date().toISOString(),
+          error: `Nomad API returned status ${error.response.status}`,
+        }
+      }
+      const message =
+        error instanceof Error ? error.message : 'Unknown error contacting Nomad API'
+      logger.error(`[CollectionUpdateService] Failed to check for updates: ${message}`)
+      return {
+        updates: [],
+        checked_at: new Date().toISOString(),
+        error: `Failed to contact Nomad API: ${message}`,
+      }
+    }
+  }
+
+  async applyUpdate(
+    update: ResourceUpdateInfo
+  ): Promise<{ success: boolean; jobId?: string; error?: string }> {
+    // Check if a download is already in progress for this URL
+    const existingJob = await RunDownloadJob.getByUrl(update.download_url)
+    if (existingJob) {
+      const state = await existingJob.getState()
+      if (state === 'active' || state === 'waiting' || state === 'delayed') {
+        return {
+          success: false,
+          error: `A download is already in progress for ${update.resource_id}`,
+        }
+      }
+    }
+
+    const filename = this.buildFilename(update)
+    const filepath = this.buildFilepath(update, filename)
+
+    const result = await RunDownloadJob.dispatch({
+      url: update.download_url,
+      filepath,
+      timeout: 30000,
+      allowedMimeTypes:
+        update.resource_type === 'zim' ? ZIM_MIME_TYPES : PMTILES_MIME_TYPES,
+      forceNew: true,
+      filetype: update.resource_type,
+      resourceMetadata: {
+        resource_id: update.resource_id,
+        version: update.latest_version,
+        collection_ref: null,
+      },
+    })
+
+    if (!result || !result.job) {
+      return { success: false, error: 'Failed to dispatch download job' }
+    }
+
+    logger.info(
+      `[CollectionUpdateService] Dispatched update download for ${update.resource_id}: ${update.installed_version} → ${update.latest_version}`
+    )
+
+    return { success: true, jobId: result.job.id }
+  }
+
+  async applyAllUpdates(
+    updates: ResourceUpdateInfo[]
+  ): Promise<{ results: Array<{ resource_id: string; success: boolean; jobId?: string; error?: string }> }> {
+    const results: Array<{
+      resource_id: string
+      success: boolean
+      jobId?: string
+      error?: string
+    }> = []
+
+    for (const update of updates) {
+      const result = await this.applyUpdate(update)
+      results.push({ resource_id: update.resource_id, ...result })
+    }
+
+    return { results }
+  }
+
+  private buildFilename(update: ResourceUpdateInfo): string {
+    if (update.resource_type === 'zim') {
+      return `${update.resource_id}_${update.latest_version}.zim`
+    }
+    return `${update.resource_id}_${update.latest_version}.pmtiles`
+  }
+
+  private buildFilepath(update: ResourceUpdateInfo, filename: string): string {
+    if (update.resource_type === 'zim') {
+      return join(process.cwd(), ZIM_STORAGE_PATH, filename)
+    }
+    return join(process.cwd(), MAP_STORAGE_PATH, 'pmtiles', filename)
+  }
+}

admin/app/services/container_registry_service.ts

@@ -0,0 +1,484 @@
+import logger from '@adonisjs/core/services/logger'
+import { isNewerVersion, parseMajorVersion } from '../utils/version.js'
+
+export interface ParsedImageReference {
+  registry: string
+  namespace: string
+  repo: string
+  tag: string
+  /** Full name for registry API calls: namespace/repo */
+  fullName: string
+}
+
+export interface AvailableUpdate {
+  tag: string
+  isLatest: boolean
+  releaseUrl?: string
+}
+
+interface TokenCacheEntry {
+  token: string
+  expiresAt: number
+}
+
+const SEMVER_TAG_PATTERN = /^v?(\d+\.\d+(?:\.\d+)?)$/
+const PLATFORM_SUFFIXES = ['-arm64', '-amd64', '-alpine', '-slim', '-cuda', '-rocm']
+const REJECTED_TAGS = new Set(['latest', 'nightly', 'edge', 'dev', 'beta', 'alpha', 'canary', 'rc', 'test', 'debug'])
+
+export class ContainerRegistryService {
+  private tokenCache = new Map<string, TokenCacheEntry>()
+  private sourceUrlCache = new Map<string, string | null>()
+  private releaseTagPrefixCache = new Map<string, string>()
+
+  /**
+   * Parse a Docker image reference string into its components.
+   */
+  parseImageReference(image: string): ParsedImageReference {
+    let registry: string
+    let remainder: string
+    let tag = 'latest'
+
+    // Split off the tag
+    const lastColon = image.lastIndexOf(':')
+    if (lastColon > -1 && !image.substring(lastColon).includes('/')) {
+      tag = image.substring(lastColon + 1)
+      image = image.substring(0, lastColon)
+    }
+
+    // Determine registry vs image path
+    const parts = image.split('/')
+
+    if (parts.length === 1) {
+      // e.g. "nginx" → Docker Hub library image
+      registry = 'registry-1.docker.io'
+      remainder = `library/${parts[0]}`
+    } else if (parts.length === 2 && !parts[0].includes('.') && !parts[0].includes(':')) {
+      // e.g. "ollama/ollama" → Docker Hub user image
+      registry = 'registry-1.docker.io'
+      remainder = image
+    } else {
+      // e.g. "ghcr.io/kiwix/kiwix-serve" → custom registry
+      registry = parts[0]
+      remainder = parts.slice(1).join('/')
+    }
+
+    const namespaceParts = remainder.split('/')
+    const repo = namespaceParts.pop()!
+    const namespace = namespaceParts.join('/')
+
+    return {
+      registry,
+      namespace,
+      repo,
+      tag,
+      fullName: remainder,
+    }
+  }
+
+  /**
+   * Get an anonymous auth token for the given registry and repository.
+   * NOTE: This could be expanded in the future to support private repo authentication
+   */
+  private async getToken(registry: string, fullName: string): Promise<string> {
+    const cacheKey = `${registry}/${fullName}`
+    const cached = this.tokenCache.get(cacheKey)
+    if (cached && cached.expiresAt > Date.now()) {
+      return cached.token
+    }
+
+    let tokenUrl: string
+    if (registry === 'registry-1.docker.io') {
+      tokenUrl = `https://auth.docker.io/token?service=registry.docker.io&scope=repository:${fullName}:pull`
+    } else if (registry === 'ghcr.io') {
+      tokenUrl = `https://ghcr.io/token?service=ghcr.io&scope=repository:${fullName}:pull`
+    } else {
+      // For other registries, try the standard v2 token endpoint
+      tokenUrl = `https://${registry}/token?service=${registry}&scope=repository:${fullName}:pull`
+    }
+
+    const response = await this.fetchWithRetry(tokenUrl)
+    if (!response.ok) {
+      throw new Error(`Failed to get auth token from ${registry}: ${response.status}`)
+    }
+
+    const data = (await response.json()) as { token?: string; access_token?: string }
+    const token = data.token || data.access_token || ''
+
+    if (!token) {
+      throw new Error(`No token returned from ${registry}`)
+    }
+
+    // Cache for 5 minutes (tokens usually last longer, but be conservative)
+    this.tokenCache.set(cacheKey, {
+      token,
+      expiresAt: Date.now() + 5 * 60 * 1000,
+    })
+
+    return token
+  }
+
+  /**
+   * List all tags for a given image from the registry.
+   */
+  async listTags(parsed: ParsedImageReference): Promise<string[]> {
+    const token = await this.getToken(parsed.registry, parsed.fullName)
+    const allTags: string[] = []
+    let url = `https://${parsed.registry}/v2/${parsed.fullName}/tags/list?n=1000`
+
+    while (url) {
+      const response = await this.fetchWithRetry(url, {
+        headers: { Authorization: `Bearer ${token}` },
+      })
+
+      if (!response.ok) {
+        throw new Error(`Failed to list tags for ${parsed.fullName}: ${response.status}`)
+      }
+
+      const data = (await response.json()) as { tags?: string[] }
+      if (data.tags) {
+        allTags.push(...data.tags)
+      }
+
+      // Handle pagination via Link header
+      const linkHeader = response.headers.get('link')
+      if (linkHeader) {
+        const match = linkHeader.match(/<([^>]+)>;\s*rel="next"/)
+        url = match ? match[1] : ''
+      } else {
+        url = ''
+      }
+    }
+
+    return allTags
+  }
+
+  /**
+   * Check if a specific tag supports the given architecture by fetching its manifest.
+   */
+  async checkArchSupport(parsed: ParsedImageReference, tag: string, hostArch: string): Promise<boolean> {
+    try {
+      const token = await this.getToken(parsed.registry, parsed.fullName)
+      const url = `https://${parsed.registry}/v2/${parsed.fullName}/manifests/${tag}`
+
+      const response = await this.fetchWithRetry(url, {
+        headers: {
+          Authorization: `Bearer ${token}`,
+          Accept: [
+            'application/vnd.oci.image.index.v1+json',
+            'application/vnd.docker.distribution.manifest.list.v2+json',
+            'application/vnd.oci.image.manifest.v1+json',
+            'application/vnd.docker.distribution.manifest.v2+json',
+          ].join(', '),
+        },
+      })
+
+      if (!response.ok) return true // If we can't check, assume it's compatible
+
+      const manifest = (await response.json()) as {
+        mediaType?: string
+        manifests?: Array<{ platform?: { architecture?: string } }>
+      }
+      const mediaType = manifest.mediaType || response.headers.get('content-type') || ''
+
+      // Manifest list — check if any platform matches
+      if (
+        mediaType.includes('manifest.list') ||
+        mediaType.includes('image.index') ||
+        manifest.manifests
+      ) {
+        const manifests = manifest.manifests || []
+        return manifests.some(
+          (m: any) => m.platform && m.platform.architecture === hostArch
+        )
+      }
+
+      // Single manifest — assume compatible (can't easily determine arch without fetching config blob)
+      return true
+    } catch (error) {
+      logger.warn(`[ContainerRegistryService] Error checking arch for ${tag}: ${error.message}`)
+      return true // Assume compatible on error
+    }
+  }
+
+  /**
+   * Extract the source repository URL from an image's OCI labels.
+   * Uses the standardized `org.opencontainers.image.source` label.
+   * Result is cached per image (not per tag).
+   */
+  async getSourceUrl(parsed: ParsedImageReference): Promise<string | null> {
+    const cacheKey = `${parsed.registry}/${parsed.fullName}`
+    if (this.sourceUrlCache.has(cacheKey)) {
+      return this.sourceUrlCache.get(cacheKey)!
+    }
+
+    try {
+      const token = await this.getToken(parsed.registry, parsed.fullName)
+
+      // First get the manifest to find the config blob digest
+      const manifestUrl = `https://${parsed.registry}/v2/${parsed.fullName}/manifests/${parsed.tag}`
+      const manifestRes = await this.fetchWithRetry(manifestUrl, {
+        headers: {
+          Authorization: `Bearer ${token}`,
+          Accept: [
+            'application/vnd.oci.image.manifest.v1+json',
+            'application/vnd.docker.distribution.manifest.v2+json',
+            'application/vnd.oci.image.index.v1+json',
+            'application/vnd.docker.distribution.manifest.list.v2+json',
+          ].join(', '),
+        },
+      })
+
+      if (!manifestRes.ok) {
+        this.sourceUrlCache.set(cacheKey, null)
+        return null
+      }
+
+      const manifest = (await manifestRes.json()) as {
+        config?: { digest?: string }
+        manifests?: Array<{ digest?: string; mediaType?: string; platform?: { architecture?: string } }>
+      }
+
+      // If this is a manifest list, pick the first manifest to get the config
+      let configDigest = manifest.config?.digest
+      if (!configDigest && manifest.manifests?.length) {
+        const firstManifest = manifest.manifests[0]
+        if (firstManifest.digest) {
+          const childRes = await this.fetchWithRetry(
+            `https://${parsed.registry}/v2/${parsed.fullName}/manifests/${firstManifest.digest}`,
+            {
+              headers: {
+                Authorization: `Bearer ${token}`,
+                Accept: 'application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json',
+              },
+            }
+          )
+          if (childRes.ok) {
+            const childManifest = (await childRes.json()) as { config?: { digest?: string } }
+            configDigest = childManifest.config?.digest
+          }
+        }
+      }
+
+      if (!configDigest) {
+        this.sourceUrlCache.set(cacheKey, null)
+        return null
+      }
+
+      // Fetch the config blob to read labels
+      const blobUrl = `https://${parsed.registry}/v2/${parsed.fullName}/blobs/${configDigest}`
+      const blobRes = await this.fetchWithRetry(blobUrl, {
+        headers: { Authorization: `Bearer ${token}` },
+      })
+
+      if (!blobRes.ok) {
+        this.sourceUrlCache.set(cacheKey, null)
+        return null
+      }
+
+      const config = (await blobRes.json()) as {
+        config?: { Labels?: Record<string, string> }
+      }
+
+      const sourceUrl = config.config?.Labels?.['org.opencontainers.image.source'] || null
+      this.sourceUrlCache.set(cacheKey, sourceUrl)
+      return sourceUrl
+    } catch (error) {
+      logger.warn(`[ContainerRegistryService] Failed to get source URL for ${cacheKey}: ${error.message}`)
+      this.sourceUrlCache.set(cacheKey, null)
+      return null
+    }
+  }
+
+  /**
+   * Detect whether a GitHub/GitLab repo uses a 'v' prefix on release tags.
+   * Probes the GitHub API with the current tag to determine the convention,
+   * then caches the result per source URL.
+   */
+  async detectReleaseTagPrefix(sourceUrl: string, sampleTag: string): Promise<string> {
+    if (this.releaseTagPrefixCache.has(sourceUrl)) {
+      return this.releaseTagPrefixCache.get(sourceUrl)!
+    }
+
+    try {
+      const url = new URL(sourceUrl)
+      if (url.hostname !== 'github.com') {
+        this.releaseTagPrefixCache.set(sourceUrl, '')
+        return ''
+      }
+
+      const cleanPath = url.pathname.replace(/\.git$/, '').replace(/\/$/, '')
+      const strippedTag = sampleTag.replace(/^v/, '')
+      const vTag = `v${strippedTag}`
+
+      // Try both variants against GitHub's API — the one that 200s tells us the convention
+      // Try v-prefixed first since it's more common
+      const vRes = await this.fetchWithRetry(
+        `https://api.github.com/repos${cleanPath}/releases/tags/${vTag}`,
+        { headers: { Accept: 'application/vnd.github.v3+json', 'User-Agent': 'ProjectNomad' } },
+        1
+      )
+      if (vRes.ok) {
+        this.releaseTagPrefixCache.set(sourceUrl, 'v')
+        return 'v'
+      }
+
+      const plainRes = await this.fetchWithRetry(
+        `https://api.github.com/repos${cleanPath}/releases/tags/${strippedTag}`,
+        { headers: { Accept: 'application/vnd.github.v3+json', 'User-Agent': 'ProjectNomad' } },
+        1
+      )
+      if (plainRes.ok) {
+        this.releaseTagPrefixCache.set(sourceUrl, '')
+        return ''
+      }
+    } catch {
+      // On error, fall through to default
+    }
+
+    // Default: no prefix modification
+    this.releaseTagPrefixCache.set(sourceUrl, '')
+    return ''
+  }
+
+  /**
+   * Build a release URL for a specific tag given a source repository URL and
+   * the detected release tag prefix convention.
+   * Supports GitHub and GitLab URL patterns.
+   */
+  buildReleaseUrl(sourceUrl: string, tag: string, releaseTagPrefix: string): string | undefined {
+    try {
+      const url = new URL(sourceUrl)
+      if (url.hostname === 'github.com' || url.hostname.includes('gitlab')) {
+        const cleanPath = url.pathname.replace(/\.git$/, '').replace(/\/$/, '')
+        const strippedTag = tag.replace(/^v/, '')
+        const releaseTag = releaseTagPrefix ? `${releaseTagPrefix}${strippedTag}` : strippedTag
+        return `${url.origin}${cleanPath}/releases/tag/${releaseTag}`
+      }
+    } catch {
+      // Invalid URL, skip
+    }
+    return undefined
+  }
+
+  /**
+   * Filter and sort tags to find compatible updates for a service.
+   */
+  filterCompatibleUpdates(
+    tags: string[],
+    currentTag: string,
+    majorVersion: number
+  ): string[] {
+    return tags
+      .filter((tag) => {
+        // Must match semver pattern
+        if (!SEMVER_TAG_PATTERN.test(tag)) return false
+
+        // Reject known non-version tags
+        if (REJECTED_TAGS.has(tag.toLowerCase())) return false
+
+        // Reject platform suffixes
+        if (PLATFORM_SUFFIXES.some((suffix) => tag.toLowerCase().endsWith(suffix))) return false
+
+        // Must be same major version
+        if (parseMajorVersion(tag) !== majorVersion) return false
+
+        // Must be newer than current
+        return isNewerVersion(tag, currentTag)
+      })
+      .sort((a, b) => (isNewerVersion(a, b) ? -1 : 1)) // Newest first
+  }
+
+  /**
+   * High-level method to get available updates for a service.
+   * Returns a sorted list of compatible newer versions (newest first).
+   */
+  async getAvailableUpdates(
+    containerImage: string,
+    hostArch: string,
+    fallbackSourceRepo?: string | null
+  ): Promise<AvailableUpdate[]> {
+    const parsed = this.parseImageReference(containerImage)
+    const currentTag = parsed.tag
+
+    if (currentTag === 'latest') {
+      logger.warn(
+        `[ContainerRegistryService] Cannot check updates for ${containerImage} — using :latest tag`
+      )
+      return []
+    }
+
+    const majorVersion = parseMajorVersion(currentTag)
+
+    // Fetch tags and source URL in parallel
+    const [tags, ociSourceUrl] = await Promise.all([
+      this.listTags(parsed),
+      this.getSourceUrl(parsed),
+    ])
+
+    // OCI label takes precedence, fall back to DB-stored source_repo
+    const sourceUrl = ociSourceUrl || fallbackSourceRepo || null
+
+    const compatible = this.filterCompatibleUpdates(tags, currentTag, majorVersion)
+
+    // Detect release tag prefix convention (e.g. 'v' vs no prefix) if we have a source URL
+    let releaseTagPrefix = ''
+    if (sourceUrl) {
+      releaseTagPrefix = await this.detectReleaseTagPrefix(sourceUrl, currentTag)
+    }
+
+    // Check architecture support for the top candidates (limit checks to save API calls)
+    const maxArchChecks = 10
+    const results: AvailableUpdate[] = []
+
+    for (const tag of compatible.slice(0, maxArchChecks)) {
+      const supported = await this.checkArchSupport(parsed, tag, hostArch)
+      if (supported) {
+        results.push({
+          tag,
+          isLatest: results.length === 0,
+          releaseUrl: sourceUrl ? this.buildReleaseUrl(sourceUrl, tag, releaseTagPrefix) : undefined,
+        })
+      }
+    }
+
+    // For remaining tags (beyond arch check limit), include them but mark as not latest
+    for (const tag of compatible.slice(maxArchChecks)) {
+      results.push({
+        tag,
+        isLatest: false,
+        releaseUrl: sourceUrl ? this.buildReleaseUrl(sourceUrl, tag, releaseTagPrefix) : undefined,
+      })
+    }
+
+    return results
+  }
+
+  /**
+   * Fetch with retry and exponential backoff for rate limiting.
+   */
+  private async fetchWithRetry(
+    url: string,
+    init?: RequestInit,
+    maxRetries = 3
+  ): Promise<Response> {
+    for (let attempt = 0; attempt <= maxRetries; attempt++) {
+      const response = await fetch(url, init)
+
+      if (response.status === 429 && attempt < maxRetries) {
+        const retryAfter = response.headers.get('retry-after')
+        const delay = retryAfter
+          ? parseInt(retryAfter, 10) * 1000
+          : Math.pow(2, attempt) * 1000
+        logger.warn(
+          `[ContainerRegistryService] Rate limited on ${url}, retrying in ${delay}ms`
+        )
+        await new Promise((resolve) => setTimeout(resolve, delay))
+        continue
+      }
+
+      return response
+    }
+
+    throw new Error(`Failed to fetch ${url} after ${maxRetries} retries`)
+  }
+}

admin/app/services/docker_service.ts

@@ -6,11 +6,14 @@ import transmit from '@adonisjs/transmit/services/main'
 import { doResumableDownloadWithRetry } from '../utils/downloads.js'
 import { join } from 'path'
 import { ZIM_STORAGE_PATH } from '../utils/fs.js'
+import { KiwixLibraryService } from './kiwix_library_service.js'
 import { SERVICE_NAMES } from '../../constants/service_names.js'
 import { exec } from 'child_process'
 import { promisify } from 'util'
 // import { readdir } from 'fs/promises'
 import KVStore from '#models/kv_store'
+import { BROADCAST_CHANNELS } from '../../constants/broadcast.js'
+import { KIWIX_LIBRARY_CMD } from '../../constants/kiwix.js'
 
 @inject()
 export class DockerService {
@@ -18,6 +21,9 @@ export class DockerService {
   private activeInstallations: Set<string> = new Set()
   public static NOMAD_NETWORK = 'project-nomad_default'
 
+  private _servicesStatusCache: { data: { service_name: string; status: string }[]; expiresAt: number } | null = null
+  private _servicesStatusInflight: Promise<{ service_name: string; status: string }[]> | null = null
+
   constructor() {
     // Support both Linux (production) and Windows (development with Docker Desktop)
     const isWindows = process.platform === 'win32'
@@ -55,6 +61,7 @@ export class DockerService {
       const dockerContainer = this.docker.getContainer(container.Id)
       if (action === 'stop') {
         await dockerContainer.stop()
+        this.invalidateServicesStatusCache()
         return {
           success: true,
           message: `Service ${serviceName} stopped successfully`,
@@ -62,7 +69,18 @@ export class DockerService {
       }
 
       if (action === 'restart') {
+        if (serviceName === SERVICE_NAMES.KIWIX) {
+          const isLegacy = await this.isKiwixOnLegacyConfig()
+          if (isLegacy) {
+            logger.info('[DockerService] Kiwix on legacy glob config — running migration instead of restart.')
+            await this.migrateKiwixToLibraryMode()
+            this.invalidateServicesStatusCache()
+            return { success: true, message: 'Kiwix migrated to library mode successfully.' }
+          }
+        }
+
         await dockerContainer.restart()
+        this.invalidateServicesStatusCache()
 
         return {
           success: true,
@@ -79,6 +97,7 @@ export class DockerService {
         }
 
         await dockerContainer.start()
+        this.invalidateServicesStatusCache()
 
         return {
           success: true,
@@ -90,7 +109,7 @@ export class DockerService {
         success: false,
         message: `Invalid action: ${action}. Use 'start', 'stop', or 'restart'.`,
       }
-    } catch (error) {
+    } catch (error: any) {
       logger.error(`Error starting service ${serviceName}: ${error.message}`)
       return {
         success: false,
@@ -101,19 +120,43 @@ export class DockerService {
 
   /**
    * Fetches the status of all Docker containers related to Nomad services. (those prefixed with 'nomad_')
+   * Results are cached for 5 seconds and concurrent callers share a single in-flight request,
+   * preventing Docker socket congestion during rapid page navigation.
    */
-  async getServicesStatus(): Promise<
-    {
-      service_name: string
-      status: string
-    }[]
-  > {
+  async getServicesStatus(): Promise<{ service_name: string; status: string }[]> {
+    const now = Date.now()
+    if (this._servicesStatusCache && now < this._servicesStatusCache.expiresAt) {
+      return this._servicesStatusCache.data
+    }
+    if (this._servicesStatusInflight) return this._servicesStatusInflight
+
+    this._servicesStatusInflight = this._fetchServicesStatus().then((data) => {
+      this._servicesStatusCache = { data, expiresAt: Date.now() + 5000 }
+      this._servicesStatusInflight = null
+      return data
+    }).catch((err) => {
+      this._servicesStatusInflight = null
+      throw err
+    })
+    return this._servicesStatusInflight
+  }
+
+  /**
+   * Invalidates the services status cache. Call this after any container state change
+   * (start, stop, restart, install, uninstall) so the next read reflects reality.
+   */
+  invalidateServicesStatusCache() {
+    this._servicesStatusCache = null
+    this._servicesStatusInflight = null
+  }
+
+  private async _fetchServicesStatus(): Promise<{ service_name: string; status: string }[]> {
     try {
       const containers = await this.docker.listContainers({ all: true })
       const containerMap = new Map<string, Docker.ContainerInfo>()
       containers.forEach((container) => {
-        const name = container.Names[0].replace('/', '')
-        if (name.startsWith('nomad_')) {
+        const name = container.Names[0]?.replace('/', '')
+        if (name && name.startsWith('nomad_')) {
           containerMap.set(name, container)
         }
       })
@@ -122,7 +165,7 @@ export class DockerService {
         service_name: name,
         status: container.State,
       }))
-    } catch (error) {
+    } catch (error: any) {
       logger.error(`Error fetching services status: ${error.message}`)
       return []
     }
@@ -139,6 +182,11 @@ export class DockerService {
       return null
     }
 
+    if (serviceName === SERVICE_NAMES.OLLAMA) {
+      const remoteUrl = await KVStore.getValue('ai.remoteOllamaUrl')
+      if (remoteUrl) return remoteUrl
+    }
+
     const service = await Service.query()
       .where('service_name', serviceName)
       .andWhere('installed', true)
@@ -306,7 +354,7 @@ export class DockerService {
             `No existing container found, proceeding with installation...`
           )
         }
-      } catch (error) {
+      } catch (error: any) {
         logger.warn(`Error during container cleanup: ${error.message}`)
         this._broadcast(serviceName, 'cleanup-warning', `Warning during cleanup: ${error.message}`)
       }
@@ -325,7 +373,7 @@ export class DockerService {
             const volume = this.docker.getVolume(vol.Name)
             await volume.remove({ force: true })
             this._broadcast(serviceName, 'volume-removed', `Removed volume: ${vol.Name}`)
-          } catch (error) {
+          } catch (error: any) {
             logger.warn(`Failed to remove volume ${vol.Name}: ${error.message}`)
           }
         }
@@ -333,7 +381,7 @@ export class DockerService {
         if (serviceVolumes.length === 0) {
           this._broadcast(serviceName, 'no-volumes', `No volumes found to clear`)
         }
-      } catch (error) {
+      } catch (error: any) {
         logger.warn(`Error during volume cleanup: ${error.message}`)
         this._broadcast(
           serviceName,
@@ -346,6 +394,7 @@ export class DockerService {
       service.installed = false
       service.installation_status = 'installing'
       await service.save()
+      this.invalidateServicesStatusCache()
 
       // Step 5: Recreate the container
       this._broadcast(serviceName, 'recreating', `Recreating container...`)
@@ -361,7 +410,7 @@ export class DockerService {
         success: true,
         message: `Service ${serviceName} force reinstall initiated successfully. You can receive updates via server-sent events.`,
       }
-    } catch (error) {
+    } catch (error: any) {
       logger.error(`Force reinstall failed for ${serviceName}: ${error.message}`)
       await this._cleanupFailedInstallation(serviceName)
       return {
@@ -453,13 +502,13 @@ export class DockerService {
       let gpuHostConfig = containerConfig?.HostConfig || {}
 
       if (service.service_name === SERVICE_NAMES.OLLAMA) {
-        const gpuType = await this._detectGPUType()
+        const gpuResult = await this._detectGPUType()
 
-        if (gpuType === 'nvidia') {
+        if (gpuResult.type === 'nvidia') {
           this._broadcast(
             service.service_name,
             'gpu-config',
-            `NVIDIA GPU detected. Configuring container with GPU support...`
+            `NVIDIA container runtime detected. Configuring container with GPU support...`
           )
 
           // Add GPU support for NVIDIA
@@ -473,35 +522,23 @@ export class DockerService {
               },
             ],
           }
-        } else if (gpuType === 'amd') {
-          // this._broadcast(
-          //   service.service_name,
-          //   'gpu-config',
-          //   `AMD GPU detected. Using ROCm image and configuring container with GPU support...`
-          // )
-
-          // // Use ROCm image for AMD
-          // finalImage = 'ollama/ollama:rocm'
-
-          // // Dynamically discover and add AMD GPU devices
-          // const amdDevices = await this._discoverAMDDevices()
-          // if (!amdDevices || amdDevices.length === 0) {
-          //   this._broadcast(
-          //     service.service_name,
-          //     'gpu-config-error',
-          //     `Failed to discover AMD GPU devices. Proceeding with CPU-only configuration...`
-          //   )
-          //   gpuHostConfig = { ...gpuHostConfig } // No GPU devices added
-          //   logger.warn(`[DockerService] No AMD GPU devices discovered for Ollama`)
-          // } else {
-          //   gpuHostConfig = {
-          //     ...gpuHostConfig,
-          //     Devices: amdDevices,
-          //   }
-          //   logger.info(
-          //     `[DockerService] Configured ${amdDevices.length} AMD GPU devices for Ollama`
-          //   )
-          // }
+        } else if (gpuResult.type === 'amd') {
+          this._broadcast(
+            service.service_name,
+            'gpu-config',
+            `AMD GPU detected. ROCm GPU acceleration is not yet supported in this version — proceeding with CPU-only configuration. GPU support for AMD will be available in a future update.`
+          )
+          logger.warn('[DockerService] AMD GPU detected but ROCm support is not yet enabled. Using CPU-only configuration.')
+          // TODO: Re-enable AMD GPU support once ROCm image and device discovery are validated.
+          // When re-enabling:
+          //   1. Switch image to 'ollama/ollama:rocm'
+          //   2. Restore _discoverAMDDevices() to map /dev/kfd and /dev/dri/* into the container
+        } else if (gpuResult.toolkitMissing) {
+          this._broadcast(
+            service.service_name,
+            'gpu-config',
+            `NVIDIA GPU detected but NVIDIA Container Toolkit is not installed. Using CPU-only configuration. Install the toolkit and reinstall AI Assistant for GPU acceleration: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html`
+          )
         } else {
           this._broadcast(
             service.service_name,
@@ -511,6 +548,15 @@ export class DockerService {
         }
       }
 
+      const ollamaEnv: string[] = []
+      if (service.service_name === SERVICE_NAMES.OLLAMA) {
+        ollamaEnv.push('OLLAMA_NO_CLOUD=1')
+        const flashAttentionEnabled = await KVStore.getValue('ai.ollamaFlashAttention')
+        if (flashAttentionEnabled !== false) {
+          ollamaEnv.push('OLLAMA_FLASH_ATTENTION=1')
+        }
+      }
+
       this._broadcast(
         service.service_name,
         'creating',
@@ -519,11 +565,16 @@ export class DockerService {
       const container = await this.docker.createContainer({
         Image: finalImage,
         name: service.service_name,
+        Labels: {
+          ...(containerConfig?.Labels ?? {}),
+          'com.docker.compose.project': 'project-nomad-managed',
+          'io.project-nomad.managed': 'true',
+        },
         ...(containerConfig?.User && { User: containerConfig.User }),
         HostConfig: gpuHostConfig,
         ...(containerConfig?.WorkingDir && { WorkingDir: containerConfig.WorkingDir }),
         ...(containerConfig?.ExposedPorts && { ExposedPorts: containerConfig.ExposedPorts }),
-        ...(containerConfig?.Env && { Env: containerConfig.Env }),
+        Env: [...(containerConfig?.Env ?? []), ...ollamaEnv],
         ...(service.container_command ? { Cmd: service.container_command.split(' ') } : {}),
         // Ensure container is attached to the Nomad docker network in production
         ...(process.env.NODE_ENV === 'production' && {
@@ -550,14 +601,15 @@ export class DockerService {
       service.installed = true
       service.installation_status = 'idle'
       await service.save()
+      this.invalidateServicesStatusCache()
 
       // Remove from active installs tracking
       this.activeInstallations.delete(service.service_name)
 
       // If Ollama was just installed, trigger Nomad docs discovery and embedding
       if (service.service_name === SERVICE_NAMES.OLLAMA) {
-        logger.info('[DockerService] Ollama installation complete. Enabling chat suggestions by default.')
-        await KVStore.setValue('chat.suggestionsEnabled', "true")
+        logger.info('[DockerService] Ollama installation complete. Default behavior is to not enable chat suggestions.')
+        await KVStore.setValue('chat.suggestionsEnabled', false)
 
         logger.info('[DockerService] Ollama installation complete. Triggering Nomad docs discovery...')
         
@@ -575,7 +627,7 @@ export class DockerService {
         'completed',
         `Service ${service.service_name} installation completed successfully.`
       )
-    } catch (error) {
+    } catch (error: any) {
       this._broadcast(
         service.service_name,
         'error',
@@ -591,7 +643,7 @@ export class DockerService {
     try {
       const containers = await this.docker.listContainers({ all: true })
       return containers.some((container) => container.Names.includes(`/${serviceName}`))
-    } catch (error) {
+    } catch (error: any) {
       logger.error(`Error checking if service container exists: ${error.message}`)
       return false
     }
@@ -608,11 +660,10 @@ export class DockerService {
       }
 
       const dockerContainer = this.docker.getContainer(container.Id)
-      await dockerContainer.stop()
-      await dockerContainer.remove()
+      await dockerContainer.remove({ force: true })
 
       return { success: true, message: `Service ${serviceName} container removed successfully` }
-    } catch (error) {
+    } catch (error: any) {
       logger.error(`Error removing service container: ${error.message}`)
       return {
         success: false,
@@ -627,8 +678,8 @@ export class DockerService {
      * We'll download the lightweight mini Wikipedia Top 100 zim file for this purpose.
      **/
     const WIKIPEDIA_ZIM_URL =
-      'https://github.com/Crosstalk-Solutions/project-nomad/raw/refs/heads/master/install/wikipedia_en_100_mini_2025-06.zim'
-    const filename = 'wikipedia_en_100_mini_2025-06.zim'
+      'https://github.com/Crosstalk-Solutions/project-nomad/raw/refs/heads/main/install/wikipedia_en_100_mini_2026-01.zim'
+    const filename = 'wikipedia_en_100_mini_2026-01.zim'
     const filepath = join(process.cwd(), ZIM_STORAGE_PATH, filename)
     logger.info(`[DockerService] Kiwix Serve pre-install: Downloading ZIM file to ${filepath}`)
 
@@ -660,7 +711,12 @@ export class DockerService {
         'preinstall',
         `Downloaded Wikipedia ZIM file to ${filepath}`
       )
-    } catch (error) {
+
+      // Generate the initial kiwix library XML before the container is created
+      const kiwixLibraryService = new KiwixLibraryService()
+      await kiwixLibraryService.rebuildFromDisk()
+      this._broadcast(SERVICE_NAMES.KIWIX, 'preinstall', 'Generated kiwix library XML.')
+    } catch (error: any) {
       this._broadcast(
         SERVICE_NAMES.KIWIX,
         'preinstall-error',
@@ -678,8 +734,12 @@ export class DockerService {
         await service.save()
       }
       this.activeInstallations.delete(serviceName)
+
+      // Ensure any partially created container is removed
+      await this._removeServiceContainer(serviceName)
+
       logger.info(`[DockerService] Cleaned up failed installation for ${serviceName}`)
-    } catch (error) {
+    } catch (error: any) {
       logger.error(
         `[DockerService] Failed to cleanup installation for ${serviceName}: ${error.message}`
       )
@@ -687,44 +747,192 @@ export class DockerService {
   }
 
   /**
-   * Detect GPU type (NVIDIA or AMD) on the system.
-   * Returns 'nvidia', 'amd', or 'none'.
+   * Checks whether the running kiwix container is using the legacy glob-pattern command
+   * (`*.zim --address=all`) rather than the library-file command. Used to detect containers
+   * that need to be migrated to library mode.
    */
-  private async _detectGPUType(): Promise<'nvidia' | 'amd' | 'none'> {
+  async isKiwixOnLegacyConfig(): Promise<boolean> {
     try {
+      const containers = await this.docker.listContainers({ all: true })
+      const info = containers.find((c) => c.Names.includes(`/${SERVICE_NAMES.KIWIX}`))
+      if (!info) return false
+
+      const inspected = await this.docker.getContainer(info.Id).inspect()
+      const cmd: string[] = inspected.Config?.Cmd ?? []
+      return cmd.some((arg) => arg.includes('*.zim'))
+    } catch (err: any) {
+      logger.warn(`[DockerService] Could not inspect kiwix container: ${err.message}`)
+      return false
+    }
+  }
+
+  /**
+   * Migrates the kiwix container from legacy glob mode (`*.zim`) to library mode
+   * (`--library /data/kiwix-library.xml --monitorLibrary`).
+   *
+   * This is a non-destructive recreation: ZIM files and volumes are preserved.
+   * The container is stopped, removed, and recreated with the correct library-mode command.
+   * This function is authoritative: it writes the correct command to the DB itself rather than
+   * trusting the DB to have been pre-updated by a separate migration.
+   */
+  async migrateKiwixToLibraryMode(): Promise<void> {
+    if (this.activeInstallations.has(SERVICE_NAMES.KIWIX)) {
+      logger.warn('[DockerService] Kiwix migration already in progress, skipping duplicate call.')
+      return
+    }
+
+    this.activeInstallations.add(SERVICE_NAMES.KIWIX)
+
+    try {
+      // Step 1: Build/update the XML from current disk state
+      this._broadcast(SERVICE_NAMES.KIWIX, 'migrating', 'Migrating kiwix to library mode...')
+      const kiwixLibraryService = new KiwixLibraryService()
+      await kiwixLibraryService.rebuildFromDisk()
+      this._broadcast(SERVICE_NAMES.KIWIX, 'migrating', 'Built kiwix library XML from existing ZIM files.')
+
+      // Step 2: Stop and remove old container (leave ZIM volumes intact)
+      const containers = await this.docker.listContainers({ all: true })
+      const containerInfo = containers.find((c) => c.Names.includes(`/${SERVICE_NAMES.KIWIX}`))
+      if (containerInfo) {
+        const oldContainer = this.docker.getContainer(containerInfo.Id)
+        if (containerInfo.State === 'running') {
+          await oldContainer.stop({ t: 10 }).catch((e: any) =>
+            logger.warn(`[DockerService] Kiwix stop warning during migration: ${e.message}`)
+          )
+        }
+        await oldContainer.remove({ force: true }).catch((e: any) =>
+          logger.warn(`[DockerService] Kiwix remove warning during migration: ${e.message}`)
+        )
+      }
+
+      // Step 3: Read the service record and authoritatively set the correct command.
+      // Do NOT rely on prior DB state — we write container_command here so the record
+      // stays consistent regardless of whether the DB migration ran.
+      const service = await Service.query().where('service_name', SERVICE_NAMES.KIWIX).first()
+      if (!service) {
+        throw new Error('Kiwix service record not found in DB during migration')
+      }
+
+      service.container_command = KIWIX_LIBRARY_CMD
+      service.installed = false
+      service.installation_status = 'installing'
+      await service.save()
+
+      const containerConfig = this._parseContainerConfig(service.container_config)
+
+      // Step 4: Recreate container directly (skipping _createContainer to avoid re-downloading
+      // the bootstrap ZIM — ZIM files already exist on disk)
+      this._broadcast(SERVICE_NAMES.KIWIX, 'migrating', 'Recreating kiwix container with library mode config...')
+      const newContainer = await this.docker.createContainer({
+        Image: service.container_image,
+        name: service.service_name,
+        HostConfig: containerConfig?.HostConfig ?? {},
+        ...(containerConfig?.ExposedPorts && { ExposedPorts: containerConfig.ExposedPorts }),
+        Cmd: KIWIX_LIBRARY_CMD.split(' '),
+        ...(process.env.NODE_ENV === 'production' && {
+          NetworkingConfig: {
+            EndpointsConfig: {
+              [DockerService.NOMAD_NETWORK]: {},
+            },
+          },
+        }),
+      })
+
+      await newContainer.start()
+
+      service.installed = true
+      service.installation_status = 'idle'
+      await service.save()
+      this.activeInstallations.delete(SERVICE_NAMES.KIWIX)
+
+      this._broadcast(SERVICE_NAMES.KIWIX, 'migrated', 'Kiwix successfully migrated to library mode.')
+      logger.info('[DockerService] Kiwix migration to library mode complete.')
+    } catch (error: any) {
+      logger.error(`[DockerService] Kiwix migration failed: ${error.message}`)
+      await this._cleanupFailedInstallation(SERVICE_NAMES.KIWIX)
+      throw error
+    }
+  }
+
+  /**
+   * Detect GPU type and toolkit availability.
+   * Primary: Check Docker runtimes via docker.info() (works from inside containers).
+   * Fallback: lspci for host-based installs and AMD detection.
+   */
+  private async _detectGPUType(): Promise<{ type: 'nvidia' | 'amd' | 'none'; toolkitMissing?: boolean }> {
+    try {
+      // Primary: Check Docker daemon for nvidia runtime (works from inside containers)
+      try {
+        const dockerInfo = await this.docker.info()
+        const runtimes = dockerInfo.Runtimes || {}
+        if ('nvidia' in runtimes) {
+          logger.info('[DockerService] NVIDIA container runtime detected via Docker API')
+          await this._persistGPUType('nvidia')
+          return { type: 'nvidia' }
+        }
+      } catch (error: any) {
+        logger.warn(`[DockerService] Could not query Docker info for GPU runtimes: ${error.message}`)
+      }
+
+      // Fallback: lspci for host-based installs (not available inside Docker)
       const execAsync = promisify(exec)
 
-      // Check for NVIDIA GPU
+      // Check for NVIDIA GPU via lspci
       try {
         const { stdout: nvidiaCheck } = await execAsync(
           'lspci 2>/dev/null | grep -i nvidia || true'
         )
         if (nvidiaCheck.trim()) {
-          logger.info('[DockerService] NVIDIA GPU detected')
-          return 'nvidia'
+          // GPU hardware found but no nvidia runtime — toolkit not installed
+          logger.warn('[DockerService] NVIDIA GPU detected via lspci but NVIDIA Container Toolkit is not installed')
+          return { type: 'none', toolkitMissing: true }
         }
-      } catch (error) {
-        // Continue to AMD check
+      } catch (error: any) {
+        // lspci not available (likely inside Docker container), continue
       }
 
-      // Check for AMD GPU
+      // Check for AMD GPU via lspci — restrict to display controller classes to avoid
+      // false positives from AMD CPU host bridges, PCI bridges, and chipset devices.
       try {
         const { stdout: amdCheck } = await execAsync(
-          'lspci 2>/dev/null | grep -iE "amd|radeon" || true'
+          'lspci 2>/dev/null | grep -iE "VGA|3D controller|Display" | grep -iE "amd|radeon" || true'
         )
         if (amdCheck.trim()) {
-          logger.info('[DockerService] AMD GPU detected')
-          return 'amd'
+          logger.info('[DockerService] AMD GPU detected via lspci')
+          await this._persistGPUType('amd')
+          return { type: 'amd' }
         }
-      } catch (error) {
-        // No GPU detected
+      } catch (error: any) {
+        // lspci not available, continue
+      }
+
+      // Last resort: check if we previously detected a GPU and it's likely still present.
+      // This handles cases where live detection fails transiently (e.g., Docker daemon
+      // hiccup, runtime temporarily unavailable) but the hardware hasn't changed.
+      try {
+        const savedType = await KVStore.getValue('gpu.type')
+        if (savedType === 'nvidia' || savedType === 'amd') {
+          logger.info(`[DockerService] No GPU detected live, but KV store has '${savedType}' from previous detection. Using saved value.`)
+          return { type: savedType as 'nvidia' | 'amd' }
+        }
+      } catch {
+        // KV store not available, continue
       }
 
       logger.info('[DockerService] No GPU detected')
-      return 'none'
-    } catch (error) {
+      return { type: 'none' }
+    } catch (error: any) {
       logger.warn(`[DockerService] Error detecting GPU type: ${error.message}`)
-      return 'none'
+      return { type: 'none' }
+    }
+  }
+
+  private async _persistGPUType(type: 'nvidia' | 'amd'): Promise<void> {
+    try {
+      await KVStore.setValue('gpu.type', type)
+      logger.info(`[DockerService] Persisted GPU type '${type}' to KV store`)
+    } catch (error: any) {
+      logger.warn(`[DockerService] Failed to persist GPU type: ${error.message}`)
     }
   }
 
@@ -784,8 +992,227 @@ export class DockerService {
   //   }
   // }
 
+  /**
+   * Update a service container to a new image version while preserving volumes and data.
+   * Includes automatic rollback if the new container fails health checks.
+   */
+  async updateContainer(
+    serviceName: string,
+    targetVersion: string
+  ): Promise<{ success: boolean; message: string }> {
+    try {
+      const service = await Service.query().where('service_name', serviceName).first()
+      if (!service) {
+        return { success: false, message: `Service ${serviceName} not found` }
+      }
+      if (!service.installed) {
+        return { success: false, message: `Service ${serviceName} is not installed` }
+      }
+      if (this.activeInstallations.has(serviceName)) {
+        return { success: false, message: `Service ${serviceName} already has an operation in progress` }
+      }
+
+      this.activeInstallations.add(serviceName)
+
+      // Compute new image string
+      const currentImage = service.container_image
+      const imageBase = currentImage.includes(':')
+        ? currentImage.substring(0, currentImage.lastIndexOf(':'))
+        : currentImage
+      const newImage = `${imageBase}:${targetVersion}`
+
+      // Step 1: Pull new image
+      this._broadcast(serviceName, 'update-pulling', `Pulling image ${newImage}...`)
+      const pullStream = await this.docker.pull(newImage)
+      await new Promise((res) => this.docker.modem.followProgress(pullStream, res))
+
+      // Step 2: Find and stop existing container
+      this._broadcast(serviceName, 'update-stopping', `Stopping current container...`)
+      const containers = await this.docker.listContainers({ all: true })
+      const existingContainer = containers.find((c) => c.Names.includes(`/${serviceName}`))
+
+      if (!existingContainer) {
+        this.activeInstallations.delete(serviceName)
+        return { success: false, message: `Container for ${serviceName} not found` }
+      }
+
+      const oldContainer = this.docker.getContainer(existingContainer.Id)
+
+      // Inspect to capture full config before stopping
+      const inspectData = await oldContainer.inspect()
+
+      if (existingContainer.State === 'running') {
+        await oldContainer.stop({ t: 15 })
+      }
+
+      // Step 3: Rename old container as safety net
+      const oldName = `${serviceName}_old`
+      await oldContainer.rename({ name: oldName })
+
+      // Step 4: Create new container with inspected config + new image
+      this._broadcast(serviceName, 'update-creating', `Creating updated container...`)
+
+      const hostConfig = inspectData.HostConfig || {}
+
+      // Re-run GPU detection for Ollama so updates always reflect the current GPU environment.
+      // This handles cases where the NVIDIA Container Toolkit was installed after the initial
+      // Ollama setup, and ensures DeviceRequests are always built fresh rather than relying on
+      // round-tripping the Docker inspect format back into the create API.
+      let updatedDeviceRequests: any[] | undefined = undefined
+      if (serviceName === SERVICE_NAMES.OLLAMA) {
+        const gpuResult = await this._detectGPUType()
+
+        if (gpuResult.type === 'nvidia') {
+          this._broadcast(
+            serviceName,
+            'update-gpu-config',
+            `NVIDIA container runtime detected. Configuring updated container with GPU support...`
+          )
+          updatedDeviceRequests = [
+            {
+              Driver: 'nvidia',
+              Count: -1,
+              Capabilities: [['gpu']],
+            },
+          ]
+        } else if (gpuResult.type === 'amd') {
+          this._broadcast(
+            serviceName,
+            'update-gpu-config',
+            `AMD GPU detected. ROCm GPU acceleration is not yet supported — using CPU-only configuration.`
+          )
+        } else if (gpuResult.toolkitMissing) {
+          this._broadcast(
+            serviceName,
+            'update-gpu-config',
+            `NVIDIA GPU detected but NVIDIA Container Toolkit is not installed. Using CPU-only configuration. Install the toolkit and reinstall AI Assistant for GPU acceleration: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html`
+          )
+        } else {
+          this._broadcast(serviceName, 'update-gpu-config', `No GPU detected. Using CPU-only configuration.`)
+        }
+      }
+
+      const newContainerConfig: any = {
+        Image: newImage,
+        name: serviceName,
+        Env: inspectData.Config?.Env || undefined,
+        Cmd: inspectData.Config?.Cmd || undefined,
+        ExposedPorts: inspectData.Config?.ExposedPorts || undefined,
+        WorkingDir: inspectData.Config?.WorkingDir || undefined,
+        User: inspectData.Config?.User || undefined,
+        HostConfig: {
+          Binds: hostConfig.Binds || undefined,
+          PortBindings: hostConfig.PortBindings || undefined,
+          RestartPolicy: hostConfig.RestartPolicy || undefined,
+          DeviceRequests: serviceName === SERVICE_NAMES.OLLAMA ? updatedDeviceRequests : (hostConfig.DeviceRequests || undefined),
+          Devices: hostConfig.Devices || undefined,
+        },
+        NetworkingConfig: inspectData.NetworkSettings?.Networks
+          ? {
+              EndpointsConfig: Object.fromEntries(
+                Object.keys(inspectData.NetworkSettings.Networks).map((net) => [net, {}])
+              ),
+            }
+          : undefined,
+      }
+
+      // Remove undefined values from HostConfig
+      Object.keys(newContainerConfig.HostConfig).forEach((key) => {
+        if (newContainerConfig.HostConfig[key] === undefined) {
+          delete newContainerConfig.HostConfig[key]
+        }
+      })
+
+      let newContainer: any
+      try {
+        newContainer = await this.docker.createContainer(newContainerConfig)
+      } catch (createError: any) {
+        // Rollback: rename old container back
+        this._broadcast(serviceName, 'update-rollback', `Failed to create new container: ${createError.message}. Rolling back...`)
+        const rollbackContainer = this.docker.getContainer((await this.docker.listContainers({ all: true })).find((c) => c.Names.includes(`/${oldName}`))!.Id)
+        await rollbackContainer.rename({ name: serviceName })
+        await rollbackContainer.start()
+        this.activeInstallations.delete(serviceName)
+        return { success: false, message: `Failed to create updated container: ${createError.message}` }
+      }
+
+      // Step 5: Start new container
+      this._broadcast(serviceName, 'update-starting', `Starting updated container...`)
+      await newContainer.start()
+
+      // Step 6: Health check — verify container stays running for 5 seconds
+      await new Promise((resolve) => setTimeout(resolve, 5000))
+      const newContainerInfo = await newContainer.inspect()
+
+      if (newContainerInfo.State?.Running) {
+        // Healthy — clean up old container
+        try {
+          const oldContainerRef = this.docker.getContainer(
+            (await this.docker.listContainers({ all: true })).find((c) =>
+              c.Names.includes(`/${oldName}`)
+            )?.Id || ''
+          )
+          await oldContainerRef.remove({ force: true })
+        } catch {
+          // Old container may already be gone
+        }
+
+        // Update DB
+        service.container_image = newImage
+        service.available_update_version = null
+        await service.save()
+
+        this.activeInstallations.delete(serviceName)
+        this._broadcast(
+          serviceName,
+          'update-complete',
+          `Successfully updated ${serviceName} to ${targetVersion}`
+        )
+        return { success: true, message: `Service ${serviceName} updated to ${targetVersion}` }
+      } else {
+        // Unhealthy — rollback
+        this._broadcast(
+          serviceName,
+          'update-rollback',
+          `New container failed health check. Rolling back to previous version...`
+        )
+
+        try {
+          await newContainer.stop({ t: 5 }).catch(() => {})
+          await newContainer.remove({ force: true })
+        } catch {
+          // Best effort cleanup
+        }
+
+        // Restore old container
+        const oldContainers = await this.docker.listContainers({ all: true })
+        const oldRef = oldContainers.find((c) => c.Names.includes(`/${oldName}`))
+        if (oldRef) {
+          const rollbackContainer = this.docker.getContainer(oldRef.Id)
+          await rollbackContainer.rename({ name: serviceName })
+          await rollbackContainer.start()
+        }
+
+        this.activeInstallations.delete(serviceName)
+        return {
+          success: false,
+          message: `Update failed: new container did not stay running. Rolled back to previous version.`,
+        }
+      }
+    } catch (error: any) {
+      this.activeInstallations.delete(serviceName)
+      this._broadcast(
+        serviceName,
+        'update-rollback',
+        `Update failed: ${error.message}`
+      )
+      logger.error(`[DockerService] Update failed for ${serviceName}: ${error.message}`)
+      return { success: false, message: `Update failed: ${error.message}` }
+    }
+  }
+
   private _broadcast(service: string, status: string, message: string) {
-    transmit.broadcast('service-installation', {
+    transmit.broadcast(BROADCAST_CHANNELS.SERVICE_INSTALLATION, {
       service_name: service,
       timestamp: new Date().toISOString(),
       status,
@@ -807,7 +1234,7 @@ export class DockerService {
       }
 
       return JSON.parse(toParse)
-    } catch (error) {
+    } catch (error: any) {
       logger.error(`Failed to parse container configuration: ${error.message}`)
       throw new Error(`Invalid container configuration: ${error.message}`)
     }
@@ -824,7 +1251,7 @@ export class DockerService {
 
       // Check if any image has a RepoTag that matches the requested image
       return images.some((image) => image.RepoTags && image.RepoTags.includes(imageName))
-    } catch (error) {
+    } catch (error: any) {
       logger.warn(`Error checking if image exists: ${error.message}`)
       // If run into an error, assume the image does not exist
       return false

admin/app/services/docs_service.ts

@@ -3,9 +3,20 @@ import { streamToString } from '../../util/docs.js'
 import { getFile, getFileStatsIfExists, listDirectoryContentsRecursive } from '../utils/fs.js'
 import path from 'path'
 import InternalServerErrorException from '#exceptions/internal_server_error_exception'
+import logger from '@adonisjs/core/services/logger'
 
 export class DocsService {
   private docsPath = path.join(process.cwd(), 'docs')
+
+  private static readonly DOC_ORDER: Record<string, number> = {
+    'home': 1,
+    'getting-started': 2,
+    'use-cases': 3,
+    'faq': 4,
+    'about': 5,
+    'release-notes': 6,
+  }
+
   async getDocs() {
     const contents = await listDirectoryContentsRecursive(this.docsPath)
     const files: Array<{ title: string; slug: string }> = []
@@ -20,7 +31,11 @@ export class DocsService {
       }
     }
 
-    return files.sort((a, b) => a.title.localeCompare(b.title))
+    return files.sort((a, b) => {
+      const orderA = DocsService.DOC_ORDER[a.slug] ?? 999
+      const orderB = DocsService.DOC_ORDER[b.slug] ?? 999
+      return orderA - orderB
+    })
   }
 
   parse(content: string) {
@@ -32,13 +47,13 @@ export class DocsService {
       // Filter out attribute-undefined errors which may be caused by emojis and special characters
       const criticalErrors = errors.filter((e) => e.error.id !== 'attribute-undefined')
       if (criticalErrors.length > 0) {
-        console.error('Markdoc validation errors:', errors.map((e) => JSON.stringify(e.error)).join(', '))
+        logger.error('Markdoc validation errors:', errors.map((e) => JSON.stringify(e.error)).join(', '))
         throw new Error('Markdoc validation failed')
       }
 
       return Markdoc.transform(ast, config)
     } catch (error) {
-      console.log('Error parsing Markdoc content:', error)
+      logger.error('Error parsing Markdoc content:', error)
       throw new InternalServerErrorException(`Error parsing content: ${(error as Error).message}`)
     }
   }
@@ -51,12 +66,19 @@ export class DocsService {
 
       const filename = _filename.endsWith('.md') ? _filename : `${_filename}.md`
 
-      const fileExists = await getFileStatsIfExists(path.join(this.docsPath, filename))
+      // Prevent path traversal — resolved path must stay within the docs directory
+      const basePath = path.resolve(this.docsPath)
+      const fullPath = path.resolve(path.join(this.docsPath, filename))
+      if (!fullPath.startsWith(basePath + path.sep)) {
+        throw new Error('Invalid document slug')
+      }
+
+      const fileExists = await getFileStatsIfExists(fullPath)
       if (!fileExists) {
         throw new Error(`File not found: ${filename}`)
       }
 
-      const fileStream = await getFile(path.join(this.docsPath, filename), 'stream')
+      const fileStream = await getFile(fullPath, 'stream')
       if (!fileStream) {
         throw new Error(`Failed to read file stream: ${filename}`)
       }
@@ -67,9 +89,17 @@ export class DocsService {
     }
   }
 
+  private static readonly TITLE_OVERRIDES: Record<string, string> = {
+    'faq': 'FAQ',
+  }
+
   private prettify(filename: string) {
+    const slug = filename.replace(/\.md$/, '')
+    if (DocsService.TITLE_OVERRIDES[slug]) {
+      return DocsService.TITLE_OVERRIDES[slug]
+    }
     // Remove hyphens, underscores, and file extension
-    const cleaned = filename.replace(/_/g, ' ').replace(/\.md$/, '').replace(/-/g, ' ')
+    const cleaned = slug.replace(/_/g, ' ').replace(/-/g, ' ')
     // Convert to Title Case
     const titleCased = cleaned.replace(/\b\w/g, (char) => char.toUpperCase())
     return titleCased.charAt(0).toUpperCase() + titleCased.slice(1)
@@ -115,6 +145,58 @@ export class DocsService {
             class: { type: String }
           }
         },
+        table: {
+          render: 'Table',
+        },
+        thead: {
+          render: 'TableHead',
+        },
+        tbody: {
+          render: 'TableBody',
+        },
+        tr: {
+          render: 'TableRow',
+        },
+        th: {
+          render: 'TableHeader',
+        },
+        td: {
+          render: 'TableCell',
+        },
+        paragraph: {
+          render: 'Paragraph',
+        },
+        image: {
+          render: 'Image',
+          attributes: {
+            src: { type: String, required: true },
+            alt: { type: String },
+            title: { type: String },
+          },
+        },
+        link: {
+          render: 'Link',
+          attributes: {
+            href: { type: String, required: true },
+            title: { type: String },
+          },
+        },
+        fence: {
+          render: 'CodeBlock',
+          attributes: {
+            content: { type: String },
+            language: { type: String },
+          },
+        },
+        code: {
+          render: 'InlineCode',
+          attributes: {
+            content: { type: String },
+          },
+        },
+        hr: {
+          render: 'HorizontalRule',
+        },
       },
     }
   }

admin/app/services/download_service.ts

@@ -1,25 +1,198 @@
 import { inject } from '@adonisjs/core'
 import { QueueService } from './queue_service.js'
 import { RunDownloadJob } from '#jobs/run_download_job'
-import { DownloadJobWithProgress } from '../../types/downloads.js'
+import { DownloadModelJob } from '#jobs/download_model_job'
+import { DownloadJobWithProgress, DownloadProgressData } from '../../types/downloads.js'
 import { normalize } from 'path'
+import { deleteFileIfExists } from '../utils/fs.js'
 
 @inject()
 export class DownloadService {
   constructor(private queueService: QueueService) {}
 
-  async listDownloadJobs(filetype?: string): Promise<DownloadJobWithProgress[]> {
-    const queue = this.queueService.getQueue(RunDownloadJob.queue)
-    const jobs = await queue.getJobs(['waiting', 'active', 'delayed'])
+  private parseProgress(progress: any): { percent: number; downloadedBytes?: number; totalBytes?: number; lastProgressTime?: number } {
+    if (typeof progress === 'object' && progress !== null && 'percent' in progress) {
+      const p = progress as DownloadProgressData
+      return {
+        percent: p.percent,
+        downloadedBytes: p.downloadedBytes,
+        totalBytes: p.totalBytes,
+        lastProgressTime: p.lastProgressTime,
+      }
+    }
+    // Backward compat: plain integer from in-flight jobs during upgrade
+    return { percent: parseInt(String(progress), 10) || 0 }
+  }
 
-    return jobs
-      .map((job) => ({
+  async listDownloadJobs(filetype?: string): Promise<DownloadJobWithProgress[]> {
+    // Get regular file download jobs (zim, map, etc.) — query each state separately so we can
+    // tag each job with its actual BullMQ state rather than guessing from progress data.
+    const queue = this.queueService.getQueue(RunDownloadJob.queue)
+    type FileJobState = 'waiting' | 'active' | 'delayed' | 'failed'
+
+    const [waitingJobs, activeJobs, delayedJobs, failedJobs] = await Promise.all([
+      queue.getJobs(['waiting']),
+      queue.getJobs(['active']),
+      queue.getJobs(['delayed']),
+      queue.getJobs(['failed']),
+    ])
+
+    const taggedFileJobs: Array<{ job: (typeof waitingJobs)[0]; state: FileJobState }> = [
+      ...waitingJobs.map((j) => ({ job: j, state: 'waiting' as const })),
+      ...activeJobs.map((j) => ({ job: j, state: 'active' as const })),
+      ...delayedJobs.map((j) => ({ job: j, state: 'delayed' as const })),
+      ...failedJobs.map((j) => ({ job: j, state: 'failed' as const })),
+    ]
+
+    const fileDownloads = taggedFileJobs.map(({ job, state }) => {
+      const parsed = this.parseProgress(job.progress)
+      return {
         jobId: job.id!.toString(),
         url: job.data.url,
-        progress: parseInt(job.progress.toString(), 10),
+        progress: parsed.percent,
         filepath: normalize(job.data.filepath),
         filetype: job.data.filetype,
+        title: job.data.title || undefined,
+        downloadedBytes: parsed.downloadedBytes,
+        totalBytes: parsed.totalBytes || job.data.totalBytes || undefined,
+        lastProgressTime: parsed.lastProgressTime,
+        status: state,
+        failedReason: job.failedReason || undefined,
+      }
+    })
+
+    // Get Ollama model download jobs
+    const modelQueue = this.queueService.getQueue(DownloadModelJob.queue)
+    const modelJobs = await modelQueue.getJobs(['waiting', 'active', 'delayed', 'failed'])
+
+    const modelDownloads = modelJobs.map((job) => ({
+      jobId: job.id!.toString(),
+      url: job.data.modelName || 'Unknown Model', // Use model name as url
+      progress: parseInt(job.progress.toString(), 10),
+      filepath: job.data.modelName || 'Unknown Model', // Use model name as filepath
+      filetype: 'model',
+      status: (job.failedReason ? 'failed' : 'active') as 'active' | 'failed',
+      failedReason: job.failedReason || undefined,
     }))
-      .filter((job) => !filetype || job.filetype === filetype)
+
+    const allDownloads = [...fileDownloads, ...modelDownloads]
+
+    // Filter by filetype if specified
+    const filtered = allDownloads.filter((job) => !filetype || job.filetype === filetype)
+
+    // Sort: active downloads first (by progress desc), then failed at the bottom
+    return filtered.sort((a, b) => {
+      if (a.status === 'failed' && b.status !== 'failed') return 1
+      if (a.status !== 'failed' && b.status === 'failed') return -1
+      return b.progress - a.progress
+    })
+  }
+
+  async removeFailedJob(jobId: string): Promise<void> {
+    for (const queueName of [RunDownloadJob.queue, DownloadModelJob.queue]) {
+      const queue = this.queueService.getQueue(queueName)
+      const job = await queue.getJob(jobId)
+      if (job) {
+        try {
+          await job.remove()
+        } catch {
+          // Job may be locked by the worker after cancel. Remove the stale lock and retry.
+          try {
+            const client = await queue.client
+            await client.del(`bull:${queueName}:${jobId}:lock`)
+            await job.remove()
+          } catch {
+            // Last resort: already removed or truly stuck
+          }
+        }
+        return
+      }
+    }
+  }
+
+  async cancelJob(jobId: string): Promise<{ success: boolean; message: string }> {
+    const queue = this.queueService.getQueue(RunDownloadJob.queue)
+    const job = await queue.getJob(jobId)
+
+    if (!job) {
+      // Job already completed (removeOnComplete: true) or doesn't exist
+      return { success: true, message: 'Job not found (may have already completed)' }
+    }
+
+    const filepath = job.data.filepath
+
+    // Signal the worker process to abort the download via Redis
+    await RunDownloadJob.signalCancel(jobId)
+
+    // Also try in-memory abort (works if worker is in same process)
+    RunDownloadJob.abortControllers.get(jobId)?.abort('user-cancel')
+    RunDownloadJob.abortControllers.delete(jobId)
+
+    // Poll for terminal state (up to 4s at 250ms intervals) — cooperates with BullMQ's lifecycle
+    // instead of force-removing an active job and losing the worker's failure/cleanup path.
+    const POLL_INTERVAL_MS = 250
+    const POLL_TIMEOUT_MS = 4000
+    const deadline = Date.now() + POLL_TIMEOUT_MS
+    let reachedTerminal = false
+
+    while (Date.now() < deadline) {
+      await new Promise((resolve) => setTimeout(resolve, POLL_INTERVAL_MS))
+      try {
+        const state = await job.getState()
+        if (state === 'failed' || state === 'completed' || state === 'unknown') {
+          reachedTerminal = true
+          break
+        }
+      } catch {
+        reachedTerminal = true // getState() throws if job is already gone
+        break
+      }
+    }
+
+    if (!reachedTerminal) {
+      console.warn(`[DownloadService] cancelJob: job ${jobId} did not reach terminal state within timeout, removing anyway`)
+    }
+
+    // Remove the BullMQ job
+    try {
+      await job.remove()
+    } catch {
+      // Lock contention fallback: clear lock and retry once
+      try {
+        const client = await queue.client
+        await client.del(`bull:${RunDownloadJob.queue}:${jobId}:lock`)
+        const updatedJob = await queue.getJob(jobId)
+        if (updatedJob) await updatedJob.remove()
+      } catch {
+        // Best effort - job will be cleaned up on next dismiss attempt
+      }
+    }
+
+    // Delete the partial file from disk
+    if (filepath) {
+      try {
+        await deleteFileIfExists(filepath)
+        // Also try .tmp in case PR #448 staging is merged
+        await deleteFileIfExists(filepath + '.tmp')
+      } catch {
+        // File may not exist yet (waiting job)
+      }
+    }
+
+    // If this was a Wikipedia download, update selection status to failed
+    // (the worker's failed event may not fire if we removed the job first)
+    if (job.data.filetype === 'zim' && job.data.url?.includes('wikipedia_en_')) {
+      try {
+        const { DockerService } = await import('#services/docker_service')
+        const { ZimService } = await import('#services/zim_service')
+        const dockerService = new DockerService()
+        const zimService = new ZimService(dockerService)
+        await zimService.onWikipediaDownloadComplete(job.data.url, false)
+      } catch {
+        // Best effort
+      }
+    }
+
+    return { success: true, message: 'Download cancelled and partial file deleted' }
   }
 }

admin/app/services/kiwix_library_service.ts

@@ -0,0 +1,285 @@
+import { XMLBuilder, XMLParser } from 'fast-xml-parser'
+import { readFile, writeFile, rename, readdir } from 'fs/promises'
+import { join } from 'path'
+import { Archive } from '@openzim/libzim'
+import { KIWIX_LIBRARY_XML_PATH, ZIM_STORAGE_PATH, ensureDirectoryExists } from '../utils/fs.js'
+import logger from '@adonisjs/core/services/logger'
+import { randomUUID } from 'node:crypto'
+
+const CONTAINER_DATA_PATH = '/data'
+const XML_DECLARATION = '<?xml version="1.0" encoding="UTF-8"?>\n'
+
+interface KiwixBook {
+  id: string
+  path: string
+  title: string
+  description?: string
+  language?: string
+  creator?: string
+  publisher?: string
+  name?: string
+  flavour?: string
+  tags?: string
+  faviconMimeType?: string
+  favicon?: string
+  date?: string
+  articleCount?: number
+  mediaCount?: number
+  size?: number
+}
+
+export class KiwixLibraryService {
+  getLibraryFilePath(): string {
+    return join(process.cwd(), KIWIX_LIBRARY_XML_PATH)
+  }
+
+  containerLibraryPath(): string {
+    return '/data/kiwix-library.xml'
+  }
+
+  private _filenameToTitle(filename: string): string {
+    const withoutExt = filename.endsWith('.zim') ? filename.slice(0, -4) : filename
+    const parts = withoutExt.split('_')
+    // Drop last segment if it looks like a date (YYYY-MM)
+    const lastPart = parts[parts.length - 1]
+    const isDate = /^\d{4}-\d{2}$/.test(lastPart)
+    const titleParts = isDate && parts.length > 1 ? parts.slice(0, -1) : parts
+    return titleParts.map((p) => p.charAt(0).toUpperCase() + p.slice(1)).join(' ')
+  }
+
+  /**
+   * Reads all kiwix-manage-compatible metadata from a ZIM file, including the internal UUID,
+   * rich text fields, and the base64-encoded favicon. Kiwix-serve uses the UUID for OPDS
+   * catalog entries and illustration URLs (/catalog/v2/illustration/{uuid}).
+   *
+   * Returns null on any error so callers can fall back gracefully.
+   */
+  private _readZimMetadata(zimFilePath: string): Partial<KiwixBook> | null {
+    try {
+      const archive = new Archive(zimFilePath)
+
+      const getMeta = (key: string): string | undefined => {
+        try {
+          return archive.getMetadata(key) || undefined
+        } catch {
+          return undefined
+        }
+      }
+
+      let favicon: string | undefined
+      let faviconMimeType: string | undefined
+      try {
+        if (archive.illustrationSizes.size > 0) {
+          const size = archive.illustrationSizes.has(48)
+            ? 48
+            : ([...archive.illustrationSizes][0] as number)
+          const item = archive.getIllustrationItem(size)
+          favicon = item.data.data.toString('base64')
+          faviconMimeType = item.mimetype || undefined
+        }
+      } catch {
+        // ZIM has no illustration — that's fine
+      }
+
+      const rawFilesize =
+        typeof archive.filesize === 'bigint' ? Number(archive.filesize) : archive.filesize
+
+      return {
+        id: archive.uuid || undefined,
+        title: getMeta('Title'),
+        description: getMeta('Description'),
+        language: getMeta('Language'),
+        creator: getMeta('Creator'),
+        publisher: getMeta('Publisher'),
+        name: getMeta('Name'),
+        flavour: getMeta('Flavour'),
+        tags: getMeta('Tags'),
+        date: getMeta('Date'),
+        articleCount: archive.articleCount,
+        mediaCount: archive.mediaCount,
+        size: Math.floor(rawFilesize / 1024),
+        favicon,
+        faviconMimeType,
+      }
+    } catch {
+      return null
+    }
+  }
+
+  private _buildXml(books: KiwixBook[]): string {
+    const builder = new XMLBuilder({
+      ignoreAttributes: false,
+      attributeNamePrefix: '@_',
+      format: true,
+      suppressEmptyNode: false,
+    })
+
+    const obj: Record<string, any> = {
+      library: {
+        '@_version': '20110515',
+        ...(books.length > 0 && {
+          book: books.map((b) => ({
+            '@_id': b.id,
+            '@_path': b.path,
+            '@_title': b.title,
+            ...(b.description !== undefined && { '@_description': b.description }),
+            ...(b.language !== undefined && { '@_language': b.language }),
+            ...(b.creator !== undefined && { '@_creator': b.creator }),
+            ...(b.publisher !== undefined && { '@_publisher': b.publisher }),
+            ...(b.name !== undefined && { '@_name': b.name }),
+            ...(b.flavour !== undefined && { '@_flavour': b.flavour }),
+            ...(b.tags !== undefined && { '@_tags': b.tags }),
+            ...(b.faviconMimeType !== undefined && { '@_faviconMimeType': b.faviconMimeType }),
+            ...(b.favicon !== undefined && { '@_favicon': b.favicon }),
+            ...(b.date !== undefined && { '@_date': b.date }),
+            ...(b.articleCount !== undefined && { '@_articleCount': b.articleCount }),
+            ...(b.mediaCount !== undefined && { '@_mediaCount': b.mediaCount }),
+            ...(b.size !== undefined && { '@_size': b.size }),
+          })),
+        }),
+      },
+    }
+
+    return XML_DECLARATION + builder.build(obj)
+  }
+
+  private async _atomicWrite(content: string): Promise<void> {
+    const filePath = this.getLibraryFilePath()
+    const tmpPath = `${filePath}.tmp.${randomUUID()}`
+    await writeFile(tmpPath, content, 'utf-8')
+    await rename(tmpPath, filePath)
+  }
+
+  private _parseExistingBooks(xmlContent: string): KiwixBook[] {
+    const parser = new XMLParser({
+      ignoreAttributes: false,
+      attributeNamePrefix: '@_',
+      isArray: (name) => name === 'book',
+    })
+
+    const parsed = parser.parse(xmlContent)
+    const books: any[] = parsed?.library?.book ?? []
+
+    return books
+      .map((b) => ({
+        id: b['@_id'] ?? '',
+        path: b['@_path'] ?? '',
+        title: b['@_title'] ?? '',
+        description: b['@_description'],
+        language: b['@_language'],
+        creator: b['@_creator'],
+        publisher: b['@_publisher'],
+        name: b['@_name'],
+        flavour: b['@_flavour'],
+        tags: b['@_tags'],
+        faviconMimeType: b['@_faviconMimeType'],
+        favicon: b['@_favicon'],
+        date: b['@_date'],
+        articleCount:
+          b['@_articleCount'] !== undefined ? Number(b['@_articleCount']) : undefined,
+        mediaCount: b['@_mediaCount'] !== undefined ? Number(b['@_mediaCount']) : undefined,
+        size: b['@_size'] !== undefined ? Number(b['@_size']) : undefined,
+      }))
+      .filter((b) => b.id && b.path)
+  }
+
+  async rebuildFromDisk(opts?: { excludeFilenames?: string[] }): Promise<void> {
+    const dirPath = join(process.cwd(), ZIM_STORAGE_PATH)
+    await ensureDirectoryExists(dirPath)
+
+    let entries: string[] = []
+    try {
+      entries = await readdir(dirPath)
+    } catch {
+      entries = []
+    }
+
+    const excludeSet = new Set(opts?.excludeFilenames ?? [])
+    const zimFiles = entries.filter((name) => name.endsWith('.zim') && !excludeSet.has(name))
+
+    const books: KiwixBook[] = zimFiles.map((filename) => {
+      const meta = this._readZimMetadata(join(dirPath, filename))
+      const containerPath = `${CONTAINER_DATA_PATH}/${filename}`
+      return {
+        ...meta,
+        // Override fields that must be derived locally, not from ZIM metadata
+        id: meta?.id ?? filename.slice(0, -4),
+        path: containerPath,
+        title: meta?.title ?? this._filenameToTitle(filename),
+      }
+    })
+
+    const xml = this._buildXml(books)
+    await this._atomicWrite(xml)
+    logger.info(`[KiwixLibraryService] Rebuilt library XML with ${books.length} book(s).`)
+  }
+
+  async addBook(filename: string): Promise<void> {
+    const zimFilename = filename.endsWith('.zim') ? filename : `${filename}.zim`
+    const containerPath = `${CONTAINER_DATA_PATH}/${zimFilename}`
+
+    const filePath = this.getLibraryFilePath()
+    let existingBooks: KiwixBook[] = []
+
+    try {
+      const content = await readFile(filePath, 'utf-8')
+      existingBooks = this._parseExistingBooks(content)
+    } catch (err: any) {
+      if (err.code === 'ENOENT') {
+        // XML doesn't exist yet — rebuild from disk; the completed download is already there
+        await this.rebuildFromDisk()
+        return
+      }
+      throw err
+    }
+
+    if (existingBooks.some((b) => b.path === containerPath)) {
+      logger.info(`[KiwixLibraryService] ${zimFilename} already in library, skipping.`)
+      return
+    }
+
+    const fullPath = join(process.cwd(), ZIM_STORAGE_PATH, zimFilename)
+    const meta = this._readZimMetadata(fullPath)
+
+    existingBooks.push({
+      ...meta,
+      id: meta?.id ?? zimFilename.slice(0, -4),
+      path: containerPath,
+      title: meta?.title ?? this._filenameToTitle(zimFilename),
+    })
+
+    const xml = this._buildXml(existingBooks)
+    await this._atomicWrite(xml)
+    logger.info(`[KiwixLibraryService] Added ${zimFilename} to library XML.`)
+  }
+
+  async removeBook(filename: string): Promise<void> {
+    const zimFilename = filename.endsWith('.zim') ? filename : `${filename}.zim`
+    const containerPath = `${CONTAINER_DATA_PATH}/${zimFilename}`
+
+    const filePath = this.getLibraryFilePath()
+    let existingBooks: KiwixBook[] = []
+
+    try {
+      const content = await readFile(filePath, 'utf-8')
+      existingBooks = this._parseExistingBooks(content)
+    } catch (err: any) {
+      if (err.code === 'ENOENT') {
+        logger.warn(`[KiwixLibraryService] Library XML not found, nothing to remove.`)
+        return
+      }
+      throw err
+    }
+
+    const filtered = existingBooks.filter((b) => b.path !== containerPath)
+
+    if (filtered.length === existingBooks.length) {
+      logger.info(`[KiwixLibraryService] ${zimFilename} not found in library, nothing to remove.`)
+      return
+    }
+
+    const xml = this._buildXml(filtered)
+    await this._atomicWrite(xml)
+    logger.info(`[KiwixLibraryService] Removed ${zimFilename} from library XML.`)
+  }
+}

admin/app/services/map_service.ts

@@ -1,6 +1,5 @@
 import { BaseStylesFile, MapLayer } from '../../types/maps.js'
 import {
-  DownloadCollectionOperation,
   DownloadRemoteSuccessCallback,
   FileEntry,
 } from '../../types/files.js'
@@ -14,16 +13,23 @@ import {
   getFile,
   ensureDirectoryExists,
 } from '../utils/fs.js'
-import { join } from 'path'
+import { join, resolve, sep } from 'path'
 import urlJoin from 'url-join'
-import axios from 'axios'
 import { RunDownloadJob } from '#jobs/run_download_job'
 import logger from '@adonisjs/core/services/logger'
-import { CuratedCollectionsFile, CuratedCollectionWithStatus } from '../../types/downloads.js'
-import CuratedCollection from '#models/curated_collection'
-import vine from '@vinejs/vine'
-import { curatedCollectionsFileSchema } from '#validators/curated_collections'
-import CuratedCollectionResource from '#models/curated_collection_resource'
+import InstalledResource from '#models/installed_resource'
+import { CollectionManifestService } from './collection_manifest_service.js'
+import type { CollectionWithStatus, MapsSpec } from '../../types/collections.js'
+
+const PROTOMAPS_BUILDS_METADATA_URL = 'https://build-metadata.protomaps.dev/builds.json'
+const PROTOMAPS_BUILD_BASE_URL = 'https://build.protomaps.com'
+
+export interface ProtomapsBuildInfo {
+  url: string
+  date: string
+  size: number
+  key: string
+}
 
 const BASE_ASSETS_MIME_TYPES = [
   'application/gzip',
@@ -31,15 +37,11 @@ const BASE_ASSETS_MIME_TYPES = [
   'application/octet-stream',
 ]
 
-const COLLECTIONS_URL =
-  'https://github.com/Crosstalk-Solutions/project-nomad/raw/refs/heads/master/collections/maps.json'
-
 const PMTILES_ATTRIBUTION =
   '<a href="https://github.com/protomaps/basemaps">Protomaps</a> © <a href="https://openstreetmap.org">OpenStreetMap</a>'
 const PMTILES_MIME_TYPES = ['application/vnd.pmtiles', 'application/octet-stream']
 
 interface IMapService {
-  downloadCollection: DownloadCollectionOperation
   downloadRemoteSuccessCallback: DownloadRemoteSuccessCallback
 }
 
@@ -99,34 +101,33 @@ export class MapService implements IMapService {
     return true
   }
 
-  async downloadCollection(slug: string) {
-    const collection = await CuratedCollection.query()
-      .where('slug', slug)
-      .andWhere('type', 'map')
-      .first()
-    if (!collection) {
-      return null
-    }
+  async downloadCollection(slug: string): Promise<string[] | null> {
+    const manifestService = new CollectionManifestService()
+    const spec = await manifestService.getSpecWithFallback<MapsSpec>('maps')
+    if (!spec) return null
 
-    const resources = await collection.related('resources').query().where('downloaded', false)
-    if (resources.length === 0) {
-      return null
-    }
+    const collection = spec.collections.find((c) => c.slug === slug)
+    if (!collection) return null
+
+    // Filter out already installed
+    const installed = await InstalledResource.query().where('resource_type', 'map')
+    const installedIds = new Set(installed.map((r) => r.resource_id))
+    const toDownload = collection.resources.filter((r) => !installedIds.has(r.id))
+
+    if (toDownload.length === 0) return null
 
-    const downloadUrls = resources.map((res) => res.url)
     const downloadFilenames: string[] = []
 
-    for (const url of downloadUrls) {
-      const existing = await RunDownloadJob.getByUrl(url)
+    for (const resource of toDownload) {
+      const existing = await RunDownloadJob.getActiveByUrl(resource.url)
       if (existing) {
-        logger.warn(`[MapService] Download already in progress for URL ${url}, skipping.`)
+        logger.warn(`[MapService] Download already in progress for URL ${resource.url}, skipping.`)
         continue
       }
 
-      // Extract the filename from the URL
-      const filename = url.split('/').pop()
+      const filename = resource.url.split('/').pop()
       if (!filename) {
-        logger.warn(`[MapService] Could not determine filename from URL ${url}, skipping.`)
+        logger.warn(`[MapService] Could not determine filename from URL ${resource.url}, skipping.`)
         continue
       }
 
@@ -134,12 +135,18 @@ export class MapService implements IMapService {
       const filepath = join(process.cwd(), this.mapStoragePath, 'pmtiles', filename)
 
       await RunDownloadJob.dispatch({
-        url,
+        url: resource.url,
         filepath,
         timeout: 30000,
         allowedMimeTypes: PMTILES_MIME_TYPES,
         forceNew: true,
         filetype: 'map',
+        title: (resource as any).title || undefined,
+        resourceMetadata: {
+          resource_id: resource.id,
+          version: resource.version,
+          collection_ref: slug,
+        },
       })
     }
 
@@ -147,11 +154,33 @@ export class MapService implements IMapService {
   }
 
   async downloadRemoteSuccessCallback(urls: string[], _: boolean) {
-    const resources = await CuratedCollectionResource.query().whereIn('url', urls)
-    for (const resource of resources) {
-      resource.downloaded = true
-      await resource.save()
-      logger.info(`[MapService] Marked resource as downloaded: ${resource.url}`)
+    // Create InstalledResource entries for downloaded map files
+    for (const url of urls) {
+      const filename = url.split('/').pop()
+      if (!filename) continue
+
+      const parsed = CollectionManifestService.parseMapFilename(filename)
+      if (!parsed) continue
+
+      const filepath = join(process.cwd(), this.mapStoragePath, 'pmtiles', filename)
+      const stats = await getFileStatsIfExists(filepath)
+
+      try {
+        const { DateTime } = await import('luxon')
+        await InstalledResource.updateOrCreate(
+          { resource_id: parsed.resource_id, resource_type: 'map' },
+          {
+            version: parsed.version,
+            url: url,
+            file_path: filepath,
+            file_size_bytes: stats ? Number(stats.size) : null,
+            installed_at: DateTime.now(),
+          }
+        )
+        logger.info(`[MapService] Created InstalledResource entry for: ${parsed.resource_id}`)
+      } catch (error) {
+        logger.error(`[MapService] Failed to create InstalledResource for ${filename}:`, error)
+      }
     }
   }
 
@@ -161,7 +190,7 @@ export class MapService implements IMapService {
       throw new Error(`Invalid PMTiles file URL: ${url}. URL must end with .pmtiles`)
     }
 
-    const existing = await RunDownloadJob.getByUrl(url)
+    const existing = await RunDownloadJob.getActiveByUrl(url)
     if (existing) {
       throw new Error(`Download already in progress for URL ${url}`)
     }
@@ -182,6 +211,12 @@ export class MapService implements IMapService {
       )
     }
 
+    // Parse resource metadata
+    const parsedFilename = CollectionManifestService.parseMapFilename(filename)
+    const resourceMetadata = parsedFilename
+      ? { resource_id: parsedFilename.resource_id, version: parsedFilename.version, collection_ref: null }
+      : undefined
+
     // Dispatch background job
     const result = await RunDownloadJob.dispatch({
       url,
@@ -190,6 +225,7 @@ export class MapService implements IMapService {
       allowedMimeTypes: PMTILES_MIME_TYPES,
       forceNew: true,
       filetype: 'map',
+      resourceMetadata,
     })
 
     if (!result.job) {
@@ -219,6 +255,7 @@ export class MapService implements IMapService {
       }
 
       // Perform a HEAD request to get the content length
+      const { default: axios } = await import('axios')
       const response = await axios.head(url)
 
       if (response.status !== 200) {
@@ -229,12 +266,12 @@ export class MapService implements IMapService {
       const size = contentLength ? parseInt(contentLength, 10) : 0
 
       return { filename, size }
-    } catch (error) {
+    } catch (error: any) {
       return { message: `Preflight check failed: ${error.message}` }
     }
   }
 
-  async generateStylesJSON(host: string | null = null): Promise<BaseStylesFile> {
+  async generateStylesJSON(host: string | null = null, protocol: string = 'http'): Promise<BaseStylesFile> {
     if (!(await this.checkBaseAssetsExist())) {
       throw new Error('Base map assets are missing from storage/maps')
     }
@@ -255,8 +292,8 @@ export class MapService implements IMapService {
     * e.g. user is accessing from "example.com", but we would by default generate "localhost:8080/..." so maps would
     * fail to load.
     */
-    const sources = this.generateSourcesArray(host, regions)
-    const baseUrl = this.getPublicFileBaseUrl(host, this.basemapsAssetsDir)
+    const sources = this.generateSourcesArray(host, regions, protocol)
+    const baseUrl = this.getPublicFileBaseUrl(host, this.basemapsAssetsDir, protocol)
 
     const styles = await this.generateStylesFile(
       rawStyles,
@@ -268,45 +305,14 @@ export class MapService implements IMapService {
     return styles
   }
 
-  async listCuratedCollections(): Promise<CuratedCollectionWithStatus[]> {
-    const collections = await CuratedCollection.query().where('type', 'map').preload('resources')
-    return collections.map((collection) => ({
-      ...(collection.serialize() as CuratedCollection),
-      all_downloaded: collection.resources.every((res) => res.downloaded),
-    }))
+  async listCuratedCollections(): Promise<CollectionWithStatus[]> {
+    const manifestService = new CollectionManifestService()
+    return manifestService.getMapCollectionsWithStatus()
   }
 
   async fetchLatestCollections(): Promise<boolean> {
-    try {
-      const response = await axios.get<CuratedCollectionsFile>(COLLECTIONS_URL)
-
-      const validated = await vine.validate({
-        schema: curatedCollectionsFileSchema,
-        data: response.data,
-      })
-
-      for (const collection of validated.collections) {
-        const collectionResult = await CuratedCollection.updateOrCreate(
-          { slug: collection.slug },
-          {
-            ...collection,
-            type: 'map',
-          }
-        )
-        logger.info(`[MapService] Upserted curated collection: ${collection.slug}`)
-
-        await collectionResult.related('resources').createMany(collection.resources)
-        logger.info(
-          `[MapService] Upserted ${collection.resources.length} resources for collection: ${collection.slug}`
-        )
-      }
-
-      return true
-    } catch (error) {
-      console.error(error)
-      logger.error(`[MapService] Failed to download latest Kiwix collections:`, error)
-      return false
-    }
+    const manifestService = new CollectionManifestService()
+    return manifestService.fetchAndCacheSpec('maps')
   }
 
   async ensureBaseAssets(): Promise<boolean> {
@@ -347,13 +353,15 @@ export class MapService implements IMapService {
     return await listDirectoryContentsRecursive(this.baseDirPath)
   }
 
-  private generateSourcesArray(host: string | null, regions: FileEntry[]): BaseStylesFile['sources'][] {
+  private generateSourcesArray(host: string | null, regions: FileEntry[], protocol: string = 'http'): BaseStylesFile['sources'][] {
     const sources: BaseStylesFile['sources'][] = []
-    const baseUrl = this.getPublicFileBaseUrl(host, 'pmtiles')
+    const baseUrl = this.getPublicFileBaseUrl(host, 'pmtiles', protocol)
 
     for (const region of regions) {
       if (region.type === 'file' && region.name.endsWith('.pmtiles')) {
-        const regionName = region.name.replace('.pmtiles', '')
+        // Strip .pmtiles and date suffix (e.g. "alaska_2025-12" -> "alaska") for stable source names
+        const parsed = CollectionManifestService.parseMapFilename(region.name)
+        const regionName = parsed ? parsed.resource_id : region.name.replace('.pmtiles', '')
         const source: BaseStylesFile['sources'] = {}
         const sourceUrl = urlJoin(baseUrl, region.name)
 
@@ -401,13 +409,89 @@ export class MapService implements IMapService {
     return template
   }
 
-  async delete(file: string): Promise<void> {
-    let fileName = file
-    if (!fileName.endsWith('.zim')) {
-      fileName += '.zim'
+  async getGlobalMapInfo(): Promise<ProtomapsBuildInfo> {
+    const { default: axios } = await import('axios')
+    const response = await axios.get(PROTOMAPS_BUILDS_METADATA_URL, { timeout: 15000 })
+    const builds = response.data as Array<{ key: string; size: number }>
+
+    if (!builds || builds.length === 0) {
+      throw new Error('No protomaps builds found')
     }
 
-    const fullPath = join(this.baseDirPath, fileName)
+    // Latest build first
+    const sorted = builds.sort((a, b) => b.key.localeCompare(a.key))
+    const latest = sorted[0]
+
+    const dateStr = latest.key.replace('.pmtiles', '')
+    const date = `${dateStr.slice(0, 4)}-${dateStr.slice(4, 6)}-${dateStr.slice(6, 8)}`
+
+    return {
+      url: `${PROTOMAPS_BUILD_BASE_URL}/${latest.key}`,
+      date,
+      size: latest.size,
+      key: latest.key,
+    }
+  }
+
+  async downloadGlobalMap(): Promise<{ filename: string; jobId?: string }> {
+    const info = await this.getGlobalMapInfo()
+
+    const existing = await RunDownloadJob.getByUrl(info.url)
+    if (existing) {
+      throw new Error(`Download already in progress for URL ${info.url}`)
+    }
+
+    const basePath = resolve(join(this.baseDirPath, 'pmtiles'))
+    const filepath = resolve(join(basePath, info.key))
+
+    // Prevent path traversal — resolved path must stay within the storage directory
+    if (!filepath.startsWith(basePath + sep)) {
+      throw new Error('Invalid filename')
+    }
+
+    // First, ensure base assets are present - the global map depends on them
+    const baseAssetsExist = await this.ensureBaseAssets()
+    if (!baseAssetsExist) {
+      throw new Error(
+        'Base map assets are missing and could not be downloaded. Please check your connection and try again.'
+      )
+    }
+
+    // forceNew: false so retries resume partial downloads
+    const result = await RunDownloadJob.dispatch({
+      url: info.url,
+      filepath,
+      timeout: 30000,
+      allowedMimeTypes: PMTILES_MIME_TYPES,
+      forceNew: false,
+      filetype: 'map',
+    })
+
+    if (!result.job) {
+      throw new Error('Failed to dispatch download job')
+    }
+
+    logger.info(`[MapService] Dispatched global map download job ${result.job.id}`)
+
+    return {
+      filename: info.key,
+      jobId: result.job?.id,
+    }
+  }
+
+  async delete(file: string): Promise<void> {
+    let fileName = file
+    if (!fileName.endsWith('.pmtiles')) {
+      fileName += '.pmtiles'
+    }
+
+    const basePath = resolve(join(this.baseDirPath, 'pmtiles'))
+    const fullPath = resolve(join(basePath, fileName))
+
+    // Prevent path traversal — resolved path must stay within the storage directory
+    if (!fullPath.startsWith(basePath + sep)) {
+      throw new Error('Invalid filename')
+    }
 
     const exists = await getFileStatsIfExists(fullPath)
     if (!exists) {
@@ -415,12 +499,32 @@ export class MapService implements IMapService {
     }
 
     await deleteFileIfExists(fullPath)
+
+    // Clean up InstalledResource entry
+    const parsed = CollectionManifestService.parseMapFilename(fileName)
+    if (parsed) {
+      await InstalledResource.query()
+        .where('resource_id', parsed.resource_id)
+        .where('resource_type', 'map')
+        .delete()
+      logger.info(`[MapService] Deleted InstalledResource entry for: ${parsed.resource_id}`)
+    }
   }
 
-  /*
-   * Gets the appropriate public URL for a map asset depending on environment
+  /**
+   * Gets the appropriate public URL for a map asset depending on environment. The host and protocol that the user
+   * is accessing the maps from must match the host and protocol used in the generated URLs, otherwise maps will fail to load.
+   * If you make changes to this function, you need to ensure it handles all the following cases correctly:
+   * - No host provided (should default to localhost or env URL)
+   * - Host provided as full URL (e.g. "http://example.com:8080")
+   * - Host provided as host:port (e.g. "example.com:8080")
+   * - Host provided as bare hostname (e.g. "example.com")
+   * @param specifiedHost - the host as provided by the user/request, can be null or in various formats (full URL, host:port, bare hostname)
+   * @param childPath - the path to append to the base URL (e.g. "basemaps-assets", "pmtiles")
+   * @param protocol - the protocol to use in the generated URL (e.g. "http", "https"), defaults to "http"
+   * @returns the public URL for the map asset
    */
-  private getPublicFileBaseUrl(specifiedHost: string | null, childPath: string): string {
+  private getPublicFileBaseUrl(specifiedHost: string | null, childPath: string, protocol: string = 'http'): string {
     function getHost() {
       try {
         const localUrlRaw = env.get('URL')
@@ -433,8 +537,25 @@ export class MapService implements IMapService {
       }
     }
 
-    const host = specifiedHost || getHost()
-    const withProtocol = host.startsWith('http') ? host : `http://${host}`
+    function specifiedHostOrDefault() {
+      if (specifiedHost === null) {
+        return getHost()
+      }
+      // Try as a full URL first (e.g. "http://example.com:8080")
+      try {
+        const specifiedUrl = new URL(specifiedHost)
+        if (specifiedUrl.host) return specifiedUrl.host
+      } catch {}
+      // Try as a bare host or host:port (e.g. "nomad-box:8080", "192.168.1.1:8080", "example.com")
+      try {
+        const specifiedUrl = new URL(`http://${specifiedHost}`)
+        if (specifiedUrl.host) return specifiedUrl.host
+      } catch {}
+      return getHost()
+    }
+
+    const host = specifiedHostOrDefault();
+    const withProtocol = `${protocol}://${host}`
     const baseUrlPath =
       process.env.NODE_ENV === 'production' ? childPath : urlJoin(this.mapStoragePath, childPath)
 

admin/app/services/ollama_service.ts

@@ -1,5 +1,7 @@
 import { inject } from '@adonisjs/core'
-import { ChatRequest, Ollama } from 'ollama'
+import OpenAI from 'openai'
+import type { ChatCompletionChunk, ChatCompletionMessageParam } from 'openai/resources/chat/completions.js'
+import type { Stream } from 'openai/streaming.js'
 import { NomadOllamaModel } from '../../types/ollama.js'
 import { FALLBACK_RECOMMENDED_OLLAMA_MODELS } from '../../constants/ollama.js'
 import fs from 'node:fs/promises'
@@ -9,51 +11,97 @@ import axios from 'axios'
 import { DownloadModelJob } from '#jobs/download_model_job'
 import { SERVICE_NAMES } from '../../constants/service_names.js'
 import transmit from '@adonisjs/transmit/services/main'
+import Fuse, { IFuseOptions } from 'fuse.js'
+import { BROADCAST_CHANNELS } from '../../constants/broadcast.js'
+import env from '#start/env'
+import { NOMAD_API_DEFAULT_BASE_URL } from '../../constants/misc.js'
+import KVStore from '#models/kv_store'
 
-const NOMAD_MODELS_API_BASE_URL = 'https://api.projectnomad.us/api/v1/ollama/models'
+const NOMAD_MODELS_API_PATH = '/api/v1/ollama/models'
 const MODELS_CACHE_FILE = path.join(process.cwd(), 'storage', 'ollama-models-cache.json')
 const CACHE_MAX_AGE_MS = 24 * 60 * 60 * 1000 // 24 hours
 
+export type NomadInstalledModel = {
+  name: string
+  size: number
+  digest?: string
+  details?: Record<string, any>
+}
+
+export type NomadChatResponse = {
+  message: { content: string; thinking?: string }
+  done: boolean
+  model: string
+}
+
+export type NomadChatStreamChunk = {
+  message: { content: string; thinking?: string }
+  done: boolean
+}
+
+type ChatInput = {
+  model: string
+  messages: Array<{ role: 'system' | 'user' | 'assistant'; content: string }>
+  think?: boolean | 'medium'
+  stream?: boolean
+  numCtx?: number
+}
+
 @inject()
 export class OllamaService {
-  private ollama: Ollama | null = null
-  private ollamaInitPromise: Promise<void> | null = null
+  private openai: OpenAI | null = null
+  private baseUrl: string | null = null
+  private initPromise: Promise<void> | null = null
+  private isOllamaNative: boolean | null = null
 
-  constructor() { }
+  constructor() {}
 
-  private async _initializeOllamaClient() {
-    if (!this.ollamaInitPromise) {
-      this.ollamaInitPromise = (async () => {
+  private async _initialize() {
+    if (!this.initPromise) {
+      this.initPromise = (async () => {
+        // Check KVStore for a custom base URL (remote Ollama, LM Studio, llama.cpp, etc.)
+        const customUrl = (await KVStore.getValue('ai.remoteOllamaUrl')) as string | null
+        if (customUrl && customUrl.trim()) {
+          this.baseUrl = customUrl.trim().replace(/\/$/, '')
+        } else {
+          // Fall back to the local Ollama container managed by Docker
           const dockerService = new (await import('./docker_service.js')).DockerService()
-        const qdrantUrl = await dockerService.getServiceURL(SERVICE_NAMES.OLLAMA)
-        if (!qdrantUrl) {
+          const ollamaUrl = await dockerService.getServiceURL(SERVICE_NAMES.OLLAMA)
+          if (!ollamaUrl) {
             throw new Error('Ollama service is not installed or running.')
           }
-        this.ollama = new Ollama({ host: qdrantUrl })
+          this.baseUrl = ollamaUrl.trim().replace(/\/$/, '')
+        }
+
+        this.openai = new OpenAI({
+          apiKey: 'nomad', // Required by SDK; not validated by Ollama/LM Studio/llama.cpp
+          baseURL: `${this.baseUrl}/v1`,
+        })
       })()
     }
-    return this.ollamaInitPromise
+    return this.initPromise
   }
 
   private async _ensureDependencies() {
-    if (!this.ollama) {
-      await this._initializeOllamaClient()
+    if (!this.openai) {
+      await this._initialize()
     }
   }
 
   /**
-   * Downloads a model from the Ollama service with progress tracking. Where possible,
-   * one should dispatch a background job instead of calling this method directly to avoid long blocking.
-   * @param model Model name to download
-   * @returns Success status and message
+   * Downloads a model from Ollama with progress tracking. Only works with Ollama backends.
+   * Use dispatchModelDownload() for background job processing where possible.
    */
-  async downloadModel(model: string, progressCallback?: (percent: number) => void): Promise<{ success: boolean; message: string }> {
-    try {
+  async downloadModel(
+    model: string,
+    progressCallback?: (percent: number) => void
+  ): Promise<{ success: boolean; message: string; retryable?: boolean }> {
     await this._ensureDependencies()
-      if (!this.ollama) {
-        throw new Error('Ollama client is not initialized.')
+    if (!this.baseUrl) {
+      return { success: false, message: 'AI service is not initialized.' }
     }
 
+    try {
       // See if model is already installed
       const installedModels = await this.getModels()
       if (installedModels && installedModels.some((m) => m.name === model)) {
@@ -61,32 +109,67 @@ export class OllamaService {
         return { success: true, message: 'Model is already installed.' }
       }
 
-      // Returns AbortableAsyncIterator<ProgressResponse>
-      const downloadStream = await this.ollama.pull({
-        model,
-        stream: true,
+      // Model pulling is an Ollama-only operation. Non-Ollama backends (LM Studio, llama.cpp, etc.)
+      // return HTTP 200 for unknown endpoints, so the pull would appear to succeed but do nothing.
+      if (this.isOllamaNative === false) {
+        logger.warn(
+          `[OllamaService] Non-Ollama backend detected — skipping model pull for "${model}". Load the model manually in your AI host.`
+        )
+        return {
+          success: false,
+          message: `Model "${model}" is not available in your AI host. Please load it manually (model pulling is only supported for Ollama backends).`,
+        }
+      }
+
+      // Stream pull via Ollama native API
+      const pullResponse = await axios.post(
+        `${this.baseUrl}/api/pull`,
+        { model, stream: true },
+        { responseType: 'stream', timeout: 0 }
+      )
+
+      await new Promise<void>((resolve, reject) => {
+        let buffer = ''
+        pullResponse.data.on('data', (chunk: Buffer) => {
+          buffer += chunk.toString()
+          const lines = buffer.split('\n')
+          buffer = lines.pop() || ''
+          for (const line of lines) {
+            if (!line.trim()) continue
+            try {
+              const parsed = JSON.parse(line)
+              if (parsed.completed && parsed.total) {
+                const percent = parseFloat(((parsed.completed / parsed.total) * 100).toFixed(2))
+                this.broadcastDownloadProgress(model, percent)
+                if (progressCallback) progressCallback(percent)
+              }
+            } catch {
+              // ignore parse errors on partial lines
+            }
+          }
+        })
+        pullResponse.data.on('end', resolve)
+        pullResponse.data.on('error', reject)
       })
-
-      for await (const chunk of downloadStream) {
-        if (chunk.completed && chunk.total) {
-          const percent = ((chunk.completed / chunk.total) * 100).toFixed(2)
-          const percentNum = parseFloat(percent)
-
-          this.broadcastDownloadProgress(model, percentNum)
-          if (progressCallback) {
-            progressCallback(percentNum)
-          }
-        }
-      }
 
       logger.info(`[OllamaService] Model "${model}" downloaded successfully.`)
       return { success: true, message: 'Model downloaded successfully.' }
     } catch (error) {
+      const errorMessage = error instanceof Error ? error.message : String(error)
       logger.error(
-        `[OllamaService] Failed to download model "${model}": ${error instanceof Error ? error.message : error
-        }`
+        `[OllamaService] Failed to download model "${model}": ${errorMessage}`
       )
-      return { success: false, message: 'Failed to download model.' }
+
+      // Check for version mismatch (Ollama 412 response)
+      const isVersionMismatch = errorMessage.includes('newer version of Ollama')
+      const userMessage = isVersionMismatch
+        ? 'This model requires a newer version of Ollama. Please update AI Assistant from the Apps page.'
+        : `Failed to download model: ${errorMessage}`
+
+      // Broadcast failure to connected clients so UI can show the error
+      this.broadcastDownloadError(model, userMessage)
+
+      return { success: false, message: userMessage, retryable: !isVersionMismatch }
     }
   }
 
@@ -114,78 +197,290 @@ export class OllamaService {
     }
   }
 
-  public async getClient() {
+  public async chat(chatRequest: ChatInput): Promise<NomadChatResponse> {
     await this._ensureDependencies()
-    return this.ollama!
+    if (!this.openai) {
+      throw new Error('AI client is not initialized.')
     }
 
-  public async chat(chatRequest: ChatRequest & { stream?: boolean }) {
-    await this._ensureDependencies()
-    if (!this.ollama) {
-      throw new Error('Ollama client is not initialized.')
-    }
-    return await this.ollama.chat({
-      ...chatRequest,
+    const params: any = {
+      model: chatRequest.model,
+      messages: chatRequest.messages as ChatCompletionMessageParam[],
       stream: false,
-    })
+    }
+    if (chatRequest.think) {
+      params.think = chatRequest.think
+    }
+    if (chatRequest.numCtx) {
+      params.num_ctx = chatRequest.numCtx
     }
 
-  public async deleteModel(modelName: string) {
+    const response = await this.openai.chat.completions.create(params)
+    const choice = response.choices[0]
+
+    return {
+      message: {
+        content: choice.message.content ?? '',
+        thinking: (choice.message as any).thinking ?? undefined,
+      },
+      done: true,
+      model: response.model,
+    }
+  }
+
+  public async chatStream(chatRequest: ChatInput): Promise<AsyncIterable<NomadChatStreamChunk>> {
     await this._ensureDependencies()
-    if (!this.ollama) {
-      throw new Error('Ollama client is not initialized.')
+    if (!this.openai) {
+      throw new Error('AI client is not initialized.')
     }
 
-    return await this.ollama.delete({
-      model: modelName,
-    })
+    const params: any = {
+      model: chatRequest.model,
+      messages: chatRequest.messages as ChatCompletionMessageParam[],
+      stream: true,
+    }
+    if (chatRequest.think) {
+      params.think = chatRequest.think
+    }
+    if (chatRequest.numCtx) {
+      params.num_ctx = chatRequest.numCtx
     }
 
-  public async getModels(includeEmbeddings = false) {
+    const stream = (await this.openai.chat.completions.create(params)) as unknown as Stream<ChatCompletionChunk>
+
+    // Returns how many trailing chars of `text` could be the start of `tag`
+    function partialTagSuffix(tag: string, text: string): number {
+      for (let len = Math.min(tag.length - 1, text.length); len >= 1; len--) {
+        if (text.endsWith(tag.slice(0, len))) return len
+      }
+      return 0
+    }
+
+    async function* normalize(): AsyncGenerator<NomadChatStreamChunk> {
+      // Stateful parser for <think>...</think> tags that may be split across chunks.
+      // Ollama provides thinking natively via delta.thinking; OpenAI-compatible backends
+      // (LM Studio, llama.cpp, etc.) embed them inline in delta.content.
+      let tagBuffer = ''
+      let inThink = false
+
+      for await (const chunk of stream) {
+        const delta = chunk.choices[0]?.delta
+        const nativeThinking: string = (delta as any)?.thinking ?? ''
+        const rawContent: string = delta?.content ?? ''
+
+        // Parse <think> tags out of the content stream
+        tagBuffer += rawContent
+        let parsedContent = ''
+        let parsedThinking = ''
+
+        while (tagBuffer.length > 0) {
+          if (inThink) {
+            const closeIdx = tagBuffer.indexOf('</think>')
+            if (closeIdx !== -1) {
+              parsedThinking += tagBuffer.slice(0, closeIdx)
+              tagBuffer = tagBuffer.slice(closeIdx + 8)
+              inThink = false
+            } else {
+              const hold = partialTagSuffix('</think>', tagBuffer)
+              parsedThinking += tagBuffer.slice(0, tagBuffer.length - hold)
+              tagBuffer = tagBuffer.slice(tagBuffer.length - hold)
+              break
+            }
+          } else {
+            const openIdx = tagBuffer.indexOf('<think>')
+            if (openIdx !== -1) {
+              parsedContent += tagBuffer.slice(0, openIdx)
+              tagBuffer = tagBuffer.slice(openIdx + 7)
+              inThink = true
+            } else {
+              const hold = partialTagSuffix('<think>', tagBuffer)
+              parsedContent += tagBuffer.slice(0, tagBuffer.length - hold)
+              tagBuffer = tagBuffer.slice(tagBuffer.length - hold)
+              break
+            }
+          }
+        }
+
+        yield {
+          message: {
+            content: parsedContent,
+            thinking: nativeThinking + parsedThinking,
+          },
+          done: chunk.choices[0]?.finish_reason !== null && chunk.choices[0]?.finish_reason !== undefined,
+        }
+      }
+    }
+
+    return normalize()
+  }
+
+  public async checkModelHasThinking(modelName: string): Promise<boolean> {
     await this._ensureDependencies()
-    if (!this.ollama) {
-      throw new Error('Ollama client is not initialized.')
+    if (!this.baseUrl) return false
+
+    try {
+      const response = await axios.post(
+        `${this.baseUrl}/api/show`,
+        { model: modelName },
+        { timeout: 5000 }
+      )
+      return Array.isArray(response.data?.capabilities) && response.data.capabilities.includes('thinking')
+    } catch {
+      // Non-Ollama backends don't expose /api/show — assume no thinking support
+      return false
+    }
+  }
+
+  public async deleteModel(modelName: string): Promise<{ success: boolean; message: string }> {
+    await this._ensureDependencies()
+    if (!this.baseUrl) {
+      return { success: false, message: 'AI service is not initialized.' }
+    }
+
+    try {
+      await axios.delete(`${this.baseUrl}/api/delete`, {
+        data: { model: modelName },
+        timeout: 10000,
+      })
+      return { success: true, message: `Model "${modelName}" deleted.` }
+    } catch (error) {
+      logger.error(
+        `[OllamaService] Failed to delete model "${modelName}": ${error instanceof Error ? error.message : error}`
+      )
+      return { success: false, message: 'Failed to delete model. This may not be an Ollama backend.' }
+    }
+  }
+
+  /**
+   * Generate embeddings for the given input strings.
+   * Tries the Ollama native /api/embed endpoint first, falls back to /v1/embeddings.
+   */
+  public async embed(model: string, input: string[]): Promise<{ embeddings: number[][] }> {
+    await this._ensureDependencies()
+    if (!this.baseUrl || !this.openai) {
+      throw new Error('AI service is not initialized.')
+    }
+
+    try {
+      // Prefer Ollama native endpoint (supports batch input natively)
+      const response = await axios.post(
+        `${this.baseUrl}/api/embed`,
+        { model, input },
+        { timeout: 60000 }
+      )
+      // Some backends (e.g. LM Studio) return HTTP 200 for unknown endpoints with an incompatible
+      // body — validate explicitly before accepting the result.
+      if (!Array.isArray(response.data?.embeddings)) {
+        throw new Error('Invalid /api/embed response — missing embeddings array')
+      }
+      return { embeddings: response.data.embeddings }
+    } catch {
+      // Fall back to OpenAI-compatible /v1/embeddings
+      // Explicitly request float format — some backends (e.g. LM Studio) don't reliably
+      // implement the base64 encoding the OpenAI SDK requests by default.
+      logger.info('[OllamaService] /api/embed unavailable, falling back to /v1/embeddings')
+      const results = await this.openai.embeddings.create({ model, input, encoding_format: 'float' })
+      return { embeddings: results.data.map((e) => e.embedding as number[]) }
+    }
+  }
+
+  public async getModels(includeEmbeddings = false): Promise<NomadInstalledModel[]> {
+    await this._ensureDependencies()
+    if (!this.baseUrl) {
+      throw new Error('AI service is not initialized.')
+    }
+
+    try {
+      // Prefer the Ollama native endpoint which includes size and metadata
+      const response = await axios.get(`${this.baseUrl}/api/tags`, { timeout: 5000 })
+      // LM Studio returns HTTP 200 for unknown endpoints with an incompatible body — validate explicitly
+      if (!Array.isArray(response.data?.models)) {
+        throw new Error('Not an Ollama-compatible /api/tags response')
+      }
+      this.isOllamaNative = true
+      const models: NomadInstalledModel[] = response.data.models
+      if (includeEmbeddings) return models
+      return models.filter((m) => !m.name.includes('embed'))
+    } catch {
+      // Fall back to the OpenAI-compatible /v1/models endpoint (LM Studio, llama.cpp, etc.)
+      this.isOllamaNative = false
+      logger.info('[OllamaService] /api/tags unavailable, falling back to /v1/models')
+      try {
+        const modelList = await this.openai!.models.list()
+        const models: NomadInstalledModel[] = modelList.data.map((m) => ({ name: m.id, size: 0 }))
+        if (includeEmbeddings) return models
+        return models.filter((m) => !m.name.includes('embed'))
+      } catch (err) {
+        logger.error(
+          `[OllamaService] Failed to list models: ${err instanceof Error ? err.message : err}`
+        )
+        return []
       }
-    const response = await this.ollama.list()
-    if (includeEmbeddings) {
-      return response.models
     }
-    // Filter out embedding models
-    return response.models.filter((model) => !model.name.includes('embed'))
   }
 
   async getAvailableModels(
-    { sort, recommendedOnly }: { sort?: 'pulls' | 'name'; recommendedOnly?: boolean } = {
+    {
+      sort,
+      recommendedOnly,
+      query,
+      limit,
+      force,
+    }: {
+      sort?: 'pulls' | 'name'
+      recommendedOnly?: boolean
+      query: string | null
+      limit?: number
+      force?: boolean
+    } = {
       sort: 'pulls',
       recommendedOnly: false,
+      query: null,
+      limit: 15,
     }
-  ): Promise<NomadOllamaModel[] | null> {
+  ): Promise<{ models: NomadOllamaModel[]; hasMore: boolean } | null> {
     try {
-      const models = await this.retrieveAndRefreshModels(sort)
+      const models = await this.retrieveAndRefreshModels(sort, force)
       if (!models) {
-        // If we fail to get models from the API, return the fallback recommended models
         logger.warn(
           '[OllamaService] Returning fallback recommended models due to failure in fetching available models'
         )
-        return FALLBACK_RECOMMENDED_OLLAMA_MODELS
+        return {
+          models: FALLBACK_RECOMMENDED_OLLAMA_MODELS,
+          hasMore: false,
+        }
       }
 
       if (!recommendedOnly) {
-        return models
+        const filteredModels = query ? this.fuseSearchModels(models, query) : models
+        return {
+          models: filteredModels.slice(0, limit || 15),
+          hasMore: filteredModels.length > (limit || 15),
+        }
       }
 
-      // If recommendedOnly is true, only return the first three models (if sorted by pulls, these will be the top 3)
       const sortedByPulls = sort === 'pulls' ? models : this.sortModels(models, 'pulls')
       const firstThree = sortedByPulls.slice(0, 3)
 
-      // Only return the first tag of each of these models (should be the most lightweight variant)
       const recommendedModels = firstThree.map((model) => {
         return {
           ...model,
           tags: model.tags && model.tags.length > 0 ? [model.tags[0]] : [],
         }
       })
-      return recommendedModels
+
+      if (query) {
+        const filteredRecommendedModels = this.fuseSearchModels(recommendedModels, query)
+        return {
+          models: filteredRecommendedModels,
+          hasMore: filteredRecommendedModels.length > (limit || 15),
+        }
+      }
+
+      return {
+        models: recommendedModels,
+        hasMore: recommendedModels.length > (limit || 15),
+      }
     } catch (error) {
       logger.error(
         `[OllamaService] Failed to get available models: ${error instanceof Error ? error.message : error}`
@@ -195,17 +490,26 @@ export class OllamaService {
   }
 
   private async retrieveAndRefreshModels(
-    sort?: 'pulls' | 'name'
+    sort?: 'pulls' | 'name',
+    force?: boolean
   ): Promise<NomadOllamaModel[] | null> {
     try {
+      if (!force) {
         const cachedModels = await this.readModelsFromCache()
         if (cachedModels) {
           logger.info('[OllamaService] Using cached available models data')
           return this.sortModels(cachedModels, sort)
         }
+      } else {
+        logger.info('[OllamaService] Force refresh requested, bypassing cache')
+      }
 
       logger.info('[OllamaService] Fetching fresh available models from API')
-      const response = await axios.get(NOMAD_MODELS_API_BASE_URL)
+
+      const baseUrl = env.get('NOMAD_API_URL') || NOMAD_API_DEFAULT_BASE_URL
+      const fullUrl = new URL(NOMAD_MODELS_API_PATH, baseUrl).toString()
+
+      const response = await axios.get(fullUrl)
       if (!response.data || !Array.isArray(response.data.models)) {
         logger.warn(
           `[OllamaService] Invalid response format when fetching available models: ${JSON.stringify(response.data)}`
@@ -213,14 +517,20 @@ export class OllamaService {
         return null
       }
 
-      const models = response.data.models as NomadOllamaModel[]
+      const rawModels = response.data.models as NomadOllamaModel[]
 
-      await this.writeModelsToCache(models)
-      return this.sortModels(models, sort)
+      const noCloud = rawModels
+        .map((model) => ({
+          ...model,
+          tags: model.tags.filter((tag) => !tag.cloud),
+        }))
+        .filter((model) => model.tags.length > 0)
+
+      await this.writeModelsToCache(noCloud)
+      return this.sortModels(noCloud, sort)
     } catch (error) {
       logger.error(
-        `[OllamaService] Failed to retrieve models from Nomad API: ${error instanceof Error ? error.message : error
-        }`
+        `[OllamaService] Failed to retrieve models from Nomad API: ${error instanceof Error ? error.message : error}`
       )
       return null
     }
@@ -246,7 +556,6 @@ export class OllamaService {
 
       return models
     } catch (error) {
-      // Cache doesn't exist or is invalid
       if ((error as NodeJS.ErrnoException).code !== 'ENOENT') {
         logger.warn(
           `[OllamaService] Error reading cache: ${error instanceof Error ? error.message : error}`
@@ -270,7 +579,6 @@ export class OllamaService {
 
   private sortModels(models: NomadOllamaModel[], sort?: 'pulls' | 'name'): NomadOllamaModel[] {
     if (sort === 'pulls') {
-      // Sort by estimated pulls (it should be a string like "1.2K", "500", "4M" etc.)
       models.sort((a, b) => {
         const parsePulls = (pulls: string) => {
           const multiplier = pulls.endsWith('K')
@@ -288,8 +596,6 @@ export class OllamaService {
       models.sort((a, b) => a.name.localeCompare(b.name))
     }
 
-    // Always sort model.tags by the size field in descending order
-    // Size is a string like '75GB', '8.5GB', '2GB' etc. Smaller models first
     models.forEach((model) => {
       if (model.tags && Array.isArray(model.tags)) {
         model.tags.sort((a, b) => {
@@ -302,7 +608,7 @@ export class OllamaService {
                   ? 1
                   : size.endsWith('TB')
                     ? 1_000
-                    : 0 // Unknown size format
+                    : 0
             return parseFloat(size) * multiplier
           }
           return parseSize(a.size) - parseSize(b.size)
@@ -313,12 +619,33 @@ export class OllamaService {
     return models
   }
 
+  private broadcastDownloadError(model: string, error: string) {
+    transmit.broadcast(BROADCAST_CHANNELS.OLLAMA_MODEL_DOWNLOAD, {
+      model,
+      percent: -1,
+      error,
+      timestamp: new Date().toISOString(),
+    })
+  }
+
   private broadcastDownloadProgress(model: string, percent: number) {
-    transmit.broadcast('model-download', {
+    transmit.broadcast(BROADCAST_CHANNELS.OLLAMA_MODEL_DOWNLOAD, {
       model,
       percent,
       timestamp: new Date().toISOString(),
     })
     logger.info(`[OllamaService] Download progress for model "${model}": ${percent}%`)
   }
+
+  private fuseSearchModels(models: NomadOllamaModel[], query: string): NomadOllamaModel[] {
+    const options: IFuseOptions<NomadOllamaModel> = {
+      ignoreDiacritics: true,
+      keys: ['name', 'description', 'tags.name'],
+      threshold: 0.3,
+    }
+
+    const fuse = new Fuse(models, options)
+
+    return fuse.search(query).map((result) => result.item)
+  }
 }

admin/app/services/system_service.ts

@@ -4,14 +4,22 @@ import { DockerService } from '#services/docker_service'
 import { ServiceSlim } from '../../types/services.js'
 import logger from '@adonisjs/core/services/logger'
 import si from 'systeminformation'
-import { NomadDiskInfo, NomadDiskInfoRaw, SystemInformationResponse } from '../../types/system.js'
-import { readFileSync } from 'fs'
-import path, { join } from 'path'
+import {
+  GpuHealthStatus,
+  NomadDiskInfo,
+  NomadDiskInfoRaw,
+  SystemInformationResponse,
+} from '../../types/system.js'
+import { SERVICE_NAMES } from '../../constants/service_names.js'
+import { readFileSync } from 'node:fs'
+import path, { join } from 'node:path'
 import { getAllFilesystems, getFile } from '../utils/fs.js'
 import axios from 'axios'
 import env from '#start/env'
 import KVStore from '#models/kv_store'
-import { KVStoreKey } from '../../types/kv_store.js'
+import { KV_STORE_SCHEMA, KVStoreKey } from '../../types/kv_store.js'
+import { isNewerVersion } from '../utils/version.js'
+import { invalidateAssistantNameCache } from '../../config/inertia.js'
 
 @inject()
 export class SystemService {
@@ -21,8 +29,8 @@ export class SystemService {
   constructor(private dockerService: DockerService) {}
 
   async checkServiceInstalled(serviceName: string): Promise<boolean> {
-    const services = await this.getServices({ installedOnly: true });
-    return services.some(service => service.service_name === serviceName);
+    const services = await this.getServices({ installedOnly: true })
+    return services.some((service) => service.service_name === serviceName)
   }
 
   async getInternetStatus(): Promise<boolean> {
@@ -64,8 +72,142 @@ export class SystemService {
     return false
   }
 
+  async getNvidiaSmiInfo(): Promise<
+    | Array<{ vendor: string; model: string; vram: number }>
+    | { error: string }
+    | 'OLLAMA_NOT_FOUND'
+    | 'BAD_RESPONSE'
+    | 'UNKNOWN_ERROR'
+  > {
+    try {
+      const containers = await this.dockerService.docker.listContainers({ all: false })
+      const ollamaContainer = containers.find((c) => c.Names.includes(`/${SERVICE_NAMES.OLLAMA}`))
+      if (!ollamaContainer) {
+        logger.info(
+          'Ollama container not found for nvidia-smi info retrieval. This is expected if Ollama is not installed.'
+        )
+        return 'OLLAMA_NOT_FOUND'
+      }
+
+      // Execute nvidia-smi inside the Ollama container to get GPU info
+      const container = this.dockerService.docker.getContainer(ollamaContainer.Id)
+      const exec = await container.exec({
+        Cmd: ['nvidia-smi', '--query-gpu=name,memory.total', '--format=csv,noheader,nounits'],
+        AttachStdout: true,
+        AttachStderr: true,
+        Tty: true,
+      })
+
+      // Read the output stream with a timeout to prevent hanging if nvidia-smi fails
+      const stream = await exec.start({ Tty: true })
+      const output = await new Promise<string>((resolve) => {
+        let data = ''
+        const timeout = setTimeout(() => resolve(data), 5000)
+        stream.on('data', (chunk: Buffer) => {
+          data += chunk.toString()
+        })
+        stream.on('end', () => {
+          clearTimeout(timeout)
+          resolve(data)
+        })
+      })
+
+      // Remove any non-printable characters and trim the output
+      const cleaned = Array.from(output)
+        .filter((character) => character.charCodeAt(0) > 8)
+        .join('')
+        .trim()
+      if (
+        cleaned &&
+        !cleaned.toLowerCase().includes('error') &&
+        !cleaned.toLowerCase().includes('not found')
+      ) {
+        // Split by newlines to handle multiple GPUs installed
+        const lines = cleaned.split('\n').filter((line) => line.trim())
+
+        // Map each line out to a useful structure for us
+        const gpus = lines.map((line) => {
+          const parts = line.split(',').map((s) => s.trim())
+          return {
+            vendor: 'NVIDIA',
+            model: parts[0] || 'NVIDIA GPU',
+            vram: parts[1] ? Number.parseInt(parts[1], 10) : 0,
+          }
+        })
+
+        return gpus.length > 0 ? gpus : 'BAD_RESPONSE'
+      }
+
+      // If we got output but looks like an error, consider it a bad response from nvidia-smi
+      return 'BAD_RESPONSE'
+    } catch (error) {
+      logger.error('Error getting nvidia-smi info:', error)
+      if (error instanceof Error && error.message) {
+        return { error: error.message }
+      }
+      return 'UNKNOWN_ERROR'
+    }
+  }
+
+  async getExternalOllamaGpuInfo(): Promise<Array<{
+    vendor: string
+    model: string
+    vram: number
+  }> | null> {
+    try {
+      // If a remote Ollama URL is configured, use it directly without requiring a local container
+      const remoteOllamaUrl = await KVStore.getValue('ai.remoteOllamaUrl')
+      if (!remoteOllamaUrl) {
+        const containers = await this.dockerService.docker.listContainers({ all: false })
+        const ollamaContainer = containers.find((c) => c.Names.includes(`/${SERVICE_NAMES.OLLAMA}`))
+        if (!ollamaContainer) {
+          return null
+        }
+
+        const actualImage = (ollamaContainer.Image || '').toLowerCase()
+        if (actualImage.includes('ollama/ollama') || actualImage.startsWith('ollama:')) {
+          return null
+        }
+      }
+
+      const ollamaUrl = remoteOllamaUrl || (await this.dockerService.getServiceURL(SERVICE_NAMES.OLLAMA))
+      if (!ollamaUrl) {
+        return null
+      }
+
+      await axios.get(new URL('/api/tags', ollamaUrl).toString(), { timeout: 3000 })
+
+      let vramMb = 0
+      try {
+        const psResponse = await axios.get(new URL('/api/ps', ollamaUrl).toString(), {
+          timeout: 3000,
+        })
+        const loadedModels = Array.isArray(psResponse.data?.models) ? psResponse.data.models : []
+        const largestAllocation = loadedModels.reduce(
+          (max: number, model: { size_vram?: number | string }) =>
+            Math.max(max, Number(model.size_vram) || 0),
+          0
+        )
+        vramMb = largestAllocation > 0 ? Math.round(largestAllocation / (1024 * 1024)) : 0
+      } catch {}
+
+      return [
+        {
+          vendor: 'NVIDIA',
+          model: 'NVIDIA GPU (external Ollama)',
+          vram: vramMb,
+        },
+      ]
+    } catch (error) {
+      logger.info(
+        `[SystemService] External Ollama GPU probe failed: ${error instanceof Error ? error.message : error}`
+      )
+      return null
+    }
+  }
+
   async getServices({ installedOnly = true }: { installedOnly?: boolean }): Promise<ServiceSlim[]> {
-    await this._syncContainersWithDatabase() // Sync up before fetching to ensure we have the latest status
+    const statuses = await this._syncContainersWithDatabase() // Sync and reuse the fetched status list
 
     const query = Service.query()
       .orderBy('display_order', 'asc')
@@ -80,7 +222,9 @@ export class SystemService {
         'description',
         'icon',
         'powered_by',
-        'display_order'
+        'display_order',
+        'container_image',
+        'available_update_version'
       )
       .where('is_dependency_service', false)
     if (installedOnly) {
@@ -92,8 +236,6 @@ export class SystemService {
       return []
     }
 
-    const statuses = await this.dockerService.getServicesStatus()
-
     const toReturn: ServiceSlim[] = []
 
     for (const service of services) {
@@ -110,6 +252,8 @@ export class SystemService {
         ui_location: service.ui_location || '',
         powered_by: service.powered_by,
         display_order: service.display_order,
+        container_image: service.container_image,
+        available_update_version: service.available_update_version,
       })
     }
 
@@ -143,13 +287,14 @@ export class SystemService {
 
   async getSystemInfo(): Promise<SystemInformationResponse | undefined> {
     try {
-      const [cpu, mem, os, currentLoad, fsSize, uptime] = await Promise.all([
+      const [cpu, mem, os, currentLoad, fsSize, uptime, graphics] = await Promise.all([
         si.cpu(),
         si.mem(),
         si.osInfo(),
         si.currentLoad(),
         si.fsSize(),
         si.time(),
+        si.graphics(),
       ])
 
       let diskInfo: NomadDiskInfoRaw | undefined
@@ -172,6 +317,90 @@ export class SystemService {
         logger.error('Error reading disk info file:', error)
       }
 
+      // GPU health tracking — detect when host has NVIDIA GPU but Ollama can't access it
+      let gpuHealth: GpuHealthStatus = {
+        status: 'no_gpu',
+        hasNvidiaRuntime: false,
+        ollamaGpuAccessible: false,
+      }
+
+      // Query Docker API for host-level info (hostname, OS, GPU runtime)
+      // si.osInfo() returns the container's info inside Docker, not the host's
+      try {
+        const dockerInfo = await this.dockerService.docker.info()
+
+        if (dockerInfo.Name) {
+          os.hostname = dockerInfo.Name
+        }
+        if (dockerInfo.OperatingSystem) {
+          os.distro = dockerInfo.OperatingSystem
+        }
+        if (dockerInfo.KernelVersion) {
+          os.kernel = dockerInfo.KernelVersion
+        }
+
+        // If si.graphics() returned no controllers (common inside Docker),
+        // fall back to nvidia runtime + nvidia-smi detection
+        if (!graphics.controllers || graphics.controllers.length === 0) {
+          const runtimes = dockerInfo.Runtimes || {}
+          if ('nvidia' in runtimes) {
+            gpuHealth.hasNvidiaRuntime = true
+            const nvidiaInfo = await this.getNvidiaSmiInfo()
+            if (Array.isArray(nvidiaInfo)) {
+              graphics.controllers = nvidiaInfo.map((gpu) => ({
+                model: gpu.model,
+                vendor: gpu.vendor,
+                bus: '',
+                vram: gpu.vram,
+                vramDynamic: false, // assume false here, we don't actually use this field for our purposes.
+              }))
+              gpuHealth.status = 'ok'
+              gpuHealth.ollamaGpuAccessible = true
+            } else if (nvidiaInfo === 'OLLAMA_NOT_FOUND') {
+              // No local Ollama container — check if a remote Ollama URL is configured
+              const externalOllamaGpu = await this.getExternalOllamaGpuInfo()
+              if (externalOllamaGpu) {
+                graphics.controllers = externalOllamaGpu.map((gpu) => ({
+                  model: gpu.model,
+                  vendor: gpu.vendor,
+                  bus: '',
+                  vram: gpu.vram,
+                  vramDynamic: false,
+                }))
+                gpuHealth.status = 'ok'
+                gpuHealth.ollamaGpuAccessible = true
+              } else {
+                gpuHealth.status = 'ollama_not_installed'
+              }
+            } else {
+              const externalOllamaGpu = await this.getExternalOllamaGpuInfo()
+              if (externalOllamaGpu) {
+                graphics.controllers = externalOllamaGpu.map((gpu) => ({
+                  model: gpu.model,
+                  vendor: gpu.vendor,
+                  bus: '',
+                  vram: gpu.vram,
+                  vramDynamic: false,
+                }))
+                gpuHealth.status = 'ok'
+                gpuHealth.ollamaGpuAccessible = true
+              } else {
+                gpuHealth.status = 'passthrough_failed'
+                logger.warn(
+                  `NVIDIA runtime detected but GPU passthrough failed: ${typeof nvidiaInfo === 'string' ? nvidiaInfo : JSON.stringify(nvidiaInfo)}`
+                )
+              }
+            }
+          }
+        } else {
+          // si.graphics() returned controllers (host install, not Docker) — GPU is working
+          gpuHealth.status = 'ok'
+          gpuHealth.ollamaGpuAccessible = true
+        }
+      } catch {
+        // Docker info query failed, skip host-level enrichment
+      }
+
       return {
         cpu,
         mem,
@@ -180,6 +409,8 @@ export class SystemService {
         currentLoad,
         fsSize,
         uptime,
+        graphics,
+        gpuHealth,
       }
     } catch (error) {
       logger.error('Error getting system info:', error)
@@ -187,7 +418,7 @@ export class SystemService {
     }
   }
 
-  async checkLatestVersion(): Promise<{
+  async checkLatestVersion(force?: boolean): Promise<{
     success: boolean
     updateAvailable: boolean
     currentVersion: string
@@ -195,25 +426,50 @@ export class SystemService {
     message?: string
   }> {
     try {
+      const currentVersion = SystemService.getAppVersion()
+      const cachedUpdateAvailable = await KVStore.getValue('system.updateAvailable')
+      const cachedLatestVersion = await KVStore.getValue('system.latestVersion')
+
+      // Use cached values if not forcing a fresh check.
+      // the CheckUpdateJob will update these values every 12 hours
+      if (!force) {
+        return {
+          success: true,
+          updateAvailable: cachedUpdateAvailable ?? false,
+          currentVersion,
+          latestVersion: cachedLatestVersion || '',
+        }
+      }
+
+      const earlyAccess = (await KVStore.getValue('system.earlyAccess')) ?? false
+
+      let latestVersion: string
+      if (earlyAccess) {
+        const response = await axios.get(
+          'https://api.github.com/repos/Crosstalk-Solutions/project-nomad/releases',
+          { headers: { Accept: 'application/vnd.github+json' }, timeout: 5000 }
+        )
+        if (!response?.data?.length) throw new Error('No releases found')
+        latestVersion = response.data[0].tag_name.replace(/^v/, '').trim()
+      } else {
         const response = await axios.get(
           'https://api.github.com/repos/Crosstalk-Solutions/project-nomad/releases/latest',
-        {
-          headers: { Accept: 'application/vnd.github+json' },
-          timeout: 5000,
-        }
+          { headers: { Accept: 'application/vnd.github+json' }, timeout: 5000 }
         )
-
-      if (!response || !response.data?.tag_name) {
-        throw new Error('Invalid response from GitHub API')
+        if (!response?.data?.tag_name) throw new Error('Invalid response from GitHub API')
+        latestVersion = response.data.tag_name.replace(/^v/, '').trim()
       }
 
-      const latestVersion = response.data.tag_name.replace(/^v/, '') // Remove leading 'v' if present
-      const currentVersion = SystemService.getAppVersion()
-
       logger.info(`Current version: ${currentVersion}, Latest version: ${latestVersion}`)
 
-      // NOTE: this will always return true in dev environment! See getAppVersion()
-      const updateAvailable = latestVersion !== currentVersion
+      const updateAvailable =
+        process.env.NODE_ENV === 'development'
+          ? false
+          : isNewerVersion(latestVersion, currentVersion.trim(), earlyAccess)
+
+      // Cache the results in KVStore for frontend checks
+      await KVStore.setValue('system.updateAvailable', updateAvailable)
+      await KVStore.setValue('system.latestVersion', latestVersion)
 
       return {
         success: true,
@@ -261,8 +517,129 @@ export class SystemService {
     }
   }
 
+  async getDebugInfo(): Promise<string> {
+    const appVersion = SystemService.getAppVersion()
+    const environment = process.env.NODE_ENV || 'unknown'
+
+    const [systemInfo, services, internetStatus, versionCheck] = await Promise.all([
+      this.getSystemInfo(),
+      this.getServices({ installedOnly: false }),
+      this.getInternetStatus().catch(() => null),
+      this.checkLatestVersion().catch(() => null),
+    ])
+
+    const lines: string[] = [
+      'Project NOMAD Debug Info',
+      '========================',
+      `App Version: ${appVersion}`,
+      `Environment: ${environment}`,
+    ]
+
+    if (systemInfo) {
+      const { cpu, mem, os, disk, fsSize, uptime, graphics } = systemInfo
+
+      lines.push('')
+      lines.push('System:')
+      if (os.distro) lines.push(`  OS: ${os.distro}`)
+      if (os.hostname) lines.push(`  Hostname: ${os.hostname}`)
+      if (os.kernel) lines.push(`  Kernel: ${os.kernel}`)
+      if (os.arch) lines.push(`  Architecture: ${os.arch}`)
+      if (uptime?.uptime) lines.push(`  Uptime: ${this._formatUptime(uptime.uptime)}`)
+
+      lines.push('')
+      lines.push('Hardware:')
+      if (cpu.brand) {
+        lines.push(`  CPU: ${cpu.brand} (${cpu.cores} cores)`)
+      }
+      if (mem.total) {
+        const total = this._formatBytes(mem.total)
+        const used = this._formatBytes(mem.total - (mem.available || 0))
+        const available = this._formatBytes(mem.available || 0)
+        lines.push(`  RAM: ${total} total, ${used} used, ${available} available`)
+      }
+      if (graphics.controllers && graphics.controllers.length > 0) {
+        for (const gpu of graphics.controllers) {
+          const vram = gpu.vram ? ` (${gpu.vram} MB VRAM)` : ''
+          lines.push(`  GPU: ${gpu.model}${vram}`)
+        }
+      } else {
+        lines.push('  GPU: None detected')
+      }
+
+      // Disk info — try disk array first, fall back to fsSize
+      const diskEntries = disk.filter((d) => d.totalSize > 0)
+      if (diskEntries.length > 0) {
+        for (const d of diskEntries) {
+          const size = this._formatBytes(d.totalSize)
+          const type = d.tran?.toUpperCase() || (d.rota ? 'HDD' : 'SSD')
+          lines.push(`  Disk: ${size}, ${Math.round(d.percentUsed)}% used, ${type}`)
+        }
+      } else if (fsSize.length > 0) {
+        const realFs = fsSize.filter((f) => f.fs.startsWith('/dev/'))
+        const seen = new Set<number>()
+        for (const f of realFs) {
+          if (seen.has(f.size)) continue
+          seen.add(f.size)
+          lines.push(`  Disk: ${this._formatBytes(f.size)}, ${Math.round(f.use)}% used`)
+        }
+      }
+    }
+
+    const installed = services.filter((s) => s.installed)
+    lines.push('')
+    if (installed.length > 0) {
+      lines.push('Installed Services:')
+      for (const svc of installed) {
+        lines.push(`  ${svc.friendly_name} (${svc.service_name}): ${svc.status}`)
+      }
+    } else {
+      lines.push('Installed Services: None')
+    }
+
+    if (internetStatus !== null) {
+      lines.push('')
+      lines.push(`Internet Status: ${internetStatus ? 'Online' : 'Offline'}`)
+    }
+
+    if (versionCheck?.success) {
+      const updateMsg = versionCheck.updateAvailable
+        ? `Yes (${versionCheck.latestVersion} available)`
+        : `No (${versionCheck.currentVersion} is latest)`
+      lines.push(`Update Available: ${updateMsg}`)
+    }
+
+    return lines.join('\n')
+  }
+
+  private _formatUptime(seconds: number): string {
+    const days = Math.floor(seconds / 86400)
+    const hours = Math.floor((seconds % 86400) / 3600)
+    const minutes = Math.floor((seconds % 3600) / 60)
+    if (days > 0) return `${days}d ${hours}h ${minutes}m`
+    if (hours > 0) return `${hours}h ${minutes}m`
+    return `${minutes}m`
+  }
+
+  private _formatBytes(bytes: number, decimals = 1): string {
+    if (bytes === 0) return '0 Bytes'
+    const k = 1024
+    const sizes = ['Bytes', 'KB', 'MB', 'GB', 'TB']
+    const i = Math.floor(Math.log(bytes) / Math.log(k))
+    return Number.parseFloat((bytes / Math.pow(k, i)).toFixed(decimals)) + ' ' + sizes[i]
+  }
+
   async updateSetting(key: KVStoreKey, value: any): Promise<void> {
-    await KVStore.setValue(key, value);
+    if (
+      (value === '' || value === undefined || value === null) &&
+      KV_STORE_SCHEMA[key] === 'string'
+    ) {
+      await KVStore.clearValue(key)
+    } else {
+      await KVStore.setValue(key, value)
+    }
+    if (key === 'ai.assistantCustomName') {
+      invalidateAssistantNameCache()
+    }
   }
 
   /**
@@ -270,8 +647,9 @@ export class SystemService {
    * It will mark services as not installed if their corresponding containers do not exist, regardless of their running state.
    * Handles cases where a container might have been manually removed, ensuring the database reflects the actual existence of containers.
    * Containers that exist but are stopped, paused, or restarting will still be considered installed.
+   * Returns the fetched service status list so callers can reuse it without a second Docker API call.
    */
-  private async _syncContainersWithDatabase() {
+  private async _syncContainersWithDatabase(): Promise<{ service_name: string; status: string }[]> {
     try {
       const allServices = await Service.all()
       const serviceStatusList = await this.dockerService.getServicesStatus()
@@ -284,6 +662,11 @@ export class SystemService {
         if (service.installed) {
           // If marked as installed but container doesn't exist, mark as not installed
           if (!containerExists) {
+            // Exception: remote Ollama is configured without a local container — don't reset it
+            if (service.service_name === SERVICE_NAMES.OLLAMA) {
+              const remoteUrl = await KVStore.getValue('ai.remoteOllamaUrl')
+              if (remoteUrl) continue
+            }
             logger.warn(
               `Service ${service.service_name} is marked as installed but container does not exist. Marking as not installed.`
             )
@@ -303,8 +686,11 @@ export class SystemService {
           }
         }
       }
+
+      return serviceStatusList
     } catch (error) {
       logger.error('Error syncing containers with database:', error)
+      return []
     }
   }
 
@@ -315,10 +701,21 @@ export class SystemService {
       return []
     }
 
+    // Deduplicate: same device path mounted in multiple places (Docker bind-mounts)
+    // Keep the entry with the largest size — that's the real partition
+    const deduped = new Map<string, NomadDiskInfoRaw['fsSize'][0]>()
+    for (const entry of fsSize) {
+      const existing = deduped.get(entry.fs)
+      if (!existing || entry.size > existing.size) {
+        deduped.set(entry.fs, entry)
+      }
+    }
+    const dedupedFsSize = Array.from(deduped.values())
+
     return diskLayout.blockdevices
       .filter((disk) => disk.type === 'disk') // Only physical disks
       .map((disk) => {
-        const filesystems = getAllFilesystems(disk, fsSize)
+        const filesystems = getAllFilesystems(disk, dedupedFsSize)
 
         // Across all partitions
         const totalUsed = filesystems.reduce((sum, p) => sum + (p.used || 0), 0)

admin/app/services/system_update_service.ts

@@ -2,6 +2,7 @@ import logger from '@adonisjs/core/services/logger'
 import { readFileSync, existsSync } from 'fs'
 import { writeFile } from 'fs/promises'
 import { join } from 'path'
+import KVStore from '#models/kv_store'
 
 interface UpdateStatus {
   stage: 'idle' | 'starting' | 'pulling' | 'pulled' | 'recreating' | 'complete' | 'error'
@@ -29,13 +30,17 @@ export class SystemUpdateService {
         }
       }
 
+      // Determine the Docker image tag to install.
+      const latestVersion = await KVStore.getValue('system.latestVersion')
+
       const requestData = {
         requested_at: new Date().toISOString(),
         requester: 'admin-api',
+        target_tag: latestVersion ? `v${latestVersion}` : 'latest',
       }
 
       await writeFile(SystemUpdateService.REQUEST_FILE, JSON.stringify(requestData, null, 2))
-      logger.info('[SystemUpdateService]: System update requested - sidecar will process shortly')
+      logger.info(`[SystemUpdateService]: System update requested (target tag: ${requestData.target_tag}) - sidecar will process shortly`)
 
       return {
         success: true,

admin/app/services/zim_extraction_service.ts

@@ -0,0 +1,310 @@
+import { Archive, Entry } from '@openzim/libzim'
+import * as cheerio from 'cheerio'
+import { HTML_SELECTORS_TO_REMOVE, NON_CONTENT_HEADING_PATTERNS } from '../../constants/zim_extraction.js'
+import logger from '@adonisjs/core/services/logger'
+import { ExtractZIMChunkingStrategy, ExtractZIMContentOptions, ZIMContentChunk, ZIMArchiveMetadata } from '../../types/zim.js'
+import { randomUUID } from 'node:crypto'
+import { access } from 'node:fs/promises'
+
+export class ZIMExtractionService {
+
+    private extractArchiveMetadata(archive: Archive): ZIMArchiveMetadata {
+        try {
+            return {
+                title: archive.getMetadata('Title') || archive.getMetadata('Name') || 'Unknown',
+                creator: archive.getMetadata('Creator') || 'Unknown',
+                publisher: archive.getMetadata('Publisher') || 'Unknown',
+                date: archive.getMetadata('Date') || 'Unknown',
+                language: archive.getMetadata('Language') || 'Unknown',
+                description: archive.getMetadata('Description') || '',
+            }
+        } catch (error) {
+            logger.warn('[ZIMExtractionService]: Could not extract all metadata, using defaults', error)
+            return {
+                title: 'Unknown',
+                creator: 'Unknown',
+                publisher: 'Unknown',
+                date: 'Unknown',
+                language: 'Unknown',
+                description: '',
+            }
+        }
+    }
+
+    /**
+     * Breaks out a ZIM file's entries into their structured content form
+     * to facilitate better indexing and retrieval.
+     * Returns enhanced chunks with full article context and metadata.
+     * 
+     * @param filePath - Path to the ZIM file
+     * @param opts - Options including maxArticles, strategy, onProgress, startOffset, and batchSize
+     */
+    async extractZIMContent(filePath: string, opts: ExtractZIMContentOptions = {}): Promise<ZIMContentChunk[]> {
+        try {
+            logger.info(`[ZIMExtractionService]: Processing ZIM file at path: ${filePath}`)
+            
+            // defensive - check if file still exists before opening
+            // could have been deleted by another process or batch
+            try {
+                await access(filePath)
+            } catch (error) {
+                logger.error(`[ZIMExtractionService]: ZIM file not accessible: ${filePath}`)
+                throw new Error(`ZIM file not found or not accessible: ${filePath}`)
+            }
+            
+            const archive = new Archive(filePath)
+
+            // Extract archive-level metadata once
+            const archiveMetadata = this.extractArchiveMetadata(archive)
+            logger.info(`[ZIMExtractionService]: Archive metadata - Title: ${archiveMetadata.title}, Language: ${archiveMetadata.language}`)
+
+            let articlesProcessed = 0
+            let articlesSkipped = 0
+            const processedPaths = new Set<string>()
+            const toReturn: ZIMContentChunk[] = []
+
+            // Support batch processing to avoid lock timeouts on large ZIM files
+            const startOffset = opts.startOffset || 0
+            const batchSize = opts.batchSize || (opts.maxArticles || Infinity)
+
+            for (const entry of archive.iterByPath()) {
+                // Skip articles until we reach the start offset
+                if (articlesSkipped < startOffset) {
+                    if (this.isArticleEntry(entry) && !processedPaths.has(entry.path)) {
+                        articlesSkipped++
+                    }
+                    continue
+                }
+
+                if (articlesProcessed >= batchSize) {
+                    break
+                }
+
+                if (!this.isArticleEntry(entry)) {
+                    logger.debug(`[ZIMExtractionService]: Skipping non-article entry at path: ${entry.path}`)
+                    continue
+                }
+
+                if (processedPaths.has(entry.path)) {
+                    logger.debug(`[ZIMExtractionService]: Skipping duplicate entry at path: ${entry.path}`)
+                    continue
+                }
+                processedPaths.add(entry.path)
+
+                const item = entry.item
+                const blob = item.data
+                const html = this.getCleanedHTMLString(blob.data)
+
+                const strategy = opts.strategy || this.chooseChunkingStrategy(html);
+                logger.debug(`[ZIMExtractionService]: Chosen chunking strategy for path ${entry.path}: ${strategy}`)
+
+                // Generate a unique document ID. All chunks from same article will share it
+                const documentId = randomUUID()
+                const articleTitle = entry.title || entry.path
+
+                let chunks: ZIMContentChunk[]
+
+                if (strategy === 'structured') {
+                    const structured = this.extractStructuredContent(html)
+                    chunks = structured.sections.map(s => ({
+                        text: s.text,
+                        articleTitle,
+                        articlePath: entry.path,
+                        sectionTitle: s.heading,
+                        fullTitle: `${articleTitle} - ${s.heading}`,
+                        hierarchy: `${articleTitle} > ${s.heading}`,
+                        sectionLevel: s.level,
+                        documentId,
+                        archiveMetadata,
+                        strategy,
+                    }))
+                } else {
+                    // Simple strategy - entire article as one chunk
+                    const text = this.extractTextFromHTML(html) || ''
+                    chunks = [{
+                        text,
+                        articleTitle,
+                        articlePath: entry.path,
+                        sectionTitle: articleTitle, // Same as article for simple strategy
+                        fullTitle: articleTitle,
+                        hierarchy: articleTitle,
+                        documentId,
+                        archiveMetadata,
+                        strategy,
+                    }]
+                }
+
+                logger.debug(`Extracted ${chunks.length} chunks from article at path: ${entry.path} using strategy: ${strategy}`)
+
+                const nonEmptyChunks = chunks.filter(c => c.text.trim().length > 0)
+                logger.debug(`After filtering empty chunks, ${nonEmptyChunks.length} chunks remain for article at path: ${entry.path}`)
+                toReturn.push(...nonEmptyChunks)
+                articlesProcessed++
+
+                if (opts.onProgress) {
+                    opts.onProgress(articlesProcessed, archive.articleCount)
+                }
+            }
+
+            logger.info(`[ZIMExtractionService]: Completed processing ZIM file. Total articles processed: ${articlesProcessed}`)
+            logger.debug("Final structured content sample:", toReturn.slice(0, 3).map(c => ({
+                articleTitle: c.articleTitle,
+                sectionTitle: c.sectionTitle,
+                hierarchy: c.hierarchy,
+                textPreview: c.text.substring(0, 100)
+            })))
+            logger.debug("Total structured sections extracted:", toReturn.length)
+            return toReturn
+        } catch (error) {
+            logger.error('Error processing ZIM file:', error)
+            throw error
+        }
+    }
+
+    private chooseChunkingStrategy(html: string, options = {
+        forceStrategy: null as ExtractZIMChunkingStrategy | null,
+    }): ExtractZIMChunkingStrategy {
+        const {
+            forceStrategy = null,
+        } = options;
+
+        if (forceStrategy) return forceStrategy;
+
+        // Use a simple analysis to determin if the HTML has any meaningful structure
+        // that we can leverage for better chunking. If not, we'll just chunk it as one big piece of text.
+        return this.hasStructuredHeadings(html) ? 'structured' : 'simple';
+    }
+
+    private getCleanedHTMLString(buff: Buffer<ArrayBufferLike>): string {
+        const rawString = buff.toString('utf-8');
+        const $ = cheerio.load(rawString);
+
+        HTML_SELECTORS_TO_REMOVE.forEach((selector) => {
+            $(selector).remove()
+        });
+
+        return $.html();
+    }
+
+    private extractTextFromHTML(html: string): string | null {
+        try {
+            const $ = cheerio.load(html)
+
+            // Search body first, then root if body is absent
+            const text = $('body').length ? $('body').text() : $.root().text()
+
+            return text.replace(/\s+/g, ' ').replace(/\n\s*\n/g, '\n').trim()
+        } catch (error) {
+            logger.error('Error extracting text from HTML:', error)
+            return null
+        }
+    }
+
+    private extractStructuredContent(html: string) {
+        const $ = cheerio.load(html);
+
+        const title = $('h1').first().text().trim() || $('title').text().trim();
+
+        // Extract sections with their headings and heading levels
+        const sections: Array<{ heading: string; text: string; level: number }> = [];
+        let currentSection = { heading: 'Introduction', content: [] as string[], level: 2 };
+
+        $('body').children().each((_, element) => {
+            const $el = $(element);
+            const tagName = element.tagName?.toLowerCase();
+
+            if (['h2', 'h3', 'h4'].includes(tagName)) {
+                // Save current section if it has content
+                if (currentSection.content.length > 0) {
+                    sections.push({
+                        heading: currentSection.heading,
+                        text: currentSection.content.join(' ').replace(/\s+/g, ' ').trim(),
+                        level: currentSection.level,
+                    });
+                }
+                // Start new section
+                const level = parseInt(tagName.substring(1)); // Extract number from h2, h3, h4
+                currentSection = {
+                    heading: $el.text().replace(/\[edit\]/gi, '').trim(),
+                    content: [],
+                    level,
+                };
+            } else if (['p', 'ul', 'ol', 'dl', 'table'].includes(tagName)) {
+                const text = $el.text().trim();
+                if (text.length > 0) {
+                    currentSection.content.push(text);
+                }
+            }
+        });
+
+        // Push the last section if it has content
+        if (currentSection.content.length > 0) {
+            sections.push({
+                heading: currentSection.heading,
+                text: currentSection.content.join(' ').replace(/\s+/g, ' ').trim(),
+                level: currentSection.level,
+            });
+        }
+
+        return {
+            title,
+            sections,
+            fullText: sections.map(s => `${s.heading}\n${s.text}`).join('\n\n'),
+        };
+    }
+
+    private hasStructuredHeadings(html: string): boolean {
+        const $ = cheerio.load(html);
+
+        const headings = $('h2, h3').toArray();
+
+        // Consider it structured if it has at least 2 headings to break content into meaningful sections
+        if (headings.length < 2) return false;
+
+        // Check that headings have substantial content between them
+        let sectionsWithContent = 0;
+
+        for (const heading of headings) {
+            const $heading = $(heading);
+            const headingText = $heading.text().trim();
+
+            // Skip empty or very short headings, likely not meaningful
+            if (headingText.length < 3) continue;
+
+            // Skip common non-content headings
+            if (NON_CONTENT_HEADING_PATTERNS.some(pattern => pattern.test(headingText))) {
+                continue;
+            }
+
+            // Content until next heading
+            let contentLength = 0;
+            let $next = $heading.next();
+
+            while ($next.length && !$next.is('h1, h2, h3, h4')) {
+                contentLength += $next.text().trim().length;
+                $next = $next.next();
+            }
+
+            // Consider it a real section if it has at least 100 chars of content
+            if (contentLength >= 100) {
+                sectionsWithContent++;
+            }
+        }
+
+        // Require at least 2 sections with substantial content
+        return sectionsWithContent >= 2;
+    }
+
+    private isArticleEntry(entry: Entry): boolean {
+        try {
+            if (entry.isRedirect) return false;
+
+            const item = entry.item;
+            const mimeType = item.mimetype;
+
+            return mimeType === 'text/html' || mimeType === 'application/xhtml+xml';
+        } catch {
+            return false;
+        }
+    }
+}

admin/app/services/zim_service.ts

@@ -16,35 +16,24 @@ import {
   listDirectoryContents,
   ZIM_STORAGE_PATH,
 } from '../utils/fs.js'
-import { join } from 'path'
-import { CuratedCategory, CuratedCollectionWithStatus, CuratedCollectionsFile, WikipediaOption, WikipediaState } from '../../types/downloads.js'
+import { join, resolve, sep } from 'path'
+import { WikipediaOption, WikipediaState } from '../../types/downloads.js'
 import vine from '@vinejs/vine'
-import { curatedCategoriesFileSchema, curatedCollectionsFileSchema, wikipediaOptionsFileSchema } from '#validators/curated_collections'
-import CuratedCollection from '#models/curated_collection'
-import CuratedCollectionResource from '#models/curated_collection_resource'
-import InstalledTier from '#models/installed_tier'
+import { wikipediaOptionsFileSchema } from '#validators/curated_collections'
 import WikipediaSelection from '#models/wikipedia_selection'
-import ZimFileMetadata from '#models/zim_file_metadata'
+import InstalledResource from '#models/installed_resource'
 import { RunDownloadJob } from '#jobs/run_download_job'
-import { DownloadCollectionOperation, DownloadRemoteSuccessCallback } from '../../types/files.js'
 import { SERVICE_NAMES } from '../../constants/service_names.js'
+import { CollectionManifestService } from './collection_manifest_service.js'
+import { KiwixLibraryService } from './kiwix_library_service.js'
+import type { CategoryWithStatus } from '../../types/collections.js'
 
 const ZIM_MIME_TYPES = ['application/x-zim', 'application/x-openzim', 'application/octet-stream']
-const CATEGORIES_URL = 'https://raw.githubusercontent.com/Crosstalk-Solutions/project-nomad/refs/heads/master/collections/kiwix-categories.json'
-const COLLECTIONS_URL =
-  'https://github.com/Crosstalk-Solutions/project-nomad/raw/refs/heads/master/collections/kiwix.json'
-const WIKIPEDIA_OPTIONS_URL = 'https://raw.githubusercontent.com/Crosstalk-Solutions/project-nomad/refs/heads/master/collections/wikipedia.json'
-
-
-
-interface IZimService {
-  downloadCollection: DownloadCollectionOperation
-  downloadRemoteSuccessCallback: DownloadRemoteSuccessCallback
-}
+const WIKIPEDIA_OPTIONS_URL = 'https://raw.githubusercontent.com/Crosstalk-Solutions/project-nomad/refs/heads/main/collections/wikipedia.json'
 
 @inject()
-export class ZimService implements IZimService {
-  constructor(private dockerService: DockerService) {}
+export class ZimService {
+  constructor(private dockerService: DockerService) { }
 
   async list() {
     const dirPath = join(process.cwd(), ZIM_STORAGE_PATH)
@@ -53,24 +42,8 @@ export class ZimService implements IZimService {
     const all = await listDirectoryContents(dirPath)
     const files = all.filter((item) => item.name.endsWith('.zim'))
 
-    // Fetch metadata for all files
-    const metadataRecords = await ZimFileMetadata.all()
-    const metadataMap = new Map(metadataRecords.map((m) => [m.filename, m]))
-
-    // Enrich files with metadata
-    const enrichedFiles = files.map((file) => {
-      const metadata = metadataMap.get(file.name)
     return {
-        ...file,
-        title: metadata?.title || null,
-        summary: metadata?.summary || null,
-        author: metadata?.author || null,
-        size_bytes: metadata?.size_bytes || null,
-      }
-    })
-
-    return {
-      files: enrichedFiles,
+      files,
     }
   }
 
@@ -165,16 +138,13 @@ export class ZimService implements IZimService {
     }
   }
 
-  async downloadRemote(
-    url: string,
-    metadata?: { title: string; summary?: string; author?: string; size_bytes?: number }
-  ): Promise<{ filename: string; jobId?: string }> {
+  async downloadRemote(url: string, metadata?: { title?: string; summary?: string; author?: string; size_bytes?: number }): Promise<{ filename: string; jobId?: string }> {
     const parsed = new URL(url)
     if (!parsed.pathname.endsWith('.zim')) {
       throw new Error(`Invalid ZIM file URL: ${url}. URL must end with .zim`)
     }
 
-    const existing = await RunDownloadJob.getByUrl(url)
+    const existing = await RunDownloadJob.getActiveByUrl(url)
     if (existing) {
       throw new Error('A download for this URL is already in progress')
     }
@@ -187,19 +157,11 @@ export class ZimService implements IZimService {
 
     const filepath = join(process.cwd(), ZIM_STORAGE_PATH, filename)
 
-    // Store metadata if provided
-    if (metadata) {
-      await ZimFileMetadata.updateOrCreate(
-        { filename },
-        {
-          title: metadata.title,
-          summary: metadata.summary || null,
-          author: metadata.author || null,
-          size_bytes: metadata.size_bytes || null,
-        }
-      )
-      logger.info(`[ZimService] Stored metadata for ZIM file: ${filename}`)
-    }
+    // Parse resource metadata for the download job
+    const parsedFilename = CollectionManifestService.parseZimFilename(filename)
+    const resourceMetadata = parsedFilename
+      ? { resource_id: parsedFilename.resource_id, version: parsedFilename.version, collection_ref: null }
+      : undefined
 
     // Dispatch a background download job
     const result = await RunDownloadJob.dispatch({
@@ -209,6 +171,9 @@ export class ZimService implements IZimService {
       allowedMimeTypes: ZIM_MIME_TYPES,
       forceNew: true,
       filetype: 'zim',
+      title: metadata?.title,
+      totalBytes: metadata?.size_bytes,
+      resourceMetadata,
     })
 
     if (!result || !result.job) {
@@ -223,44 +188,66 @@ export class ZimService implements IZimService {
     }
   }
 
-  async downloadCollection(slug: string) {
-    const collection = await CuratedCollection.query().where('slug', slug).andWhere('type', 'zim').first()
-    if (!collection) {
-      return null
+  async listCuratedCategories(): Promise<CategoryWithStatus[]> {
+    const manifestService = new CollectionManifestService()
+    return manifestService.getCategoriesWithStatus()
   }
 
-    const resources = await collection.related('resources').query().where('downloaded', false)
-    if (resources.length === 0) {
-      return null
+  async downloadCategoryTier(categorySlug: string, tierSlug: string): Promise<string[] | null> {
+    const manifestService = new CollectionManifestService()
+    const spec = await manifestService.getSpecWithFallback<import('../../types/collections.js').ZimCategoriesSpec>('zim_categories')
+    if (!spec) {
+      throw new Error('Could not load ZIM categories spec')
     }
 
-    const downloadUrls = resources.map((res) => res.url)
+    const category = spec.categories.find((c) => c.slug === categorySlug)
+    if (!category) {
+      throw new Error(`Category not found: ${categorySlug}`)
+    }
+
+    const tier = category.tiers.find((t) => t.slug === tierSlug)
+    if (!tier) {
+      throw new Error(`Tier not found: ${tierSlug}`)
+    }
+
+    const allResources = CollectionManifestService.resolveTierResources(tier, category.tiers)
+
+    // Filter out already installed
+    const installed = await InstalledResource.query().where('resource_type', 'zim')
+    const installedIds = new Set(installed.map((r) => r.resource_id))
+    const toDownload = allResources.filter((r) => !installedIds.has(r.id))
+
+    if (toDownload.length === 0) return null
+
     const downloadFilenames: string[] = []
 
-    for (const url of downloadUrls) {
-      const existing = await RunDownloadJob.getByUrl(url)
-      if (existing) {
-        logger.warn(`[ZimService] Download already in progress for URL ${url}, skipping.`)
+    for (const resource of toDownload) {
+      const existingJob = await RunDownloadJob.getActiveByUrl(resource.url)
+      if (existingJob) {
+        logger.warn(`[ZimService] Download already in progress for ${resource.url}, skipping.`)
         continue
       }
 
-      // Extract the filename from the URL
-      const filename = url.split('/').pop()
-      if (!filename) {
-        logger.warn(`[ZimService] Could not determine filename from URL ${url}, skipping.`)
-        continue
-      }
+      const filename = resource.url.split('/').pop()
+      if (!filename) continue
 
       downloadFilenames.push(filename)
       const filepath = join(process.cwd(), ZIM_STORAGE_PATH, filename)
 
       await RunDownloadJob.dispatch({
-        url,
+        url: resource.url,
         filepath,
         timeout: 30000,
         allowedMimeTypes: ZIM_MIME_TYPES,
         forceNew: true,
         filetype: 'zim',
+        title: (resource as any).title || undefined,
+        totalBytes: (resource as any).size_mb ? (resource as any).size_mb * 1024 * 1024 : undefined,
+        resourceMetadata: {
+          resource_id: resource.id,
+          version: resource.version,
+          collection_ref: categorySlug,
+        },
       })
     }
 
@@ -275,95 +262,92 @@ export class ZimService implements IZimService {
       }
     }
     
-    // Restart KIWIX container to pick up new ZIM file
+    // Update the kiwix library XML after all downloaded ZIM files are in place.
+    // This covers all ZIM types including Wikipedia. Rebuilding once from disk
+    // avoids repeated XML parse/write cycles and reduces the chance of write races
+    // when multiple download jobs complete concurrently.
+    const kiwixLibraryService = new KiwixLibraryService()
+    try {
+      await kiwixLibraryService.rebuildFromDisk()
+    } catch (err) {
+      logger.error('[ZimService] Failed to rebuild kiwix library from disk:', err)
+    }
+
     if (restart) {
+      // Check if there are any remaining ZIM download jobs before restarting
+      const { QueueService } = await import('./queue_service.js')
+      const queueService = new QueueService()
+      const queue = queueService.getQueue('downloads')
+
+      // Get all active and waiting jobs
+      const [activeJobs, waitingJobs] = await Promise.all([
+        queue.getActive(),
+        queue.getWaiting(),
+      ])
+
+      // Filter out completed jobs (progress === 100) to avoid race condition
+      // where this job itself is still in the active queue
+      const activeIncompleteJobs = activeJobs.filter((job) => {
+        const progress = typeof job.progress === 'object' && job.progress !== null
+          ? (job.progress as any).percent
+          : typeof job.progress === 'number' ? job.progress : 0
+        return progress < 100
+      })
+
+      // Check if any remaining incomplete jobs are ZIM downloads
+      const allJobs = [...activeIncompleteJobs, ...waitingJobs]
+      const hasRemainingZimJobs = allJobs.some((job) => job.data.filetype === 'zim')
+
+      if (hasRemainingZimJobs) {
+        logger.info('[ZimService] Skipping container restart - more ZIM downloads pending')
+      } else {
+        // If kiwix is already running in library mode, --monitorLibrary will pick up
+        // the XML change automatically — no restart needed.
+        const isLegacy = await this.dockerService.isKiwixOnLegacyConfig()
+        if (!isLegacy) {
+          logger.info('[ZimService] Kiwix is in library mode — XML updated, no container restart needed.')
+        } else {
+          // Legacy config: restart (affectContainer will trigger migration instead)
+          logger.info('[ZimService] No more ZIM downloads pending - restarting KIWIX container')
           await this.dockerService
             .affectContainer(SERVICE_NAMES.KIWIX, 'restart')
             .catch((error) => {
-          logger.error(`[ZimService] Failed to restart KIWIX container:`, error) // Don't stop the download completion, just log the error.
+              logger.error(`[ZimService] Failed to restart KIWIX container:`, error)
             })
         }
-
-    // Mark any curated collection resources with this download URL as downloaded
-    const resources = await CuratedCollectionResource.query().whereIn('url', urls)
-    for (const resource of resources) {
-      resource.downloaded = true
-      await resource.save()
       }
     }
 
-  async listCuratedCategories(): Promise<CuratedCategory[]> {
+    // Create InstalledResource entries for downloaded files
+    for (const url of urls) {
+      // Skip Wikipedia files (managed separately)
+      if (url.includes('wikipedia_en_')) continue
+
+      const filename = url.split('/').pop()
+      if (!filename) continue
+
+      const parsed = CollectionManifestService.parseZimFilename(filename)
+      if (!parsed) continue
+
+      const filepath = join(process.cwd(), ZIM_STORAGE_PATH, filename)
+      const stats = await getFileStatsIfExists(filepath)
+
       try {
-      const response = await axios.get(CATEGORIES_URL)
-      const data = response.data
-
-      const validated = await vine.validate({
-        schema: curatedCategoriesFileSchema,
-        data,
-      });
-
-      // Look up installed tiers for all categories
-      const installedTiers = await InstalledTier.all()
-      const installedTierMap = new Map(
-        installedTiers.map((t) => [t.category_slug, t.tier_slug])
-      )
-
-      // Add installedTierSlug to each category
-      return validated.categories.map((category) => ({
-        ...category,
-        installedTierSlug: installedTierMap.get(category.slug),
-      }))
-    } catch (error) {
-      logger.error(`[ZimService] Failed to fetch curated categories:`, error)
-      throw new Error('Failed to fetch curated categories or invalid format was received')
-    }
-  }
-
-  async saveInstalledTier(categorySlug: string, tierSlug: string): Promise<void> {
-    await InstalledTier.updateOrCreate(
-      { category_slug: categorySlug },
-      { tier_slug: tierSlug }
-    )
-    logger.info(`[ZimService] Saved installed tier: ${categorySlug} -> ${tierSlug}`)
-  }
-
-  async listCuratedCollections(): Promise<CuratedCollectionWithStatus[]> {
-    const collections = await CuratedCollection.query().where('type', 'zim').preload('resources')
-    return collections.map((collection) => ({
-      ...(collection.serialize() as CuratedCollection),
-      all_downloaded: collection.resources.every((res) => res.downloaded),
-    }))
-  }
-
-  async fetchLatestCollections(): Promise<boolean> {
-    try {
-      const response = await axios.get<CuratedCollectionsFile>(COLLECTIONS_URL)
-
-      const validated = await vine.validate({
-        schema: curatedCollectionsFileSchema,
-        data: response.data,
-      })
-
-      for (const collection of validated.collections) {
-        const collectionResult = await CuratedCollection.updateOrCreate(
-          { slug: collection.slug },
+        const { DateTime } = await import('luxon')
+        await InstalledResource.updateOrCreate(
+          { resource_id: parsed.resource_id, resource_type: 'zim' },
           {
-            ...collection,
-            type: 'zim',
+            version: parsed.version,
+            url: url,
+            file_path: filepath,
+            file_size_bytes: stats ? Number(stats.size) : null,
+            installed_at: DateTime.now(),
           }
         )
-        logger.info(`[ZimService] Upserted curated collection: ${collection.slug}`)
-
-        await collectionResult.related('resources').createMany(collection.resources)
-        logger.info(
-          `[ZimService] Upserted ${collection.resources.length} resources for collection: ${collection.slug}`
-        )
-      }
-
-      return true
+        logger.info(`[ZimService] Created InstalledResource entry for: ${parsed.resource_id}`)
       } catch (error) {
-      logger.error(`[ZimService] Failed to download latest Kiwix collections:`, error)
-      return false
+        logger.error(`[ZimService] Failed to create InstalledResource for ${filename}:`, error)
+      }
     }
   }
 
@@ -373,7 +357,13 @@ export class ZimService implements IZimService {
       fileName += '.zim'
     }
 
-    const fullPath = join(process.cwd(), ZIM_STORAGE_PATH, fileName)
+    const basePath = resolve(join(process.cwd(), ZIM_STORAGE_PATH))
+    const fullPath = resolve(join(basePath, fileName))
+
+    // Prevent path traversal — resolved path must stay within the storage directory
+    if (!fullPath.startsWith(basePath + sep)) {
+      throw new Error('Invalid filename')
+    }
 
     const exists = await getFileStatsIfExists(fullPath)
     if (!exists) {
@@ -382,9 +372,21 @@ export class ZimService implements IZimService {
 
     await deleteFileIfExists(fullPath)
 
-    // Clean up metadata
-    await ZimFileMetadata.query().where('filename', fileName).delete()
-    logger.info(`[ZimService] Deleted metadata for ZIM file: ${fileName}`)
+    // Remove from kiwix library XML so --monitorLibrary stops serving the deleted file
+    const kiwixLibraryService = new KiwixLibraryService()
+    await kiwixLibraryService.removeBook(fileName).catch((err) => {
+      logger.error(`[ZimService] Failed to remove ${fileName} from kiwix library:`, err)
+    })
+
+    // Clean up InstalledResource entry
+    const parsed = CollectionManifestService.parseZimFilename(fileName)
+    if (parsed) {
+      await InstalledResource.query()
+        .where('resource_id', parsed.resource_id)
+        .where('resource_type', 'zim')
+        .delete()
+      logger.info(`[ZimService] Deleted InstalledResource entry for: ${parsed.resource_id}`)
+    }
   }
 
   // Wikipedia selector methods
@@ -487,7 +489,7 @@ export class ZimService implements IZimService {
     }
 
     // Check if already downloading
-    const existingJob = await RunDownloadJob.getByUrl(selectedOption.url)
+    const existingJob = await RunDownloadJob.getActiveByUrl(selectedOption.url)
     if (existingJob) {
       return { success: false, message: 'Download already in progress' }
     }
@@ -526,6 +528,8 @@ export class ZimService implements IZimService {
       allowedMimeTypes: ZIM_MIME_TYPES,
       forceNew: true,
       filetype: 'zim',
+      title: selectedOption.name,
+      totalBytes: selectedOption.size_mb ? selectedOption.size_mb * 1024 * 1024 : undefined,
     })
 
     if (!result || !result.job) {

admin/app/utils/downloads.ts

@@ -88,10 +88,29 @@ export async function doResumableDownload({
     let lastProgressTime = Date.now()
     let lastDownloadedBytes = startByte
 
+    // Stall detection: if no data arrives for 5 minutes, abort the download
+    const STALL_TIMEOUT_MS = 5 * 60 * 1000
+    let stallTimer: ReturnType<typeof setTimeout> | null = null
+
+    const clearStallTimer = () => {
+      if (stallTimer) {
+        clearTimeout(stallTimer)
+        stallTimer = null
+      }
+    }
+
+    const resetStallTimer = () => {
+      clearStallTimer()
+      stallTimer = setTimeout(() => {
+        cleanup(new Error('Download stalled - no data received for 5 minutes'))
+      }, STALL_TIMEOUT_MS)
+    }
+
     // Progress tracking stream to monitor data flow
     const progressStream = new Transform({
       transform(chunk: Buffer, _: any, callback: Function) {
         downloadedBytes += chunk.length
+        resetStallTimer()
 
         // Update progress tracking
         const now = Date.now()
@@ -118,6 +137,7 @@ export async function doResumableDownload({
 
     // Handle errors and cleanup
     const cleanup = (error?: Error) => {
+      clearStallTimer()
       progressStream.destroy()
       response.data.destroy()
       writeStream.destroy()
@@ -136,6 +156,7 @@ export async function doResumableDownload({
     })
 
     writeStream.on('finish', async () => {
+      clearStallTimer()
       if (onProgress) {
         onProgress({
           downloadedBytes,
@@ -151,7 +172,8 @@ export async function doResumableDownload({
       resolve(filepath)
     })
 
-    // Pipe: response -> progressStream -> writeStream
+    // Start stall timer and pipe: response -> progressStream -> writeStream
+    resetStallTimer()
     response.data.pipe(progressStream).pipe(writeStream)
   })
 }

admin/app/utils/fs.ts

@@ -5,6 +5,7 @@ import { createReadStream } from 'fs'
 import { LSBlockDevice, NomadDiskInfoRaw } from '../../types/system.js'
 
 export const ZIM_STORAGE_PATH = '/storage/zim'
+export const KIWIX_LIBRARY_XML_PATH = '/storage/zim/kiwix-library.xml'
 
 export async function listDirectoryContents(path: string): Promise<FileEntry[]> {
   const entries = await readdir(path, { withFileTypes: true })
@@ -49,7 +50,7 @@ export async function listDirectoryContentsRecursive(path: string): Promise<File
 export async function ensureDirectoryExists(path: string): Promise<void> {
   try {
     await stat(path)
-  } catch (error) {
+  } catch (error: any) {
     if (error.code === 'ENOENT') {
       await mkdir(path, { recursive: true })
     }
@@ -73,7 +74,7 @@ export async function getFile(
       return createReadStream(path)
     }
     return await readFile(path)
-  } catch (error) {
+  } catch (error: any) {
     if (error.code === 'ENOENT') {
       return null
     }
@@ -90,7 +91,7 @@ export async function getFileStatsIfExists(
       size: stats.size,
       modifiedTime: stats.mtime,
     }
-  } catch (error) {
+  } catch (error: any) {
     if (error.code === 'ENOENT') {
       return null
     }
@@ -101,7 +102,7 @@ export async function getFileStatsIfExists(
 export async function deleteFileIfExists(path: string): Promise<void> {
   try {
     await unlink(path)
-  } catch (error) {
+  } catch (error: any) {
     if (error.code !== 'ENOENT') {
       throw error
     }
@@ -138,21 +139,20 @@ export function matchesDevice(fsPath: string, deviceName: string): boolean {
   // Remove /dev/ and /dev/mapper/ prefixes
   const normalized = fsPath.replace('/dev/mapper/', '').replace('/dev/', '')
 
-  // Direct match
+  // Direct match (covers /dev/sda1 ↔ sda1, /dev/nvme0n1p1 ↔ nvme0n1p1)
   if (normalized === deviceName) {
     return true
   }
 
-  // LVM volumes use dashes instead of slashes
-  // e.g., ubuntu--vg-ubuntu--lv matches the device name
-  if (fsPath.includes(deviceName)) {
+  // LVM/device-mapper: e.g., /dev/mapper/ubuntu--vg-ubuntu--lv contains "ubuntu--lv"
+  if (fsPath.startsWith('/dev/mapper/') && fsPath.includes(deviceName)) {
     return true
   }
 
   return false
 }
 
-export function determineFileType(filename: string): 'image' | 'pdf' | 'text' | 'unknown' {
+export function determineFileType(filename: string): 'image' | 'pdf' | 'text' | 'epub' | 'zim' | 'unknown' {
   const ext = path.extname(filename).toLowerCase()
   if (['.jpg', '.jpeg', '.png', '.gif', '.bmp', '.tiff', '.webp'].includes(ext)) {
     return 'image'
@@ -160,6 +160,10 @@ export function determineFileType(filename: string): 'image' | 'pdf' | 'text' |
     return 'pdf'
   } else if (['.txt', '.md', '.docx', '.rtf'].includes(ext)) {
     return 'text'
+  } else if (ext === '.epub') {
+    return 'epub'
+  } else if (ext === '.zim') {
+    return 'zim'
   } else {
     return 'unknown'
   }

admin/app/utils/version.ts

@@ -0,0 +1,49 @@
+/**
+ * Compare two semantic version strings to determine if the first is newer than the second.
+ * @param version1 - The version to check (e.g., "1.25.0")
+ * @param version2 - The current version (e.g., "1.24.0")
+ * @returns true if version1 is newer than version2
+ */
+export function isNewerVersion(version1: string, version2: string, includePreReleases = false): boolean {
+  const normalize = (v: string) => v.replace(/^v/, '')
+  const [base1, pre1] = normalize(version1).split('-')
+  const [base2, pre2] = normalize(version2).split('-')
+
+  // If pre-releases are not included and version1 is a pre-release, don't consider it newer
+  if (!includePreReleases && pre1) {
+    return false
+  }
+
+  const v1Parts = base1.split('.').map((p) => parseInt(p, 10) || 0)
+  const v2Parts = base2.split('.').map((p) => parseInt(p, 10) || 0)
+
+  const maxLen = Math.max(v1Parts.length, v2Parts.length)
+  for (let i = 0; i < maxLen; i++) {
+    const a = v1Parts[i] || 0
+    const b = v2Parts[i] || 0
+    if (a > b) return true
+    if (a < b) return false
+  }
+
+  // Base versions equal — GA > RC, RC.n+1 > RC.n
+  if (!pre1 && pre2) return true // v1 is GA, v2 is RC → v1 is newer
+  if (pre1 && !pre2) return false // v1 is RC, v2 is GA → v2 is newer
+  if (!pre1 && !pre2) return false // both GA, equal
+
+  // Both prerelease: compare numeric suffix (e.g. "rc.2" vs "rc.1")
+  const pre1Num = parseInt(pre1.split('.')[1], 10) || 0
+  const pre2Num = parseInt(pre2.split('.')[1], 10) || 0
+  return pre1Num > pre2Num
+}
+
+/**
+ * Parse the major version number from a tag string.
+ * Strips the 'v' prefix if present.
+ * @param tag - Version tag (e.g., "v3.8.1", "10.19.4")
+ * @returns The major version number
+ */
+export function parseMajorVersion(tag: string): number {
+  const normalized = tag.replace(/^v/, '')
+  const major = parseInt(normalized.split('.')[0], 10)
+  return isNaN(major) ? 0 : major
+}

admin/app/validators/common.ts

@@ -1,12 +1,41 @@
 import vine from '@vinejs/vine'
 
+/**
+ * Checks whether a URL points to a loopback or link-local address.
+ * Used to prevent SSRF — the server should not fetch from localhost
+ * or link-local/metadata endpoints (e.g. cloud instance metadata at 169.254.x.x).
+ *
+ * RFC1918 private ranges (10.x, 172.16-31.x, 192.168.x) are intentionally
+ * ALLOWED because NOMAD is a LAN appliance and users may host content
+ * mirrors on their local network.
+ *
+ * Throws an error if the URL is a loopback or link-local address.
+ */
+export function assertNotPrivateUrl(urlString: string): void {
+  const parsed = new URL(urlString)
+  const hostname = parsed.hostname.toLowerCase()
+
+  const blockedPatterns = [
+    /^localhost$/,
+    /^127\.\d+\.\d+\.\d+$/,
+    /^0\.0\.0\.0$/,
+    /^169\.254\.\d+\.\d+$/, // Link-local / cloud metadata
+    /^\[::1\]$/,
+    /^\[?fe80:/i, // IPv6 link-local
+    /^\[::ffff:/i, // IPv4-mapped IPv6 (e.g. [::ffff:7f00:1] = 127.0.0.1)
+    /^\[::\]$/, // IPv6 all-zeros (equivalent to 0.0.0.0)
+  ]
+
+  if (blockedPatterns.some((re) => re.test(hostname))) {
+    throw new Error(`Download URL must not point to a loopback or link-local address: ${hostname}`)
+  }
+}
+
 export const remoteDownloadValidator = vine.compile(
   vine.object({
     url: vine
       .string()
-      .url({
-        require_tld: false, // Allow local URLs
-      })
+      .url({ require_tld: false }) // Allow LAN URLs (e.g. http://my-nas:8080/file.zim)
       .trim(),
   })
 )
@@ -15,9 +44,7 @@ export const remoteDownloadWithMetadataValidator = vine.compile(
   vine.object({
     url: vine
       .string()
-      .url({
-        require_tld: false, // Allow local URLs
-      })
+      .url({ require_tld: false }) // Allow LAN URLs
       .trim(),
     metadata: vine
       .object({
@@ -34,9 +61,7 @@ export const remoteDownloadValidatorOptional = vine.compile(
   vine.object({
     url: vine
       .string()
-      .url({
-        require_tld: false, // Allow local URLs
-      })
+      .url({ require_tld: false }) // Allow LAN URLs
       .trim()
       .optional(),
   })
@@ -56,7 +81,7 @@ export const downloadCollectionValidator = vine.compile(
   })
 )
 
-export const saveInstalledTierValidator = vine.compile(
+export const downloadCategoryTierValidator = vine.compile(
   vine.object({
     categorySlug: vine.string().trim().minLength(1),
     tierSlug: vine.string().trim().minLength(1),
@@ -68,3 +93,21 @@ export const selectWikipediaValidator = vine.compile(
     optionId: vine.string().trim().minLength(1),
   })
 )
+
+const resourceUpdateInfoBase = vine.object({
+  resource_id: vine.string().trim().minLength(1),
+  resource_type: vine.enum(['zim', 'map'] as const),
+  installed_version: vine.string().trim(),
+  latest_version: vine.string().trim().minLength(1),
+  download_url: vine.string().url({ require_tld: false }).trim(),
+})
+
+export const applyContentUpdateValidator = vine.compile(resourceUpdateInfoBase)
+
+export const applyAllContentUpdatesValidator = vine.compile(
+  vine.object({
+    updates: vine
+      .array(resourceUpdateInfoBase)
+      .minLength(1),
+  })
+)

admin/app/validators/curated_collections.ts

@@ -1,30 +1,20 @@
 import vine from '@vinejs/vine'
 
-export const curatedCollectionResourceValidator = vine.object({
+// ---- Versioned resource validators (with id + version) ----
+
+export const specResourceValidator = vine.object({
+  id: vine.string(),
+  version: vine.string(),
   title: vine.string(),
   description: vine.string(),
   url: vine.string().url(),
   size_mb: vine.number().min(0).optional(),
 })
 
-export const curatedCollectionValidator = vine.object({
-  slug: vine.string(),
-  name: vine.string(),
-  description: vine.string(),
-  icon: vine.string(),
-  language: vine.string().minLength(2).maxLength(5),
-  resources: vine.array(curatedCollectionResourceValidator).minLength(1),
-})
+// ---- ZIM Categories spec (versioned) ----
 
-export const curatedCollectionsFileSchema = vine.object({
-  collections: vine.array(curatedCollectionValidator).minLength(1),
-})
-
-/**
- * For validating the categories file, which has a different structure than the collections file
- * since it includes tiers within each category. 
- */
-export const curatedCategoriesFileSchema = vine.object({
+export const zimCategoriesSpecSchema = vine.object({
+  spec_version: vine.string(),
   categories: vine.array(
     vine.object({
       name: vine.string(),
@@ -39,16 +29,47 @@ export const curatedCategoriesFileSchema = vine.object({
           description: vine.string(),
           recommended: vine.boolean().optional(),
           includesTier: vine.string().optional(),
-          resources: vine.array(curatedCollectionResourceValidator),
+          resources: vine.array(specResourceValidator),
         })
       ),
     })
   ),
 })
 
-/**
- * For validating the Wikipedia options file
- */
+// ---- Maps spec (versioned) ----
+
+export const mapsSpecSchema = vine.object({
+  spec_version: vine.string(),
+  collections: vine.array(
+    vine.object({
+      slug: vine.string(),
+      name: vine.string(),
+      description: vine.string(),
+      icon: vine.string(),
+      language: vine.string().minLength(2).maxLength(5),
+      resources: vine.array(specResourceValidator).minLength(1),
+    })
+  ).minLength(1),
+})
+
+// ---- Wikipedia spec (versioned) ----
+
+export const wikipediaSpecSchema = vine.object({
+  spec_version: vine.string(),
+  options: vine.array(
+    vine.object({
+      id: vine.string(),
+      name: vine.string(),
+      description: vine.string(),
+      size_mb: vine.number().min(0),
+      url: vine.string().url().nullable(),
+      version: vine.string().nullable(),
+    })
+  ).minLength(1),
+})
+
+// ---- Wikipedia validators (used by ZimService) ----
+
 export const wikipediaOptionSchema = vine.object({
   id: vine.string(),
   name: vine.string(),

admin/app/validators/ollama.ts

@@ -10,6 +10,7 @@ export const chatSchema = vine.compile(
       })
     ),
     stream: vine.boolean().optional(),
+    sessionId: vine.number().positive().optional(),
   })
 )
 
@@ -17,5 +18,8 @@ export const getAvailableModelsSchema = vine.compile(
   vine.object({
     sort: vine.enum(['pulls', 'name'] as const).optional(),
     recommendedOnly: vine.boolean().optional(),
+    query: vine.string().trim().optional(),
+    limit: vine.number().positive().optional(),
+    force: vine.boolean().optional(),
   })
 )

admin/app/validators/rag.ts

@@ -5,3 +5,9 @@ export const getJobStatusSchema = vine.compile(
     filePath: vine.string(),
   })
 )
+
+export const deleteFileSchema = vine.compile(
+  vine.object({
+    source: vine.string(),
+  })
+)

admin/app/validators/settings.ts

@@ -1,8 +1,11 @@
 import vine from "@vinejs/vine";
 import { SETTINGS_KEYS } from "../../constants/kv_store.js";
 
+export const getSettingSchema = vine.compile(vine.object({
+    key: vine.enum(SETTINGS_KEYS),
+}))
 
 export const updateSettingSchema = vine.compile(vine.object({
     key: vine.enum(SETTINGS_KEYS),
-    value: vine.any(),
+    value: vine.any().optional(),
 }))

admin/app/validators/system.ts

@@ -18,3 +18,16 @@ export const subscribeToReleaseNotesValidator = vine.compile(
     email: vine.string().email().trim(),
   })
 )
+
+export const checkLatestVersionValidator = vine.compile(
+  vine.object({
+    force: vine.boolean().optional(), // Optional flag to force bypassing cache and checking for updates immediately
+  })
+)
+
+export const updateServiceValidator = vine.compile(
+  vine.object({
+    service_name: vine.string().trim(),
+    target_version: vine.string().trim(),
+  })
+)

admin/bin/server.ts

@@ -36,6 +36,15 @@ new Ignitor(APP_ROOT, { importer: IMPORTER })
     })
     app.listen('SIGTERM', () => app.terminate())
     app.listenIf(app.managedByPm2, 'SIGINT', () => app.terminate())
+    app.ready(async () => {
+      try {
+        const collectionManifestService = new (await import('#services/collection_manifest_service')).CollectionManifestService()
+        await collectionManifestService.reconcileFromFilesystem()
+      } catch (error) {
+        // Catch and log any errors during reconciliation to prevent the server from crashing
+        console.error('Error during collection manifest reconciliation:', error)
+      }
+    })
   })
   .httpServer()
   .start()

admin/commands/queue/work.ts

@@ -6,6 +6,8 @@ import { RunDownloadJob } from '#jobs/run_download_job'
 import { DownloadModelJob } from '#jobs/download_model_job'
 import { RunBenchmarkJob } from '#jobs/run_benchmark_job'
 import { EmbedFileJob } from '#jobs/embed_file_job'
+import { CheckUpdateJob } from '#jobs/check_update_job'
+import { CheckServiceUpdatesJob } from '#jobs/check_service_updates_job'
 
 export default class QueueWork extends BaseCommand {
   static commandName = 'queue:work'
@@ -59,12 +61,34 @@ export default class QueueWork extends BaseCommand {
         {
           connection: queueConfig.connection,
           concurrency: this.getConcurrencyForQueue(queueName),
+          lockDuration: 300000,
           autorun: true,
         }
       )
 
-      worker.on('failed', (job, err) => {
+      // Required to prevent Node from treating BullMQ internal errors as unhandled
+      // EventEmitter errors that crash the process.
+      worker.on('error', (err) => {
+        this.logger.error(`[${queueName}] Worker error: ${err.message}`)
+      })
+
+      worker.on('failed', async (job, err) => {
         this.logger.error(`[${queueName}] Job failed: ${job?.id}, Error: ${err.message}`)
+
+        // If this was a Wikipedia download, mark it as failed in the DB
+        if (job?.data?.filetype === 'zim' && job?.data?.url?.includes('wikipedia_en_')) {
+          try {
+            const { DockerService } = await import('#services/docker_service')
+            const { ZimService } = await import('#services/zim_service')
+            const dockerService = new DockerService()
+            const zimService = new ZimService(dockerService)
+            await zimService.onWikipediaDownloadComplete(job.data.url, false)
+          } catch (e: any) {
+            this.logger.error(
+              `[${queueName}] Failed to update Wikipedia status: ${e.message}`
+            )
+          }
+        }
       })
 
       worker.on('completed', (job) => {
@@ -75,6 +99,19 @@ export default class QueueWork extends BaseCommand {
       this.logger.info(`Worker started for queue: ${queueName}`)
     }
 
+    // Schedule nightly update checks (idempotent, will persist over restarts)
+    await CheckUpdateJob.scheduleNightly()
+    await CheckServiceUpdatesJob.scheduleNightly()
+
+    // Safety net: log unhandled rejections instead of crashing the worker process.
+    // Individual job errors are already caught by BullMQ; this catches anything that
+    // escapes (e.g. a fire-and-forget promise in a callback that rejects unexpectedly).
+    process.on('unhandledRejection', (reason) => {
+      this.logger.error(
+        `Unhandled promise rejection in worker process: ${reason instanceof Error ? reason.message : String(reason)}`
+      )
+    })
+
     // Graceful shutdown for all workers
     process.on('SIGTERM', async () => {
       this.logger.info('SIGTERM received. Shutting down workers...')
@@ -92,11 +129,15 @@ export default class QueueWork extends BaseCommand {
     handlers.set(DownloadModelJob.key, new DownloadModelJob())
     handlers.set(RunBenchmarkJob.key, new RunBenchmarkJob())
     handlers.set(EmbedFileJob.key, new EmbedFileJob())
+    handlers.set(CheckUpdateJob.key, new CheckUpdateJob())
+    handlers.set(CheckServiceUpdatesJob.key, new CheckServiceUpdatesJob())
 
     queues.set(RunDownloadJob.key, RunDownloadJob.queue)
     queues.set(DownloadModelJob.key, DownloadModelJob.queue)
     queues.set(RunBenchmarkJob.key, RunBenchmarkJob.queue)
     queues.set(EmbedFileJob.key, EmbedFileJob.queue)
+    queues.set(CheckUpdateJob.key, CheckUpdateJob.queue)
+    queues.set(CheckServiceUpdatesJob.key, CheckServiceUpdatesJob.queue)
 
     return [handlers, queues]
   }
@@ -111,6 +152,7 @@ export default class QueueWork extends BaseCommand {
       [DownloadModelJob.queue]: 2, // Lower concurrency for resource-intensive model downloads
       [RunBenchmarkJob.queue]: 1, // Run benchmarks one at a time for accurate results
       [EmbedFileJob.queue]: 2, // Lower concurrency for embedding jobs, can be resource intensive
+      [CheckUpdateJob.queue]: 1, // No need to run more than one update check at a time
       default: 3,
     }
 

admin/config/bodyparser.ts

@@ -47,7 +47,7 @@ const bodyParserConfig = defineConfig({
      * Maximum limit of data to parse including all files
      * and fields
      */
-    limit: '20mb',
+    limit: '110mb', // Set to 110MB to allow for some overhead beyond the 100MB file size limit
     types: ['multipart/form-data'],
   },
 })

admin/config/database.ts

@@ -13,7 +13,12 @@ const dbConfig = defineConfig({
         user: env.get('DB_USER'),
         password: env.get('DB_PASSWORD'),
         database: env.get('DB_DATABASE'),
-        ssl: env.get('DB_SSL') ?? true, // Default to true
+        ssl: env.get('DB_SSL') ? {} : false,
+      },
+      pool: {
+        min: 2,
+        max: 15,
+        acquireTimeoutMillis: 10000, // Fail fast (10s) instead of silently hanging for ~60s
       },
       migrations: {
         naturalSort: true,

admin/config/inertia.ts

@@ -1,7 +1,14 @@
+import KVStore from '#models/kv_store'
 import { SystemService } from '#services/system_service'
 import { defineConfig } from '@adonisjs/inertia'
 import type { InferSharedProps } from '@adonisjs/inertia/types'
 
+let _assistantNameCache: { value: string; expiresAt: number } | null = null
+
+export function invalidateAssistantNameCache() {
+  _assistantNameCache = null
+}
+
 const inertiaConfig = defineConfig({
   /**
    * Path to the Edge view that will be used as the root view for Inertia responses
@@ -14,6 +21,16 @@ const inertiaConfig = defineConfig({
   sharedData: {
     appVersion: () => SystemService.getAppVersion(),
     environment: process.env.NODE_ENV || 'production',
+    aiAssistantName: async () => {
+      const now = Date.now()
+      if (_assistantNameCache && now < _assistantNameCache.expiresAt) {
+        return _assistantNameCache.value
+      }
+      const customName = await KVStore.getValue('ai.assistantCustomName')
+      const value = (customName && customName.trim()) ? customName : 'AI Assistant'
+      _assistantNameCache = { value, expiresAt: now + 60_000 }
+      return value
+    },
   },
 
   /**

admin/config/logger.ts

@@ -13,12 +13,12 @@ const loggerConfig = defineConfig({
     app: {
       enabled: true,
       name: env.get('APP_NAME'),
-      level: env.get('LOG_LEVEL'),
+      level: env.get('NODE_ENV') === 'production' ? env.get('LOG_LEVEL') : 'debug', // default to 'debug' in non-production envs
       transport: {
         targets:
           targets()
             .pushIf(!app.inProduction, targets.pretty())
-            .pushIf(app.inProduction, targets.file({ destination: "/app/storage/logs/admin.log" }))
+            .pushIf(app.inProduction, targets.file({ destination: "/app/storage/logs/admin.log", mkdir: true }))
             .toArray(),
       },
     },

admin/config/transmit.ts

@@ -1,6 +1,14 @@
+import env from '#start/env'
 import { defineConfig } from '@adonisjs/transmit'
+import { redis } from '@adonisjs/transmit/transports'
 
 export default defineConfig({
-  pingInterval: false,
-  transport: null
+  pingInterval: '30s',
+  transport: {
+    driver: redis({
+      host: env.get('REDIS_HOST'),
+      port: env.get('REDIS_PORT'),
+      keyPrefix: 'transmit:',
+    })
+  }
 })

admin/constants/broadcast.ts

@@ -0,0 +1,7 @@
+
+export const BROADCAST_CHANNELS = {
+    BENCHMARK_PROGRESS: 'benchmark-progress',
+    OLLAMA_MODEL_DOWNLOAD: 'ollama-model-download',
+    SERVICE_INSTALLATION: 'service-installation',
+    SERVICE_UPDATES: 'service-updates',
+}

admin/constants/kiwix.ts

@@ -0,0 +1,2 @@
+
+export const KIWIX_LIBRARY_CMD = '--library /data/kiwix-library.xml --monitorLibrary --address=all'

admin/constants/kv_store.ts

@@ -1,3 +1,3 @@
 import { KVStoreKey } from "../types/kv_store.js";
 
-export const SETTINGS_KEYS: KVStoreKey[] = ['chat.suggestionsEnabled'];
+export const SETTINGS_KEYS: KVStoreKey[] = ['chat.suggestionsEnabled', 'chat.lastModel', 'ui.hasVisitedEasySetup', 'ui.theme', 'system.earlyAccess', 'ai.assistantCustomName', 'ai.remoteOllamaUrl', 'ai.ollamaFlashAttention'];

admin/constants/misc.ts

@@ -0,0 +1,2 @@
+
+export const NOMAD_API_DEFAULT_BASE_URL = 'https://api.projectnomad.us'

admin/constants/ollama.ts

@@ -18,6 +18,8 @@ export const FALLBACK_RECOMMENDED_OLLAMA_MODELS: NomadOllamaModel[] = [
         size: '5.1 GB',
         context: '128k',
         input: 'Text',
+        cloud: false,
+        thinking: false
       },
     ],
   },
@@ -35,6 +37,8 @@ export const FALLBACK_RECOMMENDED_OLLAMA_MODELS: NomadOllamaModel[] = [
         size: '1.1 GB',
         context: '128k',
         input: 'Text',
+        cloud: false,
+        thinking: true
       },
     ],
   },
@@ -51,11 +55,25 @@ export const FALLBACK_RECOMMENDED_OLLAMA_MODELS: NomadOllamaModel[] = [
         size: '581 MB',
         context: '128k',
         input: 'Text',
+        cloud: false,
+        thinking: false
       },
     ],
   },
 ]
 
+export const DEFAULT_QUERY_REWRITE_MODEL = 'qwen2.5:3b' // default to qwen2.5 for query rewriting with good balance of text task performance and resource usage
+
+/**
+ * Adaptive RAG context limits based on model size.
+ * Smaller models get overwhelmed with too much context, so we cap it.
+ */
+export const RAG_CONTEXT_LIMITS: { maxParams: number; maxResults: number; maxTokens: number }[] = [
+  { maxParams: 3, maxResults: 2, maxTokens: 1000 },   // 1-3B models
+  { maxParams: 8, maxResults: 4, maxTokens: 2500 },   // 4-8B models
+  { maxParams: Infinity, maxResults: 5, maxTokens: 0 }, // 13B+ (no cap)
+]
+
 export const SYSTEM_PROMPTS = {
   default: `
  Format all responses using markdown for better readability. Vanilla markdown or GitHub-flavored markdown is preferred.
@@ -75,9 +93,9 @@ IMPORTANT INSTRUCTIONS:
 1. If the user's question is directly related to the context above, use this information to provide accurate, detailed answers.
 2. Always cite or reference the context when using it (e.g., "According to the information available..." or "Based on the knowledge base...").
 3. If the context is only partially relevant, combine it with your general knowledge but be clear about what comes from the knowledge base.
-4. If the context is not relevant to the user's question, you can respond using your general knowledge without forcing the context into your answer.
+4. If the context is not relevant to the user's question, you can respond using your general knowledge without forcing the context into your answer. Do not mention the context if it's not relevant.
 5. Never fabricate information that isn't in the context or your training data.
-6. If you're unsure or the context doesn't contain enough information, acknowledge the limitations.
+6. If you're unsure or you don't have enough information to answer the user's question, acknowledge the limitations.
 
 Format your response using markdown for readability.
 `,
@@ -104,5 +122,36 @@ The suggestions should be in title case.
 Ensure that your suggestions are comma-seperated with no conjunctions like "and" or "or".
 Do not use line breaks, new lines, or extra spacing to separate the suggestions.
 Format: suggestion1, suggestion2, suggestion3
+`,
+  title_generation: `You are a title generator. Given the start of a conversation, generate a concise, descriptive title under 50 characters. Return ONLY the title text with no quotes, punctuation wrapping, or extra formatting.`,
+  query_rewrite: `
+You are a query rewriting assistant. Your task is to reformulate the user's latest question to include relevant context from the conversation history.
+
+Given the conversation history, rewrite the user's latest question to be a standalone, context-aware search query that will retrieve the most relevant information.
+
+Rules:
+1. Keep the rewritten query concise (under 150 words)
+2. Include key entities, topics, and context from previous messages
+3. Make it a clear, searchable query
+4. Do NOT answer the question - only rewrite the user's query to be more effective for retrieval
+5. Output ONLY the rewritten query, nothing else
+
+Examples:
+
+Conversation:
+User: "How do I install Gentoo?"
+Assistant: [detailed installation guide]
+User: "Is an internet connection required to install?"
+
+Rewritten Query: "Is an internet connection required to install Gentoo Linux?"
+
+---
+
+Conversation:
+User: "What's the best way to preserve meat?"
+Assistant: [preservation methods]
+User: "How long does it last?"
+
+Rewritten Query: "How long does preserved meat last using curing or smoking methods?"
 `,
 }

admin/constants/zim_extraction.ts

@@ -0,0 +1,48 @@
+
+export const HTML_SELECTORS_TO_REMOVE = [
+    'script',
+    'style',
+    'nav',
+    'header',
+    'footer',
+    'noscript',
+    'iframe',
+    'svg',
+    '.navbox',
+    '.sidebar',
+    '.infobox',
+    '.mw-editsection',
+    '.reference',
+    '.reflist',
+    '.toc',
+    '.noprint',
+    '.mw-jump-link',
+    '.mw-headline-anchor',
+    '[role="navigation"]',
+    '.navbar',
+    '.hatnote',
+    '.ambox',
+    '.sistersitebox',
+    '.portal',
+    '#coordinates',
+    '.geo-nondefault',
+    '.authority-control',
+]
+
+// Common heading names that usually don't have meaningful content under them
+export const NON_CONTENT_HEADING_PATTERNS = [
+    /^see also$/i,
+    /^references$/i,
+    /^external links$/i,
+    /^further reading$/i,
+    /^notes$/i,
+    /^bibliography$/i,
+    /^navigation$/i,
+]
+
+/**
+ * Batch size for processing ZIM articles to prevent lock timeout errors.
+ * Processing 50 articles at a time balances throughput with job duration.
+ * Typical processing time: 2-5 minutes per batch depending on article complexity.
+ */
+export const ZIM_BATCH_SIZE = 50

admin/database/migrations/1770269324176_add_unique_constraint_to_curated_collection_resources_table.ts

@@ -0,0 +1,19 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+export default class extends BaseSchema {
+  protected tableName = 'curated_collection_resources'
+
+  async up() {
+    this.schema.alterTable(this.tableName, (table) => {
+      table.unique(['curated_collection_slug', 'url'], {
+        indexName: 'curated_collection_resources_unique',
+      })
+    })
+  }
+
+  async down() {
+    this.schema.alterTable(this.tableName, (table) => {
+      table.dropUnique(['curated_collection_slug', 'url'], 'curated_collection_resources_unique')
+    })
+  }
+}

admin/database/migrations/1770273423670_drop_installed_tiers_table.ts

@@ -0,0 +1,20 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+export default class extends BaseSchema {
+  protected tableName = 'installed_tiers'
+
+  async up() {
+    this.schema.dropTableIfExists(this.tableName)
+  }
+
+  async down() {
+    // Recreate the table if we need to rollback
+    this.schema.createTable(this.tableName, (table) => {
+      table.increments('id')
+      table.string('category_slug').notNullable().unique()
+      table.string('tier_slug').notNullable()
+      table.timestamp('created_at', { useTz: true })
+      table.timestamp('updated_at', { useTz: true })
+    })
+  }
+}

admin/database/migrations/1770849108030_create_create_collection_manifests_table.ts

@@ -0,0 +1,18 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+export default class extends BaseSchema {
+  protected tableName = 'collection_manifests'
+
+  async up() {
+    this.schema.createTable(this.tableName, (table) => {
+      table.string('type').primary() // 'zim_categories' | 'maps' | 'wikipedia'
+      table.string('spec_version').notNullable()
+      table.json('spec_data').notNullable()
+      table.timestamp('fetched_at').notNullable()
+    })
+  }
+
+  async down() {
+    this.schema.dropTable(this.tableName)
+  }
+}

admin/database/migrations/1770849119787_create_create_installed_resources_table.ts

@@ -0,0 +1,25 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+export default class extends BaseSchema {
+  protected tableName = 'installed_resources'
+
+  async up() {
+    this.schema.createTable(this.tableName, (table) => {
+      table.increments('id').primary()
+      table.string('resource_id').notNullable()
+      table.enum('resource_type', ['zim', 'map']).notNullable()
+      table.string('collection_ref').nullable()
+      table.string('version').notNullable()
+      table.string('url').notNullable()
+      table.string('file_path').notNullable()
+      table.bigInteger('file_size_bytes').nullable()
+      table.timestamp('installed_at').notNullable()
+
+      table.unique(['resource_id', 'resource_type'])
+    })
+  }
+
+  async down() {
+    this.schema.dropTable(this.tableName)
+  }
+}

admin/database/migrations/1770850092871_create_drop_legacy_curated_tables_table.ts

@@ -0,0 +1,13 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+export default class extends BaseSchema {
+  async up() {
+    this.schema.dropTableIfExists('curated_collection_resources')
+    this.schema.dropTableIfExists('curated_collections')
+    this.schema.dropTableIfExists('zim_file_metadata')
+  }
+
+  async down() {
+    // These tables are legacy and intentionally not recreated
+  }
+}

admin/database/migrations/1771000000001_add_update_fields_to_services.ts

@@ -0,0 +1,21 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+export default class extends BaseSchema {
+  protected tableName = 'services'
+
+  async up() {
+    this.schema.alterTable(this.tableName, (table) => {
+      table.string('source_repo', 255).nullable()
+      table.string('available_update_version', 50).nullable()
+      table.timestamp('update_checked_at').nullable()
+    })
+  }
+
+  async down() {
+    this.schema.alterTable(this.tableName, (table) => {
+      table.dropColumn('source_repo')
+      table.dropColumn('available_update_version')
+      table.dropColumn('update_checked_at')
+    })
+  }
+}

admin/database/migrations/1771000000002_pin_latest_service_images.ts

@@ -0,0 +1,61 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+export default class extends BaseSchema {
+  protected tableName = 'services'
+
+  async up() {
+    this.defer(async (db) => {
+      // Pin :latest images to specific versions
+      await db
+        .from(this.tableName)
+        .where('container_image', 'ghcr.io/gchq/cyberchef:latest')
+        .update({ container_image: 'ghcr.io/gchq/cyberchef:10.19.4' })
+
+      await db
+        .from(this.tableName)
+        .where('container_image', 'dullage/flatnotes:latest')
+        .update({ container_image: 'dullage/flatnotes:v5.5.4' })
+
+      await db
+        .from(this.tableName)
+        .where('container_image', 'treehouses/kolibri:latest')
+        .update({ container_image: 'treehouses/kolibri:0.12.8' })
+
+      // Populate source_repo for services whose images lack the OCI source label
+      const sourceRepos: Record<string, string> = {
+        nomad_kiwix_server: 'https://github.com/kiwix/kiwix-tools',
+        nomad_ollama: 'https://github.com/ollama/ollama',
+        nomad_qdrant: 'https://github.com/qdrant/qdrant',
+        nomad_cyberchef: 'https://github.com/gchq/CyberChef',
+        nomad_flatnotes: 'https://github.com/dullage/flatnotes',
+        nomad_kolibri: 'https://github.com/learningequality/kolibri',
+      }
+
+      for (const [serviceName, repoUrl] of Object.entries(sourceRepos)) {
+        await db
+          .from(this.tableName)
+          .where('service_name', serviceName)
+          .update({ source_repo: repoUrl })
+      }
+    })
+  }
+
+  async down() {
+    this.defer(async (db) => {
+      await db
+        .from(this.tableName)
+        .where('container_image', 'ghcr.io/gchq/cyberchef:10.19.4')
+        .update({ container_image: 'ghcr.io/gchq/cyberchef:latest' })
+
+      await db
+        .from(this.tableName)
+        .where('container_image', 'dullage/flatnotes:v5.5.4')
+        .update({ container_image: 'dullage/flatnotes:latest' })
+
+      await db
+        .from(this.tableName)
+        .where('container_image', 'treehouses/kolibri:0.12.8')
+        .update({ container_image: 'treehouses/kolibri:latest' })
+    })
+  }
+}

admin/database/migrations/1771100000001_migrate_kiwix_to_library_mode.ts

@@ -0,0 +1,29 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+export default class extends BaseSchema {
+  protected tableName = 'services'
+
+  async up() {
+    this.defer(async (db) => {
+      await db
+        .from(this.tableName)
+        .where('service_name', 'nomad_kiwix_server')
+        .whereRaw('`container_command` LIKE ?', ['%*.zim%'])
+        .update({
+          container_command: '--library /data/kiwix-library.xml --monitorLibrary --address=all',
+        })
+    })
+  }
+
+  async down() {
+    this.defer(async (db) => {
+      await db
+        .from(this.tableName)
+        .where('service_name', 'nomad_kiwix_server')
+        .where('container_command', '--library /data/kiwix-library.xml --monitorLibrary --address=all')
+        .update({
+          container_command: '*.zim --address=all',
+        })
+    })
+  }
+}

admin/database/migrations/1771200000001_create_map_markers_table.ts

@@ -0,0 +1,25 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+export default class extends BaseSchema {
+  protected tableName = 'map_markers'
+
+  async up() {
+    this.schema.createTable(this.tableName, (table) => {
+      table.increments('id')
+      table.string('name').notNullable()
+      table.double('longitude').notNullable()
+      table.double('latitude').notNullable()
+      table.string('color', 20).notNullable().defaultTo('orange')
+      table.string('marker_type', 20).notNullable().defaultTo('pin')
+      table.string('route_id').nullable()
+      table.integer('route_order').nullable()
+      table.text('notes').nullable()
+      table.timestamp('created_at')
+      table.timestamp('updated_at')
+    })
+  }
+
+  async down() {
+    this.schema.dropTable(this.tableName)
+  }
+}

admin/database/seeders/service_seeder.ts

@@ -3,6 +3,7 @@ import { BaseSeeder } from '@adonisjs/lucid/seeders'
 import { ModelAttributes } from '@adonisjs/lucid/types/model'
 import env from '#start/env'
 import { SERVICE_NAMES } from '../../constants/service_names.js'
+import { KIWIX_LIBRARY_CMD } from '../../constants/kiwix.js'
 
 export default class ServiceSeeder extends BaseSeeder {
   // Use environment variable with fallback to production default
@@ -12,7 +13,7 @@ export default class ServiceSeeder extends BaseSeeder {
   )
   private static DEFAULT_SERVICES: Omit<
     ModelAttributes<Service>,
-    'created_at' | 'updated_at' | 'metadata' | 'id'
+    'created_at' | 'updated_at' | 'metadata' | 'id' | 'available_update_version' | 'update_checked_at'
   >[] = [
     {
       service_name: SERVICE_NAMES.KIWIX,
@@ -23,7 +24,8 @@ export default class ServiceSeeder extends BaseSeeder {
         'Offline access to Wikipedia, medical references, how-to guides, and encyclopedias',
       icon: 'IconBooks',
       container_image: 'ghcr.io/kiwix/kiwix-serve:3.8.1',
-      container_command: '*.zim --address=all',
+      source_repo: 'https://github.com/kiwix/kiwix-tools',
+      container_command: KIWIX_LIBRARY_CMD,
       container_config: JSON.stringify({
         HostConfig: {
           RestartPolicy: { Name: 'unless-stopped' },
@@ -46,6 +48,7 @@ export default class ServiceSeeder extends BaseSeeder {
       description: 'Vector database for storing and searching embeddings',
       icon: 'IconRobot',
       container_image: 'qdrant/qdrant:v1.16',
+      source_repo: 'https://github.com/qdrant/qdrant',
       container_command: null,
       container_config: JSON.stringify({
         HostConfig: {
@@ -68,7 +71,8 @@ export default class ServiceSeeder extends BaseSeeder {
       display_order: 3,
       description: 'Local AI chat that runs entirely on your hardware - no internet required',
       icon: 'IconWand',
-      container_image: 'ollama/ollama:0.15.2',
+      container_image: 'ollama/ollama:0.18.1',
+      source_repo: 'https://github.com/ollama/ollama',
       container_command: 'serve',
       container_config: JSON.stringify({
         HostConfig: {
@@ -91,7 +95,8 @@ export default class ServiceSeeder extends BaseSeeder {
       display_order: 11,
       description: 'Swiss Army knife for data encoding, encryption, and analysis',
       icon: 'IconChefHat',
-      container_image: 'ghcr.io/gchq/cyberchef:latest',
+      container_image: 'ghcr.io/gchq/cyberchef:10.22.1',
+      source_repo: 'https://github.com/gchq/CyberChef',
       container_command: null,
       container_config: JSON.stringify({
         HostConfig: {
@@ -113,7 +118,8 @@ export default class ServiceSeeder extends BaseSeeder {
       display_order: 10,
       description: 'Simple note-taking app with local storage',
       icon: 'IconNotes',
-      container_image: 'dullage/flatnotes:latest',
+      container_image: 'dullage/flatnotes:v5.5.4',
+      source_repo: 'https://github.com/dullage/flatnotes',
       container_command: null,
       container_config: JSON.stringify({
         HostConfig: {
@@ -137,7 +143,8 @@ export default class ServiceSeeder extends BaseSeeder {
       display_order: 2,
       description: 'Interactive learning platform with video courses and exercises',
       icon: 'IconSchool',
-      container_image: 'treehouses/kolibri:latest',
+      container_image: 'treehouses/kolibri:0.12.8',
+      source_repo: 'https://github.com/learningequality/kolibri',
       container_command: null,
       container_config: JSON.stringify({
         HostConfig: {

admin/docs/about.md

@@ -1,5 +1,17 @@
 # About Project N.O.M.A.D.
 
-Project N.O.M.A.D. (Node for Offline Media, Archives, and Data; "Nomad" for short) is a project started in 2025 by Chris Sherwood of [Crosstalk Solutions, LLC.](https://crosstalksolutions.com). The goal of the project is not to create just another utility for storing offline resources, but rather to allow users to run their own ultime "survival computer".
+Project N.O.M.A.D. (Node for Offline Media, Archives, and Data; "Nomad" for short) is a project started in 2025 by Chris Sherwood of [Crosstalk Solutions, LLC](https://crosstalksolutions.com). The goal of the project is not to create just another utility for storing offline resources, but rather to allow users to run their own ultimate "survival computer".
 
-While many similar offline survival computers are designed to be run on bare-minimum, lightweight hardware, Project N.O.M.A.D. is quite the opposite. To install and run the available AI tools, we highly encourage the use of a beefy, GPU-backed device to make the most of your install.
+While many similar offline survival computers are designed to be run on bare-minimum, lightweight hardware, Project N.O.M.A.D. is quite the opposite. To install and run the available AI tools, we highly encourage the use of a beefy, GPU-backed device to make the most of your install. See the [Hardware Guide](https://www.projectnomad.us/hardware) for detailed build recommendations at three price points.
+
+Since its initial release, NOMAD has grown to include built-in AI chat with a Knowledge Base for document-aware responses, a System Benchmark with a community leaderboard, curated content collections with tiered options, and an Easy Setup Wizard to get new users up and running quickly.
+
+Project N.O.M.A.D. is open source, released under the [Apache License 2.0](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/LICENSE).
+
+## Links
+
+- **Website:** [www.projectnomad.us](https://www.projectnomad.us)
+- **Hardware Guide:** [www.projectnomad.us/hardware](https://www.projectnomad.us/hardware)
+- **Discord:** [Join the Community](https://discord.com/invite/crosstalksolutions)
+- **GitHub:** [Crosstalk-Solutions/project-nomad](https://github.com/Crosstalk-Solutions/project-nomad)
+- **Benchmark Leaderboard:** [benchmark.projectnomad.us](https://benchmark.projectnomad.us)

admin/docs/api-reference.md

@@ -0,0 +1,200 @@
+# API Reference
+
+N.O.M.A.D. exposes a REST API for all operations. All endpoints are under `/api/` and return JSON.
+
+---
+
+## Conventions
+
+**Base URL:** `http://<your-server>/api`
+
+**Responses:**
+- Success responses include `{ "success": true }` and an HTTP 2xx status
+- Error responses return the appropriate HTTP status (400, 404, 409, 500) with an error message
+- Long-running operations (downloads, benchmarks, embeddings) return 201 or 202 with a job/benchmark ID for polling
+
+**Async pattern:** Submit a job → receive an ID → poll a status endpoint until complete.
+
+---
+
+## Health
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/health` | Returns `{ "status": "ok" }` |
+
+---
+
+## System
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/system/info` | CPU, memory, disk, and platform info |
+| GET | `/api/system/internet-status` | Check internet connectivity |
+| GET | `/api/system/debug-info` | Detailed debug information |
+| GET | `/api/system/latest-version` | Check for the latest N.O.M.A.D. version |
+| POST | `/api/system/update` | Trigger a system update |
+| GET | `/api/system/update/status` | Get update progress |
+| GET | `/api/system/update/logs` | Get update operation logs |
+| GET | `/api/system/settings` | Get a setting value (query param: `key`) |
+| PATCH | `/api/system/settings` | Update a setting (`{ key, value }`) |
+| POST | `/api/system/subscribe-release-notes` | Subscribe an email to release notes |
+
+### Services
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/system/services` | List all services with status |
+| POST | `/api/system/services/install` | Install a service |
+| POST | `/api/system/services/force-reinstall` | Force reinstall a service |
+| POST | `/api/system/services/affect` | Start, stop, or restart a service (body: `{ name, action }`) |
+| POST | `/api/system/services/check-updates` | Check for available service updates |
+| POST | `/api/system/services/update` | Update a service to a specific version |
+| GET | `/api/system/services/:name/available-versions` | List available versions for a service |
+
+---
+
+## AI Chat
+
+### Models
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/ollama/models` | List available models (supports filtering, sorting, pagination) |
+| GET | `/api/ollama/installed-models` | List locally installed models |
+| POST | `/api/ollama/models` | Download a model (async, returns job) |
+| DELETE | `/api/ollama/models` | Delete an installed model |
+
+### Chat
+
+| Method | Path | Description |
+|--------|------|-------------|
+| POST | `/api/ollama/chat` | Send a chat message. Supports streaming (SSE) and RAG context injection. Body: `{ model, messages, stream?, useRag? }` |
+| GET | `/api/chat/suggestions` | Get suggested chat prompts |
+
+### Remote Ollama
+
+| Method | Path | Description |
+|--------|------|-------------|
+| POST | `/api/ollama/configure-remote` | Configure a remote Ollama or LM Studio instance |
+| GET | `/api/ollama/remote-status` | Check remote Ollama connection status |
+
+### Chat Sessions
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/chat/sessions` | List all chat sessions |
+| POST | `/api/chat/sessions` | Create a new session |
+| GET | `/api/chat/sessions/:id` | Get a session with its messages |
+| PUT | `/api/chat/sessions/:id` | Update session metadata (title, etc.) |
+| DELETE | `/api/chat/sessions/:id` | Delete a session |
+| DELETE | `/api/chat/sessions/all` | Delete all sessions |
+| POST | `/api/chat/sessions/:id/messages` | Add a message to a session |
+
+**Streaming:** The `/api/ollama/chat` endpoint supports Server-Sent Events (SSE) when `stream: true` is passed. Connect using `EventSource` or `fetch` with a streaming reader.
+
+---
+
+## Knowledge Base (RAG)
+
+Upload documents to enable AI-powered retrieval during chat.
+
+| Method | Path | Description |
+|--------|------|-------------|
+| POST | `/api/rag/upload` | Upload a file for embedding (async, 202 response) |
+| GET | `/api/rag/files` | List stored RAG files |
+| DELETE | `/api/rag/files` | Delete a file (query param: `source`) |
+| GET | `/api/rag/active-jobs` | List active embedding jobs |
+| GET | `/api/rag/job-status` | Get status for a specific file embedding job |
+| GET | `/api/rag/failed-jobs` | List failed embedding jobs |
+| DELETE | `/api/rag/failed-jobs` | Clean up failed jobs and delete associated files |
+| POST | `/api/rag/sync` | Scan storage and sync database with filesystem |
+
+---
+
+## ZIM Files (Offline Content)
+
+ZIM files provide offline Wikipedia, books, and other content via Kiwix.
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/zim/list` | List locally stored ZIM files |
+| GET | `/api/zim/list-remote` | List remote ZIM files (paginated, supports search) |
+| GET | `/api/zim/curated-categories` | List curated categories with Essential/Standard/Comprehensive tiers |
+| POST | `/api/zim/download-remote` | Download a remote ZIM file (async) |
+| POST | `/api/zim/download-category-tier` | Download a full category tier |
+| DELETE | `/api/zim/:filename` | Delete a local ZIM file |
+
+### Wikipedia
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/zim/wikipedia` | Get current Wikipedia selection state |
+| POST | `/api/zim/wikipedia/select` | Select a Wikipedia edition and tier |
+
+---
+
+## Maps
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/maps/regions` | List available map regions |
+| GET | `/api/maps/styles` | Get map styles JSON |
+| GET | `/api/maps/curated-collections` | List curated map collections |
+| POST | `/api/maps/fetch-latest-collections` | Fetch latest collection metadata from source |
+| POST | `/api/maps/download-base-assets` | Download base map assets |
+| POST | `/api/maps/download-remote` | Download a remote map file (async) |
+| POST | `/api/maps/download-remote-preflight` | Check download size/info before starting |
+| POST | `/api/maps/download-collection` | Download an entire collection by slug (async) |
+| DELETE | `/api/maps/:filename` | Delete a local map file |
+
+---
+
+## Downloads
+
+Manage background download jobs for maps, ZIM files, and models.
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/downloads/jobs` | List all download jobs |
+| GET | `/api/downloads/jobs/:filetype` | List jobs filtered by type (`zim`, `map`, etc.) |
+| DELETE | `/api/downloads/jobs/:jobId` | Cancel and remove a download job |
+
+---
+
+## Benchmarks
+
+| Method | Path | Description |
+|--------|------|-------------|
+| POST | `/api/benchmark/run` | Run a benchmark (`full`, `system`, or `ai`; can be async) |
+| POST | `/api/benchmark/run/system` | Run system-only benchmark |
+| POST | `/api/benchmark/run/ai` | Run AI-only benchmark |
+| GET | `/api/benchmark/status` | Get current benchmark status (`idle` or `running`) |
+| GET | `/api/benchmark/results` | Get all benchmark results |
+| GET | `/api/benchmark/results/latest` | Get the most recent result |
+| GET | `/api/benchmark/results/:id` | Get a specific result |
+| POST | `/api/benchmark/submit` | Submit a result to the central repository |
+| POST | `/api/benchmark/builder-tag` | Update builder tag metadata for a result |
+| GET | `/api/benchmark/comparison` | Get comparison stats from the repository |
+| GET | `/api/benchmark/settings` | Get benchmark settings |
+| POST | `/api/benchmark/settings` | Update benchmark settings |
+
+---
+
+## Easy Setup & Content Updates
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/easy-setup/curated-categories` | List curated content categories for setup wizard |
+| POST | `/api/manifests/refresh` | Refresh manifest caches (`zim_categories`, `maps`, `wikipedia`) |
+| POST | `/api/content-updates/check` | Check for available collection updates |
+| POST | `/api/content-updates/apply` | Apply a single content update |
+| POST | `/api/content-updates/apply-all` | Apply multiple content updates |
+
+---
+
+## Documentation
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/docs/list` | List all available documentation files |

admin/docs/faq.md

@@ -13,10 +13,12 @@ No — that's the whole point. Once your content is downloaded, everything works
 
 ### What hardware do I need?
 N.O.M.A.D. is designed for capable hardware, especially if you want to use the AI features. Recommended:
-- Modern multi-core CPU
+- Modern multi-core CPU (AMD Ryzen 7 with Radeon graphics is the community sweet spot)
 - 16GB+ RAM (32GB+ for best AI performance)
-- SSD storage (size depends on content — 500GB minimum, 2TB+ recommended)
-- GPU recommended for faster AI responses
+- SSD storage (size depends on content — 500GB minimum, 1TB+ recommended)
+- NVIDIA or AMD GPU recommended for faster AI responses
+
+**For detailed build recommendations at three price points ($150–$1,000+), see the [Hardware Guide](https://www.projectnomad.us/hardware).**
 
 ### How much storage do I need?
 It depends on what you download:
@@ -34,10 +36,12 @@ Start with essentials and add more as needed.
 
 ### How do I add more Wikipedia content?
 1. Go to **Settings** (hamburger menu → Settings)
-2. Click **ZIM Manager**
-3. Browse available content
+2. Click **Content Explorer**
+3. Browse available Wikipedia packages
 4. Click Download on items you want
 
+You can also use the **Content Explorer** to browse all available ZIM content beyond Wikipedia.
+
 ### How do I add more educational courses?
 1. Open **Kolibri**
 2. Sign in as an admin
@@ -48,7 +52,46 @@ Start with essentials and add more as needed.
 Content is as current as when it was last downloaded. Wikipedia snapshots are typically updated monthly. Check the file names or descriptions for dates.
 
 ### Can I add my own files?
-Currently, N.O.M.A.D. uses standard content formats (ZIM files for Kiwix, Kolibri channels for education). Custom content support may be added in future versions.
+Yes — with the Knowledge Base. Upload PDFs, text files, and other documents to the [Knowledge Base](/knowledge-base), and the AI can reference them when answering your questions. This uses semantic search to find relevant information from your uploaded files.
+
+For Kiwix content, N.O.M.A.D. uses standard ZIM files. For educational content, Kolibri uses its own channel format.
+
+### What are curated collection tiers?
+When selecting content in the Easy Setup wizard or Content Explorer, collections are organized into three tiers:
+- **Essential** — Core content for the category (smallest download)
+- **Standard** — Essential plus additional useful content
+- **Comprehensive** — Everything available for the category (largest download)
+
+This helps you balance content coverage against storage usage.
+
+---
+
+## AI Questions
+
+### How do I use the AI chat?
+1. Go to [AI Chat](/chat) from the Command Center
+2. Type your question or request
+3. The AI responds in conversational style
+
+The AI must be installed first — enable it during Easy Setup or install it from the [Apps](/settings/apps) page.
+
+### How do I upload documents to the Knowledge Base?
+1. Go to **[Knowledge Base →](/knowledge-base)**
+2. Upload your documents (PDFs, text files, etc.)
+3. Documents are processed and indexed automatically
+4. Ask questions in AI Chat — the AI will reference your uploaded documents when relevant
+
+You can also remove documents from the Knowledge Base when they're no longer needed.
+
+NOMAD documentation is automatically added to the Knowledge Base when the AI Assistant is installed.
+
+### What is the System Benchmark?
+The System Benchmark tests your hardware performance and generates a NOMAD Score — a weighted composite of CPU, memory, disk, and AI performance. You can create a Builder Tag (a NOMAD-themed identity like "Tactical-Llama-1234") and share your results with the [community leaderboard](https://benchmark.projectnomad.us).
+
+Go to **[System Benchmark →](/settings/benchmark)** to run one.
+
+### What is the Early Access Channel?
+The Early Access Channel lets you opt in to receive release candidate builds with the latest features and improvements before they hit stable releases. You can enable or disable it from **Settings → Check for Updates**. Early access builds may contain bugs — if you prefer stability, stay on the stable channel.
 
 ---
 
@@ -74,10 +117,61 @@ The Maps feature requires downloaded map data. If you see a blank area:
 ### AI responses are slow
 
 Local AI requires significant computing power. To improve speed:
+- **Add a GPU** — An NVIDIA GPU with the NVIDIA Container Toolkit can improve AI speed by 10-20x or more
 - Close other applications on the server
 - Ensure adequate cooling (overheating causes throttling)
 - Consider using a smaller/faster AI model if available
-- Add a GPU if your hardware supports it
+
+### How do I enable GPU acceleration for AI?
+
+N.O.M.A.D. automatically detects NVIDIA GPUs when the NVIDIA Container Toolkit is installed on the host system. To set up GPU acceleration:
+
+1. **Install an NVIDIA GPU** in your server (if not already present)
+2. **Install the NVIDIA Container Toolkit** on the host — follow the [official installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
+3. **Reinstall the AI Assistant** — Go to [Apps](/settings/apps), find AI Assistant, and click **Force Reinstall**
+
+N.O.M.A.D. will detect the GPU during installation and configure the AI to use it automatically. You'll see "NVIDIA container runtime detected" in the installation progress.
+
+**Tip:** Run a [System Benchmark](/settings/benchmark) before and after to see the difference. GPU-accelerated systems typically see 100+ tokens per second vs 10-15 on CPU only.
+
+### I added/changed my GPU but AI is still slow
+
+When you add or swap a GPU, N.O.M.A.D. needs to reconfigure the AI container to use it:
+
+1. Make sure the **NVIDIA Container Toolkit** is installed on the host
+2. Go to **[Apps](/settings/apps)**
+3. Find the **AI Assistant** and click **Force Reinstall**
+
+Force Reinstall recreates the AI container with GPU support enabled. Without this step, the AI continues to run on CPU only.
+
+### I see a "GPU passthrough not working" warning
+
+N.O.M.A.D. checks whether your GPU is actually accessible inside the AI container. If a GPU is detected on the host but isn't working inside the container, you'll see a warning banner on the System Information and AI Settings pages. Click the **"Fix: Reinstall AI Assistant"** button to recreate the container with proper GPU access. This preserves your downloaded AI models.
+
+### AI Chat not available
+
+The AI Chat page requires the AI Assistant to be installed first:
+1. Go to **[Apps](/settings/apps)**
+2. Install the **AI Assistant**
+3. Wait for the installation to complete
+4. The AI Chat will then be accessible from the home screen or [Chat](/chat)
+
+### Knowledge Base upload stuck
+
+If a document upload appears stuck in the Knowledge Base:
+1. Check that the AI Assistant is running in **Settings → Apps**
+2. Large documents take time to process — wait a few minutes
+3. Try uploading a smaller document to verify the system is working
+4. Check **Settings → System** for any error messages
+
+### Benchmark won't submit to leaderboard
+
+To share results with the community leaderboard:
+- You must run a **Full Benchmark** (not System Only or AI Only)
+- The benchmark must include AI results (AI Assistant must be installed and working)
+- Your score must be higher than any previous submission from the same hardware
+
+If submission fails, check the error message for details.
 
 ### "Service unavailable" or connection errors
 
@@ -129,7 +223,7 @@ Yes, while you have internet access. Updates include:
 
 ### How do I update content (Wikipedia, etc.)?
 Content updates are separate from software updates:
-1. Go to **Settings → ZIM Manager**
+1. Go to **Settings → Content Manager** or **Content Explorer**
 2. Check for newer versions of your installed content
 3. Download updated versions as needed
 
@@ -163,7 +257,7 @@ sudo bash /opt/project-nomad/update_nomad.sh
 
 **Uninstall N.O.M.A.D.:**
 ```bash
-curl -fsSL https://raw.githubusercontent.com/Crosstalk-Solutions/project-nomad/refs/heads/master/install/uninstall_nomad.sh -o uninstall_nomad.sh
+curl -fsSL https://raw.githubusercontent.com/Crosstalk-Solutions/project-nomad/refs/heads/main/install/uninstall_nomad.sh -o uninstall_nomad.sh
 sudo bash uninstall_nomad.sh
 ```
 *Warning: This cannot be undone. All data will be deleted.*
@@ -179,17 +273,20 @@ Yes. N.O.M.A.D. runs entirely on your hardware. Your searches, AI conversations,
 By default, N.O.M.A.D. is accessible on your local network. Anyone on the same network can access it. For public networks, consider additional security measures.
 
 ### Does the AI send data anywhere?
-No. The AI runs completely locally. Your conversations are not sent to any external service.
+No. The AI runs completely locally. Your conversations are not sent to any external service. The AI chat is built into the Command Center — there's no separate service to configure.
 
 ---
 
 ## Getting More Help
 
 ### The AI can help
-Try asking Open WebUI for help. The local AI can answer questions about many topics, including technical troubleshooting.
+Try asking a question in [AI Chat](/chat). The local AI can answer questions about many topics, including technical troubleshooting. If you've uploaded NOMAD documentation to the Knowledge Base, it can also help with NOMAD-specific questions.
 
 ### Check the documentation
 You're in the docs now. Use the menu to find specific topics.
 
+### Join the community
+Get help from other NOMAD users on **[Discord](https://discord.com/invite/crosstalksolutions)**.
+
 ### Release Notes
 See what's changed in each version: **[Release Notes](/docs/release-notes)**

admin/docs/getting-started.md

@@ -1,95 +1,34 @@
 # Getting Started with N.O.M.A.D.
 
-This guide will help you install and set up your N.O.M.A.D. server.
+This guide will help you get the most out of your N.O.M.A.D. server.
 
 ---
 
-## Installation
+## Easy Setup Wizard
 
-### System Requirements
-
-N.O.M.A.D. runs on any **Debian-based Linux** system (Ubuntu recommended). The installation is terminal-based, and everything is accessed through a web browser — no desktop environment needed.
-
-**Minimum Specs** (Command Center only):
-- 2 GHz dual-core processor
-- 4 GB RAM
-- 5 GB free storage
-- Internet connection (for initial install)
-
-**Recommended Specs** (with AI features):
-- AMD Ryzen 7 / Intel Core i7 or better
-- 32 GB RAM
-- NVIDIA RTX 3060 or better (more VRAM = larger AI models)
-- 250 GB+ free storage (SSD preferred)
-
-The Command Center itself is lightweight — your hardware requirements depend on which tools and content you choose to install.
-
-### Install N.O.M.A.D.
-
-Open a terminal and run these two commands:
-
-```bash
-curl -fsSL https://raw.githubusercontent.com/Crosstalk-Solutions/project-nomad/refs/heads/master/install/install_nomad.sh -o install_nomad.sh
-```
-
-```bash
-sudo bash install_nomad.sh
-```
-
-That's it. Once the install finishes, open a browser and go to:
-
-- **Same machine:** `http://localhost:8080`
-- **Other devices on your network:** `http://YOUR_SERVER_IP:8080`
-
-### About Internet & Privacy
-
-N.O.M.A.D. is designed for offline use. Internet is only needed:
-- During initial installation
-- When downloading additional content
-
-There is **zero telemetry** — your data stays on your device.
-
-### About Security
-
-N.O.M.A.D. has no built-in authentication — it's designed to be open and accessible. If you expose it on a network, consider using firewall rules to control which ports are accessible.
-
----
-
-## After Installation
-
-### 1. Run the Easy Setup Wizard
-
-If this is your first time using N.O.M.A.D., the Easy Setup wizard will help you:
-- Choose which apps to enable
-- Download map regions for your area
-- Select knowledge collections (Wikipedia, medical references, etc.)
+If this is your first time using N.O.M.A.D., the Easy Setup wizard will help you get everything configured.
 
 **[Launch Easy Setup →](/easy-setup)**
 
+![Easy Setup Wizard — Step 1: Choose your capabilities](/docs/easy-setup-step1.webp)
+
 The wizard walks you through four simple steps:
-1. **Apps** — Choose additional tools like CyberChef or FlatNotes
+1. **Capabilities** — Choose what to enable: Information Library, AI Assistant, Education Platform, Maps, Data Tools, and Notes
 2. **Maps** — Select geographic regions for offline maps
-3. **ZIM Files** — Choose reference collections (Wikipedia, medical, survival guides)
+3. **Content** — Choose curated content collections with Essential, Standard, or Comprehensive tiers
+
+![Content tiers — Essential, Standard, and Comprehensive](/docs/easy-setup-tiers.webp)
 4. **Review** — Confirm your selections and start downloading
 
-### 2. Wait for Downloads to Complete
-
-Depending on what you selected, downloads may take a while. You can:
-- Monitor progress in the Settings area
-- Continue using features that are already installed
-- Leave your server running overnight for large downloads
-
-### 3. Explore Your Content
-
-Once downloads complete, you're ready to go. Your content works offline whenever you need it.
+Depending on what you selected, downloads may take a while. You can monitor progress in the Settings area, continue using features that are already installed, or leave your server running overnight for large downloads.
 
 ---
 
 ## Understanding Your Tools
 
-### Kiwix — Your Offline Library
+### Information Library — Offline Knowledge (Kiwix)
 
-Kiwix stores compressed versions of websites and references that work without internet.
+The Information Library stores compressed versions of websites and references that work without internet.
 
 **What's included:**
 - Full Wikipedia (millions of articles)
@@ -98,15 +37,15 @@ Kiwix stores compressed versions of websites and references that work without in
 - Classic books from Project Gutenberg
 
 **How to use it:**
-1. Click **Kiwix** from the Command Center home screen or [Apps](/settings/apps) page
+1. Click **Information Library** from the Command Center home screen or [Apps](/settings/apps) page
 2. Choose a collection (like Wikipedia)
 3. Search or browse just like the regular website
 
 ---
 
-### Kolibri — Offline Education
+### Education Platform — Offline Courses (Kolibri)
 
-Kolibri provides complete educational courses that work offline.
+The Education Platform provides complete educational courses that work offline.
 
 **What's included:**
 - Khan Academy video courses
@@ -115,7 +54,7 @@ Kolibri provides complete educational courses that work offline.
 - Works for all ages
 
 **How to use it:**
-1. Click **Kolibri** from the Command Center home screen or [Apps](/settings/apps) page
+1. Click **Education Platform** from the Command Center home screen or [Apps](/settings/apps) page
 2. Sign in or create a learner account
 3. Browse courses and start learning
 
@@ -123,28 +62,61 @@ Kolibri provides complete educational courses that work offline.
 
 ---
 
-### Open WebUI — Your AI Assistant
+### AI Assistant — Built-in Chat
 
-Chat with a local AI that runs entirely on your server — no internet needed.
+![AI Chat interface](/docs/ai-chat.webp)
+
+N.O.M.A.D. includes a built-in AI chat interface powered by Ollama. It runs entirely on your server — no internet needed, no data sent anywhere.
 
 **What can it do:**
 - Answer questions on any topic
 - Explain complex concepts simply
 - Help with writing and editing
-- Brainstorm ideas
-- Assist with problem-solving
+- Reference your uploaded documents via the Knowledge Base
+- Brainstorm ideas and assist with problem-solving
 
 **How to use it:**
-1. Click **Open WebUI** from the Command Center home screen or [Apps](/settings/apps) page
+1. Click **AI Chat** from the Command Center or go to [Chat](/chat)
 2. Type your question or request
 3. The AI responds in conversational style
 
 **Tip:** Be specific in your questions. Instead of "tell me about plants," try "what vegetables grow well in shade?"
 
+**Note:** The AI Assistant must be installed first. Enable it during Easy Setup or install it from the [Apps](/settings/apps) page.
+
+**GPU Acceleration:** If your server has an NVIDIA GPU with the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) installed, N.O.M.A.D. will automatically use it for AI — dramatically faster responses (10-20x improvement). If you add a GPU later, go to [Apps](/settings/apps) and **Force Reinstall** the AI Assistant to enable it.
+
+---
+
+### Knowledge Base — Document-Aware AI
+
+![Knowledge Base upload interface](/docs/knowledge-base.webp)
+
+The Knowledge Base lets you upload documents so the AI can reference them when answering your questions. It uses semantic search (RAG via Qdrant) to find relevant information from your uploaded files.
+
+**Supported file types:**
+- PDFs, text files, and other document formats
+- NOMAD documentation is automatically loaded when the AI Assistant is installed
+
+**How to use it:**
+1. Go to **[Knowledge Base →](/knowledge-base)**
+2. Upload your documents (PDFs, text files, etc.)
+3. Documents are processed and indexed automatically
+4. Ask questions in AI Chat — the AI will reference your uploaded documents when relevant
+5. Remove documents you no longer need — they'll be deleted from the index and local storage
+
+**Use cases:**
+- Upload emergency plans for quick reference during a crisis
+- Load technical manuals and SOPs for offline work sites
+- Add curriculum guides for homeschooling
+- Store research papers for academic work
+
 ---
 
 ### Maps — Offline Navigation
 
+![Offline maps viewer](/docs/maps.webp)
+
 View maps without internet. Download the regions you need before going offline.
 
 **How to use it:**
@@ -170,10 +142,38 @@ View maps without internet. Download the regions you need before going offline.
 As your needs change, you can add more content anytime:
 
 - **More apps:** Settings → Apps
-- **More references:** Settings → ZIM Manager
+- **More references:** Settings → Content Explorer or Content Manager
 - **More map regions:** Settings → Maps Manager
 - **More educational content:** Through Kolibri's built-in content browser
 
+### Wikipedia Selector
+
+![Content Explorer — browse and download Wikipedia packages and curated collections](/docs/content-explorer.webp)
+
+N.O.M.A.D. includes a dedicated Wikipedia content management tool for browsing and downloading Wikipedia packages.
+
+**How to use it:**
+1. Go to **[Content Explorer →](/settings/zim/remote-explorer)**
+2. Browse available Wikipedia packages by language and size
+3. Select and download the packages you want
+
+**Note:** Selecting a different Wikipedia package replaces the previously downloaded version. Only one Wikipedia selection is active at a time.
+
+### System Benchmark
+
+![System Benchmark with NOMAD Score and Builder Tag](/docs/benchmark.webp)
+
+Test your hardware performance and see how your NOMAD build stacks up against the community.
+
+**How to use it:**
+1. Go to **[System Benchmark →](/settings/benchmark)**
+2. Choose a benchmark type: Full, System Only, or AI Only
+3. View your NOMAD Score (a weighted composite of CPU, memory, disk, and AI performance)
+4. Create a Builder Tag (your NOMAD-themed identity, like "Tactical-Llama-1234")
+5. Share your results with the [community leaderboard](https://benchmark.projectnomad.us)
+
+**Note:** Only Full Benchmarks with AI data can be shared to the community leaderboard.
+
 ### Keeping Things Updated
 
 While you have internet, periodically check for updates:
@@ -184,6 +184,8 @@ While you have internet, periodically check for updates:
 
 Content updates (Wikipedia, maps, etc.) can be managed separately from software updates.
 
+**Early Access Channel:** Want the latest features before they hit stable? Enable the Early Access Channel from the Check for Updates page to receive release candidate builds. You can switch back to stable anytime.
+
 ### Monitoring System Health
 
 Check on your server anytime:
@@ -215,7 +217,7 @@ Check storage usage in **Settings → System**.
 ### Getting Help
 
 - **In-app docs:** You're reading them now
-- **AI assistant:** Ask Open WebUI for help with almost anything
+- **AI assistant:** Ask a question in [AI Chat](/chat)
 - **Release notes:** See what's new in each version
 
 ---
@@ -224,9 +226,10 @@ Check storage usage in **Settings → System**.
 
 You're ready to use N.O.M.A.D. Here are some things to try:
 
-1. **Look something up** — Search for a topic in Kiwix
-2. **Learn something** — Start a Khan Academy course in Kolibri
-3. **Ask a question** — Chat with the AI in Open WebUI
+1. **Look something up** — Search for a topic in the Information Library
+2. **Learn something** — Start a Khan Academy course in the Education Platform
+3. **Ask a question** — Chat with the AI in [AI Chat](/chat)
 4. **Explore maps** — Find your neighborhood in the Maps viewer
+5. **Upload a document** — Add a PDF to the [Knowledge Base](/knowledge-base) and ask the AI about it
 
 Enjoy your offline knowledge server!

admin/docs/home.md

@@ -8,28 +8,40 @@ Your personal offline knowledge server is ready to use.
 
 Think of it as having Wikipedia, Khan Academy, an AI assistant, and offline maps all in one place, running on hardware you control.
 
+![Command Center Dashboard](/docs/dashboard.webp)
+
 ## What Can You Do?
 
 ### Browse Offline Knowledge
 Access millions of Wikipedia articles, medical references, how-to guides, and ebooks — all stored locally on your server. No internet required.
 
-*Launch Kiwix from the [Apps](/settings/apps) page or the home screen.*
+*Launch the Information Library from the home screen or the [Apps](/settings/apps) page.*
 
 ### Learn Something New
 Khan Academy courses covering math, science, economics, and more. Complete with videos and exercises, all available offline.
 
-*Launch Kolibri from the [Apps](/settings/apps) page or the home screen.*
+*Launch the Education Platform from the home screen or the [Apps](/settings/apps) page.*
 
 ### Chat with AI
-Ask questions, get explanations, brainstorm ideas, or get help with writing. Your local AI assistant works completely offline.
+Ask questions, get explanations, brainstorm ideas, or get help with writing. Your local AI assistant works completely offline — and you can upload documents to the Knowledge Base for document-aware responses.
 
-*Launch Open WebUI from the [Apps](/settings/apps) page or the home screen.*
+**[Open AI Chat →](/chat)**
+
+### Upload Documents to the Knowledge Base
+Upload PDFs, text files, and other documents for the AI to reference. The Knowledge Base uses semantic search to find relevant information from your uploaded documents when you ask questions.
+
+**[Open Knowledge Base →](/knowledge-base)**
 
 ### View Offline Maps
 Navigate and explore maps without an internet connection. Download regions you need before going offline.
 
 **[Open Maps →](/maps)**
 
+### Benchmark Your Hardware
+Run a System Benchmark to see how your hardware performs and compare your NOMAD Score with the community leaderboard.
+
+**[Open Benchmark →](/settings/benchmark)**
+
 ---
 
 ## Getting Started
@@ -46,11 +58,15 @@ Or explore the **[Getting Started Guide](/docs/getting-started)** for a walkthro
 
 | I want to... | Go here |
 |--------------|---------|
-| Download more content | [Install Apps](/apps) |
-| Add Wikipedia/reference content | [ZIM Manager](/settings/zim-manager) |
-| Download map regions | [Maps Manager](/settings/maps-manager) |
-| Check for updates | [System Update](/settings/updates) |
-| View system status | [Settings](/settings) |
+| Chat with the AI | [AI Chat →](/chat) |
+| Upload documents for AI | [Knowledge Base →](/knowledge-base) |
+| Download more content | [Install Apps →](/settings/apps) |
+| Add Wikipedia/reference content | [Content Explorer →](/settings/zim/remote-explorer) |
+| Manage installed content | [Content Manager →](/settings/zim) |
+| Download map regions | [Maps Manager →](/settings/maps) |
+| Run a benchmark | [System Benchmark →](/settings/benchmark) |
+| Check for updates | [System Update →](/settings/update) |
+| View system status | [System Info →](/settings/system) |
 
 ---
 
@@ -64,4 +80,4 @@ N.O.M.A.D. works best when kept up to date while you have internet access. This
 
 When you go offline, you'll have everything you need — the last synced versions of all your content.
 
-**[Check for Updates →](/settings/updates)**
+**[Check for Updates →](/settings/update)**

admin/docs/release-notes.md

@@ -1,10 +1,453 @@
 # Release Notes
 
+## Version 1.31.0 - April 3, 2026
+
+### Features
+- **AI Assistant**: Added support for remote OpenAI-compatible hosts (e.g. Ollama, LM Studio, etc.) to support running models on seperate hardware from the Command Center host. Thanks @hestela for the contribution!
+- **AI Assistant**: Disabled Ollama Cloud support (not compatible with NOMAD's architecture) and added support for flash_attn to improve performance of compatible models. Thanks @hestela for the contribution!
+- **Information Library (Kiwix)**: The Kiwix container now uses an XML library file approach instead of a glob-based approach to inform the Kiwix container of available ZIM files. This allows for much more robust handling of ZIM files and avoids issues with the container failing to start due to incomplete/corrupt ZIM files being present in the storage directory. Thanks @jakeaturner for the contribution!
+- **RAG**: Added support for EPUB file embedding into the Knowledge Base. Thanks @arn6694 for the contribution!
+- **RAG**: Added support for multiple file uploads (<=5, 100mb each) to the Knowledge Base. Thanks @jakeaturner for the contribution!
+- **Maps**: Added support for customizable location markers on the map with database persistence. Thanks @chriscrosstalk for the contribution!
+- **Maps**: The global map file can now be downloaded directly from PMTiles for users who want to the full map and/or regions outside of the U.S. that haven't been added to the curated collections yet. Thanks @bgauger for the contribution!
+- **Maps**: Added a scale bar to the map viewer with imperial and metric options. Thanks @chriscrosstalk for the contribution!
+- **Downloads**: Added support/improvements for rich progress, friendly names, cancellation, and live status updates for active downloads in the UI. Thanks @chriscrosstalk for the contribution!
+- **UI**: Converted all PNGs to WEBP for reduced image sizes and improved performance. Thanks @hestela for the contribution!
+- **UI**: Added an Installed Models section to AI Assistant settings. Thanks @chriscrosstalk for the contribution!
+
+### Bug Fixes
+- **Maps**: The maps API endpoints now properly check for "X-Forwarded-Proto" to support scenarios where the Command Center is behind a reverse proxy that terminates TLS. Thanks @davidgross for the fix!
+- **Maps**: Fixed an issue where the maps API endpoints could fail with an internal error if a hostname was used to access the Command Center instead of an IP address or localhost. Thanks @jakeaturner for the fix!
+- **Queue**: Increased the BullMQ lockDuration to prevent jobs from being killed prematurely on slower systems. Thanks @bgauger for the contribution!
+- **Queue**: Added better handling for very large downloads and user-initated cancellations. Thanks @bgauger for the contribution!
+- **Install**: The install script now checks for the presence of gpg (required for NVIDIA toolkit install) and automatically attempts to install it if it's missing. Thanks @chriscrosstalk for the fix!
+- **Security**: Added key validation to the settings read API endpoint. Thanks @LuisMIguelFurlanettoSousa for the fix!
+- **Security**: Improved URL validation logic for ZIM downloads to prevent SSRF vulnerabilities. Thanks @sebastiondev for the fix!
+- **UI**: Fixed the activity feed height in Easy Setup and added automatic scrolling to the latest message during installation. Thanks @chriscrosstalk for the contribution!
+
+### Improvements
+
+- **Dependencies**: Updated various dependencies to close security vulnerabilities and improve stability
+- **Docker**: NOMAD now adds 'com.docker.compose.project': 'project-nomad-managed' and 'io.project-nomad.managed': 'true' labels to all containers installed via the Command Center to improve compatibility with other Docker management tools and make it easier to identify and manage NOMAD containers. Thanks @techyogi for the contribution!
+- **Docs**: Added a simple API reference for power users and developers. Thanks @hestela for the contribution!
+- **Docs**: Re-formatted the Quick Install command into multiple lines for better readability in the README. Thanks @samsara-02 for the contribution!
+- **Docs**: Updated the CONTRIBUTING and FAQ guides with the latest information and clarified some common questions. Thanks @jakeaturner for the contribution!
+- **Ops**: Bumped GitHub Actions to their latest versions. Thanks @salmanmkc for the contribution!
+- **Performance**: Shrunk the bundle size of the Command Center UI significantly by optimizing dependencies and tree-shaking, resulting in faster load times and a snappier user experience. Thanks @jakeaturner for the contribution!
+- **Performance**: Implemented gzip compression by default for all HTTP registered routes from the Command Center backend to further improve performance, especially on slower connections. The DISABLE_COMPRESSION environment variable can be used to turn off this feature if needed. Thanks @jakeaturner for the contribution!
+- **Performance**: Added light caching of certain Docker socket interactions and custom AI Assistant name resolution to improve performance and reduce redundant calls to the Docker API. Thanks @jakeaturner for the contribution!
+- **Performance**: Switched to Inertia router navigation calls where appropriate to take advantage of Inertia's built-in caching and performance optimizations for a smoother user experience. Thanks @jakeaturner for the contribution!
+
+## Version 1.30.3 - March 25, 2026
+
+### Features
+
+### Bug Fixes
+- **Benchmark**: Fixed an issue where CPU and Disk Write scores could be displayed as 0 if the measured values was less than half of the reference mark. Thanks @bortlesboat for the fix!
+- **Content Manager**: Fixed a missing API client method that was causing ZIM file deletions to fail. Thanks @LuisMIguelFurlanettoSousa for the fix!
+- **Install**: Fixed an issue where the install script could incorrectly report the Docker NVIDIA runtime as missing. Thanks @brenex for the fix!
+- **Support the Project**: Fixed a broken link to Rogue Support. Thanks @chriscrosstalk for the fix!
+
+### Improvements
+- **AI Assistant**: Improved error reporting and handling for model downloads. Thanks @chriscrosstalk for the contribution!
+- **AI Assistant**: Bumped the default version of Ollama installed to v0.18.1 to take advantage of the latest performance improvements and bug fixes.
+- **Apps**: Improved error reporting and handling for service installation failures. Thanks @trek-e for the contribution!
+- **Collections**: Updated various curated collection links to their latest versions. Thanks @builder555 for the contribution!
+- **Cyberchef**: Bumped the default version of CyberChef installed to v10.22.1 to take advantage of the latest features and bug fixes.
+- **Docs**: Added a link to the step-by-step installation guide and video tutorial. Thanks @chriscrosstalk for the contribution!
+- **Install**: Increased the retries limit for the MySQL service in Docker Compose to improve stability during installation on systems with slower performance. Thanks @dx4956 for the contribution!
+- **Install**: Fixed an issue where stale data could cause credentials mismatch in MySQL on reinstall. Thanks @chriscrosstalk for the fix!
+
+## Version 1.30.0 - March 20, 2026
+
+### Features
+- **Night Ops**: Added our most requested feature — a dark mode theme for the Command Center interface! Activate it from the footer and enjoy the sleek new look during your late-night missions. Thanks @chriscrosstalk for the contribution!
+- **Debug Info**: Added a new "Debug Info" modal accessible from the footer that provides detailed system and application information for troubleshooting and support. Thanks @chriscrosstalk for the contribution!
+- **Support the Project**: Added a new "Support the Project" page in settings with links to community resources, donation options, and ways to contribute.
+- **Install**: The main Nomad image is now fully self-contained and directly usable with Docker Compose, allowing for more flexible and customizable installations without relying on external scripts. The image remains fully backwards compatible with existing installations, and the install script has been updated to reflect the simpler deployment process.
+
+### Bug Fixes
+- **Settings**: Storage usage display now prefers real block devices over tempfs. Thanks @Bortlesboat for the fix!
+- **Settings**: Fixed an issue where device matching and mount entry deduplication logic could cause incorrect storage usage reporting and missing devices in storage displays.
+- **Maps**: The Maps page now respects the request protocol (http vs https) to ensure map tiles load correctly. Thanks @davidgross for the bug report!
+- **Knowledge Base**: Fixed an issue where file embedding jobs could cause a retry storm if the Ollama service was unavailable. Thanks @skyam25 for the bug report!
+- **Curated Collections**: Fixed some broken links in the curated collections definitions (maps and ZIM files) that were causing some resources to fail to download.
+- **Easy Setup**: Fixed an issue where the "Start Here" badge would persist even after visiting the Easy Setup Wizard for the first time. Thanks @chriscrosstalk for the fix!
+- **UI**: Fixed an issue where the loading spinner could look strange in certain use cases.
+- **System Updates**: Fixed an issue where the update banner would persist even after the system was updated successfully. Thanks @chriscrosstalk for the fix!
+- **Performance**: Various small memory leak fixes and performance improvements across the UI to ensure a smoother experience.
+
+### Improvements
+- **Ollama**: Improved GPU detection logic to ensure the latest GPU config is always passed to the Ollama container on update
+- **Ollama**: The detected GPU type is now persisted in the database for more reliable configuration and troubleshooting across updates and restarts. Thanks @chriscrosstalk for the contribution!
+- **Downloads**: Users can now dismiss failed download notifications to reduce clutter in the UI. Thanks @chriscrosstalk for the contribution!
+- **Logging**: Changed the default log level to "info" to reduce noise and focus on important messages. Thanks @traxeon for the suggestion!
+- **Logging**: Nomad's internal logger now creates it's own log directory on startup if it doesn't already exist to prevent errors on fresh installs where the logs directory hasn't been created yet.
+- **Dozzle**: Dozzle shell access and container actions are now disabled by default. Thanks @traxeon for the recommendation!
+- **MySQL & Redis**: Removed port exposure to host by default for improved security. Ports can still be exposed manually if needed. Thanks @traxeon for the recommendation!
+- **Dependencies**: Various dependency updates to close security vulnerabilities and improve stability
+- **Utility Scripts**: Added a check for the expected Docker Compose version (v2) in all utility scripts to provide clearer error messages and guidance if the environment is not set up correctly.
+- **Utility Scripts**: Added an additional warning to the installation script to inform about potential overwriting of existing customized configurations and the importance of backing up data before running the installation script again.
+- **Documentation**: Updated installation instructions to reflect the new option for manual deployment via Docker Compose without the install script.
+
+
+## Version 1.29.0 - March 11, 2026
+
+### Features
+- **AI Assistant**: Added improved user guidance for troubleshooting GPU pass-through issues
+- **AI Assistant**: The last used model is now automatically selected when a new chat is started
+- **Settings**: Nomad now automatically performs nightly checks for available app updates, and users can select and apply updates from the Apps page in Settings
+
+### Bug Fixes
+- **Settings**: Fixed an issue where the AI Assistant settings page would be shown in navigation even if the AI Assistant was not installed, thus causing 404 errors when clicked
+- **Security**: Path traversal and SSRF mitigations
+- **AI Assistant**: Fixed an issue that was causing intermittent failures saving chat session titles
+
+### Improvements
+- **AI Assistant**: Extensive performance improvements and improved RAG intelligence/context usage
+
+## Version 1.28.0 - March 5, 2026
+
+### Features
+- **RAG**: Added support for viewing active embedding jobs in the processing queue and improved job progress tracking with more granular status updates
+- **RAG**: Added support for removing documents from the knowledge base (deletion from Qdrant and local storage)
+
+### Bug Fixes
+- **Install**: Fixed broken url's in install script and updated to prompt for Apache 2.0 license acceptance
+- **Docs**: Updated legal notices to reflect Apache 2.0 license and added Qdrant attribution
+- **Dependencies**: Various minor dependency updates to close security vulnerabilities
+
+### Improvements
+- **License**: Added Apache 2.0 license file to repository for clarity and legal compliance
+
+## Version 1.27.0 - March 4, 2026
+
+### Features
+- **Settings**: Added pagination support for Ollama model list
+- **Early Access Channel**: Allows users to opt in to receive early access builds with the latest features and improvements before they hit stable releases
+
+### Bug Fixes
+
+### Improvements
+- **AI Assistant**: Improved chat performance by optimizing query rewriting and response streaming logic
+- **CI/CD**: Updated release workflows to support release candidate versions
+- **KV Store**: Improved type safety in KV store implementation
+
+## Version 1.26.0 - February 19, 2026
+
+### Features
+- **AI Assistant**: Added support for showing reasoning stream for models with thinking capabilities
+- **AI Assistant**: Added support for response streaming for improved UX
+
+### Bug Fixes
+
+### Improvements
+
+
+## Version 1.25.2 - February 18, 2026
+
+### Features
+
+### Bug Fixes
+- **AI Assistant**: Fixed an error from chat suggestions when no Ollama models are installed
+- **AI Assistant**: Improved discrete GPU detection logic
+- **UI**: Legacy links to /docs and /knowledge-base now gracefully redirect to the correct pages instead of showing 404 errors
+
+### Improvements
+- **AI Assistant**: Chat suggestions are now disabled by default to avoid overwhelming smaller hardware setups
+
+## Version 1.25.1 - February 12, 2026
+
+### Features
+
+### Bug Fixes
+- **Settings**: Fix potential stale cache issue when checking for system updates
+- **Settings**: Improve user guidance during system updates
+
+### Improvements
+
+
+## Version 1.25.0 - February 12, 2026
+
+### Features
+- **Collections**: Complete overhaul of collection management with dynamic manifests, database tracking of installed resources, and improved UI for managing ZIM files and map assets
+- **Collections**: Added support for checking if newer versions of installed resources are available based on manifest data
+### Bug Fixes
+- **Benchmark**: Improved error handling and status code propagation for better user feedback on submission failures
+- **Benchmark**: Fix a race condition in the sysbench container management that could lead to benchmark test failures
+
+### Improvements
+
+---
+
+## Version 1.24.0 - February 10, 2026
+
+### 🚀 Features
+
+- **AI Assistant**: Query rewriting for enhanced context retrieval
+- **AI Assistant**: Allow manual scan and resync of Knowledge Base
+- **AI Assistant**: Integrated Knowledge Base UI into AI Assistant page
+- **AI Assistant**: ZIM content embedding into Knowledge Base
+- **Downloads**: Display model download progress
+- **System**: Cron job for automatic update checks
+- **Docs**: Polished documentation rendering with desert-themed components
+
+### 🐛 Bug Fixes
+
+- **AI Assistant**: Chat suggestion performance improvements
+- **AI Assistant**: Inline code rendering
+- **GPU**: Detect NVIDIA GPUs via Docker API instead of lspci
+- **Install**: Improve Docker GPU configuration
+- **System**: Correct memory usage percentage calculation
+- **System**: Show host OS, hostname, and GPU instead of container info
+- **Collections**: Correct devdocs ZIM filenames in Computing & Technology
+- **Downloads**: Sort active downloads by progress descending
+- **Docs**: Fix multiple broken internal links and route references
+
+### ✨ Improvements
+
+- **Docs**: Overhauled in-app documentation with sidebar ordering
+- **Docs**: Updated README with feature overview
+- **GPU**: Reusable utility for running nvidia-smi
+
+---
+
+## Version 1.23.0 - February 5, 2026
+
+### 🚀 Features
+
+- **Maps**: Maps now use full page by default
+- **Navigation**: Added "Back to Home" link on standard header pages
+- **AI**: Fuzzy search for AI models list
+- **UI**: Improved global error reporting with user notifications
+
+### 🐛 Bug Fixes
+
+- **Kiwix**: Avoid restarting the Kiwix container while download jobs are running
+- **Docker**: Ensure containers are fully removed on failed service install
+- **AI**: Filter cloud models from API response and fallback model list
+- **Curated Collections**: Prevent duplicate resources when fetching latest collections
+- **Content Tiers**: Rework tier system to dynamically determine install status on the server side
+
+### ✨ Improvements
+
+- **Docs**: Added pretty rendering for markdown tables in documentation pages
+
+---
+
+## Version 1.22.0 - February 4, 2026
+
+### 🚀 Features
+
+- **Content Manager**: Display friendly names (Title and Summary) instead of raw filenames for ZIM files
+- **AI Knowledge Base**: Automatically add NOMAD documentation to AI Knowledge Base on install
+
+### 🐛 Bug Fixes
+
+- **Maps**: Ensure map asset URLs resolve correctly when accessed via hostname
+- **Wikipedia**: Prevent loading spinner overlay during download
+- **Easy Setup**: Scroll to top when navigating between wizard steps
+- **AI Chat**: Hide chat button and page unless AI Assistant is actually installed
+- **Settings**: Rename confusing "Port" column to "Location" in Apps Settings
+
+### ✨ Improvements
+
+- **Ollama**: Cleanup model download logic and improve progress tracking
+
+---
+
+## Version 1.21.0 - February 2, 2026
+
+### 🚀 Features
+
+- **AI Assistant**: Built-in AI chat interface — no more separate Open WebUI app
+- **Knowledge Base**: Document upload with OCR, semantic search (RAG), and contextual AI responses via Qdrant
+- **Wikipedia Selector**: Dedicated Wikipedia content management with smart package selection
+- **GPU Support**: NVIDIA and AMD GPU passthrough for Ollama (faster AI inference)
+
+### 🐛 Bug Fixes
+
+- **Benchmark**: Detect Intel Arc Graphics on Core Ultra processors
+- **Easy Setup**: Remove built-in System Benchmark from wizard (now in Settings)
+- **Icons**: Switch to Tabler Icons for consistency, remove unused icon libraries
+- **Docker**: Avoid re-pulling existing images during install
+
+### ✨ Improvements
+
+- **Ollama**: Fallback list of recommended models if api.projectnomad.us is down
+- **Ollama/Qdrant**: Docker images pinned to specific versions for stability
+- **README**: Added website and community links
+- Removed Open WebUI as a separate installable app (replaced by built-in AI Chat)
+
+---
+
+## Version 1.20.0 - January 28, 2026
+
+### 🚀 Features
+
+- **Collections**: Expanded curated categories with more content and improved tier selection modal UX
+- **Legal**: Expanded Legal Notices and moved to bottom of Settings sidebar
+
+### 🐛 Bug Fixes
+
+- **Install**: Handle missing curl dependency on fresh Ubuntu installs
+- **Migrations**: Fix timestamp ordering for builder_tag migration
+
+---
+
+## Version 1.19.0 - January 28, 2026
+
+### 🚀 Features
+
+- **Benchmark**: Builder Tag system — claim leaderboard spots with NOMAD-themed tags (e.g., "Tactical-Llama-1234")
+- **Benchmark**: Full benchmark with AI now required for community sharing; HMAC-signed submissions
+- **Release Notes**: Subscribe to release notes via email
+- **Maps**: Automatically download base map assets if missing
+
+### 🐛 Bug Fixes
+
+- **System Info**: Fall back to fsSize when disk array is empty (fixes "No storage devices detected")
+
+---
+
+## Version 1.18.0 - January 24, 2026
+
+### 🚀 Features
+
+- **Collections**: Improved curated collections UX with persistent tier selection and submit-to-confirm workflow
+
+### 🐛 Bug Fixes
+
+- **Benchmark**: Fix AI benchmark connectivity (Docker container couldn't reach Ollama on host)
+- **Open WebUI**: Fix install status indicator
+
+### ✨ Improvements
+
+- **Docker**: Container URL resolution utility and networking improvements
+
+---
+
+## Version 1.17.0 - January 23, 2026
+
+### 🚀 Features
+
+- **System Benchmark**: Hardware scoring with NOMAD Score, circular gauges, and community leaderboard submission
+- **Dashboard**: User-friendly app names with "Powered by" open source attribution
+- **Settings**: Updated nomenclature and added tiered content collections to Settings pages
+- **Queues**: Support working all queues with a single command
+
+### 🐛 Bug Fixes
+
+- **Easy Setup**: Select valid primary disk for storage projection bar
+- **Docs**: Remove broken service links that pointed to invalid routes
+- **Notifications**: Improved styling
+- **UI**: Remove splash screen
+- **Maps**: Static path resolution fix
+
+---
+
+## Version 1.16.0 - January 20, 2026
+
+### 🚀 Features
+
+- **Apps**: Force-reinstall option for installed applications
+- **Open WebUI**: Manage Ollama models directly from Command Center
+- **Easy Setup**: Show selected AI model size in storage projection bar
+
+### ✨ Improvements
+
+- **Curated Categories**: Improved fetching from GitHub
+- **Build**: Added dockerignore file
+
+---
+
+## Version 1.15.0 - January 19, 2026
+
+### 🚀 Features
+
+- **Easy Setup Wizard**: Redesigned Step 1 with user-friendly capability cards instead of app names
+- **Tiered Collections**: Category-based content collections with Essential, Standard, and Comprehensive tiers
+- **Storage Projection Bar**: Visual disk usage indicator showing projected additions during Easy Setup
+- **Windows Support**: Docker Desktop support for local development with platform detection and NOMAD_STORAGE_PATH env var
+- **Documentation**: Comprehensive in-app documentation (Home, Getting Started, FAQ, Use Cases)
+
+### ✨ Improvements
+
+- **Easy Setup**: Renamed step 3 label from "ZIM Files" to "Content"
+- **Notifications**: Fixed auto-dismiss not working due to stale closure
+- Added Survival & Preparedness and Education & Reference content categories
+
+---
+
+## Version 1.14.0 - January 16, 2026
+
+### 🚀 Features
+
+- **Collections**: Auto-fetch latest curated collections from GitHub
+
+### 🐛 Bug Fixes
+
+- **Docker**: Improved container state management
+
+---
+
+## Version 1.13.0 - January 15, 2026
+
+### 🚀 Features
+
+- **Easy Setup Wizard**: Initial implementation of the guided first-time setup experience
+- **Maps**: Enhanced missing assets warnings
+- **Apps**: Improved app cards with custom icons
+
+### 🐛 Bug Fixes
+
+- **Curated Collections**: UI tweaks
+- **Install**: Changed admin container pull_policy to always
+
+---
+
+## Version 1.12.0 - 1.12.3 - December 24, 2025 - January 13, 2026
+
+### 🚀 Features
+
+- **System**: Check internet status on backend with custom test URL support
+
+### 🐛 Bug Fixes
+
+- **Admin**: Improved service install status management
+- **Admin**: Improved duplicate install request handling
+- **Admin**: Fixed base map assets download URL
+- **Admin**: Fixed port binding for Open WebUI
+- **Admin**: Improved memory usage indicators
+- **Admin**: Added favicons
+- **Admin**: Fixed container healthcheck
+- **Admin**: Fixed missing ZIM download API client method
+- **Install**: Fixed disk info file mount and stability
+- **Install**: Ensure update script always pulls latest images
+- **Install**: Use modern docker compose command in update script
+- **Install**: Ensure update script is executable
+- **Scripts**: Remove disk info file on uninstall
+
+---
+
+## Version 1.11.0 - 1.11.1 - December 24, 2025
+
+### 🚀 Features
+
+- **Maps**: Curated map region collections
+- **Collections**: Map region collection definitions
+
+### 🐛 Bug Fixes
+
+- **Maps**: Fixed custom pmtiles file downloads
+- **Docs**: Documentation renderer fixes
+
+---
+
 ## Version 1.10.1 - December 5, 2025
 
 ### ✨ Improvements
-1. This is a test
-- **Kiwix**: ZIM storage path
+- **Kiwix**: ZIM storage path improvements
 
 ---
 
@@ -17,7 +460,7 @@
 ### ✨ Improvements
 
 - **Install**: Add Redis env variables to compose file
-- **Kiwix**: initial download and setup
+- **Kiwix**: Initial download and setup
 
 ---
 
@@ -40,18 +483,14 @@
 
 - Alert and button styles redesign
 - System info page redesign
-- **Collections**: Curated ZIM Collections
-- **Collections**: add Preppers Library
-- **Collections**: add slug, icon, and language
-- **Collections**: store additional data with resources list
+- **Collections**: Curated ZIM Collections with slug, icon, and language support
 - Custom map and ZIM file downloads (WIP)
 - New maps system (WIP)
 
 ### ✨ Improvements
 
-- **DockerService**: cleanup old OSM stuff
-- **Install**: standardize compose file names
-- Hide query devtools in prod
+- **DockerService**: Cleanup old OSM stuff
+- **Install**: Standardize compose file names
 
 ---
 
@@ -62,17 +501,13 @@
 - Alert and button styles redesign
 - System info page redesign
 - **Collections**: Curated ZIM Collections
-- **Collections**: add Preppers Library
-- **Collections**: add slug, icon, and language
-- **Collections**: store additional data with resources list
 - Custom map and ZIM file downloads (WIP)
 - New maps system (WIP)
 
 ### ✨ Improvements
 
-- **DockerService**: cleanup old OSM stuff
-- **Install**: standardize compose file names
-- Hide query devtools in prod
+- **DockerService**: Cleanup old OSM stuff
+- **Install**: Standardize compose file names
 
 ---
 
@@ -104,8 +539,8 @@
 
 ### ✨ Improvements
 
-- **Scripts**: logs directory creation improvements
-- **Scripts**: fix type in management-compose file path
+- **Scripts**: Logs directory creation improvements
+- **Scripts**: Fix typo in management-compose file path
 
 ---
 
@@ -192,9 +627,10 @@
 
 ## Support
 
-- 📧 Email: support@projectnomad.com
-- 🐛 Bug Reports: [GitHub Issues](https://github.com/Crosstalk-Solutions/project-nomad/issues)
+- **Discord:** [Join the Community](https://discord.com/invite/crosstalksolutions) — Get help, share your builds, and connect with other NOMAD users
+- **Bug Reports:** [GitHub Issues](https://github.com/Crosstalk-Solutions/project-nomad/issues)
+- **Website:** [www.projectnomad.us](https://www.projectnomad.us)
 
 ---
 
-*For previous release notes, see our [changelog archive](https://github.com/Crosstalk-Solutions/project-nomad/releases).*
+*For the full changelog, see our [GitHub releases](https://github.com/Crosstalk-Solutions/project-nomad/releases).*