project-nomad/admin/tests/eval/README.md

# NOMAD AI Quality Harness

A reproducible way to measure whether NOMAD's RAG pipeline is getting better or
worse, and to tell a **code regression** apart from **a small model on modest
hardware being asked too much**.

This is a developer tool. There is no UI, nothing is user-facing, and none of it
runs in production.

> **Throughput is out of scope.** tokens/sec, time-to-first-token, and the NOMAD
> Score belong to `node ace benchmark:run`. Nothing here is comparable across
> machines and nothing here should ever be, because every developer and every
> user has different hardware. This measures *quality* only.

---

## The one-minute version

```bash
node ace eval:corpus --ingest                       # once, and after any corpus edit
node ace eval:retrieval --ablate                    # seconds, no chat model, deterministic
node ace eval:generation --model=<model> --all-modes  # minutes; answers "code or model?"
node ace eval:generation --model=<model> --suite=long_context  # context budgeting
```

---

## Golden suites

Each `.jsonl` under `tests/eval/goldens/` is a **suite**, named after the file.
Only `core` runs by default; anything else is opt-in via `--suite`.

That split exists for comparability. Aggregate metrics are means over whatever
ran, so quietly adding goldens to the default set would move `recall@k` and
`correctness` for reasons unrelated to the change under test — and silently
invalidate every committed baseline.

| Suite | What it is for |
|---|---|
| `core` | The default 99 questions. Retrieval and generation quality. |
| `long_context` | Conversations long enough to exercise the budget planner. |
| `off_topic` | Coherent questions the corpus cannot answer, plus distractor controls (#1341). |

`--suite a,b` runs several. `eval:corpus` always validates every suite, since a
malformed fixture should fail validation whether or not it is in the default run.

### The `off_topic` suite

Eighteen questions no document answers (general trivia, chit-chat, requests to
write something, personal asks, and domains the corpus lacks, such as gardening)
and seven controls that the `distractor-*` documents *do* answer.

Every refusal case must retrieve **nothing**. `eval:retrieval` prints a `leaked=`
rate per tag and, with `--verbose`, each leaked question with its top chunk's
boosted and cosine scores. The controls keep a fix honest: a gate that stops the
leaks by also rejecting "how do list comprehensions work in Python?" has traded
one bug for another.

The suite exists because the relevance floor looked fine on the original 28
documents (22% of these leaked) and fails on a well-stocked library (67% leaked
once Gutenberg prose and programming docs were added). A small corpus
understates this failure, so keep the distractors in.

Retrieval-only for now. The generation tier scores `expectRefusal` as "the model
declined", and "Paris" is the right answer to the capital question as long as
nothing is cited for it.

### The `long_context` suite

Four conversations testing the two opposite ways context budgeting can fail:

- **`history-retention`** — a fact stated early, then enough filler to look
  worth trimming, but still inside the window. A failure means good context was
  thrown away that would have fit.
- **`eviction`** — far more history than the window holds, with the needed fact
  in the *last* turn before the question. A failure means the model lost the
  thread of what was just said, which is the more serious of the two.

Both grade on a specific figure or arithmetic rather than prose, so a confident
non-answer cannot pass.

---

## Why the three modes matter

`eval:generation --all-modes` runs every question three ways. That is what turns
"the AI gave a bad answer" into something you can act on:

| Mode | Context the model gets | What a low score means |
|---|---|---|
| `oracle` | The golden's own documents, injected verbatim | **The model.** Retrieval was perfect by construction, so this is the model's ceiling with this prompt. |
| `e2e` | Whatever real retrieval found | The actual product experience. |
| `noretrieval` | Nothing | The model's parametric baseline — what it knows without NOMAD. |

Read the decomposition it prints:

- **`oracle` is low** → the model cannot use good context. No amount of retrieval
  work will fix it. The honest answer to the user is "run a larger model."
- **`oracle - e2e` is large** → the model *can* use good context but is not being
  given it. This is a retrieval bug, and it is ours.
- **`e2e - noretrieval` is near zero** → RAG is not contributing. Check that
  retrieval is actually reaching the prompt.

There is a fourth reference line, `--model=mock`, which needs no Ollama at all.
It answers by echoing the injected context, so it is the **extractive ceiling**:
the best a perfect model could do given the current retrieval. A real model
below the mock line is the bottleneck; a mock line that is itself low means
retrieval is.

---

## Commands

### `eval:corpus`

```bash
node ace eval:corpus --check     # validate corpus + goldens, no services needed
node ace eval:corpus --status    # fingerprint and current chunk count
node ace eval:corpus --ingest    # wipe and rebuild (always a full rebuild)
node ace eval:corpus --reset     # remove every eval chunk
```

Ingest goes through NOMAD's **real** `RagService.embedAndStoreText`, so chunk
size, the token-estimate ratio, the `search_document:` prefix, and the embedding
model are all inside the measurement. Change any of them and the score moves —
which is the point.

### Isolation: how the eval corpus stays out of your knowledge base

**The eval corpus is not a separate Qdrant collection.** It lives in the same
`nomad_knowledge_base` collection as your real documents, tagged with the
reserved payload value `collection: __nomad_eval__`. That is not a shortcut —
NOMAD "collections" *are* payload tags rather than separate Qdrant collections,
so using the tag means the harness exercises the same filter path production
chat uses, with no production code changed to accommodate it.

Three separate mechanisms keep the two apart:

**1. Writes are scoped.** Ingest only ever adds points carrying the eval tag.
`--reset` deletes by that filter and nothing else. Your documents are never
written, re-tagged, or deleted. (On the machine this was built on: 306 points
before ingest, 335 after — exactly the 29 eval chunks, nothing else moved.)

**2. Reads are filtered server-side.** Every eval query passes
`must: [{ key: 'collection', match: { value: '__nomad_eval__' } }]`, and Qdrant
applies it during search against a keyword payload index, so your documents
never compete for a result slot. Verified directly against a knowledge base
containing unrelated content — the same query returns a real user document at
similarity **0.83** unfiltered, and only eval documents at ~0.50 filtered:

```
unfiltered:  0.831  evolution_of_steam_locomotive.txt
filtered:    0.522  water-river-song.md  [eval]
```

**3. A leak would fail the run, loudly.** Every retrieved chunk's `source` is
resolved back to a corpus document by path. Anything outside
`tests/eval/corpus/` is counted as an unresolved chunk, and a non-zero count
prints an error and exits 1 rather than quietly reporting a score. The check is
on the resolved path, not the file extension — NOMAD embeds its own
`admin/docs/*.md` into the knowledge base on first run, so an extension check
would have accepted a leaked `faq.md` as the plausible document id "faq".
See `tests/unit/eval_source_guard.spec.ts`.

**The one thing the tag does not isolate** is the *physical* collection: your
documents and the eval fixtures share an HNSW index. That has no effect on
correctness (the filter is applied during search) and results were verified
byte-identical across runs, but if you want true physical separation the change
is to thread a collection name through `_ensureCollection`,
`embedAndStoreText`, and `searchSimilarDocuments`. That was deliberately not
done, because it means touching three production methods for a test-only
benefit that the payload filter already delivers.

### `eval:retrieval`

```bash
node ace eval:retrieval
node ace eval:retrieval --ablate              # is the reranker earning its complexity?
node ace eval:retrieval --threshold=0.5       # sweep the Qdrant cutoff
node ace eval:retrieval --min-final-score=0.7 # sweep the post-rerank relevance floor
node ace eval:retrieval --tag=multi-hop
node ace eval:retrieval --verbose             # show every miss and what it retrieved
node ace eval:retrieval --report              # write JSON + Markdown to reports/
node ace eval:retrieval --suite=off_topic --judge-model=llama3:8b   # measure the relevance check
node ace eval:retrieval --suite=core,off_topic --dump=pools.json    # candidate pools for offline gate sims
```

`--judge-model` runs the opt-in relevance check (`rag.relevanceCheck`) after the
floor and prints its latency and how often it failed open. It makes the run
model-dependent, so it is never part of a baseline. `--dump` writes every
golden's full pre-floor candidate pool with both scores, which lets you simulate
a new gate in a few lines of script instead of a sweep per setting.

**There are two cutoffs, on two different scores, and they do not interchange.**
`--threshold` is what Qdrant applies to the raw cosine score, before reranking;
`--min-final-score` (`RAG_MIN_FINAL_SCORE`) is the relevance floor applied to
the reranked score, and it is the one that decides whether a chunk is injected
at all. Reranking's boosts are score-scaled, so the same number means different
things on the two axes — the score-separation table prints both, labelled by
which flag each one calibrates. Read the right row.

Both tiers pin the floor to `RAG_MIN_FINAL_SCORE` and deliberately ignore the
user's `rag.minRelevance` setting, for the same reason the harness never sets
`skipRetrieval`: a slider position on one machine must not move the numbers a
baseline was recorded against.

Embedding is the only model call, and its output is stable, so **this tier is
deterministic and hardware-independent**. Two runs produce byte-identical
numbers. A movement here is a code change, full stop — which makes it the only
tier worth gating CI on.

Multi-turn goldens are scored on their raw final message, because resolving the
coreference needs the chat model and would make the tier non-deterministic. That
bucket therefore reports the honest floor; the rewrite's contribution shows up in
the generation tier instead.

### `eval:generation`

```bash
node ace eval:generation --model=mock                       # no Ollama required
node ace eval:generation --model=llama3.2:latest --all-modes
node ace eval:generation --model=llama3:8b --repeats=5 --verbose
node ace eval:generation --model=... --tag=out-of-corpus
```

Runs at `temperature: 0` with a fixed seed. That reduces variance but does not
eliminate it — llama.cpp batching and GPU scheduling still move outputs — so
`--repeats` defaults to 3 and any question whose pass/fail flips across repeats
is reported as **unstable** and excluded from gating. Do not read an unstable
question as a regression.

Before scoring, the harness evicts other resident models and burns one throwaway
generation, borrowed from `BenchmarkService` for the same reason it added them:
a cold first run behaves differently.

### `eval:matrix`

```bash
node ace eval:matrix --models=qwen2.5:0.5b,llama3.2:latest,llama3:8b --limit=25
node ace eval:matrix --models=... --promote
```

Produces the **capability table** — the artifact that answers a support ticket.
When a user reports a bad answer, look up their model:

- scoring at or near its row → the model is at its ceiling, not a bug
- scoring well below its row → their config or our code, worth investigating

### `eval:compare`

```bash
node ace eval:retrieval --report
node ace eval:compare <report.json> --promote=retrieval     # set the baseline
node ace eval:compare tests/eval/baselines/<fp>/retrieval.json <new-report.json>
```

Exits non-zero when any metric regresses beyond the tolerance (default `0.02`).
It boots no services, so CI can run it against a committed baseline without
standing up MySQL, Redis, Qdrant, or Ollama.

**It refuses to compare reports with different corpus fingerprints.** That is
correct behaviour, not a limitation: if the corpus, chunk size, or embedding
model changed, the two runs measured different things and diffing them would
manufacture a regression. Re-baseline instead.

---

## The corpus and the goldens

- `corpus/*.md` — 28 short documents across NOMAD's real domains, plus six
  `distractor-*` documents (Project Gutenberg excerpts, the Python tutorial, MDN
  pages). The distractors stand in for a library with general prose in it, where
  something is always semantically near the question. Sources and licences are
  in each file's header: Gutenberg texts are public domain, the Python tutorial
  is PSF License v2, MDN is CC-BY-SA 2.5.
- `goldens/*.jsonl` — 99 questions, one JSON object per line.

The corpus is built with deliberate traps, not just easy questions:

| Tag | What it tests |
|---|---|
| `distractor` | `water-river-song.md` is a poem that shares vocabulary with real water questions and answers none of them. This is exactly the "poetic, tangential passage" failure `SYSTEM_PROMPTS.rag_context` rule 1 defends against. |
| `near-miss` | Water-bath vs pressure canning: two documents that look alike and give opposite advice. |
| `out-of-corpus` | Questions the corpus genuinely cannot answer, including adversarial ones about topics the corpus *partly* covers (the TR-88's warranty). The right answer is to decline. |
| `fictional` | The Thornfield protocol and the TR-88 pump do not exist. No model can know them, so a correct answer proves retrieval worked rather than that the model memorised the internet. |
| `acronym` / `acronym-control` | The same question asked with an acronym and spelled out, to measure what `preprocessQuery`'s 28-entry glossary expansion actually buys. |
| `chunk-boundary` | Facts buried late in the one long document, which is the only one that splits into multiple chunks. |
| `multi-hop` | Answers requiring two documents (elevation table + boiling times). |
| `multi-turn` | A pronoun in the second turn, the only thing that exercises `rewriteQueryWithContext`. |

### Golden format

```jsonc
{
  "id": "water-boil-altitude-01",
  "query": "How long do I need to boil water at high altitude?",
  "turns": [],                                  // prior messages for multi-turn cases
  "relevantDocIds": ["water-boiling"],          // corpus filenames without .md
  "mustInclude": ["\\b(3|three) minutes?"],     // case-insensitive REGEX
  "mustNotInclude": ["distill"],
  "expectRefusal": false,                       // true for out-of-corpus
  "tags": ["single-hop", "numeric"]
}
```

`mustInclude` and `mustNotInclude` entries are **regular expressions**, so one
entry can accept "3 minutes" or "three minutes" without inflating the list.
Every pattern is compiled at load, so a bad regex fails immediately rather than
on the one run where it finally matters.

Validation is strict on purpose. A golden that lists a document not in the
corpus, or that expects a refusal while also naming relevant documents, is
rejected at load — those mistakes are otherwise invisible and just quietly lower
the score forever.

### Editing the corpus

Any edit changes the fingerprint, which invalidates every existing baseline.
That is deliberate. After editing:

```bash
node ace eval:corpus --check     # validate first
node ace eval:corpus --ingest    # rebuild
node ace eval:retrieval --report
node ace eval:compare <report.json> --promote=retrieval
```

Note the safety facts in the corpus are real. If you add documents, keep any
health, water, or food-safety content accurate — invent only clearly-fictional
non-safety things (place names, equipment model numbers) when you need something
unguessable.

---

## What the metrics mean

**Retrieval** — measured at two levels, because they answer different questions.
Document level (recall, hit rate, MRR, nDCG) collapses chunks to their source
document: *did the answer's document reach the context?* Chunk level (precision)
does not dedupe: *how much of what we injected is noise?* — five chunks from one
irrelevant document cost a small model five slots.

`nDCG` normalizes against the *known* number of relevant documents, not against
whatever was retrieved. It is the metric that catches "right documents, wrong
order", a reranking regression that leaves recall untouched while pushing the
answer to position five where a 1B model's 2-result budget will never see it.
The implementation is cross-checked against `pytrec_eval` (TREC's reference
implementation) — see `tests/unit/eval_retrieval_metrics.spec.ts`.

**Generation** — all deterministic, no judge model required:

| Metric | What it catches |
|---|---|
| `correctness` | The `mustInclude` / `mustNotInclude` assertions. |
| `refusalCorrectness` | Declining out-of-corpus questions *and* not hedging on answerable ones. This is the "Sorry, I wasn't able to find specific context" symptom, measured. |
| `leakageRate` | Narrating retrieval ("according to Context 1", "the knowledge base"), which `rag_context` rule 4 forbids. Pure regex, zero ambiguity, catches a bad prompt edit on the first run. |
| `groundedness` | Fraction of the answer's numeric claims that appear in the injected context. |
| `thinkTagLeakRate` | Reasoning tags reaching the user. Should always be 0. |
| `historyElidedRate` | Share of cases where history had to be trimmed to fit the window. |
| `chunksDroppedRate` | Share of cases where a retrieved chunk did not fit the context budget. |
| `promptTokenError` | Mean error of the token estimator against the backend's real `prompt_eval_count`. |

**Read the budgeting metrics before concluding the model got worse.** A drop in
`correctness` alongside a rise in `historyElidedRate` is not the model answering
badly — it is the model never being shown the material. Those are different
bugs with different fixes, and before these metrics existed they were
indistinguishable.

`promptTokenError` gates the estimator that every one of those trims is decided
against. NOMAD cannot tokenize exactly (no vocabulary offline, and Ollama exposes
no tokenize endpoint), so it estimates and calibrates per model from the real
counts the backend reports. If this drifts, the budget is being enforced against
a number that is no longer true. Measured at ~4.6% on llama3:8b.

**Groundedness only sees numbers, and only numbers above 10.** An answer that
fabricates a procedure or a proper noun scores a perfect 1.0. It is a
fabrication *detector*, not a faithfulness guarantee. Numbers are the right
first target for this domain — a wrong bleach dose or canner pressure is a wrong
answer with consequences — but do not read a high score as "the answer is
faithful". Small integers are excluded because "3 layers" and "step 2" appear in
any prose and would swamp the signal.

---

## Known limitations

Read these before trusting a number.

1. **The corpus is small, so retrieval recall has little headroom.** 28
   documents produce 29 chunks; retrieving the top 5 means retrieving 17% of the
   entire corpus on every query. Real NOMAD knowledge bases hold millions of
   chunks. `recall@5` therefore sits near 0.99 and cannot detect a modest
   retrieval regression. The metrics that *do* have headroom on this corpus are
   `recall@1`, `precision@k`, `nonEmptyRateOnRefusal`, and the score
   distributions. To make recall discriminating, add substantially more
   distractor documents, or ingest `install/wikipedia_en_100_mini_*.zim` under
   the eval tag as a harder tier.

2. **`oracle` is not guaranteed to beat `e2e`.** Oracle injects the golden's
   whole documents; e2e injects up to `maxResults` retrieved chunks, which can
   include a genuinely helpful extra document. Treat small inversions as noise
   unless they survive `--repeats=5`.

3. **Temperature 0 is not determinism.** The generation tier still moves between
   runs. Always report `--repeats` ≥ 3 before concluding anything, and ignore
   questions the harness flags as unstable.

4. **No LLM judge yet.** Faithfulness beyond numeric grounding, completeness,
   and answer relevance are not measured. `autoevals` (MIT, TypeScript, talks to
   Ollama through the same `/v1` endpoint `OllamaService` already uses) is the
   intended addition, reported in a separate section and never mixed into the
   deterministic scores — a weak local judge grading a weak local model is not
   something to gate on.

---

## Layout

```
tests/eval/
  corpus/         34 markdown fixtures (28 domain + 6 distractor) — the frozen knowledge base
  goldens/        one JSONL file per suite (core: 99 questions)
  baselines/      <corpus-fingerprint>/*.json — COMMITTED; the gate compares against these
  reports/        run artifacts — gitignored
```

Baselines are filed under their corpus fingerprint so it is structurally
impossible to overwrite one corpus's baseline with a run against another.

Implementation:

- `app/utils/eval/*` — pure functions (metrics, golden parsing, report diffing).
  No I/O, no models, no services. 153 tests, run with:

  ```bash
  npm run test:eval    # this harness only — should always be green
  npm run test:unit    # every tests/unit spec
  ```

  `test:unit` currently reports **7 pre-existing failures** unrelated to this
  harness: `drug_interactions`, `drug_ingest_status`, `drug_labels`, and
  `rag_retrieval_toggle` are written against `@japa/runner` rather than
  `node:test`, and `app_auto_update`, `content_auto_update`, and
  `content_auto_update_backoff` import services that need a booted application.
  Both groups belong in the Japa suite (`node ace test`, which needs MySQL and
  Redis). They fail identically before and after any change here — use
  `test:eval` when you want a signal you can trust.
- `app/services/eval_*_service.ts` — orchestration; these need Qdrant and Ollama.
- `commands/eval/*` — the CLI.
- `app/services/rag_pipeline_service.ts` — the prompt pipeline, shared with the
  chat endpoint. The harness measures production code, not a copy of it.