# Reproduce — UDI v0.3.1 · LongMemEval-V2 Small

One command, offline, no API key, no access to the protected adapter:

```bash
python3 reproduce.py
```

It recomputes the published headline numbers directly from the benchmark's
frozen official-scorer judgments and checks them against the claim.

## The claim it reproduces

| Metric | Value |
|---|---|
| Accuracy | **256 / 451 = 56.762749%** |
| Avg. memory-query latency | **0.187002 s** |
| LAFS gain vs released reference frontier | **+3.580703 pts** (59.3455 vs 55.7648) |

Tier: LongMemEval-V2 **Small** (451 questions, all scored). Operating point: `fast`.
Official benchmark commit: `6f020ac2fc3275e46c706d3406e02c3ed79b7be2`.

## What this proves — and what it does not

**Proves:** the sealed adapter produced 451 answers; the benchmark's official
scorer judged each correct/incorrect; and the published headline numbers are an
exact, faithful recomputation of those frozen judgments. Nothing was rounded up
or tuned after the fact.

**Does not prove (by itself):**
- It is **not** an independent re-run of the LLM judge — it recomputes the score
  from the judge's frozen decisions. To re-grade the answers yourself, see below.
- It is **not** an accepted leaderboard entry, and **not** independently
  replicated. It is a self-reported development result, submitted and awaiting
  maintainer review.

## What's in the box

- `data/judged_frozen.jsonl` — 451 per-question records, scoring fields only:
  `question_id, category, question_type, is_abstention_problem, question_text,
  answer_gold, response_parsed_boxed, is_unknown, score, score_bool,
  memory_query_duration_seconds`. Each record is the adapter's answer, the gold
  answer, and the official judge's verdict.
- `reproduce.py` — the recomputation + assertions.
- `MANIFEST.sha256` — content hashes of the above.

The **protected mechanism is not here and is not needed.** Memory context,
prompts, geometry, weights, and internal state are excluded by construction —
only the scored input/output/verdict layer is published, so the number is
checkable without exposing how it was produced.

## Deeper check — re-run the judge yourself

To verify the judge itself (not just the arithmetic), take the answers
(`response_parsed_boxed`) and golds (`answer_gold`) from `judged_frozen.jsonl`
and run the official evaluator from the LongMemEval-V2 harness
(`evaluation/run_eval.py`) at commit `6f020ac2`. That step needs the official
repo and a judge-model API key, and should land within scorer noise of 256/451.

## Honest read of the result

The headline is *accuracy-per-latency at the fast end*: best accuracy in the
sub-second regime, hundreds of times faster than the minute-scale agent
baselines that reach higher raw accuracy. The category breakdown printed by
`reproduce.py` shows where it is weak: the **abstention** categories (answering
"the record does not say") score far below the answerable ones. That grounded-
refusal gap is the next thing to close, and it is left visible on purpose.

## Provenance

Frozen submission package (sealed adapter side) commits, from
`BENCHMARK-MATRIX.json`:

- backend  sha256 `ef5aa239f0dfe8400a339e5d0a90e936db1653a31540273358615caabfc3c6ba`
- runtime  sha256 `eb7ca08cc027278d07757445d047c48a49fc1d08f57efe8d51ce50ca68984852`
- geometry sha256 `b1046340489a76a8ea48c6d3c8c83c16d12720664630d49d1784431e0ed7b31f`

These commit to the protected artifacts without publishing them; an authorized
reviewer under agreement can compare a disclosed copy against them.