# LoCoMo headline integrity — 87.66% is not reproducible; canonical = 86.49% (settled 2026-07-01)

**TL;DR.** The published LoCoMo SOTA headline **87.66% (1350/1540)** does not reproduce on a fresh
judge pass on **any** substrate. Fresh 7-lane W4 + fresh gpt-4.1-mini judge = **85.19%** on its own
archived 2026-06-11 substrate and **86.49%** on the current substrate. The inflation is at the
**judgment layer** (the harness's documented stale-verdict-replay bug), not substrate drift. Founder
adopted **86.49%** as the canonical, reproducible, still-SOTA number (+4.54pp over Memori 81.95,
z=4.64, p<10⁻⁵). Verify: `benchmarks/results/locomo-sota-2026-06/recount.mjs`.

## The investigation chain (each step verified)

1. **Evidence not in the repos.** The 87.66 report + raw answers/judgments lived only in the
   throwaway `hive-mind-test` repo + git-ignored local disk (`.gitignore **/benchmarks/results/*`);
   OSS showed the old 73.1%. (See `locomo-sota-evidence-drift-2026-06-30.md`.)

2. **Committed judgments recount low.** `memori-gpt41mini-ours-judgments.jsonl` @ `05f2146` (the
   report's cited input) recounts to **1313/1540 = 85.26%**, not 1350/87.66. Method validated: it
   reproduces the Mem0 (73.96%), Config-C (76.62%) and theirs-arm (82.14%) numbers exactly.

3. **The committed answers were the WRONG config** (founder's "7-lane W4?" catch). That answers file
   has `raw_detail=0, importance=0, ~3098 tok` — a **reduced 2-lane** run (distilled+semantic), not
   7-lane W4. Re-judged fresh = **85.39%**. The 87.66 needed the full 7-lane stack (raw-detail≈16,
   ~3700 tok).

4. **Regenerated the true 7-lane W4** (PROFILES+DATEWIN+EPISODIC+RAWDETAIL, uncapped) on the current
   substrate → **86.49% (1332/1540)**. Better than 2-lane (+1.1pp), validates the Pareto — but still
   1.17pp below 87.66.

5. **Substrate confound ruled out.** The 87.66 (2026-06-11) ran on minds archived as
   `minds-pre-wave3c` (2026-06-11 00:38); the current minds were rebuilt larger on 2026-06-29. Fresh
   7-lane W4 on the **archived original substrate** = **85.19%** — *lower* than current. So the gap
   is NOT substrate drift (newer substrate scores higher); 87.66 doesn't reproduce even on its own
   substrate.

6. **Cause = stale-verdict replay.** The harness note (2026-06-15,
   `RESULT-backlog-closeout`): *"judge resumes by question_id and replayed stale verdicts."* The
   original 1350-correct judgment pass included replayed/inflated verdicts; it is lost and no fresh
   judge (85.19 / 86.49) reproduces it. The `41-judge` resume-by-linecount is the mechanism —
   reusing an OUT_FILE skips fresh judging.

## Definitive numbers (all fresh gpt-4.1-mini judge, Memori verbatim prompt, N=1540)
| Configuration | Overall | vs Memori |
|---|--:|--:|
| Published claim (2026-06-11) | 87.66% (1350) | +5.71pp — *unreproducible* |
| Archived 2026-06-11 substrate, 7-lane W4 | 85.19% (1312) | +3.24pp |
| **Current substrate, 7-lane W4 (CANONICAL)** | **86.49% (1332)** | **+4.54pp, z=4.64** |
| 2-lane committed file, fresh judge | 85.39% | +3.44pp |

## Resolution (2026-07-01)
- Adopt **86.49%** as canonical. Pinned in `benchmarks/results/locomo-sota-2026-06/`
  (report + answers + judgments + `recount.mjs`), git-tracked.
- Correct 87.66→86.49 across all surfaces (docs, public `apps/www`, `BenchmarkApp`, arXiv draft,
  OSS, memory index); recompute stats (+5.71→+4.54pp, z=4.42→4.64).
- Prevent recurrence: the harness must use a fresh `OUT_TAG` per run; benchmark SOTA evidence must
  pin substrate+answers+judgments together in-repo. `hive-mind-test` is throwaway.