moving

docs/analysis/locomo-87.66-vs-85.26-integrity-2026-06-30.md

@@ -0,0 +1,55 @@
+# LoCoMo headline integrity — 87.66% is not reproducible; canonical = 86.49% (settled 2026-07-01)
+
+**TL;DR.** The published LoCoMo SOTA headline **87.66% (1350/1540)** does not reproduce on a fresh
+judge pass on **any** substrate. Fresh 7-lane W4 + fresh gpt-4.1-mini judge = **85.19%** on its own
+archived 2026-06-11 substrate and **86.49%** on the current substrate. The inflation is at the
+**judgment layer** (the harness's documented stale-verdict-replay bug), not substrate drift. Founder
+adopted **86.49%** as the canonical, reproducible, still-SOTA number (+4.54pp over Memori 81.95,
+z=4.64, p<10⁻⁵). Verify: `benchmarks/results/locomo-sota-2026-06/recount.mjs`.
+
+## The investigation chain (each step verified)
+
+1. **Evidence not in the repos.** The 87.66 report + raw answers/judgments lived only in the
+   throwaway `hive-mind-test` repo + git-ignored local disk (`.gitignore **/benchmarks/results/*`);
+   OSS showed the old 73.1%. (See `locomo-sota-evidence-drift-2026-06-30.md`.)
+
+2. **Committed judgments recount low.** `memori-gpt41mini-ours-judgments.jsonl` @ `05f2146` (the
+   report's cited input) recounts to **1313/1540 = 85.26%**, not 1350/87.66. Method validated: it
+   reproduces the Mem0 (73.96%), Config-C (76.62%) and theirs-arm (82.14%) numbers exactly.
+
+3. **The committed answers were the WRONG config** (founder's "7-lane W4?" catch). That answers file
+   has `raw_detail=0, importance=0, ~3098 tok` — a **reduced 2-lane** run (distilled+semantic), not
+   7-lane W4. Re-judged fresh = **85.39%**. The 87.66 needed the full 7-lane stack (raw-detail≈16,
+   ~3700 tok).
+
+4. **Regenerated the true 7-lane W4** (PROFILES+DATEWIN+EPISODIC+RAWDETAIL, uncapped) on the current
+   substrate → **86.49% (1332/1540)**. Better than 2-lane (+1.1pp), validates the Pareto — but still
+   1.17pp below 87.66.
+
+5. **Substrate confound ruled out.** The 87.66 (2026-06-11) ran on minds archived as
+   `minds-pre-wave3c` (2026-06-11 00:38); the current minds were rebuilt larger on 2026-06-29. Fresh
+   7-lane W4 on the **archived original substrate** = **85.19%** — *lower* than current. So the gap
+   is NOT substrate drift (newer substrate scores higher); 87.66 doesn't reproduce even on its own
+   substrate.
+
+6. **Cause = stale-verdict replay.** The harness note (2026-06-15,
+   `RESULT-backlog-closeout`): *"judge resumes by question_id and replayed stale verdicts."* The
+   original 1350-correct judgment pass included replayed/inflated verdicts; it is lost and no fresh
+   judge (85.19 / 86.49) reproduces it. The `41-judge` resume-by-linecount is the mechanism —
+   reusing an OUT_FILE skips fresh judging.
+
+## Definitive numbers (all fresh gpt-4.1-mini judge, Memori verbatim prompt, N=1540)
+| Configuration | Overall | vs Memori |
+|---|--:|--:|
+| Published claim (2026-06-11) | 87.66% (1350) | +5.71pp — *unreproducible* |
+| Archived 2026-06-11 substrate, 7-lane W4 | 85.19% (1312) | +3.24pp |
+| **Current substrate, 7-lane W4 (CANONICAL)** | **86.49% (1332)** | **+4.54pp, z=4.64** |
+| 2-lane committed file, fresh judge | 85.39% | +3.44pp |
+
+## Resolution (2026-07-01)
+- Adopt **86.49%** as canonical. Pinned in `benchmarks/results/locomo-sota-2026-06/`
+  (report + answers + judgments + `recount.mjs`), git-tracked.
+- Correct 87.66→86.49 across all surfaces (docs, public `apps/www`, `BenchmarkApp`, arXiv draft,
+  OSS, memory index); recompute stats (+5.71→+4.54pp, z=4.42→4.64).
+- Prevent recurrence: the harness must use a fresh `OUT_TAG` per run; benchmark SOTA evidence must
+  pin substrate+answers+judgments together in-repo. `hive-mind-test` is throwaway.