moving

docs/analysis/locomo-sota-evidence-drift-2026-06-30.md

@@ -0,0 +1,82 @@
+# LoCoMo SOTA evidence drift — root cause + consolidation (2026-06-30)
+
+> **UPDATE 2026-07-01:** consolidating the evidence surfaced a deeper defect — the **87.66% number
+> itself does not reproduce** (fresh 7-lane W4 = 86.49% current / 85.19% archived substrate; stale-
+> verdict-replay inflation). Canonical headline is now **86.49%**. This doc's gitignore/side-repo
+> root cause still stands; the number correction is in
+> [`locomo-87.66-vs-85.26-integrity-2026-06-30.md`](./locomo-87.66-vs-85.26-integrity-2026-06-30.md).
+
+**Symptom (founder):** "old benchmark results show, and the SOTA claim is not within waggle-os
+or hive-mind." The 87.66% LoCoMo memory-SOTA is real and verified, but looking at the canonical
+repos you see *old* numbers — the reproducible evidence isn't there.
+
+**Verdict:** Confirmed. This is a recurrence of the §7.5 substrate-drift pattern, but in the
+**evidence/results** dimension, not the code dimension — with an aggravating `.gitignore` rule
+that was *silently swallowing* the result report.
+
+---
+
+## How it was diagnosed (4-repo evidence matrix)
+
+| Repo / location | LoCoMo number visible | 87.66 evidence present? |
+|---|---|---|
+| waggle-os committed `benchmarks/results/` | April-2026 GEPA + v4–v6 manifests + `agentic-locomo-2026-04-25` | **No** |
+| waggle-os `docs/` (methodology §0, arXiv `.tex`/`.docx`, WAGGLE-CORNERSTONE) | 87.66 **prose** | Claim only — no reproducible data |
+| waggle-os on-disk `benchmarks/results/memori-phase22-RESULT.md` | **82.21%** (Phase-2.2 precursor) | No — and **git-ignored**, never committed |
+| OSS `marolinik/hive-mind` `benchmarks/locomo/RESULTS.md` + README badge | **73.1%** (N=320, Opus self-judge) | **No** |
+| `hive-mind-test` @ `05f2146` (private side repo) | **87.66%** (N=1540, same-judge) | **Yes — committed** |
+
+**Substrate code (the engine) is NOT the problem this time.** All four SOTA-critical elements —
+`inprocess-reranker.ts`, `search.ts` reranker wiring, `resolve-relative-date.ts` /
+`parse-date-window.ts` (write-time temporal dating), `raw-detail-lane.ts` — are present in
+waggle-os `main` (`packages/hive-mind-core/src/mind/`) and in the OSS mirror. The reranker was
+reverse-ported from OSS to monorepo in `f47ee8f` (2026-06-11). (Minor: the OSS reranker sits at
+an older commit `974ad7b` but is content-equivalent.)
+
+## Root cause (three compounding failures)
+
+1. **`.gitignore` swallow.** `.gitignore` line `**/benchmarks/results/*` ignores everything
+   directly under `benchmarks/results/`. The June W3.3 result report was generated there and
+   silently never committed. Older results (`gepa-faza1/…`, `agentic-locomo-2026-04-25`,
+   `manifest-v4/v5`) survive only because they were force-added / committed *before* the rule —
+   so the directory shows a stale snapshot.
+2. **Evidence produced in a throwaway side repo.** The actual 87.66 run (report + 1,540×2
+   answers + judgments) was produced and committed in `hive-mind-test`, which is a private
+   benchmark working repo — not the product monorepo and not the public OSS repo. It was never
+   forward-ported. This is the §7.5 "benchmark work in a side checkout is throwaway unless
+   reverse-ported" failure mode.
+3. **OSS public repo never refreshed.** `marolinik/hive-mind` still advertises the earlier
+   73.1% (N=320) result in `RESULTS.md` + README badge; the 87.66 number was never published
+   there even though the winning-stack *code* was (PR #14).
+
+Net effect: the SOTA *claim* (prose) shipped to the monorepo docs, but its *reproducible
+evidence* lived only on local disk (git-ignored) + a side repo. A fresh clone of either
+canonical repo shows old numbers — exactly the founder's report.
+
+> Red herring: the founder pointed at `D:/Projects/waggle-os-w4` (branch `feature/w4-port`).
+> That worktree is **226 commits behind `main`** and only 3 doc/lint commits ahead, all of whose
+> content was already re-ported to `main` (paper via `4193e68a`). It holds nothing `main` lacks.
+
+## The fix (this change)
+
+**A. Commit the canonical evidence into the product monorepo (done here).**
+`benchmarks/results/locomo-sota-2026-06/` now holds the two canonical reports (verbatim, with
+provenance headers), an `INDEX.md` (the previously-untracked SOTA single-source-of-truth), and a
+`README.md` reproduction recipe. The raw ~3.8 MB answers/judgments are intentionally **not**
+duplicated into the lean product repo — they're pointered to `hive-mind-test` + OSS.
+
+**B. Stop the silent swallow (structural drift-closure).** A `.gitignore` negation exception
+re-includes `benchmarks/results/locomo-sota-2026-06/**` so this evidence stays committed and
+future canonical SOTA evidence has a non-ignored home.
+
+**C. OSS public update — prepared, founder-gated.** Updating `marolinik/hive-mind`
+`benchmarks/locomo/RESULTS.md` + README badge from 73.1% → 87.66% is **outward-facing** (it
+pre-announces the SOTA ahead of arXiv submission, whose citation pass + endorsement are still
+open). Left as a go/no-go for the founder rather than pushed unilaterally.
+
+## Prevent recurrence (recommended follow-ups)
+- Add a one-line contract to `CLAUDE.md` §7.5 / `packages/hive-mind-core/CONTRIBUTING.md`:
+  *"Benchmark SOTA evidence MUST land in `benchmarks/results/locomo-sota-*/` (gitignore-excepted)
+  in the monorepo. `hive-mind-test` is throwaway — forward-port the report the same arc."*
+- Optionally extend `scripts/oss-drift-check.sh` to assert `RESULTS.md` headline parity between
+  the monorepo evidence dir and the OSS `benchmarks/locomo/`.