83 lines
5.4 KiB
Markdown
83 lines
5.4 KiB
Markdown
# LoCoMo SOTA evidence drift — root cause + consolidation (2026-06-30)
|
||
|
||
> **UPDATE 2026-07-01:** consolidating the evidence surfaced a deeper defect — the **87.66% number
|
||
> itself does not reproduce** (fresh 7-lane W4 = 86.49% current / 85.19% archived substrate; stale-
|
||
> verdict-replay inflation). Canonical headline is now **86.49%**. This doc's gitignore/side-repo
|
||
> root cause still stands; the number correction is in
|
||
> [`locomo-87.66-vs-85.26-integrity-2026-06-30.md`](./locomo-87.66-vs-85.26-integrity-2026-06-30.md).
|
||
|
||
**Symptom (founder):** "old benchmark results show, and the SOTA claim is not within waggle-os
|
||
or hive-mind." The 87.66% LoCoMo memory-SOTA is real and verified, but looking at the canonical
|
||
repos you see *old* numbers — the reproducible evidence isn't there.
|
||
|
||
**Verdict:** Confirmed. This is a recurrence of the §7.5 substrate-drift pattern, but in the
|
||
**evidence/results** dimension, not the code dimension — with an aggravating `.gitignore` rule
|
||
that was *silently swallowing* the result report.
|
||
|
||
---
|
||
|
||
## How it was diagnosed (4-repo evidence matrix)
|
||
|
||
| Repo / location | LoCoMo number visible | 87.66 evidence present? |
|
||
|---|---|---|
|
||
| waggle-os committed `benchmarks/results/` | April-2026 GEPA + v4–v6 manifests + `agentic-locomo-2026-04-25` | **No** |
|
||
| waggle-os `docs/` (methodology §0, arXiv `.tex`/`.docx`, WAGGLE-CORNERSTONE) | 87.66 **prose** | Claim only — no reproducible data |
|
||
| waggle-os on-disk `benchmarks/results/memori-phase22-RESULT.md` | **82.21%** (Phase-2.2 precursor) | No — and **git-ignored**, never committed |
|
||
| OSS `marolinik/hive-mind` `benchmarks/locomo/RESULTS.md` + README badge | **73.1%** (N=320, Opus self-judge) | **No** |
|
||
| `hive-mind-test` @ `05f2146` (private side repo) | **87.66%** (N=1540, same-judge) | **Yes — committed** |
|
||
|
||
**Substrate code (the engine) is NOT the problem this time.** All four SOTA-critical elements —
|
||
`inprocess-reranker.ts`, `search.ts` reranker wiring, `resolve-relative-date.ts` /
|
||
`parse-date-window.ts` (write-time temporal dating), `raw-detail-lane.ts` — are present in
|
||
waggle-os `main` (`packages/hive-mind-core/src/mind/`) and in the OSS mirror. The reranker was
|
||
reverse-ported from OSS to monorepo in `f47ee8f` (2026-06-11). (Minor: the OSS reranker sits at
|
||
an older commit `974ad7b` but is content-equivalent.)
|
||
|
||
## Root cause (three compounding failures)
|
||
|
||
1. **`.gitignore` swallow.** `.gitignore` line `**/benchmarks/results/*` ignores everything
|
||
directly under `benchmarks/results/`. The June W3.3 result report was generated there and
|
||
silently never committed. Older results (`gepa-faza1/…`, `agentic-locomo-2026-04-25`,
|
||
`manifest-v4/v5`) survive only because they were force-added / committed *before* the rule —
|
||
so the directory shows a stale snapshot.
|
||
2. **Evidence produced in a throwaway side repo.** The actual 87.66 run (report + 1,540×2
|
||
answers + judgments) was produced and committed in `hive-mind-test`, which is a private
|
||
benchmark working repo — not the product monorepo and not the public OSS repo. It was never
|
||
forward-ported. This is the §7.5 "benchmark work in a side checkout is throwaway unless
|
||
reverse-ported" failure mode.
|
||
3. **OSS public repo never refreshed.** `marolinik/hive-mind` still advertises the earlier
|
||
73.1% (N=320) result in `RESULTS.md` + README badge; the 87.66 number was never published
|
||
there even though the winning-stack *code* was (PR #14).
|
||
|
||
Net effect: the SOTA *claim* (prose) shipped to the monorepo docs, but its *reproducible
|
||
evidence* lived only on local disk (git-ignored) + a side repo. A fresh clone of either
|
||
canonical repo shows old numbers — exactly the founder's report.
|
||
|
||
> Red herring: the founder pointed at `D:/Projects/waggle-os-w4` (branch `feature/w4-port`).
|
||
> That worktree is **226 commits behind `main`** and only 3 doc/lint commits ahead, all of whose
|
||
> content was already re-ported to `main` (paper via `4193e68a`). It holds nothing `main` lacks.
|
||
|
||
## The fix (this change)
|
||
|
||
**A. Commit the canonical evidence into the product monorepo (done here).**
|
||
`benchmarks/results/locomo-sota-2026-06/` now holds the two canonical reports (verbatim, with
|
||
provenance headers), an `INDEX.md` (the previously-untracked SOTA single-source-of-truth), and a
|
||
`README.md` reproduction recipe. The raw ~3.8 MB answers/judgments are intentionally **not**
|
||
duplicated into the lean product repo — they're pointered to `hive-mind-test` + OSS.
|
||
|
||
**B. Stop the silent swallow (structural drift-closure).** A `.gitignore` negation exception
|
||
re-includes `benchmarks/results/locomo-sota-2026-06/**` so this evidence stays committed and
|
||
future canonical SOTA evidence has a non-ignored home.
|
||
|
||
**C. OSS public update — prepared, founder-gated.** Updating `marolinik/hive-mind`
|
||
`benchmarks/locomo/RESULTS.md` + README badge from 73.1% → 87.66% is **outward-facing** (it
|
||
pre-announces the SOTA ahead of arXiv submission, whose citation pass + endorsement are still
|
||
open). Left as a go/no-go for the founder rather than pushed unilaterally.
|
||
|
||
## Prevent recurrence (recommended follow-ups)
|
||
- Add a one-line contract to `CLAUDE.md` §7.5 / `packages/hive-mind-core/CONTRIBUTING.md`:
|
||
*"Benchmark SOTA evidence MUST land in `benchmarks/results/locomo-sota-*/` (gitignore-excepted)
|
||
in the monorepo. `hive-mind-test` is throwaway — forward-port the report the same arc."*
|
||
- Optionally extend `scripts/oss-drift-check.sh` to assert `RESULTS.md` headline parity between
|
||
the monorepo evidence dir and the OSS `benchmarks/locomo/`.
|