53 lines
3.0 KiB
Markdown
53 lines
3.0 KiB
Markdown
# LoCoMo Memory-SOTA — canonical evidence (86.49%, reproducible + pinned)
|
|
|
|
**Headline (defensible, same-judge, reproducible):** On LoCoMo under Memori's own published
|
|
protocol (gpt-4.1-mini answerer **and** judge, verbatim "be generous" ACCURACY_PROMPT, natural
|
|
full distribution, N=1,540), the Hive Mind 7-lane W4 substrate scores **86.49% overall
|
|
(1,332/1,540), +4.54pp over Memori's 81.95%** (one-sample z=4.64, p<10⁻⁵), leading every category.
|
|
Verify offline with `node recount.mjs`.
|
|
|
|
> **Supersedes the 87.66% claim.** 87.66% was the 2026-06-11 report; it does not reproduce on a
|
|
> fresh judge pass (85.19% on its own archived substrate, 86.49% on the current one) and was
|
|
> inflated by the stale-verdict-replay bug the harness documented on 2026-06-15. 86.49% is the
|
|
> honest, fresh, substrate-pinned number. Full story:
|
|
> `docs/analysis/locomo-87.66-vs-85.26-integrity-2026-06-30.md`.
|
|
|
|
## Contents
|
|
| File | What |
|
|
|---|---|
|
|
| `LOCOMO-SOTA-86.49-vs-Memori.md` | Canonical report — Memori head-to-head + Mem0 + reproduction. |
|
|
| `recount.mjs` | Offline verification (zero API): recounts the judgments → 1332/1540 = 86.49%. |
|
|
| `INDEX.md` | Single-source-of-truth index: every number + where each artifact lives. |
|
|
| `data/answers/…N1540.jsonl` | The 7-lane W4 per-question answers (raw_detail≈16, importance≈4.5). |
|
|
| `data/judgments/…N1540.jsonl` | The per-question judge verdicts — recount source of the 86.49%. |
|
|
|
|
## Per-category (7-lane W4, N=1,540, gpt-4.1-mini answerer+judge)
|
|
| Category | Ours | Memori (pub) | Δ | n |
|
|
|---|--:|--:|--:|--:|
|
|
| single-hop | 92.27 | 87.87 | +4.40 | 841 |
|
|
| multi-hop | 80.50 | 72.70 | +7.80 | 282 |
|
|
| temporal | 81.62 | 80.37 | +1.25 | 321 |
|
|
| open-ended | 69.79 | 63.54 | +6.25 | 96 |
|
|
| **overall** | **86.49** | **81.95** | **+4.54** | 1540 |
|
|
|
|
vs Mem0 (same-judge, our ruler): **86.49 vs 73.96, +12.53pp overall; temporal +30.84pp.**
|
|
|
|
## Substrate code that produces this (in this monorepo)
|
|
All under `packages/hive-mind-core/src/mind/`: `inprocess-reranker.ts`, `search.ts` (reranker
|
|
wiring), `resolve-relative-date.ts` + `parse-date-window.ts` (write-time temporal dating),
|
|
`raw-detail-lane.ts`, `recall-context.ts`. 7 lanes = distilled + semantic + importance-K5 +
|
|
episodic + profiles + date-window + raw-detail (CE-reranked).
|
|
|
|
## Reproduce
|
|
- **Offline (zero API):** `node recount.mjs` → asserts 1332/1540 per-category.
|
|
- **Full regen** (in `hive-mind-test/scripts/locomo`):
|
|
`PROMPT_MODE=ours PROFILES=1 DATEWIN=1 EPISODIC=1 RAWDETAIL=1 OUT_TAG=<fresh> node 40-cell-retrieval-gpt41mini.mjs`
|
|
→ `41-judge --in=…-<fresh>.jsonl`. **Use a fresh `OUT_TAG`** (reusing a judgments file replays
|
|
stale verdicts — that is what produced the bogus 87.66).
|
|
|
|
## Why this directory exists
|
|
The 86.49% claim previously lived only on local disk (git-ignored by `**/benchmarks/results/*`) +
|
|
the throwaway `hive-mind-test` repo, and its *number* had drifted from its *evidence*. This dir
|
|
pins report + answers + judgments + a recount check together, git-tracked via a `.gitignore`
|
|
negation exception. Do **not** remove that exception.
|