LoCoMo Memory-SOTA — 86.49% (7-lane W4, N=1540, same-judge vs Memori)

Reproduced & pinned 2026-07-01. Subject = gpt-4.1-mini, judge = gpt-4.1-mini + Memori's verbatim "be generous" ACCURACY_PROMPT, natural full-N distribution, overall = count-weighted micro-average. Full 7-lane W4 stack (distilled + semantic + importance-K5 + episodic + profiles + date-window + raw-detail/CE-rerank), uncapped.

Correction note. The original 2026-06-11 report claimed 87.66%. That number did not reproduce on a fresh judge pass — on its own archived 2026-06-11 substrate it re-scores 85.19%, and on the current substrate 86.49% — consistent with the stale-verdict-replay bug documented in the harness on 2026-06-15 (judge resumes by question_id and replayed stale verdicts). The original 1350-correct judgment set is lost and unreproducible. 86.49% is the honest, fresh, reproducible figure (verify with node recount.mjs). See docs/analysis/locomo-87.66-vs-85.26-integrity-2026-06-30.md.

Head-to-head vs Memori Table 1 (N=1540)

Category	Memori (Table 1)	Our substrate (gpt-4.1-mini)	Δ vs Memori	n
single-hop	87.87	92.27% (776/841)	+4.40pp	841
multi-hop	72.70	80.50% (227/282)	+7.80pp	282
temporal	80.37	81.62% (262/321)	+1.25pp	321
open-ended	63.54	69.79% (67/96)	+6.25pp	96
Overall (count-weighted)	81.95	86.49% (1332/1540)	+4.54pp	1540

One-sample proportion test vs Memori's fixed 81.95%: z = 4.64, p < 10⁻⁵ (one-sided). Memori baseline reproduced first on our own harness: nb02 = 81.98 vs published 81.95 (ruler validated).

Competitor re-run — Mem0 (same answerer + judge, our ruler)

Category	Ours (7-lane W4)	Mem0 (our ruler)	Δ
single-hop	92.27	83.59	+8.68
multi-hop	80.50	74.82	+5.68
temporal	81.62	50.78	+30.84
open-ended	69.79	64.58	+5.21
overall	86.49	73.96	+12.53

Lead every category on Mem0's own protocol; temporal by +30.84pp — write-time-dated episodic timeline vs Mem0's ingestion-time stamping. (Mem0 judgments: 1139/1540 = 73.96%, committed.)

Tokens/query

avg context_tokens ≈ 3,100 on this run (per-row context_tokens in the answers file; the 42-report tokens/query line is a known display bug — ignore it). The token-Pareto knee (Config D, −28% tokens for ~−1pp) was measured on a prior answer set and should be re-measured on this substrate before re-citing exact numbers.

Reproduce

Offline (zero API): node recount.mjs — recounts data/judgments/…-N1540.jsonl → 1332/1540.
Full regen: in hive-mind-test/scripts/locomo, PROMPT_MODE=ours PROFILES=1 DATEWIN=1 EPISODIC=1 RAWDETAIL=1 OUT_TAG=<fresh> node 40-cell-retrieval-gpt41mini.mjs then 41-judge --in=…-<fresh>.jsonl. Always use a fresh OUT_TAG — reusing an existing judgments file triggers the stale-verdict replay that produced the bogus 87.66.

Provenance (pinned together — this is the fix for the drift)

Answers: data/answers/locomo-7lane-w4-answers-N1540.jsonl (7-lane: raw_detail≈16, importance≈4.5).
Judgments: data/judgments/locomo-7lane-w4-judgments-N1540.jsonl (recount = 1332/1540).
Substrate: 10 LoCoMo workspace minds (~/.hive-mind/workspaces/proj-locomo-*), current (2026-06-29) build.
Zep + LangMem stay literature-sourced (founder decision 2026-06-16).

3.5 KiB Raw Permalink Blame History Unescape Escape