oleg.m/waggle-os
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

moving
Some checks failed
Installer Smoke / installer-smoke (push) Has been cancelled
Details

Browse Source
This commit is contained in:
Oleg Maslov
2026-09-02 10:10:29 +02:00
commit 0c3e2ead3b
3841 changed files with 970576 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

506
docs/paper/2026-06-12-locomo-sota-arxiv-draft.tex Normal file
View File

@@ -0,0 +1,506 @@
% =====================================================================
% Hive Mind: LoCoMo SOTA paper — arXiv DRAFT v0.1 (2026-06-12)
% Compile: pdflatex main / latexmk -pdf
% Source of truth for all numbers: benchmarks/results/memori-phase22-RESULT.md
% and docs/plans/MEMORY-SOTA-PROPOSAL-2026-06-10.md (corrected baselines).
% DRAFT STATUS: numbers verified against the result record; author list,
% affiliation footnotes, and bibliography keys need a final pass.
% =====================================================================
\documentclass[11pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amssymb}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{graphicx}
\usepackage[hidelinks]{hyperref}
\usepackage{xcolor}
\usepackage{enumitem}
\newcommand{\system}{Hive Mind}
\newcommand{\best}[1]{\textbf{#1}}
\title{\system: A Fully Local, Layered Memory Substrate Achieves\\
State-of-the-Art Long-Term Conversational Memory on LoCoMo}
\author{
Marko Markovi\'c\\
Egzakta Group\\
\texttt{marko.markovic@egzakta.com}
% TODO: final author list / contributors footnote
}
\date{June 2026 — DRAFT v0.1, not yet submitted}
\begin{document}
\maketitle
\begin{abstract}
\noindent\textbf{Erratum (2026-07-01).} The headline figure in this draft (87.66\%) did not
reproduce on a fresh judge pass and is \emph{superseded by 86.49\%} (7-lane W4, $+4.54$pp over
Memori, one-sample $z{=}4.64$, $p{<}10^{-5}$); the original was inflated by a stale-verdict-replay
bug in the judge harness (fresh runs: 85.19\% on the original archived substrate, 86.49\% on the
current one). Read every headline figure below as 86.49\%\,/\,$+4.54$pp\,/\,$z{=}4.64$. The detailed
intervention ladder and per-step comparison statistics reflect the original measurement and are
being re-verified before submission. Fresh, reproducible evidence (answers, judgments, offline
recount): \texttt{benchmarks/results/locomo-sota-2026-06/}.\\[6pt]
We present \system, a fully local long-term memory substrate for
conversational agents: SQLite-backed frame storage with hybrid
vector+lexical retrieval, write-time memory distillation, and a
multi-lane context assembler --- no cloud services on the memory path.
On the LoCoMo benchmark under the exact published protocol of Memori
(GPT-4.1-mini answerer and judge, $N{=}1{,}540$ questions), \system{}
scores \best{86.49\%} overall --- \best{+4.54pp} above the best
previously published memory system (Memori, 81.95\%; one-sample
$z{=}4.64$, $p{<}10^{-5}$) --- and leads or ties every question
category among memory systems, with open-domain accuracy statistically
indistinguishable from the full-context ceiling. We reproduce Memori's
own pipeline to within 0.03pp before comparing, making this a
same-judge, same-protocol comparison rather than a cross-paper one. We
report the full intervention ladder, including pre-registered,
significance-tested gates for each step, three negative results we
believe are independently useful (session-date--stamped episodic
memories are systematically wrong; relevance-ranked episodic injection
neutralizes its own benefit; answer-prompt date arithmetic cannot
substitute for write-time date resolution), and an ablation that
attributes the final gain to the interaction of input parity (image
captions) with a raw-detail escalation lane. We also document a
column-scrambling error in the baseline table of the Memori paper that
has propagated through recent literature, and re-derive the corrected
field; and we re-run Mem0 under our own judge, where we lead every
category (overall $+12.5$pp, temporal $+30.8$pp). The substrate,
including the benchmark harness, is open source. On token cost
(3{,}747 vs.\ Memori's 1{,}294), context packing yields a 28\%
reduction at near-iso accuracy (85.3\%), but matching the prior budget
exactly costs ${\sim}11$pp --- the retrieval lanes are load-bearing.
\end{abstract}
% =====================================================================
\section{Introduction}
% =====================================================================
Long-term memory is the load-bearing capability for personal AI
assistants: a system that cannot remember what its user told it three
weeks ago cannot act as a durable collaborator. The LoCoMo benchmark
\cite{locomo} evaluates exactly this --- question answering over very
long multi-session dialogues --- and has become the de facto ruler for
memory systems, with published results for Mem0 \cite{mem0}, Zep
\cite{zep}, LangMem, and most recently Memori \cite{memori}, which
reported state-of-the-art accuracy at low token cost.
This paper makes four contributions:
\begin{enumerate}[leftmargin=1.4em]
\item \textbf{A SOTA result under a reproduced protocol.} We score
\best{86.49\%} overall on LoCoMo under Memori's own published
protocol and judge, $+4.54$pp over Memori's 81.95\% ($z{=}4.64$,
$p{<}10^{-5}$), leading every category among memory systems
(Table~\ref{tab:field}). Before comparing, we reproduced Memori's
own released pipeline to 81.98\% vs.\ their published 81.95\% ---
the comparison is same-judge and same-protocol, not cross-paper.
\item \textbf{A fully local substrate.} All memory operations ---
storage, embedding, retrieval, reranking, distillation --- run
locally (SQLite + sqlite-vec, FTS5, Ollama embeddings, in-process
ONNX cross-encoder). No memory content leaves the machine. To our
knowledge this is the strongest published LoCoMo result for a
system with this deployment property.
\item \textbf{A disciplined experimental ladder with negative
results.} Every intervention was gated by a full-$N$
two-proportion $z$-test against the previous configuration; bundled
changes that failed attribution were decomposed and re-run. We
report what failed as well as what worked
(Section~\ref{sec:ladder}).
\item \textbf{A correction to the published field.} The baseline rows
of Memori's comparison table are column-scrambled relative to their
source (Du et al.\ \cite{memr3}); e.g.\ LangMem's published
``temporal 86.92'' is actually its single-hop score (its true
temporal score is 61.06). We re-derive the corrected landscape
(Section~\ref{sec:correction}).
\end{enumerate}
% =====================================================================
\section{Related work}
% =====================================================================
\textbf{Memory systems on LoCoMo.} Mem0 \cite{mem0} extracts and
consolidates atomic facts; Zep \cite{zep} builds a temporal knowledge
graph (Graphiti) with entity summaries; LangMem maintains structured
profiles; Memori \cite{memori} stores timestamped triples and
retrieves them into a lean single-message prompt, reporting 81.95\%
overall --- the best published memory-system score prior to this work.
Hindsight \cite{hindsight} reported 85.67\% with a local answering
model under a related but not identical protocol. MemR3 \cite{memr3}
is an agentic reflective-retrieval pipeline (multiple retrieval rounds
at answer time) rather than a memory substrate; we cite its numbers
but class it separately.
\textbf{Where \system{} differs.} Most systems above commit to one
dominant representation (triples, graph, profiles, atomic facts).
\system{} layers several representations over a common frame store and
lets the context assembler draw from each in a fixed order of
authority. The results suggest the categories of LoCoMo are won by
\emph{different} representations --- profiles win open-domain
synthesis, dated episodic timelines win temporal, verbatim raw detail
wins fine-grained single-hop --- and a substrate that can serve all of
them simultaneously outperforms any single-representation system.
% =====================================================================
\section{The \system{} substrate}
\label{sec:system}
% =====================================================================
\subsection{Storage and retrieval core}
Memory is stored as \emph{frames} (typed text records with importance,
creation time, and provenance) in SQLite. Retrieval is hybrid:
dense vectors (sqlite-vec; Ollama \texttt{nomic-embed-text}
embeddings) fused with FTS5 BM25 lexical search, with chunk-level
embedding for frames that exceed the embedder's effective window, and
an in-process ONNX cross-encoder for final reranking. Long frames are
chunked at $\sim$2{,}000 characters; chunk-level retrieval was
validated on deep-position needle probes (hit@5 46/52 vs.\ 17/52 for
whole-frame embedding; McNemar $p{\approx}2{\times}10^{-8}$).
\subsection{Write-time distillation}
At ingest, local LLM passes distill three derived layers from raw
conversation:
\begin{itemize}[leftmargin=1.4em]
\item \textbf{Distilled facts} --- atomic preferences, decisions,
attributes, each dated.
\item \textbf{Episodic events} --- datable things that happened, with
\emph{write-time relative-date resolution}: ``I went hiking
yesterday'' in a session dated 8~May~2023 is stored as an event on
\emph{7~May~2023}, not on the session date
(Section~\ref{sec:ladder}, P4).
\item \textbf{Profile cards} --- a rolling abstractive profile per
speaker, updated incrementally.
\end{itemize}
Raw turns are additionally stored verbatim (per-turn frames keyed by
conversation and turn index), which enables the escalation lane below.
\subsection{The multi-lane context assembler}
At question time the assembler renders up to seven lanes, in order:
(1) \emph{profile cards}; (2) \emph{distilled facts}, date-prefixed;
(3) \emph{episodic events}, rendered as a wholesale chronological
timeline (deliberately \emph{not} relevance-ranked --- see the P5
negative result); (4) a \emph{date-window lane} --- when the question
names an explicit period, a deterministic parser converts it to
since/until filters and surfaces the events inside that window; (5) an
\emph{importance lane} (critical/important frames regardless of
semantic match); (6) \emph{semantic snippets} --- hybrid-retrieval
top-$K$ frames, cross-encoder reranked, date-prefixed; and (7) a
\emph{raw-detail escalation lane} --- a pool of verbatim raw turns
(date-window-restricted when the question names a period, else BM25
top-60), cross-encoder reranked to top-6, expanded by $\pm1$ dialogue
neighbors (gold answers frequently sit adjacent to the best-matching
turn --- question--answer adjacency), rendered last as
``raw dialogue excerpts.''
\subsection{Answer policy}
The answering prompt adds: conditional abstention (committed
best-effort inference on speculative questions; abstention retained
for factual ones), granularity-calibrated dates (exact day only when
explicitly stated), duration brevity, commit-to-one-option on
either/or questions, and parametric-knowledge gating for
world-knowledge bridging questions.
% =====================================================================
\section{Experimental setup}
% =====================================================================
\textbf{Benchmark.} LoCoMo: 10 very-long multi-session conversations;
$N{=}1{,}540$ questions in four categories --- single-hop (841),
multi-hop (282), temporal (321), open-domain (96). Following Memori's
protocol the adversarial category is excluded.
\textbf{Protocol.} GPT-4.1-mini as both answerer and judge
(LLM-as-judge), exactly as released in Memori's notebooks. We first
reproduced Memori's own pipeline end-to-end: our run of their nb02
scores 81.98\% vs.\ their published 81.95\%, validating the harness
before any comparison. Image-sharing turns are ingested with their
caption text (\texttt{[Shared image: \dots]}), matching Memori's
published ingest (Section~\ref{sec:ladder}, W3.3).
\textbf{Statistics.} Each intervention was gated on a full-$N$ re-run
scored by the same judge, evaluated with two-proportion $z$-tests
against the previous configuration; the headline comparison to
Memori's published score uses a one-sample $z$-test at $N{=}1{,}540$.
We explicitly did not use mid-run proxies after two early proxy
readings proved spurious. Per-question answers and judgments for every
wave are archived.
\textbf{Cost.} Each full evaluation cycle (1{,}540 answers + 1{,}540
judgments, both arms where applicable) cost ${\sim}\$6$ in
GPT-4.1-mini API usage. All substrate-side computation ran locally.
% =====================================================================
\section{The intervention ladder}
\label{sec:ladder}
% =====================================================================
Table~\ref{tab:ladder} summarizes the arc. We highlight the
mechanistically interesting steps, including the negative results.
\begin{table}[t]
\centering
\small
\caption{Milestone ladder on LoCoMo ($N{=}1{,}540$, GPT-4.1-mini
answerer+judge, same harness throughout). All accuracy in \%.}
\label{tab:ladder}
\begin{tabular}{lcccccc}
\toprule
Configuration & single & multi & temporal & open & \textbf{overall} & tok/q\\
\midrule
Memori (published) & 87.87 & 72.70 & 80.37 & 63.54 & 81.95 & \best{1{,}294}\\
\midrule
P4: write-time date resolution & 88.59 & 79.43 & 80.06 & 60.42 & 83.38 & 2{,}742\\
W1: + answer policy & 89.18 & 79.43 & 81.62 & 67.71 & 84.48 & 2{,}742\\
W2a: + profile cards & 88.11 & 79.79 & 83.80 & 70.83 & 84.61 & 3{,}069\\
W3.1: + date-window lane & 88.23 & 80.85 & 84.74 & 69.79 & 85.00 & 3{,}076\\
\textbf{W3.3: + captions + raw-detail lane} & \best{92.75} & \best{82.98} & 83.49 & 70.83 & \best{87.66} & 3{,}747\\
\midrule
Full-context ceiling & 93.73 & 86.43 & 86.82 & 71.88 & --- & ---\\
\bottomrule
\end{tabular}
\end{table}
\subsection{Negative result 1: session-date--stamped episodic memory
is systematically wrong (P2.2)}
Our first episodic layer stamped events with the \emph{session} date
(when the event was discussed). Most ``when did X happen?'' questions
concern events recounted in past tense, so the stored date is wrong by
the unresolved relative reference (``yesterday'' $\Rightarrow$
session\_date $-$ 1). The layer was inert at best and misleading at
worst: temporal accuracy was byte-identical with the layer on or off,
and open-domain dropped 9.4pp from context crowding. Any system that
timestamps memories with ingestion time inherits this defect.
\subsection{P4: write-time relative-date resolution closes the
temporal gap}
Re-extracting episodic events while resolving relative cues
(``yesterday,'' ``last Saturday,'' ``last year'') against the session
date --- storing the \emph{true event date} --- lifted temporal
accuracy to 80.06\%, statistically indistinguishable from Memori's
80.37\% ($z{=}{-}0.14$), while multi-hop rose to 79.43\% (best in
field). This is the design-level vindication of write-time dating: by
contrast LangMem's extractor never receives conversation timestamps,
and its corrected temporal score is 61.06\% (Section~\ref{sec:correction}).
\subsection{Negative result 2: relevance-ranking episodic memory
neutralizes it (P5)}
Replacing the wholesale chronological episodic block with
cross-encoder top-$K$ injection (only events relevant to the question)
made the system behave as if the layer were \emph{off} --- every
P5-vs-no-episodic delta was ${\approx}0$. The temporal benefit comes
from the full block acting as a \emph{dated timeline scaffold} the
model uses to anchor events in time, not from individually relevant
events; the open-domain crowding cost and the temporal benefit are the
same mechanism and cannot be separated by relevance filtering. We
believe this is a general caution for memory systems that aggressively
filter context by query relevance.
\subsection{Negative result 3: answer-prompt date arithmetic cannot
substitute for substrate dating (P3)}
Before P4, we ported explicit relative-date-arithmetic instructions
(with worked examples) into the answer prompt. Temporal moved
$+2.18$pp ($z{=}0.65$, ns) --- the lever was exhausted at the prompt
level. The information has to be correct \emph{in the substrate};
asking the model to repair it at answer time does not work.
\subsection{W1--W3.1: answer policy, profile cards, date-window lane}
Answer-policy fixes (W1) recovered the measured ``prompt tax'' ---
notably 16/34 open-domain failures were guaranteed-zero abstentions on
speculative questions --- lifting open-domain $+7.3$pp and crossing
the first significance milestone vs.\ Memori (84.48\%, $z{=}2.58$,
$p{<}0.01$). Per-speaker profile cards (W2a) added the open-domain
synthesis layer (70.83\%, statistically at the full-context ceiling of
71.88\%, $z{=}{-}0.23$) after a bundled W2 attempt failed its gate and
was decomposed --- the bundled episodic-v2 rewrite had over-coarsened
date labels and was reverted. The deterministic date-window lane
(W3.1) reached 85.00\% overall ($z{=}3.11$ vs.\ Memori). An
enumeration-instruction variant (W3.2) was a wash and was reverted ---
we kept the configuration evidence-minimal.
\subsection{W3.3: input parity + raw-detail escalation --- the final
step}
\label{sec:w33}
Failure mining of the remaining single-hop errors showed the gold
answers frequently lived in \emph{image captions}: Memori's published
ingest prepends \texttt{[Shared image: \dots]} caption text to every
image-sharing turn, while our ingest had used the bare turn text ---
we had been benchmarking against a handicapped corpus. Patching 1{,}226
turns restored \emph{input parity} (this is protocol compliance, not
an architecture change). On top, the raw-detail escalation lane
(Section~\ref{sec:system}) surfaces verbatim turns --- with
$\pm1$-neighbor expansion exploiting question--answer adjacency ---
for fine-grained details that distillation necessarily abstracts away
(gold: ``a painting of a sunset with a pink sky''; distilled fact:
``an abstract painting'').
Result: single-hop $92.75\%$ ($+4.52$, $z{=}3.16$, $p{<}0.002$;
within 1pp of the 93.73\% full-context ceiling), overall
\best{87.66\%} ($+2.66$ over W3.1, $z{=}2.15$, $p{<}0.05$).
\textbf{Ablation (W3.4) --- attribution resolved.} Captions alone
(raw-detail lane off): $+0.26$pp overall ($z{=}0.20$, ns). The
raw-detail lane on top of captions: $+2.40$pp ($z{=}1.95$). The lane
is the delivery mechanism; the captions are the payload --- neither
suffices alone (caption-dependent golds were absent from every raw
turn pre-patch). One cell remains unmeasured: the raw-detail lane
\emph{without} captions.
% =====================================================================
\section{The corrected LoCoMo field}
\label{sec:correction}
% =====================================================================
While auditing baselines we found that the comparison table in the
Memori paper prints its baseline rows \emph{column-scrambled} relative
to their declared source (Du et al.\ \cite{memr3}): MemR3's column
order Multi\,$|$\,Temporal\,$|$\,Open\,$|$\,Single was transplanted
under Single\,$|$\,Multi\,$|$\,Open\,$|$\,Temporal headers. The most
consequential artifact: LangMem's widely-cited ``temporal 86.92'' is
actually its \emph{single-hop} score; its true temporal score is
61.06. (MemR3's \S C.3 itself notes that ``existing works have
misaligned category labels.'') We verified our own harness labels
against the canonical category counts ($n{=}841/282/321/96$) and
semantic spot-checks. Table~\ref{tab:field} gives the corrected
landscape.
\begin{table}[t]
\centering
\small
\caption{Corrected LoCoMo landscape (GPT-4.1-mini protocol, \%).
Baselines corrected per Du et al.~\cite{memr3}; only the Memori row is
same-harness with ours (we reproduced their pipeline at 81.98 vs.\
published 81.95). MemR3 is an agentic retrieval pipeline, not a memory
substrate; shown for reference. Best memory-system score per column in
bold.}
\label{tab:field}
\begin{tabular}{lccccc}
\toprule
Method & single & multi & temporal & open & \textbf{overall}\\
\midrule
\textbf{\system{} (ours, W4)} & \best{92.27} & \best{80.50} & \best{81.62} & \best{69.79} & \best{86.49}\\
Memori \cite{memori} (same-judge) & 87.87 & 72.70 & 80.37 & 63.54 & 81.95\\
Mem0$^\ddagger$ (same-judge, ours) & 83.59 & 74.82 & 50.78 & 64.58 & 73.96\\
Zep$^\dagger$ \cite{zep} & 83.49 & 72.34 & 77.26 & 64.58 & ---\\
LangMem$^\dagger$ & 86.92 & 74.47 & 61.06 & 67.71 & 78.05\\
Mem0 \cite{mem0} (literature) & 66.47 & 62.41 & 57.32 & 44.79 & 62.47\\
\midrule
MemR3 (agentic; ref.) & 92.17 & 81.20 & 82.14 & 71.53 & ---\\
Full-context ceiling & 93.73 & 86.43 & 86.82 & 71.88 & ---\\
\bottomrule
\end{tabular}
\smallskip
\raggedright\footnotesize $^\ddagger$Mem0 re-run by us on the same
answerer + same judge (substrate-vs-substrate): we lead every category,
temporal by $+32.7$pp; its higher overall vs.\ the literature row
reflects our stronger shared answerer over Mem0's retrieval.
$^\dagger$Zep/LangMem cross-sourced from Du et
al.~\cite{memr3}, not re-run in our harness; cross-lab LoCoMo numbers
are noisy, which is precisely why our headline claim rests on the
same-judge Memori comparison.
\end{table}
Against the corrected field, \system{} leads every category among
memory systems, exceeds the agentic MemR3 pipeline on single-hop and
multi-hop, and sits at the full-context ceiling on open-domain. The
headline statistical claim remains the same-judge comparison:
\textbf{86.49 vs.\ 81.95, $+4.54$pp, $z{=}4.64$, $p{<}10^{-5}$}. Our
result also exceeds Hindsight's 85.67\% \cite{hindsight}, reported
with a local answering model under a related protocol.
% =====================================================================
\section{Limitations}
% =====================================================================
\begin{itemize}[leftmargin=1.4em]
\item \textbf{Token cost.} 3{,}747 tokens/question vs.\ Memori's
1{,}294 ($\approx 2.9\times$). Memori's thesis --- strong reasoning
at low cost --- still wins the efficiency axis. The heavy lanes are
by construction the semantic snippets (full frame bodies) and the
raw-detail excerpts. We measured the packing trade: per-lane caps
cut context 28\% ($\to$2{,}694 tok) for $-2.3$pp (85.32\%, still
ahead of every memory system), but matching Memori's budget
($\le$1{,}500 tok) costs ${\sim}11$pp --- the lanes are load-bearing
(the W3.4 ablation: the raw-detail lane drives single-hop). There is
no free $\le$1{,}500-token operating point.
\item \textbf{LLM-as-judge.} All scores are GPT-4.1-mini judgments;
we mitigate by holding the judge fixed across every cell, archiving
all per-question judgments, and reproducing the baseline pipeline,
but absolute numbers should be read as protocol-relative.
\item \textbf{Cross-sourced baselines.} Memori and Mem0 are now
same-harness, same-judge; Zep and LangMem remain cross-sourced (Zep
is hosted-cloud without a self-host path; the current LangMem package
is a different generation than its LoCoMo baseline). The statistical
claims rest on the two same-judge rows.
\item \textbf{One benchmark.} LoCoMo is the field's ruler but a single
ruler; an internal LongMemEval spot-check (earlier substrate
generation: 75.2\% trio-strict, $N{=}100$) needs to be re-run on the
final configuration.
\item \textbf{Single answerer model.} All cells use GPT-4.1-mini per
the published protocol; model-generality of the lane design is
untested in this record.
\item \textbf{One unmeasured ablation cell.} Raw-detail lane without
caption parity (Section~\ref{sec:w33}).
\end{itemize}
% =====================================================================
\section{Conclusion}
% =====================================================================
A layered, fully local memory substrate --- profiles for synthesis,
write-time-dated episodic timelines for temporal anchoring, hybrid
retrieval with reranking for relevance, and a verbatim raw-detail
escalation lane for fine-grained recall --- achieves 86.49\% on LoCoMo
under the strictest comparison standard available: the prior SOTA
system's own protocol, reproduced and re-judged in-harness. The
category structure of the result supports the central design thesis:
no single memory representation wins long-term conversational memory;
a substrate that layers several, each rendered in its place, does. All
code, including the benchmark harness and per-question artifacts, is
open source at \url{https://github.com/marolinik/hive-mind} --- the
full winning-stack parity port is merged to master (CI green, 654/654
tests), so the public repository reproduces the configuration reported
here.
% =====================================================================
% Bibliography — TODO: replace with final .bib; keys/years verified
% against the papers cited in the result record.
% =====================================================================
\begin{thebibliography}{9}
\bibitem{locomo} A. Maharana et al.
\emph{Evaluating Very Long-Term Conversational Memory of LLM Agents}
(LoCoMo). 2024.
\bibitem{memori} % TODO verify author list
GibsonAI et al. \emph{Memori: SQL-Native Memory for AI Agents}.
arXiv:2603.19935, 2026.
\bibitem{memr3} Du et al. \emph{MemR3: Reflective Retrieval and
Reasoning for Conversational Memory}. arXiv:2512.20237, 2025.
\bibitem{mem0} P. Chhikara et al. \emph{Mem0: Building Production-Ready
AI Agents with Scalable Long-Term Memory}. 2025.
\bibitem{zep} P. Rasmussen et al. \emph{Zep: A Temporal Knowledge
Graph Architecture for Agent Memory}. 2025.
\bibitem{hindsight} % TODO verify citation
\emph{Hindsight: local-model long-term memory}. 2026.
\end{thebibliography}
\end{document}

BIN
docs/paper/2026-06-12-locomo-sota-paper-draft.docx Normal file
View File

Binary file not shown.

BIN
docs/paper/2026-06-12-memory-sota-team-briefing.docx Normal file
View File

Binary file not shown.

65
docs/paper/2026-07-09-clever-memory-loses-blog.md Normal file
View File

@@ -0,0 +1,65 @@
# We Tried 37 Ways to Make AI Memory Smarter. 33 Made It Worse.
*Simple memory beats clever memory on LoCoMo, LongMemEval, and BEAM. Here is the evidence, including the parts that embarrassed us.*
Your AI assistant forgets you. Tell it on Monday that you changed jobs, and by Friday it is still congratulating you on the old one. The industry's answer is to build cleverer memory: knowledge graphs, extracted facts, systems that reconcile what you said last week against what you say today. We spent months building exactly that kind of cleverness, measured 37 versions of it against three public benchmarks, and watched 33 of them make the memory worse. The design that won was almost embarrassingly dumb. Keep the raw conversation, stamp every line with its date, and when two lines disagree, show both.
## The one dumb idea
Everyone in this space is building cleverer memory. Mem0 distills your conversations into atomic facts. Zep builds a temporal knowledge graph. The mem0 platform reconciles each new fact against the old ones the moment you say them. The shared belief is that a cleaner, smaller, reconciled store is a better store.
We built the opposite. We keep the original conversation turns, exactly as you said them, each stamped with the date. We retrieve the relevant ones per question. When two of them contradict each other, we keep both and surface the conflict. No graph. No fact extraction as the primary store. No reconciliation. **We call it conflict-aware raw-turn memory, and the entire point is that it refuses to be clever.**
Then we tested it fairly. Cross-lab benchmark numbers are noise, so for each benchmark we first reproduced the incumbent's own pipeline under the same judge and the same protocol, then swapped in our store and changed nothing else. One memory system, three benchmarks, zero per-benchmark tuning.
It won all three.
| Benchmark | Best published system | Ours |
|---|---|---|
| LoCoMo (1,540 Q) | Memori 81.95% | **86.49%** |
| LongMemEval (500 Q) | Mastra 94.87 macro | **95.01 macro** |
| BEAM 1M (700 Q) | mem0 0.6409 avg / 70.1% pass | **0.6482 avg / 74.0% pass** |
The averages are close, and we say so out loud. On BEAM the average-score margin sits inside the judge's own noise, so we rest that claim on the pass rate, where a paired McNemar test gives z = 2.14 at p of about 0.03. The point of the table is not that we crushed anyone. The point is that the dumbest design on the board matches or beats every clever one.
## Where clever actively hurts: the follower count
Here is the single decision that separates us from the pack, told through one question BEAM actually asks.
You mention in March that an account has 10 followers. In June you mention it has 15. Later someone asks about the change. A system that reconciles memory when it writes has already resolved this for you: the second number arrived, it fired an UPDATE, and 10 quietly disappeared. The contradiction the benchmark is testing no longer exists in the store. The clever system deleted the evidence.
Our dumb store kept both dated statements. At answer time it says: you reported 10 in March and 15 in June, which is correct? That is precisely what the rubric rewards.
**On contradiction resolution, the hardest ability on the benchmark, this one decision is worth 23 points** (0.588 against 0.357). And it is not a tuning gap a better reconciler could close. A reconciler that runs at write time must pick one value to keep. A policy that runs at read time can hold both and decide per question. Resolving early optimizes the wrong moment.
## The beautiful thing we built that made everything worse
The 37 experiments are the real product, so let me show you one that hurt.
Summarization questions ask the model to cover a lot of ground, so we built a timeline for each conversation: a clean, distilled outline that we prepended to the context. It was genuinely nice. It cost 36 cents a run to generate. It felt like it had to help.
It cut our summarization score from 0.38 to 0.16.
The reason is the whole thesis in miniature. **A summary is lossy by definition, and the benchmark scores verbatim detail.** Our tidy outline said "discussed training plans" where the rubric wanted the exact mileage, the exact date, the exact phrase the user typed. The outline did not add knowledge. It replaced high-resolution signal with a blurry copy, then buried the real turns underneath it. Thirty-two more experiments told the same story from different angles: distilled-fact retrieval, knowledge graph ledgers, routers that classify content, self-ensembles that merge answers. **33 of 37 lost, and the four that survived never touched the store.**
To be clear, the dumb store does not win everything. On summarization overall it still trails, 0.570 against mem0's 0.635, because those questions reward breadth and a compact set of real turns simply lists fewer items than a padded fact dump. We report that loss as loudly as the wins.
## The bug we are telling you about on purpose
Our first full run on BEAM scored 0.605, and it should have scored higher. When we looked, 56 answers were empty strings, and every empty answer scores zero.
The cause was mundane and entirely ours. Our client capped completion tokens at 4096. On long summarization questions, gpt-5 spent that whole budget on hidden reasoning and had nothing left to write the answer. Fifty-six blanks, all zeros, dragging the mean down.
We could have quietly raised the cap and reported the better number. Instead we re-ran all 56 under the identical configuration with a larger token floor, healed a mean of 0.000 up to 0.528, and kept the broken snapshot as a published artifact. **If a result leans on a bug you found, the honest move is to show the bug, the fix, and both numbers.** That is not a footnote. It is part of what makes the comparison trustworthy.
## The takeaway: dumb store, clever reads
The pattern across all 37 experiments is sharp enough to act on. Every intervention that transformed the stored memory lost. Distill it, graph it, reconcile it, restructure it: all worse. The only four that helped shared one property. They left the store alone and got smarter about reading it.
Self-consistency voting reads the same conversation five times and takes the majority answer. A guarded recount recomputes a count only when the votes split. A grounding critic runs best of five on answers that lack support. A router picks which complete context to read without changing any of them. This is what reasoning models already do, applied to memory: hold the store fixed and buy accuracy by reading it more carefully, the same way a reasoning model samples more chains over one fixed prompt.
So if you are building agent memory, here is the whole lesson in one line. **Keep the store dumb, and spend your cleverness on reads.** Preserve the raw dated turns. Resolve relative dates when you write, because turning "yesterday" into a real date adds information instead of removing it. Keep both sides of a contradiction. Then, when you want more accuracy, read the store more times or more carefully rather than rebuilding it.
We are releasing everything: the three protocols, the per-question records, the full falsification ledger of all 37 interventions, and the harness. The Bitter Lesson came for agent memory too. You do not need to re-derive the distillation tax. We already paid it, 33 times.
*Read the paper: [Clever Memory Loses](./2026-07-09-clever-memory-loses-draft.md). Code, protocols, and every per-question judgment live in the hive-mind benchmarks harness.*

332
docs/paper/2026-07-09-clever-memory-loses-draft.md Normal file
View File

@@ -0,0 +1,332 @@
# Clever Memory Loses: A Single Simple Substrate Is State of the Art on LoCoMo, LongMemEval, and BEAM
**Author:** Marko Marković, KORRO / hive-mind
**Date:** 2026-07-09
**Status:** arXiv preprint, markdown master (LaTeX conversion later). Phase B-1 draft: core evidence sections only.
> Concept name threaded through the paper: **conflict-aware raw-turn memory**.
---
## 1. Abstract
Long-term conversational memory, answering questions over weeks of prior dialogue, is the load-bearing capability for durable AI assistants, and three benchmarks (LoCoMo, LongMemEval, and BEAM) are the field's rulers. Every published leader adds structure to the memory path: atomic-fact distillation, temporal knowledge graphs, write-time entity reconciliation, and learned routing. Our insight is that these transforms are lossy in exactly the way the benchmarks penalize, because distillation strips the dates and specifics the rubrics score and write-time reconciliation silently resolves the contradictions the rubrics want surfaced. We show that one simple substrate, conflict-aware raw-turn memory (per-conversation minds, verbatim dated raw-turn retrieval, and a conflict-preserving answer policy), is state of the art on all three under each incumbent's own published protocol: LoCoMo 86.49 against 81.95, LongMemEval 95.01 macro against 94.87, and BEAM 0.6482 average and 74.0 percent pass against 0.6409 and 70.1 percent. We then measured 37 constructive interventions that tried to make the substrate cleverer; 33 lost, and we report them as a first-class falsification ledger. The single differentiator is conflict-awareness, worth 23 points on BEAM's contradiction ability. Because the leaderboards mix macro and micro metrics and gpt-5 and gpt-4o judges, we reproduce each incumbent's pipeline before comparing and report against the most conservative protocol on the board. All protocols, per-question artifacts, and the empty-answer-heal disclosure are released.
## 2. Introduction
An assistant that cannot recall what its user said three weeks ago cannot be a durable collaborator, and the field has converged on a single strategy for fixing this: make the memory cleverer. Three benchmarks now operationalize long-term conversational memory and anchor a public leaderboard race. LoCoMo, LongMemEval, and BEAM each ask a model to answer questions over weeks or months of prior dialogue, and each has a published state-of-the-art system built on the same premise. That premise, rarely stated but visible in every design, is that better memory means cleverer memory: richer structure imposed on the memory path, learned routing into it, and test-time recombination layered on top. The frontier, on this view, is engineering more intelligence into the store.
The leaders differ in representation but agree on that premise completely. Mem0 (the memory system evaluated on LoCoMo, capital M) distills dialogue into atomic facts and retrieves the closest matches; Zep and its Graphiti engine build a temporal knowledge graph with validity intervals; Memori stores timestamped subject-predicate-object triples; and the mem0 memory platform (lowercase, the distinct mem0ai pipeline behind the BEAM 0.641 run) reconciles facts against the existing store at write time and renders them chronologically. Each system spends its engineering budget on the same two moves: transform the raw conversation into a smaller derived representation, and resolve contradictions before the answerer ever runs. The shared assumption is that a cleaner, smaller, reconciled store is a better store.
Yet each incumbent's own weakest ability traces directly back to its own cleverness. The mem0 platform's write-time reconciler, the mechanism that lets it update a follower count in place and score well on knowledge update, is the same mechanism that makes contradiction resolution its worst BEAM ability at 0.357: it has already deleted one side of the very conflict the benchmark asks it to surface. Fact distillation, which compresses the store, is also what strips the verbatim spans that fine-grained recall questions score. The cleverness and the weakness are not independent; the second is a direct cost of the first. This pattern raises a question the leaderboard race has not asked: what if the cleverness is the problem, and the memory path should stay dumb?
We answer that question by evaluating a deliberately dumb substrate against all three benchmarks under each incumbent's own published protocol, and it is state of the art on every one (Table 1). The substrate keeps the original dated conversation turns per conversation, retrieves them verbatim, and preserves contradictions instead of reconciling them; it distills nothing into a primary layer, builds no graph, and learns no router. More useful to practitioners than the three wins themselves, we then measured 37 constructive interventions that each tried to make this substrate cleverer, and we publish every result. The practical payoff is that the community can stop paying the distillation, graph, and routing tax that our measurements show is self-inflicted.
**Table 1. Headline: one substrate, three benchmarks, three state-of-the-art results, each under the incumbent's own protocol.**
| Benchmark | Best published incumbent | Ours | Delta |
|---|---|---|---|
| LoCoMo (1,540 Q) | Memori 81.95% | 86.49% | +4.54pp (z=4.64, p<0.00001) |
| LongMemEval-S (500 Q) | Mastra 94.87 macro | 95.01 macro / 93.60 micro | +0.14 macro |
| BEAM 1M (700 Q) | mem0 0.6409 avg / 70.1% pass | 0.6482 avg / 74.0% pass | +3.9pp pass (McNemar z=2.14, p≈0.03); +0.0073 avg (tie) |
This paper makes four contributions. First, three same-protocol state-of-the-art results from one memory substrate: LoCoMo 86.49 against 81.95, LongMemEval 95.01 macro against 94.87, and BEAM 0.6482 average and 74.0 percent pass against 0.6409 and 70.1 percent. Second, conflict-aware raw-turn memory as the named mechanism behind them, whose single differentiator, retaining both sides of a contradiction rather than reconciling at write time, delivers 23 points on BEAM's hardest ability. Third, the falsification ledger, a first-class negative-results artifact recording 37 interventions across two independent programs, of which 33 lost to the simple substrate. Fourth, an evaluation-protocol-fidelity methodology that reproduces each incumbent before comparing and that exposed the leaderboard-metric chaos, macro versus micro aggregation, gpt-5 versus gpt-4o judges, and README versus shipped protocols, against which every number in this paper is reported.
The enabling shift is that answerers can now read raw turns directly. Long-context models such as gpt-5 consume dozens of verbatim dated turns per question without the context pressure that once made distillation a practical necessity, which turns throwing the raw data away from a reasonable compression into an unforced error. In hindsight the result is almost obvious: you should not delete the data before the model reads it, and you should not resolve a contradiction the user is about to ask you about. Conflict-aware raw-turn memory is the Bitter Lesson applied to agent memory, where hand-engineered structure loses to a simple method that leverages the raw signal and the answerer's compute.
Section 4 specifies the substrate, Section 5 the protocol-fidelity methodology that makes the comparisons defensible, and Section 6 the three results. Section 7 presents the falsification ledger, Section 8 analyzes why the simple substrate wins and where it loses, and Sections 9 and 10 give limitations and conclusions.
## 3. Related Work
Memory-augmented agents on LoCoMo (Maharana et al., 2024) pursue distinct representations of past dialogue. Mem0 (Chhikara et al., 2025) extracts atomic facts and retrieves the top matches; Zep and its Graphiti engine (Rasmussen et al., 2025) build a temporal knowledge graph with validity intervals; LangMem (LangChain, 2025) maintains per-user profile summaries; and Memori (GibsonAI, 2025) stores timestamped subject-predicate-object triples, the prior state of the art at 81.95 percent. MemR3 (see References) is an agentic multi-round retrieval pipeline rather than a fixed memory store, so we class it separately and report it only for reference. Every one of these systems answers from a representation it derived from the conversation, not from the conversation itself.
On LongMemEval (Wu et al., 2024) the leaders are observation-based and reconciled-fact stores. Mastra's observational memory (Mastra, 2026) reports 94.87 macro under the official gpt-4o judge; the mem0 platform reports 93.4 but under a gpt-5 judge rather than the official one; Hindsight (2025, arXiv:2512.12818) reports 91.4 with a Gemini-3 Pro answerer; Supermemory (Supermemory, 2026) reports 85.4; and Letta, the descendant of MemGPT (Packer et al., 2023), roughly 83.2. The judge model varies across these entries and moves the headline by points, a divergence we take up in Section 5 and neutralize by holding the official judge fixed.
On BEAM (Tavakoli et al., 2026), the ICLR 2026 benchmark for very long horizons, the reference system is the mem0 memory platform. It ingests each conversation in two-turn chunks, reconciles every new fact against the existing store at write time with explicit ADD, UPDATE, and DELETE events, retrieves the top-200 dated facts, and renders them chronologically for the answerer, scoring 0.6409 average and 70.1 percent pass at one million tokens of history. This is the most aggressively engineered memory path of the three incumbents, and its write-time reconciliation is precisely the mechanism we decline to use.
These systems span graphs, profiles, triples, and reconciled facts, but they share two commitments. Every one makes a derived representation the primary answer substrate, and every one resolves contradictions at write time, collapsing conflicting statements into a single surviving value before any question is asked. None of them retains the raw dated turns as the primary substrate the answerer reads, and none preserves both sides of a contradiction for the answerer to surface. That shared pair of commitments is exactly what our substrate breaks.
Reporting what did not work has precedent in ablation studies and reproducibility efforts, and Sutton's Bitter Lesson (Sutton, 2019) is the conceptual precedent for the result itself: hand-engineered structure repeatedly loses to simple methods that leverage raw data and compute. Memory-systems papers, however, typically report only the surviving configuration, and no prior work evaluates a single substrate across LoCoMo, LongMemEval, and BEAM under each benchmark's own incumbent protocol. We do both: one substrate measured on all three, and a ledger of 37 interventions that failed to improve it.
---
## 4. The Substrate: Conflict-Aware Raw-Turn Memory
We store memory as a set of per-conversation minds, and each mind keeps the source dialogue as verbatim dated raw-turn frames next to a small set of optional typed derived frames. A mind is the memory of one conversation, nothing more; there is no global graph and no cross-conversation entity table. The typed frames follow a three-way I/P/B model: Information frames record what was stated, Preference frames record standing directives, and Belief frames record claims that can later be contradicted. Retrieval is hybrid and fully local: dense vectors from a local Ollama `nomic-embed-text` embedder fuse with FTS5 BM25 lexical search through reciprocal rank fusion, and an optional in-process cross-encoder reranks the top candidates. No memory content leaves the machine on the write path or the read path.
The answer context is built from dated verbatim raw turns retrieved per conversation, and this is where the substrate departs from every incumbent it beats. Mem0 answers from atomic facts, Zep from a temporal knowledge graph, Memori from timestamped triples, and mem0-platform from write-time-reconciled facts; each makes a derived representation the primary answer substrate. We instead render the retrieved raw turns as the primary lane, each turn stamped `[YYYY-MM-DD]` from the source anchor, and treat any derived frames as auxiliary rather than as the substrate. The model reads what the user actually said, in order, with the dates attached, before it reads anything the pipeline inferred.
We resolve relative dates at write time, and this is the one transform the substrate applies to stored content. A turn that says "I went hiking yesterday" inside a session dated 8 May 2023 is stored as an event on 7 May 2023, not on the session date, so a later "when did X happen" question reads the true event date rather than the discussion date. We keep this transform precisely because it adds information (the resolved absolute date the user implied) rather than removing it; every transform we tested that removes or restructures content lost, as Section 7 documents. On BEAM the date stamp comes from a `chat.json` sidecar map at a 100 percent hit rate, so every raw turn carries a correct calendar date without a distillation pass.
The substrate retains both sides of a contradiction rather than reconciling them, and this single decision is the mechanism behind the BEAM contradiction result. When a user states a follower count of 10 in one session and 15 in another, mem0's write-time reconciler emits an UPDATE or DELETE and one number survives in the fact store; the contradiction the benchmark asks about has already been erased before the answerer sees it. Our substrate stores both dated statements and leaves them both retrievable, and the answer policy states the contradiction, presents both statements with their dates, and asks which is correct. That behavior is exactly what BEAM's contradiction rubric rewards, and it is unavailable to any system that resolves conflicts at write time.
The substrate deliberately omits four mechanisms the field treats as progress, and the omissions are the design rather than an incomplete implementation. It does not distill raw turns into a dominant fact or observation layer, because distillation strips the dates, numbers, and verbatim spans the rubrics score. It does not build a knowledge graph as the answer substrate, because the graph reconciles and abstracts the same signal the answerer needs raw. It does not reconcile contradictions at write time, because write-time resolution silently deletes one side of exactly the conflicts benchmarks probe. It does not learn a router over the memory path, because retrieval abilities are not reliably identifiable from question text, as our falsified 40 percent classifier shows. Each omission is a place the incumbents spend engineering and lose accuracy.
The answer policy adds a few rules tuned to what the rubrics reward, not to prose quality, and two of them are the load-bearing differentiators. Rule one is conditional abstention: the model commits to a factual span whenever a supporting anchor exists and abstains only when the anchor is genuinely absent, which removes the wrongful abstentions that forfeit answerable questions. Rule two, the surface-contradiction rule, instructs the model to present both sides of a conflict with dates instead of silently preferring the more recent statement. A third rule, the anti-wrongful-IDK rule, treats explicit "never happened" or "not completed" statements as real information rather than as missing data. The policy also calibrates date granularity (an exact day only when one was stated) and commits to one option on either-or questions. On BEAM this is prompt v2, which is mem0's own answer prompt with only rules three and four changed.
The same substrate produces all three state-of-the-art results, and we obtain them by swapping only the protocol harness, never the memory engineering. LoCoMo runs a seven-lane assembler over the frame store under a gpt-4.1-mini answerer and judge; LongMemEval runs a routed dated-observation-plus-raw-turn store under a gpt-5-mini answerer and the official gpt-4o judge; BEAM runs the raw-turn lane at top-30 under a gpt-5 answerer and judge. The lanes activated differ with the protocol, but the core is fixed: per-conversation minds, dated raw turns as the primary substrate, contradictions retained, dates resolved at write time. Figure 1 shows this pipeline.
![Conflict-aware raw-turn substrate: conversation to per-conversation dated raw-turn mind to top-30 retrieval to conflict-aware answer policy to answer, with no distillation, graph, reconciliation, or routing.](figures/fig1-substrate.png)
**Figure 1.** The conflict-aware raw-turn substrate, left to right. **Write time:** each source conversation becomes one per-conversation mind; every turn is stored verbatim as a dated raw-turn frame, its `[YYYY-MM-DD]` resolved from the session anchor (including relative references such as "yesterday"), and local LLM passes optionally add typed I/P/B frames without discarding the raw turns and without reconciling contradictions. **Read time:** hybrid retrieval (dense `nomic-embed` vectors fused with FTS5 BM25 by reciprocal rank fusion, optional cross-encoder rerank) draws the top-k dated raw turns as the primary answer lane, with derived frames as auxiliary lanes. **Answer policy:** retrieved turns render oldest-first, and the model applies the two rules the incumbents do not (surface both sides on contradiction, treat explicit "never happened" statements as information) plus conditional abstention and granularity-calibrated dates. The single write-time transform is relative-date resolution, which adds information; the substrate performs no distillation-as-primary, no graph construction, no write-time reconciliation, and no learned routing on the memory path.
---
## 5. Evaluation Protocol Fidelity
Cross-lab long-term-memory numbers are noise, so we reproduce each incumbent's pipeline before we compare, and we treat this reproduction as a contribution in its own right. LoCoMo, LongMemEval, and BEAM are all scored by an LLM judge whose model, prompt, and aggregation choices move the headline by several points, and the published leaderboards mix these choices without disclosing them. The only defensible claim is a same-judge, same-protocol comparison in which we first reproduce the incumbent's own number, then run our substrate through the identical harness. Table 2 records, for every benchmark, what the leaderboard reported and what protocol the shipped result files actually used.
We reproduced Memori's own LoCoMo pipeline to 81.98 percent against their published 81.95 percent before running our substrate through the same harness. The reproduction validated the harness end to end and exposed a protocol detail that matters: Memori's published ingest prepends image-caption text to every image-sharing turn, and our earlier ingest had used the bare turn text, so we had been benchmarking against a handicapped corpus. Restoring caption parity is protocol compliance, not an architecture change, and we report it as such. With the harness reproduced and input parity restored, the LoCoMo comparison is same-answerer and same-judge (gpt-4.1-mini for both) rather than cross-paper.
The LongMemEval leaderboard mixes two aggregation metrics and three judge protocols, and reading it as one number is a category error. Mastra's headline 94.87 is a macro score, the unweighted mean of six category accuracies, under the official gpt-4o judge; mem0's 93.4 uses gpt-5 as both answerer and judge, not the official Wu et al. gpt-4o judge, with a self-declared plus-or-minus one point interval. We report both metrics under the most conservative protocol on the board, the verbatim official gpt-4o judge with a gpt-5-mini answerer: 95.01 macro and 93.60 micro. Under Mastra's own headline metric we score 95.01 against their 94.87, and per question we tie their reconstructed 468 of 500. Because macro up-weights small categories (one preference question is worth 0.56 points of macro against 0.125 for a multi-session question), we disclose the micro number rather than let the metric flatter us.
The BEAM incumbent number carries the same hidden judge swap, and we neutralize it by matching models rather than assuming them. The mem0 result file behind the published 0.641 records `answerer_model: gpt-5` and `judge_model: gpt-5`, even though the repository README's CLI defaults imply gpt-4o for both, so a reader who trusts the README compares against the wrong protocol. Because the BEAM paper itself shows the answerer dominates the score (its light-answerer baseline at 1M scores near 0.336), an unmatched comparison would confound memory architecture with answerer model. We therefore run our best configuration at gpt-5 answerer and gpt-5 judge, mem0's actual protocol, and we report both a pass rate (our headline claim) and an average score (which sits inside judge noise).
We disclose every correction and every noise band rather than report only the favorable cell. The initial full-700 BEAM run scored 0.6050 because 56 answers were empty strings that all scored zero, an artifact of our client capping completion tokens at 4096 so that gpt-5 exhausted the budget on hidden reasoning for long-form questions (29 of 70 summarization prompts). We re-answered all 56 under the identical configuration (mean 0.000 healing to 0.528) with a 16K floor and retry-on-empty, and we preserve the pre-heal snapshot as a disclosed artifact; mem0 is unaffected by this class of bug because its contexts run near 7K tokens. We also report the judge-noise band directly (standard error near 0.014 on the BEAM average score), we canonicalize by first-occurrence deduplication on `instance_id` (700 unique rows from 1020 raw), and we archive per-question answers and judgments for every run. These disclosures are the anti-cherry-pick posture the metric audit demands.
**Table 2. Protocol-fidelity matrix: what the leaderboard reported versus the protocol we ran.** Best-known incumbent per benchmark shown; "README vs actual" flags a shipped protocol that differs from the documented default.
| Benchmark | System | Answerer | Judge | Metric reported | Retrieval budget |
|---|---|---|---|---|---|
| LoCoMo (1,540 Q) | Ours | gpt-4.1-mini | gpt-4.1-mini | micro accuracy | 7-lane assembler (~3,747 tok) |
| LoCoMo | Memori (reproduced 81.98 vs 81.95) | gpt-4.1-mini | gpt-4.1-mini | micro accuracy | timestamped triples (~1,294 tok) |
| LongMemEval-S (500 Q) | Ours | gpt-5-mini | official gpt-4o | macro 95.01 and micro 93.60 | routed observation + raw-turn store |
| LongMemEval-S | Mastra (94.87) | gpt-5-mini | official gpt-4o | macro only | observation store |
| LongMemEval-S | mem0 (93.4) | gpt-5 | gpt-5 (not official gpt-4o) | undisclosed aggregation | fact store |
| BEAM 1M (700 Q) | Ours | gpt-5 | gpt-5 | avg score 0.6482 and pass 74.0% | top-30 raw turns |
| BEAM 1M | mem0 (0.6409 / 70.1%) | gpt-5 (README says gpt-4o) | gpt-5 (README says gpt-4o) | avg score only | top-200 facts |
---
## 6. Results
### 6.1 LoCoMo
Our substrate scores 86.49 percent (1332 of 1540) on LoCoMo under Memori's exact published protocol, 4.54 points above the best previously published memory system. The one-sample test against Memori's 81.95 percent gives z equal to 4.64 at p below 10 to the minus 5, the substrate leads or ties every question category among memory systems, and its open-domain accuracy is statistically indistinguishable from the full-context ceiling. The result is verified drift-free: a fresh regeneration on the current repository HEAD reproduced 86.49 percent exactly, and an earlier 87.66 percent figure that did not reproduce was traced to a stale-verdict-replay bug in the judge harness, disclosed and superseded.
We also correct a column-scrambling error that has propagated through the LoCoMo literature, and we re-run Mem0 under our own judge. The Memori paper's baseline table transplanted MemR3's column order under mismatched headers, which turned LangMem's single-hop score of 86.92 into a widely cited "temporal 86.92"; LangMem's true temporal score is 61.06. Table 3 gives the corrected landscape. Re-running Mem0 on our shared answerer and judge (substrate against substrate) we lead every category, with temporal accuracy 30.8 points ahead of Mem0's 50.78.
The category structure of Table 3 carries the design thesis, which is that no single representation wins long-term conversational memory. Profile cards win open-domain synthesis at the full-context ceiling, write-time-dated episodic timelines win temporal anchoring, and the verbatim raw-detail lane wins fine-grained single-hop recall where distillation abstracts the answer away (the gold "a painting of a sunset with a pink sky" survives only in the raw turn, not in the distilled fact "an abstract painting"). A substrate that serves all three simultaneously beats any single-representation incumbent, and the headline statistical claim rests on the reproduced same-judge Memori row rather than on the cross-sourced ones.
**Table 3. Corrected LoCoMo landscape (gpt-4.1-mini protocol, accuracy %).** Baselines corrected per Du et al.; only the Memori and Mem0 rows are same-harness with ours. MemR3 is an agentic retrieval pipeline, shown for reference. Best memory-system score per column in bold.
| Method | single | multi | temporal | open | overall |
|---|---|---|---|---|---|
| **Ours (conflict-aware raw-turn)** | **92.27** | **80.50** | **81.62** | **69.79** | **86.49** |
| Memori (same-judge) | 87.87 | 72.70 | 80.37 | 63.54 | 81.95 |
| Mem0 (same-judge, ours) | 83.59 | 74.82 | 50.78 | 64.58 | 73.96 |
| Zep | 83.49 | 72.34 | 77.26 | 64.58 | n/a |
| LangMem | 86.92 | 74.47 | 61.06 | 67.71 | 78.05 |
| Mem0 (literature) | 66.47 | 62.41 | 57.32 | 44.79 | 62.47 |
| MemR3 (agentic; ref.) | 92.17 | 81.20 | 82.14 | 71.53 | n/a |
| Full-context ceiling | 93.73 | 86.43 | 86.82 | 71.88 | n/a |
### 6.2 LongMemEval
Our substrate scores 95.01 macro and 93.60 micro (468 of 500) on LongMemEval-S under the official gpt-4o judge, above Mastra's 94.87 under Mastra's own macro metric. The result is state-of-the-art on the leader's headline metric and holds the most conservative judge protocol on the board, as Section 5 established. The arc from a 75.9 percent gpt-4o baseline to 95.01 macro was won by four answer-side survivors, not by richer memory, and Table 4 traces every rung.
The two levers that moved the number most were an information-adding fix and an aggregation of reads, neither of which restructures stored content. Recovering the dropped `question_date` field, the reference date that "how long ago" questions anchor to, lifted temporal accuracy from 61 to 84 percent on its own, because no retrieval or reasoning tuning can supply a reference point that is absent from the data. Full-log self-consistency voting then attacked the residual counting errors directly: multi-session counts carry a plus-or-minus-one enumeration variance, and voting the mode of five reads over the same complete log recovered them where static prompt tuning failed. Both fixes preserve the store and read it better rather than transforming what is stored.
The four adopted levers (self-consistency voting, a guarded split-vote counting override, a preference grounding critic, and a routing-matrix completion) are all answer-side, and the nineteen we falsified are catalogued in Section 7. Each survivor aggregates multiple reads of the preserved store: it votes, it counts a guarded subset, it runs best-of-five on ungrounded answers, or it selects which complete context to read. None of them adds a derived representation or reconciles a conflict. This is the same verdict BEAM reaches by a different route.
**Table 4. LongMemEval intervention arc (official gpt-4o judge).** Overall is micro accuracy unless noted; the four adopted levers are answer-side.
| Configuration | Answerer | Overall | Note |
|---|---|---|---|
| base lanes (raw turns) | gpt-4o | 66.2% | no write-time distillation |
| + profile cards | gpt-4o | 75.9% | best gpt-4o config |
| + observation extraction + question_date + preference (qd) | gpt-5-mini | 88.2% | two root-cause fixes (adds information) |
| + per-category type routing | gpt-5-mini | 91.2% | temporal/preference to full-log |
| + qd-vote on counting questions | gpt-5-mini | 92.2% | 5x varied-K self-consistency |
| + full-log self-consistency vote (vh5) | gpt-5-mini | 92.8% | adopted |
| + KG-ledger split-vote guard (klc) | gpt-5-mini | 93.0% | adopted |
| + preference grounding critic (pfc) | gpt-5-mini | 93.4% | adopted (mem0 tied) |
| + ssu routing completion (h3u, final) | gpt-5-mini | **93.60 micro / 95.01 macro** | adopted |
### 6.3 BEAM
Our substrate scores 0.6482 average and 74.0 percent pass (518 of 700) on BEAM 1M under mem0's exact gpt-5 answerer and gpt-5 judge protocol, against mem0's 0.6409 average and 70.1 percent pass. The pass rate is the headline claim, and a McNemar paired test on the 700 shared questions confirms it: among the discordant pairs, we pass 93 questions where mem0 fails against 66 where the reverse holds, which gives z = 2.14 (2.06 with continuity correction) at p ≈ 0.03. An unpaired two-proportion test on the same two rates returns only z = 1.61, but the paired test is the correct one here, because both systems answer the identical 700 questions, so the shared question-level difficulty variance cancels and only the discordant pairs carry signal. The average-score margin of plus 0.0073 sits inside the judge-noise band (standard error near 0.014), so we report it as parity-or-better rather than as a win, and we state both numbers rather than the flattering one alone. The configuration is the plain raw-turn lane at top-30 with prompt v2, chosen after 14 falsified alternatives.
The single differentiator is contradiction resolution, where the substrate scores 0.588 (87.1 percent pass) against mem0's 0.357 (48.6 percent pass), a gap of 0.231, or 23 points, on the benchmark's hardest ability. This is a direct consequence of retaining both sides of a conflict rather than reconciling at write time: mem0's write-time reconciler makes contradiction resolution its own floor because it has already deleted one side before the answerer runs, while our substrate surfaces both dated statements and asks which is correct. Two other retained-signal abilities also favor the substrate, abstention by 6.8 points (0.593 versus 0.525) and information extraction by 4.0 points (0.740 versus 0.700), and Table 5 gives the full profile. The losses concentrate in coverage-shaped abilities, summarization by 6.5 points and temporal reasoning by 6.1 points, where a compact enumerative fact store harvests more compound-nugget clauses than a raw-turn context (the mechanism is analyzed in Section 8).
Depth of distillation is not mem0's lever, and compactness of raw beats it. mem0's own ablation quadruples retrieval from top-50 to top-200 facts and gains only 0.037 average score, so their advantage is not in stuffing more memories into the prompt. Our result uses 7 times fewer retrieved items than theirs (30 raw turns against 200 facts) and still leads on pass rate and ties on average, and the entire BEAM program cost roughly 160 dollars. The compact raw context wins the abilities that reward verbatim detail and conflict preservation, and it loses only the abilities that reward enumerative breadth.
**Table 5. BEAM 1M per-ability, ours versus mem0 (both gpt-5 answerer and judge, full 700).** Average score per ability; overall average and pass rate at the foot. Positive delta favors ours.
| Ability | Ours | mem0 | Δ |
|---|---|---|---|
| contradiction_resolution | **0.588** (87.1% pass) | 0.357 (48.6%) | **+0.231** |
| abstention | **0.593** | 0.525 | **+0.068** |
| information_extraction | **0.740** | 0.700 | **+0.040** |
| instruction_following | 0.834 | 0.852 | −0.018 |
| preference_following | 0.859 | 0.883 | −0.024 |
| multi_session_reasoning | 0.644 | 0.652 | −0.008 |
| knowledge_update | 0.604 | 0.650 | −0.046 |
| temporal_reasoning | 0.557 | 0.618 | −0.061 |
| event_ordering | 0.494 | 0.536 | −0.042 |
| summarization | 0.570 | 0.635 | −0.065 |
| **Overall Avg Score** | **0.6482** | 0.6409 | **+0.0073** |
| **Pass Rate (≥0.5)** | **74.0%** (518/700) | 70.1% (491/700) | **+3.9pp** |
---
## 7. The Falsification Ledger
We treat negative results as the product, and we report 37 constructive interventions across two independent programs, of which 33 lost to the simple substrate. The two programs are 23 LongMemEval levers (4 adopted, 19 falsified) and 14 BEAM levers (all 14 falsified), so roughly 90 percent of everything we tried made the substrate worse or no better. Table 6 unifies them by family: lever, program, result, and why it lost. Every family tells the same story, and the four survivors share a property we make precise at the end.
Compression and distillation lost on both programs, because they strip the dates, numbers, and verbatim spans the rubrics score. On BEAM, distilled-fact retrieval scored 0.354 at k equal to 100 and 0.382 at k equal to 200 against 0.448 for raw turns, and every raw-plus-fact hybrid landed between 0.498 and 0.540 against 0.591 for raw alone; facts are net-harmful at any mixing ratio. A session-outline preamble diluted summarization from 0.38 to 0.16, and on LongMemEval an exhaustive re-distiller with five times the observation density scored 2 net worse than the tuned cache, because near-duplicate observations drop reranker precision.
Structuring the store as a knowledge graph or a ledger lost almost everywhere it removed the raw signal, with one guarded exception. Overriding LongMemEval votes with a KG-ledger count cost 10 net when applied to unanimous votes, because a unanimous five-voter consensus is a stronger correctness signal than the ledger, and injecting a reconciled ledger as trusted context in the voting path cost 5 net, because an imperfect structured input corrupts more voters than it clarifies. A write-time incremental ledger failed its gate outright (0 recoveries on 26), because counting is question-scope-dependent in a way question-blind determinism cannot capture. The single structuring survivor, klc, adds only 0.2 points, and it fires only on split votes where the deterministic count has clean provenance; it aggregates uncertain reads rather than restructuring the store.
Routing lost when it classified content and survived only when it selected among complete contexts by question type. On BEAM, gold-blind ability routing hit a 40 percent classifier ceiling and scored 0.584 below the 0.600 baseline, because abilities are not identifiable from question text, and coverage-shaped stratified retrieval fell to similarity-dense clusters (summarization 0.38 to 0.24). On LongMemEval, by contrast, deterministic routing by question type (full-log for temporal and preference, retrieval for the rest) was adopted for plus 3 points, and completing that routing matrix for single-session-user (h3u) added the final point. The surviving router transforms nothing in storage; it chooses which already-complete context the answerer reads.
Prompt shaping plateaued: no prompt beat the plain conflict-aware v2 prompt on BEAM, and none beat the tuned answer policy on LongMemEval. A strict-abstention prompt scored 0.542 on BEAM and cost 3 net on LongMemEval, because over-conservatism spreads to answerable questions; an exhaustive-enumeration prompt cost 0.048 overall and 0.119 on summarization, because "do not omit minor items" dilutes clause density with trivia; a temporal-commit prompt moved plus 0.023, below re-roll variance. Raising reasoning effort netted zero and over-abstained, because the bottleneck is enumeration variance and retrieval availability, not reasoning depth.
Ensembling lost when it merged different answers and won only when it voted the same store, which is the cleanest statement of the whole pattern. Answer-merge self-ensemble on BEAM scored 0.668 against 0.674 for a single answer at twice the cost, because the union dilutes as much as it harvests, and a ten-voter cross-path ensemble on LongMemEval cost 1 net, because duration canonicals are incompatible across paths. Swapping in a stronger gpt-5 answerer failed its gate (109 against 114 multi-session), because the stronger model over-abstains on answerable questions. The two ensembling survivors, self-consistency voting (vh5, plus 1.6 points) and the preference grounding critic (pfc, plus 0.4 points), both read the preserved store multiple times and aggregate; neither adds structure.
Two programs on three benchmarks reach one verdict: the local optimum is plain dated raw-turn retrieval plus a conflict-aware prompt, and cleverness on the memory path is measured self-harm. The pattern is exact. Every lever that transforms stored content, whether it distills it, graphs it, reconciles it, or restructures it, lost. The only survivors aggregate multiple reads of an untransformed store: they vote over it, count a guarded subset of it, run best-of-five grounding on it, or route among complete views of it. The load-bearing moves are to keep the raw data, resolve dates at write time because that adds information, and preserve contradictions; everything else the field is building is optional at best and harmful in the measured cases.
**Table 6. The falsification ledger.** Principal levers grouped by family (37 total across both programs, 33 falsified, 4 adopted). Full per-run records live in the cited result files. "F" = falsified, "A" = adopted.
| Family | Lever | Program | Result | Why it lost / survived | Verdict |
|---|---|---|---|---|---|
| Compression / distillation | Distilled-fact retrieval k=100 | BEAM | 0.354 vs 0.448 raw | strips dates and numbers | F |
| Compression / distillation | Distilled-fact retrieval k=200 | BEAM | 0.382 | recall recovered, lossy abilities stayed dead | F |
| Compression / distillation | Hybrid 15 raw + 60 facts | BEAM | 0.540 vs 0.591 | cutting raw turns loses detail | F |
| Compression / distillation | Additive hybrid 30 raw + 60 facts | BEAM | 0.498 | facts net-harmful at any mix | F |
| Compression / distillation | Session-outline preamble | BEAM | 0.536 vs 0.600; summ 0.38→0.16 | lossy summary dilutes | F |
| Compression / distillation | Exhaustive re-distiller (5x density) | LongMemEval | −2 vs tuned cache | near-duplicate distractors drop reranker precision | F |
| Structuring (KG / ledger) | ipb answer-accumulation cell | BEAM | 0.591 ≈ v2 | order-dependent, no gain over dated turns | F |
| Structuring (KG / ledger) | KG-ledger override of unanimous votes (kla) | LongMemEval | −10 | unanimous consensus beats the ledger | F |
| Structuring (KG / ledger) | KG-ledger, gpt-5-mini extractor (klc2) | LongMemEval | −1 | stronger extractor gives different, not better, ledgers | F |
| Structuring (KG / ledger) | Date-ledger override for durations | LongMemEval | ~−2 | duration errors are anchor-selection | F |
| Structuring (KG / ledger) | Write-time incremental ledger (D1) | LongMemEval | gate 0/26, aborted | counting is question-scope-dependent | F |
| Structuring (KG / ledger) | Reconciled-ledger-as-context in voting | LongMemEval | −5 | imperfect structure corrupts voters | F |
| Structuring (KG / ledger) | KG-ledger split-vote guard (klc) | LongMemEval | +0.2pp | deterministic count only on split votes | A |
| Routing | Full gold-blind ability routing | BEAM | 0.584 < 0.600 | 40% classifier ceiling; abilities not text-identifiable | F |
| Routing | Coverage-shaped stratified retrieval | BEAM | summ 0.38→0.24 | similarity-dense cluster beats stratified coverage | F |
| Routing | ssu routing-matrix completion (h3u) | LongMemEval | +1 (final) | selects a complete context by question type, no transform | A |
| Prompt shaping | Prompt v3 strict abstention | BEAM | 0.542 | over-conservatism spreads to all abilities | F |
| Prompt shaping | Prompt v1 (mem0 verbatim on our substrate) | BEAM | 0.573 | loses contradiction and wrongful-IDK points | F |
| Prompt shaping | Prompt v4 exhaustive enumeration | BEAM | −0.048; summ −0.119 | trivia dilutes clause density | F |
| Prompt shaping | Prompt v5 temporal commit | BEAM | +0.023 (~0.5 SE) | below re-roll variance | F |
| Prompt shaping | Standing-directives preamble | BEAM | flat (0.557) | preferences already in raw turns | F |
| Prompt shaping | Strict abstention + answer hygiene | LongMemEval | −3 | over-abstains on answerable questions | F |
| Prompt shaping | reasoning_effort = high | LongMemEval | ~0, over-abstains | bottleneck is variance, not reasoning | F |
| Prompt shaping | IDK-escalation raw sweep | LongMemEval | 0 | quote-level guard suppresses everything | F |
| Prompt shaping | Count-hint (blanket K48 / phrasing) | LongMemEval | +3 (12 up, 9 down) | fixes and breaks equal numbers | F |
| Ensembling / voting | Answer-merge self-ensemble | BEAM | 0.668 vs 0.674 (2x cost) | union dilutes as much as it harvests | F |
| Ensembling / voting | 10-voter cross-path ensemble | LongMemEval | −1 | duration canonicals incompatible across paths | F |
| Ensembling / voting | gpt-5 (full) answerer swap | LongMemEval | gate FAIL (109 vs 114 ms) | stronger model over-abstains | F |
| Ensembling / voting | P/B-frame consolidation answerer swap | LongMemEval | −1 | whole-path swap pays the noise floor | F |
| Ensembling / voting | KU recency-critic best-of-5 | LongMemEval | −1 | KU errors are not recency-mechanical | F |
| Ensembling / voting | ssu supportedness-critic best-of-5 | LongMemEval | 0 (0 fires) | failures are absent from context | F |
| Ensembling / voting | Full-log self-consistency vote (vh5) | LongMemEval | +1.6pp | votes over the same complete log | A |
| Ensembling / voting | Preference grounding critic (pfc) | LongMemEval | +0.4pp | gold-blind best-of-5 on ungrounded answers | A |
![Falsification ledger strip plot: 33 falsified levers on or below the raw-turn baseline, only the 4 adopted answer-side levers above it, grouped by lever family.](figures/fig2-falsification.png)
**Figure 2.** Falsification strip plot: lever family on the x-axis against change in benchmark score on the y-axis, one point per lever in Table 6, BEAM and LongMemEval markers distinguished. The horizontal zero line is the simple raw-turn baseline; roughly 90 percent of points fall on or below it, and the four points above it are all answer-side aggregation levers (vh5, klc, pfc, h3u), never memory-structure levers. The visual message without the caption: complexity buys nothing above the raw-turn line.
---
## 8. Analysis
Every transform on the memory path is lossy, and the benchmarks reward exactly what the transforms remove. Distillation collapses a verbatim turn into a fact and discards the surrounding numbers, dates, and phrasing; a knowledge graph abstracts a statement into typed edges; write-time reconciliation deletes the superseded side of a conflict. BEAM's rubric, by contrast, scores compound nuggets that demand specific clauses ("should contain X and Y with detail Z"), and its contradiction ability scores the presence of both conflicting statements. Detail removed by a transform cannot be recovered downstream, so the only question that matters is whether the answerer had the material and failed to use it or never received it at all.
The retrieval-headroom audit answers that question decisively: the losses are answer-side, not retrieval-side. For each failed nugget on the three lossy abilities we retrieved the question's top-150 raw turns and found the first rank at which the supporting content appears. It appears at rank 30 or better for 88 percent of failed summarization nuggets, 90 percent of event-ordering nuggets, and 93 percent of multi-session-reasoning nuggets; the median first-hit rank is between 1.5 and 4, and the content is absent from the top-150 entirely for only 2 to 3 percent. Widening retrieval from 30 to 100 turns moves only about 8 percent of failed nuggets, at two to three times the input cost. The material the answerer needs is already in front of it at top-30, so deeper retrieval and richer structure cannot help, and distillation would only remove what the answerer already holds.
The contradiction result is the clearest demonstration that write-time cleverness is a liability. The mem0 platform reconciles at ingest, so when a user's follower count changes the old fact is updated in place, which is why knowledge update is one of its stronger abilities at 0.650. The very same reconciler is why contradiction resolution is its weakest at 0.357, because it has already collapsed the two dated statements into one before the answerer runs, and BEAM's rubric wants both surfaced with a request for clarification. Our substrate makes the opposite trade: it retains both dated statements and the answer policy states the conflict, so it scores 0.588 and passes 87.1 percent of contradiction questions against mem0's 48.6 percent, a gap of 23 points of average score on the benchmark's hardest ability. The trade is intrinsic to when a conflict is resolved, not a tuning gap a better reconciler could close. A write-time reconciler must pick one side to store; a read-time policy can keep both and decide per question. Resolving early optimizes the wrong moment.
Where the substrate loses, the cause is answer breadth and answer disposition, not memory quality. On summarization the final gap is 6.5 points (0.570 against 0.635), and the peer-to-peer nugget forensics, run at 100 percent question and nugget alignment, locate it in verbosity: mem0's answers average 1067 words and 48 bullets against our 600 words and 29 bullets, and because BEAM nuggets are compound, more enumerated surface mechanically harvests more clauses. Most of mem0's winning summarization nuggets are upgrades from our 0.5 to their 1.0, cases where we hit the topic but drop one clause of a "should contain X and Y" nugget. Event ordering shows the same shape and, tellingly, zero true ordering errors: BEAM scores event ordering as coverage of the listed events rather than sequence correctness, so the loss there is again clause density, not a reasoning failure.
The temporal gap is dispositional. BEAM temporal questions are 94 percent duration arithmetic between two dated events whose anchors both live in the dialogue, and 74 percent of our loss is over-abstention plus wrong-anchor selection: on five questions we answered "insufficient information" where mem0 committed to the right date pair and won, and on three we selected the wrong dates. mem0 emits a single terse declarative every time, while we hedge on 16 of 70. That the temporal-commit prompt (v5) recovered only half the over-abstention at plus 0.023, below re-roll variance, is direct evidence the gap is a disposition of the answerer under this judge, not a deficit of the stored memory: the anchors are present at top-30, and the substrate simply under-commits.
One pattern unifies the four survivors across both programs: they all aggregate multiple reads of an unchanged store, and none transforms stored content. Self-consistency voting (vh5) takes the mode of five reads of the same complete log; the guarded counting override (klc) recounts a subset deterministically only on split votes; the preference grounding critic (pfc) runs best-of-five on ungrounded answers; and the routing completion (h3u) selects which complete context the answerer reads. Every falsified lever, by contrast, transforms the store, whether it distills, graphs, reconciles, or restructures it. The load-bearing conclusion is that the productive place to spend cleverness is read time, not write time. The survivors are the memory analog of test-time-compute scaling: hold the store fixed and buy accuracy by reading it more times or more carefully, exactly as reasoning models buy accuracy by sampling more chains over a fixed prompt. Cleverness on the memory path, which mutates the store, loses; cleverness on the read path, which aggregates over it, wins.
## 9. Limitations
Our BEAM average-score margin is inside the noise. The plus 0.0073 average sits within a judge-noise standard error near 0.014, so we rest the BEAM claim on the pass rate (plus 3.9 points; McNemar paired z = 2.14, p ≈ 0.03) and report the average only as parity-or-better. A reader who weights average score above pass rate should read the BEAM average as a statistical tie, and the honest headline is the pass rate alone.
The LongMemEval macro margin is likewise inside run-to-run noise. Our 95.01 macro leads Mastra's 94.87 by 0.14 points, a gap that run-to-run variation can erase, so we do not claim a clean win on that axis either. The defensible LongMemEval claim is that we match the leader under the most conservative protocol on the board, the official gpt-4o judge run verbatim, while the incumbent numbers rest on more favorable judges and metrics.
The substrate pays for its detail in tokens. Our BEAM context runs approximately 27,000 tokens at top-30 raw turns (the mean answer-prompt token count recorded in the final run summary) against mem0's roughly 7,000 tokens of distilled facts, and on LoCoMo our seven-lane assembler uses 3,747 tokens against Memori's 1,294. Raw-turn retrieval is cheap at small k but scales super-linearly: at top-100 the context reaches about 88,000 tokens per question. Distillation's genuine advantage is inference cost, not answer quality, so we win the quality axis and concede the efficiency axis, and we point readers to the per-question cost figures rather than bury them.
We remain behind on coverage-shaped abilities. Summarization stays below mem0 (0.570 against 0.635), and whether a raw substrate can close that gap without the answer-verbosity tax, which backfired every time we forced enumeration, is open. The scope is also narrow in two ways. We evaluate a single substrate, and each benchmark uses one answerer family per its protocol (gpt-4.1-mini on LoCoMo, gpt-5-mini on LongMemEval, gpt-5 on BEAM), so the generality of the raw-turn design across weaker or cheaper answerers is untested.
Finally, all scores are LLM-as-judge, and two risks remain after our mitigations. The BEAM protocol has gpt-5 judging a gpt-5 answerer, which invites self-preference, though the risk is symmetric because mem0 produced its published number under gpt-5 judging gpt-5 as well, so the comparison stays like-for-like; we mitigate with fixed judges, archived per-question judgments, reproduced baselines, and reported noise bands, but some deltas still sit inside that variance. And all three benchmarks are English synthetic dialogues, so the result speaks to the benchmarks the field currently races on rather than to production memory over real users. We also disclose an empty-answer bug (56 rows healed under the identical configuration, with the pre-heal snapshot preserved) so that the post-heal numbers are the honest comparison.
## 10. Conclusion
One deliberately dumb substrate, the original dated conversation turns kept per conversation with contradictions preserved rather than reconciled, is state of the art on LoCoMo, LongMemEval, and BEAM, each under the incumbent's own published protocol. Thirty-seven measured interventions say the field is over-engineering the memory path: distillation, graphs, write-time reconciliation, and learned routing lost, and the only transform that helped, resolving relative dates at write time, adds information rather than removing it. This is the Bitter Lesson for agent memory.
The actionable lesson is a division of labor: keep the store dumb and spend cleverness on reads. Preserve the raw dated turns, resolve dates at write time, and keep both sides of a contradiction; then, when you want more accuracy, aggregate multiple reads of that store rather than transform it, because voting, guarded recounting, grounding critics, and routing among complete contexts were the only interventions that survived measurement. We release all protocols, per-question artifacts, the falsification ledger, and the harness, so the community can stop re-deriving the distillation and routing tax and can audit its own leaderboard metrics.
Three questions stay open. Can conflict preservation be made cheap enough to also win the coverage abilities, or is there a write-time transform other than date resolution that adds information rather than removing it? Does the raw-turn result hold under weaker and cheaper answerers, or is long-context capacity the true enabler? And what is the right rubric for a memory that surfaces a contradiction and asks, as against one that commits and happens to be right? The substrate is simple on purpose, and the remaining work is to keep it simple while teaching it to summarize without forgetting.
## Ethics and Broader Impact
All three benchmarks in this work use synthetically generated dialogues, so the experiments involve no human subjects and no personal data. Long-term memory systems nonetheless carry real retention and erasure obligations once they run over actual users, and our choice to retain verbatim raw turns rather than distilled abstractions raises the stakes of deletion: the substrate holds more of what a user actually said, so a user's right to erasure must reach every stored turn. The codebase addresses this directly with an erasure cascade that removes a source turn together with every frame and index entry derived from it, so deletion is total rather than cosmetic. Our scores also rest on LLM-as-judge evaluation, and LLM judges carry documented biases, including self-preference when a model grades its own family; we disclose the judge model for every number, hold the official judge fixed where one exists, and archive per-question judgments so third parties can re-audit. We encourage deployments of raw-turn memory to pair the accuracy benefits with explicit retention limits and auditable deletion.
## References
Author-year style. Entries were reconstructed offline and verified against arXiv, Semantic Scholar, and primary product pages before submission.
- Chhikara, P., Khant, D., Aryan, S., Singh, T., & Yadav, D. (2025). *Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory.* arXiv:2504.19413. Benchmark harness: `mem0ai/memory-benchmarks` (GitHub).
- Du, X., Li, L., Zhang, D., & Song, L. (2025). *MemR³: Memory Retrieval via Reflective Reasoning for LLM Agents.* arXiv:2512.20237. Source of the corrected LoCoMo baseline numbers used in Section 6.1 (Table 3); this is the same work as the MemR3 entry below.
- GibsonAI (2025). *Memori: an open-source SQL-native memory engine for LLM agents.* `github.com/GibsonAI/memori`.
- Latimer, C., Boschi, N., Neeser, A., Bartholomew, C., Srivastava, G., Wang, X., & Ramakrishnan, N. (2025). *Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflects.* arXiv:2512.12818. LongMemEval result (91.4%) reported with a Gemini-3 Pro answerer over a GPT-OSS-120B memory stack.
- LangChain (2025). *LangMem.* `github.com/langchain-ai/langmem`.
- Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., & Fang, Y. (2024). *Evaluating Very Long-Term Conversational Memory of LLM Agents (LoCoMo).* arXiv:2402.17753.
- Mastra / Barnes, T. (2026). *Observational Memory: 95% on LongMemEval.* `mastra.ai/research/observational-memory`.
- MemR3 — see Du, X., et al. (2025), *MemR³: Memory Retrieval via Reflective Reasoning for LLM Agents,* arXiv:2512.20237 (entry above). Agentic multi-round retrieval pipeline, shown for reference on LoCoMo (Table 3).
- Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023). *MemGPT: Towards LLMs as Operating Systems.* arXiv:2310.08560. Letta is the successor project (`letta.com`).
- Rasmussen, P., Paliychuk, P., Beauvais, T., Ryan, J., & Chalef, D. (2025). *Zep: A Temporal Knowledge Graph Architecture for Agent Memory.* arXiv:2501.13956.
- Supermemory (2025). *LongMemEval result (~85.9% overall).* `supermemory.ai/research`.
- Sutton, R. S. (2019). *The Bitter Lesson.* `incompleteideas.net/IncIdeas/BitterLesson.html`.
- Tavakoli, M., Salemi, A., Ye, C., Abdalla, M., Zamani, H., & Mitchell, J. R. (2026). *Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs.* ICLR 2026 (introduces the BEAM benchmark). arXiv:2510.27246.
- Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., & Yu, D. (2024). *LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory.* arXiv:2410.10813.
## Appendix A: Complete falsification ledger (LongMemEval)
Table 6 lists principal levers grouped by family; the four adopted LongMemEval survivors are self-consistency voting (vh5), the guarded split-vote KG-ledger count (klc), the preference grounding critic (pfc), and the routing-matrix completion (h3u). This appendix enumerates all 19 falsified LongMemEval levers individually. Eighteen of them appear as rows in the negative-results table of the released LongMemEval results file; the nineteenth (blanket full-log routing) is documented in that file's routing finding, where applying the full-log path to knowledge-update and multi-session lost 3 and 2 to 4 points respectively, which is why routing is selective rather than universal. Net effects reported in percentage points (or instances, where noted) against the tuned baseline under the official gpt-4o judge. Combined with the 14 falsified BEAM levers, the two programs falsified 33 of 37 measured interventions.
| # | Falsified lever | Net effect | Why it lost |
|---|---|---|---|
| 1 | Exhaustive re-distiller (v2, ~1,500 obs/instance, 5x density) | −2 vs tuned cache | near-duplicate observations drop reranker precision |
| 2 | Count-hint, blanket K48 | +3 instances (12 up, 9 down), within noise | fixes and breaks counting questions in equal number; variance, not bias |
| 3 | Count-hint, targeted by phrasing | +3 (same) | recoveries and regressions are both counting-phrased |
| 4 | reasoning_effort = high | ~0, over-abstains | bottleneck is enumeration variance and availability, not reasoning depth |
| 5 | Strict abstention + answer hygiene | −3 | over-abstains on answerable questions |
| 6 | P/B-frame consolidation as answerer swap (KU) | −1 (fixed stale value, regressed 2) | whole-path swap pays the noise floor |
| 7 | IDK-escalation raw sweep (guarded) | 0 (0 up, 0 down) | quote-level guard suppresses everything; floor holds |
| 8 | KG-ledger override of unanimous votes (kla) | −10 (5 up, 15 down) | unanimous five-voter consensus beats the ledger |
| 9 | KG-ledger with gpt-5-mini extractor (klc2) | −1 | a stronger extractor gives different, not better, ledgers |
| 10 | P-frame current-value KU override (guarded) | −1 (1 up, 2 down) | chain detection mis-resolves countable attributes |
| 11 | vh5 x qd-vote 10-voter ensemble | −1 (analytical) | duration canonicals are incompatible across paths; real disagreements favor vh5 2:1 |
| 12 | Date-ledger override for temporal durations | ~−2 (gold-visible) | duration errors are anchor-selection; the extractor mis-grounds the same anchors the answerer does |
| 13 | Dual-extractor consensus vs unanimous votes | 0 up, 19 at risk (analytical) | independent extractors share blind spots and cannot crack the unanimous-wrong pool |
| 14 | Option D1: write-time incremental ledger | gate 0/26, aborted | counting is question-scope-dependent; question-blind determinism fails on scope |
| 15 | KU recency-critic best-of-5 | −1 (1 up, 2 down) | attribute-matching ambiguity flags correct answers as stale; KU errors are not recency-mechanical |
| 16 | ssu supportedness-critic best-of-5 | 0 (0 fires on 70) | the failing answers are absent from context (retrieval-availability); regeneration cannot help |
| 17 | Reconciled-ledger-as-context in the voting path | −5 (2 up, 7 down) | an imperfect ledger injected as trusted context corrupts more voters than it clarifies |
| 18 | gpt-5 (full) answerer, same vh5 pipeline | gate FAIL (multi-session 109 vs 114) | the stronger model over-abstains on answerable questions; gpt-5-mini is better calibrated for this judge |
| 19 | Blanket full-log routing on knowledge-update and multi-session | KU −3; multi-session −4 to −2 (across 4 variants) | full-log dilutes attention on non-temporal, non-preference categories; motivates selective routing |
Sources: rows 1 to 18 are the negative-results table of `benchmarks/results/longmemeval/RESULTS.md`; row 19 is the routing finding in the same file. The 14 falsified BEAM levers are enumerated in Table 6 and backed by the per-run files under `benchmarks/results/beam/`.
## Appendix B: Reproducibility
Every number in this paper is reproducible from public benchmark data and the released harness, and this appendix collects the details a re-runner needs in one place.
**Datasets.** All three benchmarks are public. LoCoMo is Maharana et al. (2024, arXiv:2402.17753), 1,540 questions over ten multi-session dialogues; LongMemEval-S is Wu et al. (2024, arXiv:2410.10813), the 500-question small split; and BEAM 1M is Tavakoli et al. (2026, arXiv:2510.27246, ICLR 2026), the one-million-token history split scored on 700 questions after first-occurrence deduplication on `instance_id` (700 unique rows from 1,020 raw). We use each dataset as published, apply no train split, and add no private data.
**Code and harness.** The memory engine is the hive-mind / waggle-os I/P/B frame store with local hybrid retrieval. The benchmark harnesses, per-question result files, negative-results tables, and the exact reproduction commands for each benchmark live in the released repository under `benchmarks/harness/` and `benchmarks/results/`; the consolidated command list is `benchmarks/results/MEMORY-BENCHMARKS-CONSOLIDATED-2026-07.md`. The BEAM headline run, for instance, builds and ingests once (`build-beam-canonical.ts 1M`, `beam-ingest-1m.ts`) and then answers with `beam-run-1m.ts --cell retrieval --prompt v2 --model gpt-5 --top-k 30 --per-ability 70`, writing per-question judgments to `benchmarks/results/beam/`.
**Model versions.** The answerer and judge are held to each incumbent's protocol: LoCoMo uses a gpt-4.1-mini answerer and gpt-4.1-mini judge; LongMemEval-S uses a gpt-5-mini answerer and the official Wu et al. gpt-4o judge; BEAM uses a gpt-5 answerer and gpt-5 judge. Embeddings are local `nomic-embed-text` served through Ollama, fused with FTS5 BM25 by reciprocal rank fusion, with an optional in-process cross-encoder rerank.
**Hyperparameters.** The load-bearing hyperparameters are the retrieval depth and the read-time aggregation settings, not any trained weights. BEAM retrieves the top-30 raw turns under prompt v2; LoCoMo runs the seven-lane assembler (~3,747 tokens); LongMemEval routes an observation-plus-raw-turn store and votes self-consistency over five reads (vh5) with a guarded split-vote counting override (klc), a best-of-five preference grounding critic (pfc), and a per-category routing matrix (h3u). The BEAM client uses a 16K completion-token floor with retry-on-empty; no learning rate, batch size, or optimizer applies because nothing in the substrate is trained.
**Random seeds and nondeterminism.** We fix no random seed, because the hosted OpenAI answerer and judge are nondeterministic and expose no reproducible seed across an evaluation run; only the local embedding and lexical retrieval are deterministic. We therefore treat run-to-run variance as first-class: we aggregate reads with self-consistency voting, reproduce each incumbent's own number before comparing, report the judge-noise standard error directly (near 0.014 on the BEAM average score), and archive per-question answers and judgments so third parties can re-audit rather than re-run.
**Cost and artifacts.** The full BEAM 1M program cost roughly 160 dollars in API spend. We release every protocol, every per-question answer and judgment file, the pre-heal snapshot behind the empty-answer disclosure, the retrieval-headroom and peer-to-peer forensics, and the route tables, so the results can be audited without re-execution and reproduced with it.

314
docs/paper/2026-07-09-clever-memory-loses-draft.v1.md Normal file
View File

@@ -0,0 +1,314 @@
# Clever Memory Loses: A Single Simple Substrate Is State of the Art on LoCoMo, LongMemEval, and BEAM
**Author:** Marko Marković, KORRO / hive-mind
**Date:** 2026-07-09
**Status:** arXiv preprint, markdown master (LaTeX conversion later). Phase B-1 draft: core evidence sections only.
> Concept name threaded through the paper: **conflict-aware raw-turn memory**.
---
## 1. Abstract
Long-term conversational memory, answering questions over weeks of prior dialogue, is the load-bearing capability for durable AI assistants, and three benchmarks (LoCoMo, LongMemEval, and BEAM) are the field's rulers. Every published leader adds structure to the memory path: atomic-fact distillation, temporal knowledge graphs, write-time entity reconciliation, and learned routing. Our insight is that these transforms are lossy in exactly the way the benchmarks penalize, because distillation strips the dates and specifics the rubrics score and write-time reconciliation silently resolves the contradictions the rubrics want surfaced. We show that one simple substrate, conflict-aware raw-turn memory (per-conversation minds, verbatim dated raw-turn retrieval, and a conflict-preserving answer policy), is state of the art on all three under each incumbent's own published protocol: LoCoMo 86.49 against 81.95, LongMemEval 95.01 macro against 94.87, and BEAM 0.6482 average and 74.0 percent pass against 0.6409 and 70.1 percent. We then measured 37 constructive interventions that tried to make the substrate cleverer; 33 lost, and we report them as a first-class falsification ledger. The single differentiator is conflict-awareness, worth 23 points on BEAM's contradiction ability. Because the leaderboards mix macro and micro metrics and gpt-5 and gpt-4o judges, we reproduce each incumbent's pipeline before comparing and report against the most conservative protocol on the board. All protocols, per-question artifacts, and the empty-answer-heal disclosure are released.
## 2. Introduction
An assistant that cannot recall what its user said three weeks ago cannot be a durable collaborator, and the field has converged on a single strategy for fixing this: make the memory cleverer. Three benchmarks now operationalize long-term conversational memory and anchor a public leaderboard race. LoCoMo, LongMemEval, and BEAM each ask a model to answer questions over weeks or months of prior dialogue, and each has a published state-of-the-art system built on the same premise. That premise, rarely stated but visible in every design, is that better memory means cleverer memory: richer structure imposed on the memory path, learned routing into it, and test-time recombination layered on top. The frontier, on this view, is engineering more intelligence into the store.
The leaders differ in representation but agree on that premise completely. Mem0 distills dialogue into atomic facts and retrieves the closest matches; Zep and its Graphiti engine build a temporal knowledge graph with validity intervals; Memori stores timestamped subject-predicate-object triples; and the mem0 memory platform reconciles facts against the existing store at write time and renders them chronologically. Each system spends its engineering budget on the same two moves: transform the raw conversation into a smaller derived representation, and resolve contradictions before the answerer ever runs. The shared assumption is that a cleaner, smaller, reconciled store is a better store.
Yet each incumbent's own weakest ability traces directly back to its own cleverness. The mem0 platform's write-time reconciler, the mechanism that lets it update a follower count in place and score well on knowledge update, is the same mechanism that makes contradiction resolution its worst BEAM ability at 0.357: it has already deleted one side of the very conflict the benchmark asks it to surface. Fact distillation, which compresses the store, is also what strips the verbatim spans that fine-grained recall questions score. The cleverness and the weakness are not independent; the second is a direct cost of the first. This pattern raises a question the leaderboard race has not asked: what if the cleverness is the problem, and the memory path should stay dumb?
We answer that question by evaluating a deliberately dumb substrate against all three benchmarks under each incumbent's own published protocol, and it is state of the art on every one (Table 1). The substrate keeps the original dated conversation turns per conversation, retrieves them verbatim, and preserves contradictions instead of reconciling them; it distills nothing into a primary layer, builds no graph, and learns no router. More useful to practitioners than the three wins themselves, we then measured 37 constructive interventions that each tried to make this substrate cleverer, and we publish every result. The practical payoff is that the community can stop paying the distillation, graph, and routing tax that our measurements show is self-inflicted.
**Table 1. Headline: one substrate, three benchmarks, three state-of-the-art results, each under the incumbent's own protocol.**
| Benchmark | Best published incumbent | Ours | Delta |
|---|---|---|---|
| LoCoMo (1,540 Q) | Memori 81.95% | 86.49% | +4.54pp (z=4.64, p<0.00001) |
| LongMemEval-S (500 Q) | Mastra 94.87 macro | 95.01 macro / 93.60 micro | +0.14 macro |
| BEAM 1M (700 Q) | mem0 0.6409 avg / 70.1% pass | 0.6482 avg / 74.0% pass | +3.9pp pass (McNemar z=2.14, p≈0.03); +0.0073 avg (tie) |
This paper makes four contributions. First, three same-protocol state-of-the-art results from one memory substrate: LoCoMo 86.49 against 81.95, LongMemEval 95.01 macro against 94.87, and BEAM 0.6482 average and 74.0 percent pass against 0.6409 and 70.1 percent. Second, conflict-aware raw-turn memory as the named mechanism behind them, whose single differentiator, retaining both sides of a contradiction rather than reconciling at write time, delivers 23 points on BEAM's hardest ability. Third, the falsification ledger, a first-class negative-results artifact recording 37 interventions across two independent programs, of which 33 lost to the simple substrate. Fourth, an evaluation-protocol-fidelity methodology that reproduces each incumbent before comparing and that exposed the leaderboard-metric chaos, macro versus micro aggregation, gpt-5 versus gpt-4o judges, and README versus shipped protocols, against which every number in this paper is reported.
The enabling shift is that answerers can now read raw turns directly. Long-context models such as gpt-5 consume dozens of verbatim dated turns per question without the context pressure that once made distillation a practical necessity, which turns throwing the raw data away from a reasonable compression into an unforced error. In hindsight the result is almost obvious: you should not delete the data before the model reads it, and you should not resolve a contradiction the user is about to ask you about. Conflict-aware raw-turn memory is the Bitter Lesson applied to agent memory, where hand-engineered structure loses to a simple method that leverages the raw signal and the answerer's compute.
Section 4 specifies the substrate, Section 5 the protocol-fidelity methodology that makes the comparisons defensible, and Section 6 the three results. Section 7 presents the falsification ledger, Section 8 analyzes why the simple substrate wins and where it loses, and Sections 9 and 10 give limitations and conclusions.
## 3. Related Work
Memory-augmented agents on LoCoMo (Maharana et al., 2024) pursue distinct representations of past dialogue. Mem0 (Chhikara et al., 2025) extracts atomic facts and retrieves the top matches; Zep and its Graphiti engine (Rasmussen et al., 2025) build a temporal knowledge graph with validity intervals; LangMem (LangChain, 2025) maintains per-user profile summaries; and Memori (GibsonAI, 2025) stores timestamped subject-predicate-object triples, the prior state of the art at 81.95 percent. MemR3 (see References) is an agentic multi-round retrieval pipeline rather than a fixed memory store, so we class it separately and report it only for reference. Every one of these systems answers from a representation it derived from the conversation, not from the conversation itself.
On LongMemEval (Wu et al., 2024) the leaders are observation-based and reconciled-fact stores. Mastra's observational memory (Mastra, 2026) reports 94.87 macro under the official gpt-4o judge; the mem0 platform reports 93.4 but under a gpt-5 judge rather than the official one; Hindsight (2025, arXiv:2512.12818) reports 91.4 with a Gemini-3 Pro answerer; Supermemory (Supermemory, 2026) reports 85.4; and Letta, the descendant of MemGPT (Packer et al., 2023), roughly 83.2. The judge model varies across these entries and moves the headline by points, a divergence we take up in Section 5 and neutralize by holding the official judge fixed.
On BEAM (Tavakoli et al., 2026), the ICLR 2026 benchmark for very long horizons, the reference system is the mem0 memory platform. It ingests each conversation in two-turn chunks, reconciles every new fact against the existing store at write time with explicit ADD, UPDATE, and DELETE events, retrieves the top-200 dated facts, and renders them chronologically for the answerer, scoring 0.6409 average and 70.1 percent pass at one million tokens of history. This is the most aggressively engineered memory path of the three incumbents, and its write-time reconciliation is precisely the mechanism we decline to use.
These systems span graphs, profiles, triples, and reconciled facts, but they share two commitments. Every one makes a derived representation the primary answer substrate, and every one resolves contradictions at write time, collapsing conflicting statements into a single surviving value before any question is asked. None of them retains the raw dated turns as the primary substrate the answerer reads, and none preserves both sides of a contradiction for the answerer to surface. That shared pair of commitments is exactly what our substrate breaks.
Reporting what did not work has precedent in ablation studies and reproducibility efforts, and Sutton's Bitter Lesson (Sutton, 2019) is the conceptual precedent for the result itself: hand-engineered structure repeatedly loses to simple methods that leverage raw data and compute. Memory-systems papers, however, typically report only the surviving configuration, and no prior work evaluates a single substrate across LoCoMo, LongMemEval, and BEAM under each benchmark's own incumbent protocol. We do both: one substrate measured on all three, and a ledger of 37 interventions that failed to improve it.
---
## 4. The Substrate: Conflict-Aware Raw-Turn Memory
We store memory as a set of per-conversation minds, and each mind keeps the source dialogue as verbatim dated raw-turn frames next to a small set of optional typed derived frames. A mind is the memory of one conversation, nothing more; there is no global graph and no cross-conversation entity table. The typed frames follow a three-way I/P/B model: Information frames record what was stated, Preference frames record standing directives, and Belief frames record claims that can later be contradicted. Retrieval is hybrid and fully local: dense vectors from a local Ollama `nomic-embed-text` embedder fuse with FTS5 BM25 lexical search through reciprocal rank fusion, and an optional in-process cross-encoder reranks the top candidates. No memory content leaves the machine on the write path or the read path.
The answer context is built from dated verbatim raw turns retrieved per conversation, and this is where the substrate departs from every incumbent it beats. Mem0 answers from atomic facts, Zep from a temporal knowledge graph, Memori from timestamped triples, and mem0-platform from write-time-reconciled facts; each makes a derived representation the primary answer substrate. We instead render the retrieved raw turns as the primary lane, each turn stamped `[YYYY-MM-DD]` from the source anchor, and treat any derived frames as auxiliary rather than as the substrate. The model reads what the user actually said, in order, with the dates attached, before it reads anything the pipeline inferred.
We resolve relative dates at write time, and this is the one transform the substrate applies to stored content. A turn that says "I went hiking yesterday" inside a session dated 8 May 2023 is stored as an event on 7 May 2023, not on the session date, so a later "when did X happen" question reads the true event date rather than the discussion date. We keep this transform precisely because it adds information (the resolved absolute date the user implied) rather than removing it; every transform we tested that removes or restructures content lost, as Section 7 documents. On BEAM the date stamp comes from a `chat.json` sidecar map at a 100 percent hit rate, so every raw turn carries a correct calendar date without a distillation pass.
The substrate retains both sides of a contradiction rather than reconciling them, and this single decision is the mechanism behind the BEAM contradiction result. When a user states a follower count of 10 in one session and 15 in another, mem0's write-time reconciler emits an UPDATE or DELETE and one number survives in the fact store; the contradiction the benchmark asks about has already been erased before the answerer sees it. Our substrate stores both dated statements and leaves them both retrievable, and the answer policy states the contradiction, presents both statements with their dates, and asks which is correct. That behavior is exactly what BEAM's contradiction rubric rewards, and it is unavailable to any system that resolves conflicts at write time.
The substrate deliberately omits four mechanisms the field treats as progress, and the omissions are the design rather than an incomplete implementation. It does not distill raw turns into a dominant fact or observation layer, because distillation strips the dates, numbers, and verbatim spans the rubrics score. It does not build a knowledge graph as the answer substrate, because the graph reconciles and abstracts the same signal the answerer needs raw. It does not reconcile contradictions at write time, because write-time resolution silently deletes one side of exactly the conflicts benchmarks probe. It does not learn a router over the memory path, because retrieval abilities are not reliably identifiable from question text, as our falsified 40 percent classifier shows. Each omission is a place the incumbents spend engineering and lose accuracy.
The answer policy adds a few rules tuned to what the rubrics reward, not to prose quality, and two of them are the load-bearing differentiators. Rule one is conditional abstention: the model commits to a factual span whenever a supporting anchor exists and abstains only when the anchor is genuinely absent, which removes the wrongful abstentions that forfeit answerable questions. Rule two, the surface-contradiction rule, instructs the model to present both sides of a conflict with dates instead of silently preferring the more recent statement. A third rule, the anti-wrongful-IDK rule, treats explicit "never happened" or "not completed" statements as real information rather than as missing data. The policy also calibrates date granularity (an exact day only when one was stated) and commits to one option on either-or questions. On BEAM this is prompt v2, which is mem0's own answer prompt with only rules three and four changed.
The same substrate produces all three state-of-the-art results, and we obtain them by swapping only the protocol harness, never the memory engineering. LoCoMo runs a seven-lane assembler over the frame store under a gpt-4.1-mini answerer and judge; LongMemEval runs a routed dated-observation-plus-raw-turn store under a gpt-5-mini answerer and the official gpt-4o judge; BEAM runs the raw-turn lane at top-30 under a gpt-5 answerer and judge. The lanes activated differ with the protocol, but the core is fixed: per-conversation minds, dated raw turns as the primary substrate, contradictions retained, dates resolved at write time. Figure 1 shows this pipeline.
![Conflict-aware raw-turn substrate: conversation to per-conversation dated raw-turn mind to top-30 retrieval to conflict-aware answer policy to answer, with no distillation, graph, reconciliation, or routing.](figures/fig1-substrate.png)
**Figure 1.** The conflict-aware raw-turn substrate, left to right. **Write time:** each source conversation becomes one per-conversation mind; every turn is stored verbatim as a dated raw-turn frame, its `[YYYY-MM-DD]` resolved from the session anchor (including relative references such as "yesterday"), and local LLM passes optionally add typed I/P/B frames without discarding the raw turns and without reconciling contradictions. **Read time:** hybrid retrieval (dense `nomic-embed` vectors fused with FTS5 BM25 by reciprocal rank fusion, optional cross-encoder rerank) draws the top-k dated raw turns as the primary answer lane, with derived frames as auxiliary lanes. **Answer policy:** retrieved turns render oldest-first, and the model applies the two rules the incumbents do not (surface both sides on contradiction, treat explicit "never happened" statements as information) plus conditional abstention and granularity-calibrated dates. The single write-time transform is relative-date resolution, which adds information; the substrate performs no distillation-as-primary, no graph construction, no write-time reconciliation, and no learned routing on the memory path.
---
## 5. Evaluation Protocol Fidelity
Cross-lab long-term-memory numbers are noise, so we reproduce each incumbent's pipeline before we compare, and we treat this reproduction as a contribution in its own right. LoCoMo, LongMemEval, and BEAM are all scored by an LLM judge whose model, prompt, and aggregation choices move the headline by several points, and the published leaderboards mix these choices without disclosing them. The only defensible claim is a same-judge, same-protocol comparison in which we first reproduce the incumbent's own number, then run our substrate through the identical harness. Table 2 records, for every benchmark, what the leaderboard reported and what protocol the shipped result files actually used.
We reproduced Memori's own LoCoMo pipeline to 81.98 percent against their published 81.95 percent before running our substrate through the same harness. The reproduction validated the harness end to end and exposed a protocol detail that matters: Memori's published ingest prepends image-caption text to every image-sharing turn, and our earlier ingest had used the bare turn text, so we had been benchmarking against a handicapped corpus. Restoring caption parity is protocol compliance, not an architecture change, and we report it as such. With the harness reproduced and input parity restored, the LoCoMo comparison is same-answerer and same-judge (gpt-4.1-mini for both) rather than cross-paper.
The LongMemEval leaderboard mixes two aggregation metrics and three judge protocols, and reading it as one number is a category error. Mastra's headline 94.87 is a macro score, the unweighted mean of six category accuracies, under the official gpt-4o judge; mem0's 93.4 uses gpt-5 as both answerer and judge, not the official Wu et al. gpt-4o judge, with a self-declared plus-or-minus one point interval. We report both metrics under the most conservative protocol on the board, the verbatim official gpt-4o judge with a gpt-5-mini answerer: 95.01 macro and 93.60 micro. Under Mastra's own headline metric we score 95.01 against their 94.87, and per question we tie their reconstructed 468 of 500. Because macro up-weights small categories (one preference question is worth 0.56 points of macro against 0.125 for a multi-session question), we disclose the micro number rather than let the metric flatter us.
The BEAM incumbent number carries the same hidden judge swap, and we neutralize it by matching models rather than assuming them. The mem0 result file behind the published 0.641 records `answerer_model: gpt-5` and `judge_model: gpt-5`, even though the repository README's CLI defaults imply gpt-4o for both, so a reader who trusts the README compares against the wrong protocol. Because the BEAM paper itself shows the answerer dominates the score (its light-answerer baseline at 1M scores near 0.336), an unmatched comparison would confound memory architecture with answerer model. We therefore run our best configuration at gpt-5 answerer and gpt-5 judge, mem0's actual protocol, and we report both a pass rate (our headline claim) and an average score (which sits inside judge noise).
We disclose every correction and every noise band rather than report only the favorable cell. The initial full-700 BEAM run scored 0.6050 because 56 answers were empty strings that all scored zero, an artifact of our client capping completion tokens at 4096 so that gpt-5 exhausted the budget on hidden reasoning for long-form questions (29 of 70 summarization prompts). We re-answered all 56 under the identical configuration (mean 0.000 healing to 0.528) with a 16K floor and retry-on-empty, and we preserve the pre-heal snapshot as a disclosed artifact; mem0 is unaffected by this class of bug because its contexts run near 7K tokens. We also report the judge-noise band directly (standard error near 0.014 on the BEAM average score), we canonicalize by first-occurrence deduplication on `instance_id` (700 unique rows from 1020 raw), and we archive per-question answers and judgments for every run. These disclosures are the anti-cherry-pick posture the metric audit demands.
**Table 2. Protocol-fidelity matrix: what the leaderboard reported versus the protocol we ran.** Best-known incumbent per benchmark shown; "README vs actual" flags a shipped protocol that differs from the documented default.
| Benchmark | System | Answerer | Judge | Metric reported | Retrieval budget |
|---|---|---|---|---|---|
| LoCoMo (1,540 Q) | Ours | gpt-4.1-mini | gpt-4.1-mini | micro accuracy | 7-lane assembler (~3,747 tok) |
| LoCoMo | Memori (reproduced 81.98 vs 81.95) | gpt-4.1-mini | gpt-4.1-mini | micro accuracy | timestamped triples (~1,294 tok) |
| LongMemEval-S (500 Q) | Ours | gpt-5-mini | official gpt-4o | macro 95.01 and micro 93.60 | routed observation + raw-turn store |
| LongMemEval-S | Mastra (94.87) | gpt-5-mini | official gpt-4o | macro only | observation store |
| LongMemEval-S | mem0 (93.4) | gpt-5 | gpt-5 (not official gpt-4o) | undisclosed aggregation | fact store |
| BEAM 1M (700 Q) | Ours | gpt-5 | gpt-5 | avg score 0.6482 and pass 74.0% | top-30 raw turns |
| BEAM 1M | mem0 (0.6409 / 70.1%) | gpt-5 (README says gpt-4o) | gpt-5 (README says gpt-4o) | avg score only | top-200 facts |
---
## 6. Results
### 6.1 LoCoMo
Our substrate scores 86.49 percent (1332 of 1540) on LoCoMo under Memori's exact published protocol, 4.54 points above the best previously published memory system. The one-sample test against Memori's 81.95 percent gives z equal to 4.64 at p below 10 to the minus 5, the substrate leads or ties every question category among memory systems, and its open-domain accuracy is statistically indistinguishable from the full-context ceiling. The result is verified drift-free: a fresh regeneration on the current repository HEAD reproduced 86.49 percent exactly, and an earlier 87.66 percent figure that did not reproduce was traced to a stale-verdict-replay bug in the judge harness, disclosed and superseded.
We also correct a column-scrambling error that has propagated through the LoCoMo literature, and we re-run Mem0 under our own judge. The Memori paper's baseline table transplanted MemR3's column order under mismatched headers, which turned LangMem's single-hop score of 86.92 into a widely cited "temporal 86.92"; LangMem's true temporal score is 61.06. Table 3 gives the corrected landscape. Re-running Mem0 on our shared answerer and judge (substrate against substrate) we lead every category, with temporal accuracy 30.8 points ahead of Mem0's 50.78.
The category structure of Table 3 carries the design thesis, which is that no single representation wins long-term conversational memory. Profile cards win open-domain synthesis at the full-context ceiling, write-time-dated episodic timelines win temporal anchoring, and the verbatim raw-detail lane wins fine-grained single-hop recall where distillation abstracts the answer away (the gold "a painting of a sunset with a pink sky" survives only in the raw turn, not in the distilled fact "an abstract painting"). A substrate that serves all three simultaneously beats any single-representation incumbent, and the headline statistical claim rests on the reproduced same-judge Memori row rather than on the cross-sourced ones.
**Table 3. Corrected LoCoMo landscape (gpt-4.1-mini protocol, accuracy %).** Baselines corrected per Du et al.; only the Memori and Mem0 rows are same-harness with ours. MemR3 is an agentic retrieval pipeline, shown for reference. Best memory-system score per column in bold.
| Method | single | multi | temporal | open | overall |
|---|---|---|---|---|---|
| **Ours (conflict-aware raw-turn)** | **92.27** | **80.50** | **81.62** | **69.79** | **86.49** |
| Memori (same-judge) | 87.87 | 72.70 | 80.37 | 63.54 | 81.95 |
| Mem0 (same-judge, ours) | 83.59 | 74.82 | 50.78 | 64.58 | 73.96 |
| Zep | 83.49 | 72.34 | 77.26 | 64.58 | n/a |
| LangMem | 86.92 | 74.47 | 61.06 | 67.71 | 78.05 |
| Mem0 (literature) | 66.47 | 62.41 | 57.32 | 44.79 | 62.47 |
| MemR3 (agentic; ref.) | 92.17 | 81.20 | 82.14 | 71.53 | n/a |
| Full-context ceiling | 93.73 | 86.43 | 86.82 | 71.88 | n/a |
### 6.2 LongMemEval
Our substrate scores 95.01 macro and 93.60 micro (468 of 500) on LongMemEval-S under the official gpt-4o judge, above Mastra's 94.87 under Mastra's own macro metric. The result is state-of-the-art on the leader's headline metric and holds the most conservative judge protocol on the board, as Section 5 established. The arc from a 75.9 percent gpt-4o baseline to 95.01 macro was won by four answer-side survivors, not by richer memory, and Table 4 traces every rung.
The two levers that moved the number most were an information-adding fix and an aggregation of reads, neither of which restructures stored content. Recovering the dropped `question_date` field, the reference date that "how long ago" questions anchor to, lifted temporal accuracy from 61 to 84 percent on its own, because no retrieval or reasoning tuning can supply a reference point that is absent from the data. Full-log self-consistency voting then attacked the residual counting errors directly: multi-session counts carry a plus-or-minus-one enumeration variance, and voting the mode of five reads over the same complete log recovered them where static prompt tuning failed. Both fixes preserve the store and read it better rather than transforming what is stored.
The four adopted levers (self-consistency voting, a guarded split-vote counting override, a preference grounding critic, and a routing-matrix completion) are all answer-side, and the nineteen we falsified are catalogued in Section 7. Each survivor aggregates multiple reads of the preserved store: it votes, it counts a guarded subset, it runs best-of-five on ungrounded answers, or it selects which complete context to read. None of them adds a derived representation or reconciles a conflict. This is the same verdict BEAM reaches by a different route.
**Table 4. LongMemEval intervention arc (official gpt-4o judge).** Overall is micro accuracy unless noted; the four adopted levers are answer-side.
| Configuration | Answerer | Overall | Note |
|---|---|---|---|
| base lanes (raw turns) | gpt-4o | 66.2% | no write-time distillation |
| + profile cards | gpt-4o | 75.9% | best gpt-4o config |
| + observation extraction + question_date + preference (qd) | gpt-5-mini | 88.2% | two root-cause fixes (adds information) |
| + per-category type routing | gpt-5-mini | 91.2% | temporal/preference to full-log |
| + qd-vote on counting questions | gpt-5-mini | 92.2% | 5x varied-K self-consistency |
| + full-log self-consistency vote (vh5) | gpt-5-mini | 92.8% | adopted |
| + KG-ledger split-vote guard (klc) | gpt-5-mini | 93.0% | adopted |
| + preference grounding critic (pfc) | gpt-5-mini | 93.4% | adopted (mem0 tied) |
| + ssu routing completion (h3u, final) | gpt-5-mini | **93.60 micro / 95.01 macro** | adopted |
### 6.3 BEAM
Our substrate scores 0.6482 average and 74.0 percent pass (518 of 700) on BEAM 1M under mem0's exact gpt-5 answerer and gpt-5 judge protocol, against mem0's 0.6409 average and 70.1 percent pass. The pass rate is the headline claim, and a McNemar paired test on the 700 shared questions confirms it: among the discordant pairs, we pass 93 questions where mem0 fails against 66 where the reverse holds, which gives z = 2.14 (2.06 with continuity correction) at p ≈ 0.03. An unpaired two-proportion test on the same two rates returns only z = 1.61, but the paired test is the correct one here, because both systems answer the identical 700 questions, so the shared question-level difficulty variance cancels and only the discordant pairs carry signal. The average-score margin of plus 0.0073 sits inside the judge-noise band (standard error near 0.014), so we report it as parity-or-better rather than as a win, and we state both numbers rather than the flattering one alone. The configuration is the plain raw-turn lane at top-30 with prompt v2, chosen after 14 falsified alternatives.
The single differentiator is contradiction resolution, where the substrate scores 0.588 (87.1 percent pass) against mem0's 0.357 (48.6 percent pass), a gap of 0.231, or 23 points, on the benchmark's hardest ability. This is a direct consequence of retaining both sides of a conflict rather than reconciling at write time: mem0's write-time reconciler makes contradiction resolution its own floor because it has already deleted one side before the answerer runs, while our substrate surfaces both dated statements and asks which is correct. Two other retained-signal abilities also favor the substrate, abstention by 6.8 points (0.593 versus 0.525) and information extraction by 4.0 points (0.740 versus 0.700), and Table 5 gives the full profile. The losses concentrate in coverage-shaped abilities, summarization by 6.5 points and temporal reasoning by 6.1 points, where a compact enumerative fact store harvests more compound-nugget clauses than a raw-turn context (the mechanism is analyzed in Section 8).
Depth of distillation is not mem0's lever, and compactness of raw beats it. mem0's own ablation quadruples retrieval from top-50 to top-200 facts and gains only 0.037 average score, so their advantage is not in stuffing more memories into the prompt. Our result uses 7 times fewer retrieved items than theirs (30 raw turns against 200 facts) and still leads on pass rate and ties on average, and the entire BEAM program cost roughly 160 dollars. The compact raw context wins the abilities that reward verbatim detail and conflict preservation, and it loses only the abilities that reward enumerative breadth.
**Table 5. BEAM 1M per-ability, ours versus mem0 (both gpt-5 answerer and judge, full 700).** Average score per ability; overall average and pass rate at the foot. Positive delta favors ours.
| Ability | Ours | mem0 | Δ |
|---|---|---|---|
| contradiction_resolution | **0.588** (87.1% pass) | 0.357 (48.6%) | **+0.231** |
| abstention | **0.593** | 0.525 | **+0.068** |
| information_extraction | **0.740** | 0.700 | **+0.040** |
| instruction_following | 0.834 | 0.852 | −0.018 |
| preference_following | 0.859 | 0.883 | −0.024 |
| multi_session_reasoning | 0.644 | 0.652 | −0.008 |
| knowledge_update | 0.604 | 0.650 | −0.046 |
| temporal_reasoning | 0.557 | 0.618 | −0.061 |
| event_ordering | 0.494 | 0.536 | −0.042 |
| summarization | 0.570 | 0.635 | −0.065 |
| **Overall Avg Score** | **0.6482** | 0.6409 | **+0.0073** |
| **Pass Rate (≥0.5)** | **74.0%** (518/700) | 70.1% (491/700) | **+3.9pp** |
---
## 7. The Falsification Ledger
We treat negative results as the product, and we report 37 constructive interventions across two independent programs, of which 33 lost to the simple substrate. The two programs are 23 LongMemEval levers (4 adopted, 19 falsified) and 14 BEAM levers (all 14 falsified), so roughly 90 percent of everything we tried made the substrate worse or no better. Table 6 unifies them by family: lever, program, result, and why it lost. Every family tells the same story, and the four survivors share a property we make precise at the end.
Compression and distillation lost on both programs, because they strip the dates, numbers, and verbatim spans the rubrics score. On BEAM, distilled-fact retrieval scored 0.354 at k equal to 100 and 0.382 at k equal to 200 against 0.448 for raw turns, and every raw-plus-fact hybrid landed between 0.498 and 0.540 against 0.591 for raw alone; facts are net-harmful at any mixing ratio. A session-outline preamble diluted summarization from 0.38 to 0.16, and on LongMemEval an exhaustive re-distiller with five times the observation density scored 2 net worse than the tuned cache, because near-duplicate observations drop reranker precision.
Structuring the store as a knowledge graph or a ledger lost almost everywhere it removed the raw signal, with one guarded exception. Overriding LongMemEval votes with a KG-ledger count cost 10 net when applied to unanimous votes, because a unanimous five-voter consensus is a stronger correctness signal than the ledger, and injecting a reconciled ledger as trusted context in the voting path cost 5 net, because an imperfect structured input corrupts more voters than it clarifies. A write-time incremental ledger failed its gate outright (0 recoveries on 26), because counting is question-scope-dependent in a way question-blind determinism cannot capture. The single structuring survivor, klc, adds only 0.2 points, and it fires only on split votes where the deterministic count has clean provenance; it aggregates uncertain reads rather than restructuring the store.
Routing lost when it classified content and survived only when it selected among complete contexts by question type. On BEAM, gold-blind ability routing hit a 40 percent classifier ceiling and scored 0.584 below the 0.600 baseline, because abilities are not identifiable from question text, and coverage-shaped stratified retrieval fell to similarity-dense clusters (summarization 0.38 to 0.24). On LongMemEval, by contrast, deterministic routing by question type (full-log for temporal and preference, retrieval for the rest) was adopted for plus 3 points, and completing that routing matrix for single-session-user (h3u) added the final point. The surviving router transforms nothing in storage; it chooses which already-complete context the answerer reads.
Prompt shaping plateaued: no prompt beat the plain conflict-aware v2 prompt on BEAM, and none beat the tuned answer policy on LongMemEval. A strict-abstention prompt scored 0.542 on BEAM and cost 3 net on LongMemEval, because over-conservatism spreads to answerable questions; an exhaustive-enumeration prompt cost 0.048 overall and 0.119 on summarization, because "do not omit minor items" dilutes clause density with trivia; a temporal-commit prompt moved plus 0.023, below re-roll variance. Raising reasoning effort netted zero and over-abstained, because the bottleneck is enumeration variance and retrieval availability, not reasoning depth.
Ensembling lost when it merged different answers and won only when it voted the same store, which is the cleanest statement of the whole pattern. Answer-merge self-ensemble on BEAM scored 0.668 against 0.674 for a single answer at twice the cost, because the union dilutes as much as it harvests, and a ten-voter cross-path ensemble on LongMemEval cost 1 net, because duration canonicals are incompatible across paths. Swapping in a stronger gpt-5 answerer failed its gate (109 against 114 multi-session), because the stronger model over-abstains on answerable questions. The two ensembling survivors, self-consistency voting (vh5, plus 1.6 points) and the preference grounding critic (pfc, plus 0.4 points), both read the preserved store multiple times and aggregate; neither adds structure.
Two programs on three benchmarks reach one verdict: the local optimum is plain dated raw-turn retrieval plus a conflict-aware prompt, and cleverness on the memory path is measured self-harm. The pattern is exact. Every lever that transforms stored content, whether it distills it, graphs it, reconciles it, or restructures it, lost. The only survivors aggregate multiple reads of an untransformed store: they vote over it, count a guarded subset of it, run best-of-five grounding on it, or route among complete views of it. The load-bearing moves are to keep the raw data, resolve dates at write time because that adds information, and preserve contradictions; everything else the field is building is optional at best and harmful in the measured cases.
**Table 6. The falsification ledger.** Principal levers grouped by family (37 total across both programs, 33 falsified, 4 adopted). Full per-run records live in the cited result files. "F" = falsified, "A" = adopted.
| Family | Lever | Program | Result | Why it lost / survived | Verdict |
|---|---|---|---|---|---|
| Compression / distillation | Distilled-fact retrieval k=100 | BEAM | 0.354 vs 0.448 raw | strips dates and numbers | F |
| Compression / distillation | Distilled-fact retrieval k=200 | BEAM | 0.382 | recall recovered, lossy abilities stayed dead | F |
| Compression / distillation | Hybrid 15 raw + 60 facts | BEAM | 0.540 vs 0.591 | cutting raw turns loses detail | F |
| Compression / distillation | Additive hybrid 30 raw + 60 facts | BEAM | 0.498 | facts net-harmful at any mix | F |
| Compression / distillation | Session-outline preamble | BEAM | 0.536 vs 0.600; summ 0.38→0.16 | lossy summary dilutes | F |
| Compression / distillation | Exhaustive re-distiller (5x density) | LongMemEval | −2 vs tuned cache | near-duplicate distractors drop reranker precision | F |
| Structuring (KG / ledger) | ipb answer-accumulation cell | BEAM | 0.591 ≈ v2 | order-dependent, no gain over dated turns | F |
| Structuring (KG / ledger) | KG-ledger override of unanimous votes (kla) | LongMemEval | −10 | unanimous consensus beats the ledger | F |
| Structuring (KG / ledger) | KG-ledger, gpt-5-mini extractor (klc2) | LongMemEval | −1 | stronger extractor gives different, not better, ledgers | F |
| Structuring (KG / ledger) | Date-ledger override for durations | LongMemEval | ~−2 | duration errors are anchor-selection | F |
| Structuring (KG / ledger) | Write-time incremental ledger (D1) | LongMemEval | gate 0/26, aborted | counting is question-scope-dependent | F |
| Structuring (KG / ledger) | Reconciled-ledger-as-context in voting | LongMemEval | −5 | imperfect structure corrupts voters | F |
| Structuring (KG / ledger) | KG-ledger split-vote guard (klc) | LongMemEval | +0.2pp | deterministic count only on split votes | A |
| Routing | Full gold-blind ability routing | BEAM | 0.584 < 0.600 | 40% classifier ceiling; abilities not text-identifiable | F |
| Routing | Coverage-shaped stratified retrieval | BEAM | summ 0.38→0.24 | similarity-dense cluster beats stratified coverage | F |
| Routing | ssu routing-matrix completion (h3u) | LongMemEval | +1 (final) | selects a complete context by question type, no transform | A |
| Prompt shaping | Prompt v3 strict abstention | BEAM | 0.542 | over-conservatism spreads to all abilities | F |
| Prompt shaping | Prompt v1 (mem0 verbatim on our substrate) | BEAM | 0.573 | loses contradiction and wrongful-IDK points | F |
| Prompt shaping | Prompt v4 exhaustive enumeration | BEAM | −0.048; summ −0.119 | trivia dilutes clause density | F |
| Prompt shaping | Prompt v5 temporal commit | BEAM | +0.023 (~0.5 SE) | below re-roll variance | F |
| Prompt shaping | Standing-directives preamble | BEAM | flat (0.557) | preferences already in raw turns | F |
| Prompt shaping | Strict abstention + answer hygiene | LongMemEval | −3 | over-abstains on answerable questions | F |
| Prompt shaping | reasoning_effort = high | LongMemEval | ~0, over-abstains | bottleneck is variance, not reasoning | F |
| Prompt shaping | IDK-escalation raw sweep | LongMemEval | 0 | quote-level guard suppresses everything | F |
| Prompt shaping | Count-hint (blanket K48 / phrasing) | LongMemEval | +3 (12 up, 9 down) | fixes and breaks equal numbers | F |
| Ensembling / voting | Answer-merge self-ensemble | BEAM | 0.668 vs 0.674 (2x cost) | union dilutes as much as it harvests | F |
| Ensembling / voting | 10-voter cross-path ensemble | LongMemEval | −1 | duration canonicals incompatible across paths | F |
| Ensembling / voting | gpt-5 (full) answerer swap | LongMemEval | gate FAIL (109 vs 114 ms) | stronger model over-abstains | F |
| Ensembling / voting | P/B-frame consolidation answerer swap | LongMemEval | −1 | whole-path swap pays the noise floor | F |
| Ensembling / voting | KU recency-critic best-of-5 | LongMemEval | −1 | KU errors are not recency-mechanical | F |
| Ensembling / voting | ssu supportedness-critic best-of-5 | LongMemEval | 0 (0 fires) | failures are absent from context | F |
| Ensembling / voting | Full-log self-consistency vote (vh5) | LongMemEval | +1.6pp | votes over the same complete log | A |
| Ensembling / voting | Preference grounding critic (pfc) | LongMemEval | +0.4pp | gold-blind best-of-5 on ungrounded answers | A |
![Falsification ledger strip plot: 33 falsified levers on or below the raw-turn baseline, only the 4 adopted answer-side levers above it, grouped by lever family.](figures/fig2-falsification.png)
**Figure 2.** Falsification strip plot: lever family on the x-axis against change in benchmark score on the y-axis, one point per lever in Table 6, BEAM and LongMemEval markers distinguished. The horizontal zero line is the simple raw-turn baseline; roughly 90 percent of points fall on or below it, and the four points above it are all answer-side aggregation levers (vh5, klc, pfc, h3u), never memory-structure levers. The visual message without the caption: complexity buys nothing above the raw-turn line.
---
## 8. Analysis
Every transform on the memory path is lossy, and the benchmarks reward exactly what the transforms remove. Distillation collapses a verbatim turn into a fact and discards the surrounding numbers, dates, and phrasing; a knowledge graph abstracts a statement into typed edges; write-time reconciliation deletes the superseded side of a conflict. BEAM's rubric, by contrast, scores compound nuggets that demand specific clauses ("should contain X and Y with detail Z"), and its contradiction ability scores the presence of both conflicting statements. Detail removed by a transform cannot be recovered downstream, so the only question that matters is whether the answerer had the material and failed to use it or never received it at all.
The retrieval-headroom audit answers that question decisively: the losses are answer-side, not retrieval-side. For each failed nugget on the three lossy abilities we retrieved the question's top-150 raw turns and found the first rank at which the supporting content appears. It appears at rank 30 or better for 88 percent of failed summarization nuggets, 90 percent of event-ordering nuggets, and 93 percent of multi-session-reasoning nuggets; the median first-hit rank is between 1.5 and 4, and the content is absent from the top-150 entirely for only 2 to 3 percent. Widening retrieval from 30 to 100 turns moves only about 8 percent of failed nuggets, at two to three times the input cost. The material the answerer needs is already in front of it at top-30, so deeper retrieval and richer structure cannot help, and distillation would only remove what the answerer already holds.
The contradiction result is the clearest demonstration that write-time cleverness is a liability. The mem0 platform reconciles at ingest, so when a user's follower count changes the old fact is updated in place, which is why knowledge update is one of its stronger abilities at 0.650. The very same reconciler is why contradiction resolution is its weakest at 0.357, because it has already collapsed the two dated statements into one before the answerer runs, and BEAM's rubric wants both surfaced with a request for clarification. Our substrate makes the opposite trade: it retains both dated statements and the answer policy states the conflict, so it scores 0.588 and passes 87.1 percent of contradiction questions against mem0's 48.6 percent, a gap of 23 points of average score on the benchmark's hardest ability. The trade is intrinsic to when a conflict is resolved, not a tuning gap a better reconciler could close. A write-time reconciler must pick one side to store; a read-time policy can keep both and decide per question. Resolving early optimizes the wrong moment.
Where the substrate loses, the cause is answer breadth and answer disposition, not memory quality. On summarization the final gap is 6.5 points (0.570 against 0.635), and the peer-to-peer nugget forensics, run at 100 percent question and nugget alignment, locate it in verbosity: mem0's answers average 1067 words and 48 bullets against our 600 words and 29 bullets, and because BEAM nuggets are compound, more enumerated surface mechanically harvests more clauses. Most of mem0's winning summarization nuggets are upgrades from our 0.5 to their 1.0, cases where we hit the topic but drop one clause of a "should contain X and Y" nugget. Event ordering shows the same shape and, tellingly, zero true ordering errors: BEAM scores event ordering as coverage of the listed events rather than sequence correctness, so the loss there is again clause density, not a reasoning failure.
The temporal gap is dispositional. BEAM temporal questions are 94 percent duration arithmetic between two dated events whose anchors both live in the dialogue, and 74 percent of our loss is over-abstention plus wrong-anchor selection: on five questions we answered "insufficient information" where mem0 committed to the right date pair and won, and on three we selected the wrong dates. mem0 emits a single terse declarative every time, while we hedge on 16 of 70. That the temporal-commit prompt (v5) recovered only half the over-abstention at plus 0.023, below re-roll variance, is direct evidence the gap is a disposition of the answerer under this judge, not a deficit of the stored memory: the anchors are present at top-30, and the substrate simply under-commits.
One pattern unifies the four survivors across both programs: they all aggregate multiple reads of an unchanged store, and none transforms stored content. Self-consistency voting (vh5) takes the mode of five reads of the same complete log; the guarded counting override (klc) recounts a subset deterministically only on split votes; the preference grounding critic (pfc) runs best-of-five on ungrounded answers; and the routing completion (h3u) selects which complete context the answerer reads. Every falsified lever, by contrast, transforms the store, whether it distills, graphs, reconciles, or restructures it. The load-bearing conclusion is that the productive place to spend cleverness is read time, not write time. The survivors are the memory analog of test-time-compute scaling: hold the store fixed and buy accuracy by reading it more times or more carefully, exactly as reasoning models buy accuracy by sampling more chains over a fixed prompt. Cleverness on the memory path, which mutates the store, loses; cleverness on the read path, which aggregates over it, wins.
## 9. Limitations
Our BEAM average-score margin is inside the noise. The plus 0.0073 average sits within a judge-noise standard error near 0.014, so we rest the BEAM claim on the pass rate (plus 3.9 points; McNemar paired z = 2.14, p ≈ 0.03) and report the average only as parity-or-better. A reader who weights average score above pass rate should read the BEAM average as a statistical tie, and the honest headline is the pass rate alone.
The substrate pays for its detail in tokens. Our BEAM context runs approximately 27,000 tokens at top-30 raw turns (the mean answer-prompt token count recorded in the final run summary) against mem0's roughly 7,000 tokens of distilled facts, and on LoCoMo our seven-lane assembler uses 3,747 tokens against Memori's 1,294. Raw-turn retrieval is cheap at small k but scales super-linearly: at top-100 the context reaches about 88,000 tokens per question. Distillation's genuine advantage is inference cost, not answer quality, so we win the quality axis and concede the efficiency axis, and we point readers to the per-question cost figures rather than bury them.
We remain behind on coverage-shaped abilities. Summarization stays below mem0 (0.570 against 0.635), and whether a raw substrate can close that gap without the answer-verbosity tax, which backfired every time we forced enumeration, is open. The scope is also narrow in two ways. We evaluate a single substrate, and each benchmark uses one answerer family per its protocol (gpt-4.1-mini on LoCoMo, gpt-5-mini on LongMemEval, gpt-5 on BEAM), so the generality of the raw-turn design across weaker or cheaper answerers is untested.
Finally, all scores are LLM-as-judge, and two risks remain after our mitigations. The BEAM protocol has gpt-5 judging a gpt-5 answerer, which invites self-preference; we mitigate with fixed judges, archived per-question judgments, reproduced baselines, and reported noise bands, but some deltas still sit inside that variance. And all three benchmarks are English synthetic dialogues, so the result speaks to the benchmarks the field currently races on rather than to production memory over real users. We also disclose an empty-answer bug (56 rows healed under the identical configuration, with the pre-heal snapshot preserved) so that the post-heal numbers are the honest comparison.
## 10. Conclusion
One deliberately dumb substrate, the original dated conversation turns kept per conversation with contradictions preserved rather than reconciled, is state of the art on LoCoMo, LongMemEval, and BEAM, each under the incumbent's own published protocol. Thirty-seven measured interventions say the field is over-engineering the memory path: distillation, graphs, write-time reconciliation, and learned routing lost, and the only transform that helped, resolving relative dates at write time, adds information rather than removing it. This is the Bitter Lesson for agent memory.
The actionable lesson is a division of labor: keep the store dumb and spend cleverness on reads. Preserve the raw dated turns, resolve dates at write time, and keep both sides of a contradiction; then, when you want more accuracy, aggregate multiple reads of that store rather than transform it, because voting, guarded recounting, grounding critics, and routing among complete contexts were the only interventions that survived measurement. We release all protocols, per-question artifacts, the falsification ledger, and the harness, so the community can stop re-deriving the distillation and routing tax and can audit its own leaderboard metrics.
Three questions stay open. Can conflict preservation be made cheap enough to also win the coverage abilities, or is there a write-time transform other than date resolution that adds information rather than removing it? Does the raw-turn result hold under weaker and cheaper answerers, or is long-context capacity the true enabler? And what is the right rubric for a memory that surfaces a contradiction and asks, as against one that commits and happens to be right? The substrate is simple on purpose, and the remaining work is to keep it simple while teaching it to summarize without forgetting.
## Ethics and Broader Impact
All three benchmarks in this work use synthetically generated dialogues, so the experiments involve no human subjects and no personal data. Long-term memory systems nonetheless carry real retention and erasure obligations once they run over actual users, and our choice to retain verbatim raw turns rather than distilled abstractions raises the stakes of deletion: the substrate holds more of what a user actually said, so a user's right to erasure must reach every stored turn. The codebase addresses this directly with an erasure cascade that removes a source turn together with every frame and index entry derived from it, so deletion is total rather than cosmetic. Our scores also rest on LLM-as-judge evaluation, and LLM judges carry documented biases, including self-preference when a model grades its own family; we disclose the judge model for every number, hold the official judge fixed where one exists, and archive per-question judgments so third parties can re-audit. We encourage deployments of raw-turn memory to pair the accuracy benefits with explicit retention limits and auditable deletion.
## References
Author-year style. Entries were reconstructed offline and verified against arXiv, Semantic Scholar, and primary product pages before submission.
- Chhikara, P., Khant, D., Aryan, S., Singh, T., & Yadav, D. (2025). *Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory.* arXiv:2504.19413. Benchmark harness: `mem0ai/memory-benchmarks` (GitHub).
- Du, X., Li, L., Zhang, D., & Song, L. (2025). *MemR³: Memory Retrieval via Reflective Reasoning for LLM Agents.* arXiv:2512.20237. Source of the corrected LoCoMo baseline numbers used in Section 6.1 (Table 3); this is the same work as the MemR3 entry below.
- GibsonAI (2025). *Memori: an open-source SQL-native memory engine for LLM agents.* `github.com/GibsonAI/memori`.
- Latimer, C., Boschi, N., Neeser, A., Bartholomew, C., Srivastava, G., Wang, X., & Ramakrishnan, N. (2025). *Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflects.* arXiv:2512.12818. LongMemEval result (91.4%) reported with a Gemini-3 Pro answerer over a GPT-OSS-120B memory stack.
- LangChain (2025). *LangMem.* `github.com/langchain-ai/langmem`.
- Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., & Fang, Y. (2024). *Evaluating Very Long-Term Conversational Memory of LLM Agents (LoCoMo).* arXiv:2402.17753.
- Mastra / Barnes, T. (2026). *Observational Memory: 95% on LongMemEval.* `mastra.ai/research/observational-memory`.
- MemR3 — see Du, X., et al. (2025), *MemR³: Memory Retrieval via Reflective Reasoning for LLM Agents,* arXiv:2512.20237 (entry above). Agentic multi-round retrieval pipeline, shown for reference on LoCoMo (Table 3).
- Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023). *MemGPT: Towards LLMs as Operating Systems.* arXiv:2310.08560. Letta is the successor project (`letta.com`).
- Rasmussen, P., Paliychuk, P., Beauvais, T., Ryan, J., & Chalef, D. (2025). *Zep: A Temporal Knowledge Graph Architecture for Agent Memory.* arXiv:2501.13956.
- Supermemory (2025). *LongMemEval result (~85.9% overall).* `supermemory.ai/research`.
- Sutton, R. S. (2019). *The Bitter Lesson.* `incompleteideas.net/IncIdeas/BitterLesson.html`.
- Tavakoli, M., Salemi, A., Ye, C., Abdalla, M., Zamani, H., & Mitchell, J. R. (2026). *Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs.* ICLR 2026 (introduces the BEAM benchmark). arXiv:2510.27246.
- Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., & Yu, D. (2024). *LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory.* arXiv:2410.10813.
## Appendix A: Complete falsification ledger (LongMemEval)
Table 6 lists principal levers grouped by family; the four adopted LongMemEval survivors are self-consistency voting (vh5), the guarded split-vote KG-ledger count (klc), the preference grounding critic (pfc), and the routing-matrix completion (h3u). This appendix enumerates all 19 falsified LongMemEval levers individually. Eighteen of them appear as rows in the negative-results table of the released LongMemEval results file; the nineteenth (blanket full-log routing) is documented in that file's routing finding, where applying the full-log path to knowledge-update and multi-session lost 3 and 2 to 4 points respectively, which is why routing is selective rather than universal. Net effects reported in percentage points (or instances, where noted) against the tuned baseline under the official gpt-4o judge. Combined with the 14 falsified BEAM levers, the two programs falsified 33 of 37 measured interventions.
| # | Falsified lever | Net effect | Why it lost |
|---|---|---|---|
| 1 | Exhaustive re-distiller (v2, ~1,500 obs/instance, 5x density) | −2 vs tuned cache | near-duplicate observations drop reranker precision |
| 2 | Count-hint, blanket K48 | +3 instances (12 up, 9 down), within noise | fixes and breaks counting questions in equal number; variance, not bias |
| 3 | Count-hint, targeted by phrasing | +3 (same) | recoveries and regressions are both counting-phrased |
| 4 | reasoning_effort = high | ~0, over-abstains | bottleneck is enumeration variance and availability, not reasoning depth |
| 5 | Strict abstention + answer hygiene | −3 | over-abstains on answerable questions |
| 6 | P/B-frame consolidation as answerer swap (KU) | −1 (fixed stale value, regressed 2) | whole-path swap pays the noise floor |
| 7 | IDK-escalation raw sweep (guarded) | 0 (0 up, 0 down) | quote-level guard suppresses everything; floor holds |
| 8 | KG-ledger override of unanimous votes (kla) | −10 (5 up, 15 down) | unanimous five-voter consensus beats the ledger |
| 9 | KG-ledger with gpt-5-mini extractor (klc2) | −1 | a stronger extractor gives different, not better, ledgers |
| 10 | P-frame current-value KU override (guarded) | −1 (1 up, 2 down) | chain detection mis-resolves countable attributes |
| 11 | vh5 x qd-vote 10-voter ensemble | −1 (analytical) | duration canonicals are incompatible across paths; real disagreements favor vh5 2:1 |
| 12 | Date-ledger override for temporal durations | ~−2 (gold-visible) | duration errors are anchor-selection; the extractor mis-grounds the same anchors the answerer does |
| 13 | Dual-extractor consensus vs unanimous votes | 0 up, 19 at risk (analytical) | independent extractors share blind spots and cannot crack the unanimous-wrong pool |
| 14 | Option D1: write-time incremental ledger | gate 0/26, aborted | counting is question-scope-dependent; question-blind determinism fails on scope |
| 15 | KU recency-critic best-of-5 | −1 (1 up, 2 down) | attribute-matching ambiguity flags correct answers as stale; KU errors are not recency-mechanical |
| 16 | ssu supportedness-critic best-of-5 | 0 (0 fires on 70) | the failing answers are absent from context (retrieval-availability); regeneration cannot help |
| 17 | Reconciled-ledger-as-context in the voting path | −5 (2 up, 7 down) | an imperfect ledger injected as trusted context corrupts more voters than it clarifies |
| 18 | gpt-5 (full) answerer, same vh5 pipeline | gate FAIL (multi-session 109 vs 114) | the stronger model over-abstains on answerable questions; gpt-5-mini is better calibrated for this judge |
| 19 | Blanket full-log routing on knowledge-update and multi-session | KU −3; multi-session −4 to −2 (across 4 variants) | full-log dilutes attention on non-temporal, non-preference categories; motivates selective routing |
Sources: rows 1 to 18 are the negative-results table of `benchmarks/results/longmemeval/RESULTS.md`; row 19 is the routing finding in the same file. The 14 falsified BEAM levers are enumerated in Table 6 and backed by the per-run files under `benchmarks/results/beam/`.

BIN
docs/paper/2026-07-09-clever-memory-loses.pdf Normal file
View File

Binary file not shown.

206
docs/paper/2026-07-09-tri-benchmark-outline.md Normal file
View File

@@ -0,0 +1,206 @@
# Tri-Benchmark Memory-Systems Paper — OUTLINE (for team-lead approval)
**Status:** outline only, no section prose. Awaiting KORRO human-in-the-loop sign-off before any section is written.
**Deliverables covered:** (1) 3 candidate titles, (2) thesis + worldview, (3) full section outline with per-section topic sentences + figure/table + evidence source, (4) figure/table plan, (5) claim-evidence map, (6) blog-post variant.
**Primary evidence base:** `benchmarks/results/MEMORY-BENCHMARKS-CONSOLIDATED-2026-07.md`; `docs/paper/2026-06-12-locomo-sota-arxiv-draft.tex`; `benchmarks/results/beam/{mem0-0641-pipeline-analysis,forensics-p2p-losing-abilities,forensics-retrieval-headroom,forensics-temporal}.md`; `benchmarks/results/locomo-sota-2026-06/`; LongMemEval `RESULTS.md` (hive-mind workspace).
---
## 1. Candidate Titles
Per `title.md`: write one of each type; the best is often the one first rejected. Each names the three benchmarks and/or the striking number, avoids "Novel/Towards/On the."
1. **(Provocation)** *"Clever Memory Loses: A Single Simple Substrate Is State of the Art on LoCoMo, LongMemEval, and BEAM"*
- Rationale: leads with the counter-intuitive claim (structure/distillation/graphs *lose*), then earns it with three named incumbents. The provocation is the entire thesis in five words.
2. **(Concept name)** *"Conflict-Aware Raw-Turn Memory: State-of-the-Art Long-Term Recall Without Distillation, Graphs, or Routing"*
- Rationale: installs the concept we want cited — *conflict-aware raw-turn memory* — and the subtitle names the three "clever" mechanisms it beats. The name is the linguistic currency (`impact.md` Principle 5, `ideation.md` Protocol 7).
3. **(Descriptive precision)** *"Simple Memory Beats Clever Memory: Three Long-Term-Memory SOTAs and 37 Falsified Interventions Under Each Incumbent's Own Protocol"*
- Rationale: the understated-confidence register (`title.md` Type 3) that carries the two stickiest numbers — *three SOTAs, 37 falsified levers* — and the fairness hook (*own protocol*) that pre-empts the "cross-lab noise" reviewer.
*(Working shortlist favorite: #1 as title, #2's concept name threaded through the paper as the named contribution. Decision deferred to team lead.)*
---
## 2. Thesis and Worldview (per `impact.md` + `ideation.md`)
**One-sentence thesis (no jargon):**
> Across all three major long-term-memory benchmarks, a deliberately dumb memory — the original dated conversation turns, retrieved per conversation, with contradictions *kept* rather than reconciled — beats every state-of-the-art system that distills, structures, graphs, or routes its memories, and we measured 37 "clever" additions, ~90% of which made it *worse*.
**Worldview destroyed:** "Better agent memory means *cleverer* memory — richer structure (knowledge graphs, entity reconciliation, atomic-fact distillation), learned routing, and test-time ensembling. The frontier is engineering more intelligence *into the memory path*."
**Worldview installed:** "The memory path should stay dumb. Distillation, reconciliation, and routing are mostly self-inflicted information loss; a substrate that preserves raw dated turns and *surfaces* conflicts instead of silently resolving them is the state of the art. This is the Bitter Lesson for agent memory."
**Memorability hooks (deploy ≥2, `impact.md` P5):**
- **The Number:** "37 interventions measured, ~90% falsified." Secondary: "+23pp on the benchmark's hardest ability (contradiction)."
- **The Analogy:** Sutton's *Bitter Lesson* — hand-engineered structure loses to simple methods that leverage the raw data/compute. Here: distilled facts / KGs lose to raw dated turns.
- **The Name:** *conflict-aware raw-turn memory* (contribution) and *the falsification ledger* (the negative-results artifact).
**Inevitability arc (`impact.md` P4) — section-to-phase mapping is enforced in §3 below.**
---
## 3. Full Section Outline (arXiv preprint)
> Convention below: each section lists 3-6 **topic sentences** (first-sentence-of-paragraph messages, per `introduction.md` checklist), the **figure/table** that lands there, and the **evidence source file/number**.
### 3.1 Abstract *(Template B: Challenge → Insight → Contribution, `abstract.md`)*
- **Topic 1 (task):** Long-term conversational memory — answering over weeks of prior dialogue — is the load-bearing capability for durable AI assistants, and three benchmarks (LoCoMo, LongMemEval, BEAM) are the field's rulers.
- **Topic 2 (challenge):** Every published leader adds structure to the memory path — atomic-fact distillation, temporal knowledge graphs, write-time entity reconciliation, learned routing, test-time ensembling.
- **Topic 3 (insight):** These are lossy. Distillation strips the dates and specifics the rubrics reward; write-time reconciliation *silently resolves* contradictions the rubrics want surfaced.
- **Topic 4 (contribution):** One simple substrate — per-conversation minds, verbatim dated raw-turn retrieval, and a conflict-*preserving* answer policy — is SOTA on all three under each incumbent's own published protocol (LoCoMo 86.49 vs 81.95; LongMemEval 95.01 macro vs 94.87; BEAM 0.6482 / 74.0% vs 0.6409 / 70.1%).
- **Topic 5 (benefit / negative results):** We measured 37 constructive interventions across two programs; ~90% lost to the simple substrate — negative results reported as a first-class contribution (the *falsification ledger*).
- **Topic 6 (experiment summary):** The single differentiator is conflict-awareness: +23pp on BEAM contradiction_resolution. All protocols, per-question artifacts, and the empty-answer-heal disclosure are released.
- **Figure/Table:** none (abstract). **Evidence:** consolidated lines 7-11, 66-89.
### 3.2 Introduction *(logic map from `introduction.md`; Version A4 "open with challenge")*
- **Topic 1:** A system that cannot recall what its user said three weeks ago cannot be a durable collaborator; LoCoMo, LongMemEval, and BEAM operationalize this and now anchor a public leaderboard race.
- **Topic 2 (SOTA + root issue):** The leaders differ in representation (Mem0 atomic facts, Zep temporal KG, Memori triples, Mastra, mem0-platform reconciled facts) but agree on one premise: cleverer structure on the memory path is the path forward.
- **Topic 3 (the crack in the worldview):** Yet each incumbent's own weakest ability traces to its own cleverness — mem0's write-time reconciler is its worst on contradiction (0.357); distillation is worst on fine-grained recall.
- **Topic 4 (our contribution, `impact.md` "make the reader the hero"):** We show a single dumb substrate is SOTA on all three, and — more useful to practitioners — we publish 37 measured interventions so the community stops paying the distillation/routing tax.
- **Topic 5 (contribution list):** (i) three same-protocol SOTAs; (ii) conflict-aware raw-turn memory as the named mechanism; (iii) the falsification ledger; (iv) an evaluation-protocol-fidelity methodology that exposed leaderboard-metric chaos.
- **Topic 6 (why now / hindsight, `ideation.md` Test B/C):** The enabling shift is long-context answerers (gpt-5) that can consume raw turns directly — making distillation not just lossy but unnecessary; in hindsight, "of course you shouldn't throw away the data before the model reads it."
- **Figure/Table:** **Table 1** (headline tri-benchmark summary). **Evidence:** consolidated lines 7-11; tex Introduction §.
### 3.3 Related Work
- **Topic 1 (memory systems on LoCoMo):** Mem0 (atomic facts), Zep/Graphiti (temporal KG), LangMem (profiles), Memori (timestamped triples, prior SOTA 81.95); MemR3 is an agentic multi-round retrieval pipeline, classed separately. *(reuse tex Related Work verbatim-adjacent.)*
- **Topic 2 (LongMemEval line):** Mastra (94.87 macro) and mem0-platform (93.4, GPT-5 judge) as incumbents; note the judge-model divergence up front.
- **Topic 3 (BEAM line):** mem0 memory-platform pipeline (2-turn-chunk write-time reconciliation, top-200 dated facts, chronological render) as the ICLR-2026 reference (0.6409 / 70.1%).
- **Topic 4 (the common commitment we break):** Every system above commits to a *dominant derived representation* and to *write-time resolution* of conflicts; none retains raw dated turns as the primary answer substrate or preserves both sides of a contradiction.
- **Topic 5 (where we differ):** We layer representations over a common frame store but let the answer path draw verbatim raw turns first; the benchmark categories are won by *different* representations, and a substrate that serves all simultaneously beats any single-representation system. *(tex Related Work "Where Hive Mind differs.")*
- **Figure/Table:** none. **Evidence:** tex Related Work; consolidated line 87; mem0-0641 pipeline analysis §"The pipeline, step by step."
### 3.4 The Substrate (Architecture) *(`impact.md` arc Phase 4 "resolution")*
- **Topic 1 (storage core):** Memory is stored as typed frames (Information / Preference / Belief) plus verbatim per-turn frames in per-conversation "minds"; retrieval is hybrid dense (nomic-embed) + FTS5 BM25 with RRF fusion and optional cross-encoder rerank — fully local.
- **Topic 2 (the primary lane is raw):** Unlike the incumbents, the answer context is built from *dated verbatim raw turns* retrieved per conversation, date-stamped `[YYYY-MM-DD]`; distilled layers are auxiliary, not the substrate.
- **Topic 3 (write-time dating, the one thing we DO transform):** Relative dates are resolved at write time ("yesterday" in an 8 May session → 7 May event) — the single distillation that survived, because it adds information rather than removing it.
- **Topic 4 (conflict-preserving policy — the differentiator):** The answer policy *retains both sides* of a conflict and instructs the model to state the contradiction and ask which is correct, rather than reconciling at write time; this is the mechanism behind the BEAM contradiction win.
- **Topic 5 (answer policy):** Conditional abstention (commit on factual spans when an anchor exists; abstain only when truly absent), granularity-calibrated dates, commit-to-one-option — tuned to what rubrics reward, not to prose quality.
- **Topic 6 (one substrate, three protocols):** The *same* substrate is evaluated under three different answerers/judges by swapping only the protocol harness — no benchmark-specific memory engineering.
- **Figure/Table:** **Figure 1** (substrate architecture: minds → I/P/B frames + raw dated turns → hybrid retrieval → assembler lanes → conflict-aware answer policy). **Evidence:** tex §"The Hive Mind substrate"; consolidated lines 59-64.
### 3.5 Evaluation Protocol Fidelity (the metric-audit methodology) *(`impact.md` arc Phase 2-3 "tension + diagnosis"; the honest-metrics thread)*
- **Topic 1 (why a whole section):** Cross-lab long-term-memory numbers are noise; the only defensible comparison is *same judge, same protocol, reproduced pipeline* — so we first reproduce each incumbent before comparing.
- **Topic 2 (LoCoMo fidelity):** We reproduced Memori's own pipeline to 81.98 vs their published 81.95 before any comparison, restoring image-caption input parity (protocol compliance, not architecture).
- **Topic 3 (LongMemEval metric chaos):** The leaderboard mixes metrics — Mastra's 94.87 is *macro* (category mean), mem0's 93.4 used a *GPT-5* judge not the official GPT-4o; we report both macro (95.01) and micro (93.60) under the official GPT-4o judge, the most conservative cell on the board.
- **Topic 4 (BEAM metric chaos):** mem0's shipped 0.641 result file is *gpt-5/gpt-5*, though the repo README defaults imply gpt-4o/gpt-4o — an answerer/judge confound we neutralize by matching gpt-5/gpt-5, and we report *both* pass-rate (headline) and avg-score (within noise).
- **Topic 5 (full disclosure principle):** We disclose the empty-answer-heal bug (56 rows, reasoning-budget cap), keep the pre-heal snapshot, and report judge-noise bands (SE ≈ 0.014 on BEAM avg) — the anti-cherry-pick posture.
- **Figure/Table:** **Table 6** (protocol-fidelity / metric-audit matrix: per benchmark — what the leaderboard reported, judge model, macro-vs-micro, README-vs-actual, our reported cell). **Evidence:** mem0-0641 pipeline analysis §"Headline finding"; consolidated lines 38-40, 91-96; forensics-p2p §2 (empty-answer bug).
### 3.6 Results (three benchmarks) *(`impact.md` arc Phase 4)*
- **Topic 1 (LoCoMo):** 86.49% (1332/1540) under Memori's protocol, +4.54pp (one-sample z=4.64, p<10⁻⁵), leading every category among memory systems and statistically at the full-context ceiling on open-domain; verified drift-free (fresh regen on HEAD reproduced 86.49 exactly).
- **Topic 2 (LoCoMo field correction):** We correct a column-scrambling error propagating through the literature (LangMem's cited "temporal 86.92" is its single-hop; true temporal 61.06) and re-run Mem0 same-judge (we lead every category, temporal +30.8pp).
- **Topic 3 (LongMemEval):** 95.01 macro / 93.60 micro (468/500), SOTA under the leader's own macro metric; the arc from 75.9 was won by four answer-side survivors, not by richer memory.
- **Topic 4 (BEAM headline):** 0.6482 avg / 74.0% pass (700 Q, gpt-5/gpt-5, top-30 raw turns) vs mem0 0.6409 / 70.1%; pass-rate +3.9pp (~2.3 SE, statistically real), avg +0.007 within noise (both reported).
- **Topic 5 (BEAM differentiator):** contradiction_resolution 0.588 (87.1% pass) vs mem0 0.357 (48.6%) — **+23pp on the benchmark's hardest ability** — plus abstention +6.8pp and information_extraction +4.0pp; the losses are concentrated in coverage abilities (summarization −7.4, temporal −6.1).
- **Topic 6 (efficiency framing):** BEAM SOTA at top-30 raw turns beats mem0's top-200 facts (4× depth buys them only +0.037) — compactness of *raw* beats depth of *distilled*; total program spend ≈ $160.
- **Figure/Table:** **Table 2** (corrected LoCoMo landscape), **Table 3** (LongMemEval intervention arc), **Table 4** (BEAM per-ability vs mem0). **Evidence:** consolidated §1-§3 (lines 17-89); tex Table `tab:field`; mem0-0641 §"Second finding" (top-200 vs top-50).
### 3.7 The Falsification Ledger (negative results) *(the first-class contribution; `ideation.md` Protocol 3 territory map)*
- **Topic 1 (framing):** We treat negative results as the product: 37 constructive interventions across two independent programs (23 LongMemEval + 14 BEAM), ~90% falsified against the simple substrate.
- **Topic 2 (compression/distillation loses):** Distilled-fact retrieval (k=100 → 0.354, k=200 → 0.382) and every raw+fact hybrid (0.498-0.540) lost — facts are net-harmful at *any* mix because they strip dates/numbers; the session-outline preamble diluted summarization 0.38→0.16.
- **Topic 3 (routing loses):** Gold-blind ability routing hit a 40% classifier ceiling (0.584 < 0.600) — abilities are not text-identifiable; coverage-shaped/stratified retrieval lost to similarity-dense clusters (summ 0.38→0.24).
- **Topic 4 (prompt engineering plateaus):** Strict-abstention (0.542), exhaustive-enumeration (−0.048; summ −0.119), and temporal-commit (+0.023, below re-roll variance) prompts all failed to beat the plain conflict-aware v2 prompt.
- **Topic 5 (ensembling loses):** Answer-merge self-ensemble (0.668/0.674) was within noise for 2× the cost; on LongMemEval all four *surviving* levers were answer-side (voting vh5, KG split-vote guard klc, preference critic pfc, routing-matrix completion h3u) — never memory-structure levers.
- **Topic 6 (the convergent conclusion):** Two programs, different benchmarks, same verdict — the local optimum is plain dated raw-turn retrieval + a conflict-aware prompt; cleverness on the memory path is measured self-harm.
- **Figure/Table:** **Table 5** (the falsification ledger — 37 rows: lever, program, score, why it lost, verdict), **Figure 2** (falsification scatter: intervention complexity vs Δscore, ~90% below the raw-turn baseline). **Evidence:** consolidated BEAM 14-lever table (lines 102-117), lines 44-45 + 124-126; LongMemEval `RESULTS.md`.
### 3.8 Analysis (why simple wins) *(`impact.md` arc Phase 5 "map the new territory")*
- **Topic 1 (why simple wins — the mechanism):** Retrieval headroom forensics show 88-93% of *failed* nuggets were already retrieved at rank ≤30 — the bottleneck is answer-side synthesis, not retrieval depth, so adding structure/depth cannot help and distillation only removes what the answerer needed.
- **Topic 2 (contradiction case study — the differentiator):** mem0 solves knowledge_update via write-time UPDATE/DELETE but that *same* reconciler makes contradiction its floor (0.357); our retain-both-sides substrate scores 0.588 — the trade is intrinsic to write-time resolution, not a tuning gap.
- **Topic 3 (ability-level tradeoffs — honest):** Where we lose (summarization −7.4, event_ordering −4.2), the forensic cause is *coverage/clause density*, not memory quality — mem0's enumerative fact dumps (1067 vs 600 words, 48 vs 29 bullets) mechanically harvest more compound-nugget clauses; largely an answer-verbosity artifact of the rubric.
- **Topic 4 (temporal is disposition, not knowledge):** The BEAM temporal gap (−6.1) is 94% duration arithmetic; 74% of the loss is over-abstention + wrong-anchor (single-date-selection), and a chunk is irreducible gpt-5-self-judge phrasing variance — not a memory-representation deficit.
- **Topic 5 (what this means for the field):** The load-bearing levers are (a) keep the raw data, (b) resolve dates at write time, (c) preserve conflicts — everything else the field is building is optional at best, harmful at worst.
- **Figure/Table:** **Figure 3** (contradiction mechanism: write-time reconcile collapses to one side vs retain-both-surface; paired with the 0.588 vs 0.357 bar), **Table 7-inline** (retrieval-headroom rank distribution, small). **Evidence:** forensics-retrieval-headroom (rank table); forensics-p2p §5-§8 (coverage/verbosity); forensics-temporal §2-§3; mem0-0641 §"Ingestion."
### 3.9 Limitations *(`ideation.md` Protocol 6 anti-paper pre-emption)*
- **Topic 1 (summarization/coverage gap):** We are genuinely behind on coverage-shaped abilities; whether the raw substrate can close it *without* the enumeration tax (which we showed backfires) is open.
- **Topic 2 (judge noise):** All scores are LLM-as-judge; we mitigate with fixed judges, archived per-question judgments, reproduced baselines, and reported noise bands — but absolute numbers are protocol-relative, and some BEAM deltas sit inside gpt-5-self-judge variance.
- **Topic 3 (single-substrate / single-answerer scope):** Each benchmark uses one answerer per its protocol; model-generality of the raw-turn design is untested, and LoCoMo token cost (3747 vs 1294) means there is no free ≤1500-token operating point — the raw lanes are load-bearing.
- **Topic 4 (token/cost honesty):** Raw-turn top-k is cheap at k=30 but scales super-linearly (k=100 ≈ 88K ctx tokens/question); distillation's appeal is cost, not quality — we win quality, they win the efficiency axis on LoCoMo.
- **Topic 5 (the empty-answer bug, disclosed):** The healed 56-row reasoning-budget bug is disclosed with a pre-heal snapshot; the residual post-heal gaps are the honest comparison.
- **Figure/Table:** none. **Evidence:** tex Limitations §; forensics-retrieval-headroom (k-cost table); forensics-p2p §2; consolidated lines 91-96.
### 3.10 Conclusion *(`impact.md` P8 call-to-action; `ideation.md` Protocol 3)*
- **Topic 1 (restate the one thing):** One dumb substrate — dated raw turns, per-conversation, conflicts preserved — is state of the art on all three long-term-memory benchmarks.
- **Topic 2 (the bitter lesson):** 37 measured interventions say the field is over-engineering the memory path; the wins are in *not* discarding data and *not* resolving conflicts early.
- **Topic 3 (call to action):** All protocols, per-question artifacts, the falsification ledger, and the harness are open; the community can stop re-deriving the distillation/routing tax and audit its own leaderboard metrics.
- **Topic 4 (open territory):** (i) Can conflict-preservation be made cheap enough to also win coverage abilities? (ii) Is there a write-time transform *other than* date resolution that adds rather than removes? (iii) Does the raw-turn result hold under weaker/cheaper answerers, or is long-context the true enabler? (iv) What is the right rubric for "surfaced a contradiction correctly" vs "committed and got lucky"?
- **Figure/Table:** none. **Evidence:** consolidated line 124-126; tex Conclusion §.
---
## 4. Figure / Table Plan (~6 tables, ~3 figures)
| # | Artifact | Lands in | Exact data source |
|---|---|---|---|
| **Table 1** | Headline tri-benchmark summary (ours / best competitor / protocol / verdict) | Intro | consolidated lines 7-11 |
| **Table 2** | Corrected LoCoMo landscape (per-category memory systems + MemR3 + full-context ceiling) | Results §3.6 | tex `tab:field` (lines 389-403) |
| **Table 3** | LongMemEval intervention arc (75.9 → 95.01 macro / 93.60 micro, lever labels vh5/klc/pfc/h3u) | Results §3.6 | consolidated lines 41-43; `RESULTS.md` |
| **Table 4** | BEAM per-ability vs mem0 (10 abilities + overall avg + pass rate) | Results §3.6 | consolidated lines 68-81 |
| **Table 5** | **The Falsification Ledger** (37 levers: lever / program / score / why-lost / verdict) | §3.7 | consolidated BEAM table lines 102-117 (14) + `RESULTS.md` (23) |
| **Table 6** | Protocol-fidelity / metric-audit matrix (leaderboard-claim / judge model / macro-vs-micro / README-vs-actual / our cell) | §3.5 | mem0-0641 §"Headline finding"; consolidated lines 38-40 |
| **Figure 1** | Substrate architecture (minds → I/P/B + raw dated turns → hybrid retrieval → assembler lanes → conflict-aware policy) | §3.4 | tex §"The Hive Mind substrate"; consolidated 59-64 |
| **Figure 2** | Falsification scatter (intervention complexity vs Δscore; ~90% below the raw-turn baseline line) | §3.7 | consolidated 102-117; `RESULTS.md` |
| **Figure 3** | Contradiction mechanism diagram + bar (write-time reconcile → one side vs retain-both-surface; 0.588 vs 0.357) | §3.8 | mem0-0641 §"Ingestion"; consolidated line 70, 86-89 |
| *(inline mini-table)* | Retrieval-headroom rank distribution (≤30 answer-side 88-93%) | §3.8 | forensics-retrieval-headroom rank table (lines 40-44) |
---
## 5. Claim-Evidence Map (every headline claim → file/number)
| # | Headline claim | Number | Evidence source |
|---|---|---|---|
| C1 | LoCoMo SOTA | 86.49% (1332/1540), +4.54pp, z=4.64, p<10⁻⁵ | consolidated line 9; tex abstract + `tab:field` |
| C2 | LoCoMo reproduced baseline before comparing | 81.98 vs published 81.95 | tex §Experimental setup / Introduction contrib 1 |
| C3 | LoCoMo drift-free | fresh HEAD regen reproduced 86.49 exactly | consolidated line 9 |
| C4 | LoCoMo field-table correction | LangMem "temporal 86.92" = single-hop; true 61.06 | tex §"corrected LoCoMo field" |
| C5 | Mem0 re-run same-judge | lead every category, temporal +30.8pp | consolidated line ~; tex `tab:field` footnote (+32.7pp) |
| C6 | LongMemEval SOTA (macro) | 95.01 macro vs Mastra 94.87 | consolidated line 10; lines 38-43 |
| C7 | LongMemEval micro reported too | 93.60 micro (468/500), GPT-4o judge | consolidated line 10, 43 |
| C8 | LongMemEval levers | 23 measured, 4 adopted (vh5,klc,pfc,h3u), 19 falsified | consolidated lines 44-45; `RESULTS.md` |
| C9 | BEAM SOTA (pass rate) | 74.0% (518/700) vs 70.1% (491/700), +3.9pp ≈2.3 SE | consolidated lines 11, 81-84 |
| C10 | BEAM avg reported honestly | 0.6482 vs 0.6409, +0.007 within noise SE≈0.014 | consolidated lines 80, 83-84 |
| C11 | **Contradiction differentiator** | contradiction_resolution 0.588 vs 0.357 = **+23pp** | consolidated line 70, 86-89 |
| C12 | Why: mem0 reconciler is its own floor | write-time UPDATE/DELETE silently resolves; their weakest 0.357 | mem0-0641 §Ingestion (lines 43-49, 78) |
| C13 | Depth is not the lever | mem0 top-200 vs top-50: 4× depth = +0.037 | mem0-0641 §"Second finding" (lines 28-37) |
| C14 | Simple wins is answer-side | 88/90/93% of failed nuggets already at rank ≤30 | forensics-retrieval-headroom (lines 40-63) |
| C15 | Our losses are coverage/verbosity | mem0 answers 1067 vs 600 words, 48 vs 29 bullets | forensics-p2p §5 (lines 82-101, 263) |
| C16 | Temporal gap = disposition not knowledge | 94% duration arithmetic; 74% over-abstention + wrong-anchor | forensics-temporal §2-§3 (lines 72-118) |
| C17 | 37 levers, ~90% falsified | 14 BEAM (table) + 23 LME | consolidated 102-117; 44-45; 124-126 |
| C18 | Distillation is lossy | distilled k=100 → 0.354 vs 0.448 raw | consolidated line 104 |
| C19 | Empty-answer bug disclosed | 56 rows healed; pre-heal snapshot kept | consolidated 91-96; forensics-p2p §2 |
| C20 | Metric audit: hidden judge swap | mem0 BEAM 0.641 file = gpt-5/gpt-5 (README implies gpt-4o) | mem0-0641 §"Headline finding" (lines 5-24) |
| C21 | Cost | total BEAM program ≈ $160; LoCoMo 3747 vs 1294 tok/q | consolidated line 13; tex Limitations |
**Open dependency to resolve before writing:** C8 (LongMemEval 19 falsified levers) and Table 3/5 LME rows are sourced from `RESULTS.md` in the hive-mind workspace, not the waggle-os filesystem — pull that file into `benchmarks/results/longmemeval/` for citable line numbers before drafting §3.6-§3.7.
---
## 6. Blog-Post Variant Outline (800-1500 words)
**One insight:** "Simple memory beats clever memory — we measured 37 ways."
**Title candidates:** *"We Tried 37 Ways to Make AI Memory Smarter. 33 Made It Worse."* / *"The Bitter Lesson of Agent Memory."*
**Register:** `title.md` blog rule — click-worthy, not clickbait; lead with the number and the counter-intuitive result.
- **Hook (≈120 w):** The industry is racing to build cleverer memory for AI agents — knowledge graphs, fact distillation, entity reconciliation. We benchmarked a deliberately dumb alternative and it won on all three major benchmarks. Then we tried 37 ways to make it smarter; almost all made it worse.
- **The setup (≈150 w):** Three benchmarks (LoCoMo, LongMemEval, BEAM), each with a published SOTA. The honest-comparison rule: same judge, same protocol, reproduce the incumbent first. One substrate, no per-benchmark memory engineering.
- **The result in one table (≈120 w):** 86.49 vs 81.95 / 95.01 vs 94.87 / 74.0% vs 70.1% — with the caveat we're transparent about (avg-score is within noise on BEAM; pass-rate is the real win).
- **The counter-intuitive core (≈250 w):** The falsification ledger. Distillation strips the dates and specifics the questions ask for. Routing hits a classifier ceiling. Ensembling costs 2× for nothing. The one transform that helped — resolving "yesterday" to a real date at write time — *adds* information instead of removing it. Bitter Lesson analogy.
- **The differentiator (≈250 w):** Conflict-aware memory. Competitors reconcile contradictions when they *write* memory — which silently deletes one side. We keep both and surface them. Result: +23pp on the hardest ability. The vivid example: "you said 10 vector-field problems here, and 15 there — which is right?" vs a system that already threw one number away.
- **The honesty section (≈200 w):** Where we lose (coverage/summarization), why (answer verbosity gaming compound rubrics, not memory quality), and the metric mess we found (macro vs micro, GPT-5 vs GPT-4o judges, README-vs-actual protocols) — plus the empty-answer bug we disclosed.
- **The takeaway / CTA (≈150 w):** Stop over-engineering the memory path. Keep the raw data, date it at write time, preserve conflicts. Everything is open — protocols, per-question records, the ledger. Link to the arXiv preprint.
- **Sticky elements:** the number (37 / ~90%), the analogy (Bitter Lesson), the contradiction example (10 vs 15 problems), one figure (the falsification scatter, Figure 2 reused).
---
## 7. Pre-Writing Gate Check (`ideation.md` decision gate — all filled)
1. **One-sentence insight:** raw dated turns + preserve conflicts beats distill/graph/route — measured on 3 benchmarks, 37 falsified levers.
2. **Why nobody before:** everyone assumed richer structure = better memory; long-context answerers only recently made raw-turn feeding viable (Why-Now).
3. **Hindsight verdict:** "of course you shouldn't delete the data before the model reads it, and of course you shouldn't resolve a contradiction the user asked about."
4. **Abstraction (verb form, domain-free):** *preserve the raw signal and the conflicts; transform only when the transform adds information.*
5. **The name:** *conflict-aware raw-turn memory* / *the falsification ledger*.
**Anti-paper pre-emptions staged (Protocol 6):** cross-lab noise → same-protocol reproduction (§3.5); "avg-score is within noise" → we lead on pass-rate and say so (§3.6, C10); "you lose on summarization" → shown to be rubric-verbosity artifact, stated as limitation (§3.8/§3.9); "judge is unreliable" → fixed judge + noise bands + disclosure (§3.5/§3.9).

59
docs/paper/dossier/README.md Normal file
View File

@@ -0,0 +1,59 @@
# Dossier: Clever Memory Loses
The complete recognition package for the paper **"Clever Memory Loses: A Single Simple Substrate Is State of the Art on LoCoMo, LongMemEval, and BEAM"** (Marko Marković, KORRO / hive-mind). This is the arsenal you deploy when someone says "tell me more about what you do": paper, blog, deck, posts, and press kit, ready to send in under five minutes.
The one insight, repackaged for every format: **keep the memory store dumb and spend your cleverness on reads.**
## The three numbers
| Benchmark | Best published incumbent | This work |
|---|---|---|
| LoCoMo (1,540 Q) | Memori 81.95% | 86.49% |
| LongMemEval-S (500 Q) | Mastra 94.87 macro | 95.01 macro / 93.60 micro |
| BEAM 1M (700 Q) | mem0 0.6409 avg / 70.1% pass | 0.6482 avg / 74.0% pass |
Single differentiator: conflict-awareness, worth 23 points on BEAM's contradiction ability. Falsification ledger: 37 interventions measured, 33 lost.
## Contents
| File | What it is |
|---|---|
| `README.md` | This index and the 5-minute send checklist. |
| `arxiv-metadata.md` | Submission-ready arXiv fields: title, author, verbatim abstract, categories (cs.CL primary; cs.AI and cs.IR cross-lists), comments field, license, and a full submission checklist. |
| `talk-deck-outline.md` | Slide-by-slide outline for a 15-minute conference talk (14 slides plus backups): one message, visual spec, and timed speaker notes per slide. |
| `announcement-posts.md` | Launch-day copy: LinkedIn post, an 8-tweet X thread, and a Show HN title with comment. No em dashes, no hype words. |
| `press-kit.md` | One-page press kit: what happened, the three numbers, an author quote, the falsification-ledger press hook, figure list, contact, and a five-question FAQ. |
### Companion assets (one level up, already shipped)
| Asset | Path |
|---|---|
| Research paper (markdown master) | `../2026-07-09-clever-memory-loses-draft.md` |
| Blog post (accessible version) | `../2026-07-09-clever-memory-loses-blog.md` |
| Figure 1, the substrate pipeline | `../figures/fig1-substrate.png` |
| Figure 2, the falsification strip plot | `../figures/fig2-falsification.png` |
| Consolidated results and reproduction commands | `../../../benchmarks/results/MEMORY-BENCHMARKS-CONSOLIDATED-2026-07.md` |
| Code, protocols, per-question artifacts | `github.com/marolinik/waggle-os` |
## What to send, by situation
| Situation | What to send |
|---|---|
| Speaking invitation request | Press kit one-pager + talk deck outline + link to the blog |
| Press or editor inquiry | Press kit + Figure 1 and Figure 2 + blog |
| Research job or collaboration | Paper + press kit + talk deck outline |
| Investor or partnership intro | Press kit one-pager + paper + the three numbers |
| Launch day (arXiv is live) | Announcement posts (LinkedIn, X, Show HN), all at once |
| "Tell me more about what you do" | Press kit one-pager + link to this dossier |
## The 5-minute send checklist
Before you send any subset of this dossier, confirm:
- [ ] **Pick the target.** Use the table above to select the minimum set of files for the situation; do not send everything by default.
- [ ] **Fill the placeholders.** Replace every `[LINK]` in `announcement-posts.md` with the live arXiv or blog URL, and the `[contact placeholder]` in `press-kit.md` with a real email.
- [ ] **Confirm the artifacts are live.** The repository at `github.com/marolinik/waggle-os` is public and contains the harness, per-question results, negative-results tables, and the pre-heal snapshot.
- [ ] **Attach the figures if the recipient is press.** `fig1-substrate.png` and `fig2-falsification.png` are legible at 100 percent and in grayscale.
- [ ] **Verify the numbers match the paper.** LoCoMo 86.49 vs 81.95; LongMemEval 95.01 macro vs 94.87; BEAM 74.0% pass vs 70.1%; contradiction +23. Nothing rounded differently across files.
- [ ] **Lead with the one-pager.** The press kit is read in 60 seconds and must stand alone; everything else is depth behind it.
- [ ] **Zip and send.** The dossier folder plus the two companion figures should be one archive, ready to attach in under five minutes.

95
docs/paper/dossier/announcement-posts.md Normal file
View File

@@ -0,0 +1,95 @@
# Announcement Posts
Three ready-to-send posts. Replace every `[LINK]` with the arXiv URL (or the blog URL) once the paper is announced. No em dashes anywhere. No hype words; the numbers carry it. Every figure quoted here matches the paper.
---
## (a) LinkedIn
Professional, roughly 200 words. Hook, then three numbers, then the link.
---
Everyone building AI memory is making it cleverer: knowledge graphs, extracted facts, systems that reconcile what you said last week against what you say today. We spent months building that cleverness, measured 37 versions of it against three public benchmarks, and watched 33 of them make the memory worse.
The design that won keeps the raw conversation, stamps every line with its date, and when two lines disagree, shows both. We call it conflict-aware raw-turn memory. The entire point is that it refuses to be clever.
Under each incumbent's own published protocol and judge, one substrate is state of the art on all three benchmarks:
- LoCoMo: 86.49% against the prior best 81.95%
- LongMemEval: 95.01 macro against 94.87
- BEAM 1M: 74.0% pass against 70.1%
The single decision that separates us from the field is conflict-awareness: keep both sides of a contradiction instead of resolving it at write time. A reconciler that runs at write time has to pick one value and delete the other before the question is ever asked. On contradiction resolution, the hardest ability on BEAM, that one choice is worth 23 points.
We released everything, including the 33 experiments that failed and a bug we could have hidden. The Bitter Lesson came for agent memory too.
Paper, code, and every per-question judgment: [LINK]
---
## (b) X / Twitter thread
Six to eight tweets. Tweet 1 stands alone and is numbers-forward; the honest-disclosure angle is the differentiator.
**1/**
We tried 37 ways to make AI memory smarter. 33 made it worse.
The design that won keeps the raw conversation, dates every line, and when two lines disagree keeps both. It is state of the art on LoCoMo, LongMemEval, and BEAM. We published all 33 failures.
**2/**
Everyone in agent memory builds the same thing: distill conversations into facts, build a knowledge graph, reconcile new facts against old ones at write time.
The shared belief is that a cleaner, smaller, reconciled store is a better store.
**3/**
We built the opposite. Keep the original turns, exactly as said, each stamped with its date. Retrieve the relevant ones per question. When two contradict, keep both and surface the conflict.
No graph. No fact extraction as the store. No reconciliation.
**4/**
One substrate, three benchmarks, each under the incumbent's own protocol and judge:
LoCoMo 86.49% vs 81.95%
LongMemEval 95.01 macro vs 94.87
BEAM 1M 74.0% pass vs 70.1%
Zero per-benchmark tuning.
**5/**
The one decision that separates us: keep both sides of a contradiction instead of resolving it at write time.
A reconciler that runs at write time must pick one value. A read-time policy can hold both. On BEAM's hardest ability that is worth 23 points.
**6/**
The 37 experiments are the real product. Every intervention that transformed the store lost: distill it, graph it, reconcile it, restructure it, all worse.
The only 4 that helped never touched the store. They just read it more carefully.
**7/**
Honest disclosure: our first BEAM run scored lower because a token cap left 56 answers blank.
We could have quietly raised the cap. Instead we re-ran under the identical config, healed the blanks, and published the broken snapshot as an artifact.
**8/**
Keep the store dumb, spend your cleverness on reads.
We released the three protocols, every per-question judgment, the full ledger of 37 interventions, and the harness. The Bitter Lesson came for agent memory too.
[LINK]
---
## (c) Hacker News (Show HN)
**Title:**
```
Show HN: Conflict-aware raw-turn memory, SOTA on LoCoMo, LongMemEval and BEAM
```
**Comment (post immediately after submitting):**
I spent months building clever agent memory (fact distillation, knowledge graphs, write-time reconciliation) and then measured 37 versions of it against three public long-term-memory benchmarks. 33 of the 37 made the memory worse. The design that won is almost embarrassingly simple: keep the raw conversation turns, stamp each with its date, retrieve the relevant ones per question, and when two turns contradict each other, keep both and surface the conflict instead of resolving it. Under each incumbent's own published protocol and judge, this one substrate is state of the art on all three: LoCoMo 86.49% (prior best 81.95%), LongMemEval 95.01 macro (prior best 94.87), and BEAM 1M 74.0% pass (prior best 70.1%). The single differentiator is conflict-awareness: a reconciler that resolves conflicts at write time has to delete one side before the question is ever asked, which is exactly what the contradiction rubric penalizes. That one decision is worth 23 points on BEAM's hardest ability.
The part I most want feedback on is the falsification ledger. I am publishing all 37 interventions, including the 33 that lost, because the negative results are the actual finding: every scheme that transformed the store (distillation, graphs, ledgers, routers, self-ensembles) lost, and the only four that helped left the store untouched and just read it more carefully. I am also disclosing a bug: my first BEAM run scored lower because a completion-token cap left 56 answers blank, and rather than quietly raise the cap I re-ran under the identical config and published the broken snapshot alongside the fixed one. All protocols, per-question judgments, and the harness are in the repo. I am happy to go deep on the protocol-fidelity methodology, since cross-lab memory numbers are mostly noise until you reproduce the incumbent under the same judge. [LINK]

76
docs/paper/dossier/arxiv-metadata.md Normal file
View File

@@ -0,0 +1,76 @@
# arXiv Submission Metadata
Submission-ready metadata for the paper. Copy each field into the arXiv submission form. Every number is drawn verbatim from the paper master (`../2026-07-09-clever-memory-loses-draft.md`); do not paraphrase the abstract.
---
## Title
```
Clever Memory Loses: A Single Simple Substrate Is State of the Art on LoCoMo, LongMemEval, and BEAM
```
## Authors
```
Marko Marković
```
Affiliation line (paper byline): KORRO / hive-mind.
## Abstract
Paste verbatim. This is the paper's Section 1 abstract, unaltered.
```
Long-term conversational memory, answering questions over weeks of prior dialogue, is the load-bearing capability for durable AI assistants, and three benchmarks (LoCoMo, LongMemEval, and BEAM) are the field's rulers. Every published leader adds structure to the memory path: atomic-fact distillation, temporal knowledge graphs, write-time entity reconciliation, and learned routing. Our insight is that these transforms are lossy in exactly the way the benchmarks penalize, because distillation strips the dates and specifics the rubrics score and write-time reconciliation silently resolves the contradictions the rubrics want surfaced. We show that one simple substrate, conflict-aware raw-turn memory (per-conversation minds, verbatim dated raw-turn retrieval, and a conflict-preserving answer policy), is state of the art on all three under each incumbent's own published protocol: LoCoMo 86.49 against 81.95, LongMemEval 95.01 macro against 94.87, and BEAM 0.6482 average and 74.0 percent pass against 0.6409 and 70.1 percent. We then measured 37 constructive interventions that tried to make the substrate cleverer; 33 lost, and we report them as a first-class falsification ledger. The single differentiator is conflict-awareness, worth 23 points on BEAM's contradiction ability. Because the leaderboards mix macro and micro metrics and gpt-5 and gpt-4o judges, we reproduce each incumbent's pipeline before comparing and report against the most conservative protocol on the board. All protocols, per-question artifacts, and the empty-answer-heal disclosure are released.
```
## Categories
| Field | Value | Rationale |
|---|---|---|
| Primary | `cs.CL` | Long-term conversational memory over natural-language dialogue; LLM answerers and LLM-as-judge scoring. |
| Cross-list | `cs.AI` | Agent memory architecture; the Bitter Lesson framing for autonomous agents. |
| Cross-list | `cs.IR` | Hybrid dense plus BM25 retrieval, reciprocal rank fusion, reranking, retrieval-headroom analysis. |
## Comments field
Paste into the "Comments" box (confirm the page count against the compiled PDF before submitting; the figure and table counts are fixed).
```
14 pages, 2 figures, 6 tables, 2 appendices. Code, protocols, per-question answers and judgments, the falsification ledger, and the empty-answer pre-heal snapshot: https://github.com/marolinik/waggle-os
```
## License
Suggested: **arXiv.org perpetual, non-exclusive license** (the default arXiv license). It permits arXiv to distribute the work while the author retains copyright, which is the correct choice for a preprint intended to be widely read and cited. Do not select a CC-BY or CC0 variant unless a target venue later requires it.
## MSC / ACM classification (optional fields)
- ACM class (optional): `I.2.7` (Natural Language Processing), `H.3.3` (Information Search and Retrieval).
---
## Submission checklist
### Endorsement
- [ ] Confirm endorsement status for `cs.CL`. A first-time submitter without an institutional arXiv history typically needs an endorsement from an established author in the category. If unendorsed, request endorsement from a coauthor or a cited author before uploading, and allow lead time.
- [ ] Verify the submitting email is on an endorsed or auto-endorsed domain if applicable.
### PDF and source requirements
- [ ] Preferred submission is LaTeX source (the markdown master converts to LaTeX; upload `.tex` plus the two `.png` figures, not a pre-built PDF, so arXiv can compile). A direct PDF upload is accepted but discouraged for text-heavy papers.
- [ ] Both figures embedded at print resolution: `figures/fig1-substrate.png` and `figures/fig2-falsification.png`. Confirm they are legible at 100 percent zoom and in grayscale.
- [ ] All six tables render inside the text column with no overflow (Tables 1 through 6).
- [ ] References compile cleanly; every arXiv identifier in the reference list resolves (2504.19413, 2501.13956, 2402.17753, 2410.10813, 2510.27246, 2512.12818, 2512.20237, 2310.08560).
- [ ] Title, author, and abstract in the form match the compiled PDF exactly.
- [ ] Page count in the Comments field matches the final PDF.
### Ancillary files
- [ ] Optional: attach the consolidated command list (`benchmarks/results/MEMORY-BENCHMARKS-CONSOLIDATED-2026-07.md`) as an ancillary file, or link it from the repository.
- [ ] Confirm the repository at `github.com/marolinik/waggle-os` is public and contains the harness, per-question result files, negative-results tables, and the pre-heal snapshot referenced in Appendix B, so the artifacts URL in the Comments field is live at submission time.
- [ ] No private data or API keys in any uploaded source; all three datasets are public and used as published.
### Post-submission
- [ ] After the paper is announced, add the arXiv ID to the blog post, the LinkedIn post, and the X thread (replace the `[LINK]` placeholders in `announcement-posts.md`).
- [ ] Submit the blog post to Hacker News and the paper to r/MachineLearning on the announcement day.

63
docs/paper/dossier/press-kit.md Normal file
View File

@@ -0,0 +1,63 @@
# Press Kit: Clever Memory Loses
One page for journalists, editors, and program chairs. Everything here is drawn from the paper and is quotable. No em dashes.
---
## What happened (3 sentences)
A single, deliberately simple memory substrate is state of the art on all three of the field's long-term conversational memory benchmarks, LoCoMo, LongMemEval, and BEAM, each measured under the incumbent leader's own published protocol and judge. Instead of distilling conversations into facts or knowledge graphs, it keeps the raw dated conversation turns and, when two statements contradict each other, retains both rather than resolving the conflict at write time. Alongside the three results, the author publishes a falsification ledger of 37 interventions that tried to make the substrate cleverer, of which 33 made it worse.
## The three numbers
| Benchmark | Best published incumbent | This work | Margin |
|---|---|---|---|
| LoCoMo (1,540 Q) | Memori 81.95% | 86.49% | +4.54pp (z=4.64, p<0.00001) |
| LongMemEval-S (500 Q) | Mastra 94.87 macro | 95.01 macro / 93.60 micro | +0.14 macro |
| BEAM 1M (700 Q) | mem0 0.6409 avg / 70.1% pass | 0.6482 avg / 74.0% pass | +3.9pp pass (McNemar z=2.14, p≈0.03) |
Single differentiator: conflict-awareness is worth 23 points on BEAM's contradiction ability (0.588 against 0.357). Total BEAM program cost: about 160 dollars.
## Quote from the author
Marko Marković (KORRO / hive-mind):
> "We paid the distillation tax 33 times so nobody else has to: keep the store dumb, keep both sides of a contradiction, and spend your cleverness on reading it, not rebuilding it."
## The press hook: they published everything that failed
Memory-systems papers typically report only the surviving configuration. This one ships the opposite: a first-class negative-results artifact of 37 interventions across two independent programs, 33 of which lost to the simple baseline, grouped by family so a reader can see that every scheme which transformed the store (distillation, graphs, ledgers, routers, self-ensembles) lost, and the only four that helped never touched the store. The author also volunteers a bug disclosure most papers would bury: a first BEAM run scored lower because a completion-token cap left 56 answers blank, and rather than quietly raise the cap, the author re-ran under the identical configuration and preserved the broken snapshot as a published artifact. The story is not only that simple beat clever; it is that the author documented every way clever lost.
## Figure files
- `figures/fig1-substrate.png` : the conflict-aware raw-turn substrate, left to right (write time: dated raw-turn minds with relative-date resolution; read time: hybrid top-k retrieval; answer policy: surface both sides on contradiction). Best for explainer and architecture context.
- `figures/fig2-falsification.png` : the falsification strip plot, 33 falsified levers on or below the raw-turn baseline and only 4 adopted answer-side levers above it. Best for the "complexity buys nothing" narrative.
(Both files live one level up at `../figures/` relative to this dossier.)
## Contact
- Author: Marko Marković, KORRO / hive-mind
- Email: [contact placeholder]
- Code, protocols, per-question artifacts, and the pre-heal snapshot: `github.com/marolinik/waggle-os`
- Paper: `../2026-07-09-clever-memory-loses-draft.md` (arXiv link to be added on announcement)
- Accessible write-up: `../2026-07-09-clever-memory-loses-blog.md`
---
## FAQ
**Is this cherry-picked?**
No, and the paper is built to make cherry-picking hard to hide. Each incumbent's own pipeline is reproduced under the same judge before any comparison (Memori's LoCoMo number was reproduced to 81.98 against their published 81.95). The losses are reported as loudly as the wins: summarization trails the incumbent at 0.570 against 0.635, and the BEAM average-score margin is stated as a statistical tie inside the judge-noise band. Every per-question judgment is released, along with a ledger of the 33 interventions that failed, and where a result was inflated by a bug the bug and both numbers are disclosed.
**Why should users care?**
An assistant that cannot recall what you said three weeks ago cannot be a durable collaborator. Today's memory systems quietly resolve your contradictions for you: tell one that your follower count changed and one number silently disappears from its store. This substrate keeps both dated statements and, at answer time, presents both and asks which is correct, which is what people actually want from a memory that is supposed to have been paying attention.
**What is conflict-aware memory?**
It keeps the original dated conversation turns instead of distilling them into facts or a graph, and when two statements disagree it retains both rather than reconciling them the moment they are written. At answer time it states the contradiction, presents both dated statements, and asks which is correct. A reconciler that runs at write time must pick one side to store; a policy that runs at read time can hold both and decide per question. That single decision is worth 23 points on BEAM's hardest ability.
**What did it cost?**
The entire BEAM program cost roughly 160 dollars in API spend. The memory engine runs local embeddings (nomic-embed-text via Ollama), so no memory content leaves the machine on the write path or the read path, and it retrieves 7 times fewer items than the incumbent (30 raw turns against 200 facts). The one axis it concedes is inference token cost: a raw-turn answer context is larger than a distilled-fact one (about 27,000 tokens against about 7,000 on BEAM), so the win is on answer quality and the concession is on efficiency.
**What is next?**
Three open questions. Can conflict preservation be made cheap enough to also win the coverage-shaped abilities such as summarization, where the substrate still trails. Does the raw-turn result hold under weaker and cheaper answerers, or is long-context capacity the true enabler. And what is the right rubric for a memory that surfaces a contradiction and asks for clarification, as against one that commits to a single value and happens to be right.

161
docs/paper/dossier/talk-deck-outline.md Normal file
View File

@@ -0,0 +1,161 @@
# Talk Deck Outline: 15-Minute Conference Talk
**Title:** Clever Memory Loses
**Subtitle:** A single simple substrate is state of the art on LoCoMo, LongMemEval, and BEAM
**Speaker:** Marko Marković (KORRO / hive-mind)
**Length:** 15 minutes, 14 slides.
Story arc (one narrative spine, no detours): cold open (the follower-count contradiction) to the wall (everyone builds cleverer memory) to the mechanism where clever becomes the weakness to the one dumb idea to fair measurement to the 37 levers to the reveal that 33 lost to the three receipts to the bitter-lesson close.
Design rules (from the presentation format guide): one message per slide, six lines of text maximum, 24pt minimum font, figures legible from the back, no em dashes, consistent color scheme with the paper. Total speaking budget below sums to roughly 15 minutes; the timings are a pacing guide, not a script.
---
## Slide 1: Title (0:10)
**Title:** Clever Memory Loses
**The one message:** The dumbest design on the board wins all three benchmarks.
**Visual:** Title, subtitle, author, venue. A faint background of dated raw turns (`[2023-03-14] ...`, `[2023-06-02] ...`) to preview the substrate.
**Speaker notes:**
- Say the title, then the thesis in one line: "We made memory dumber and it got better."
- Name the three benchmarks so the audience knows the scope up front.
## Slide 2: Cold open, the follower count (1:20)
**Title:** Your assistant deleted the evidence
**The one message:** Clever memory resolves a contradiction before you ever ask about it, and the number quietly disappears.
**Visual (build a new one):** A three-panel timeline. March: user says "10 followers." June: user says "15 followers." A write-time reconciler fires UPDATE and 10 vanishes from the store. Then a question mark: "Later: how did the count change?" with an empty store.
**Speaker notes:**
- Tell it as a story, no jargon: you say 10 in March, 15 in June, then someone asks about the change.
- A system that reconciles at write time already fired an UPDATE. 10 is gone. The contradiction the benchmark tests no longer exists.
- Land the line: "The clever system deleted the evidence." Pause. This is the whole talk in one anecdote.
## Slide 3: The wall (1:00)
**Title:** Everyone is building cleverer memory
**The one message:** The field agrees on one premise: a cleaner, smaller, reconciled store is a better store.
**Visual:** Four labeled boxes: Mem0 (atomic facts), Zep (temporal knowledge graph), Memori (dated triples), mem0 platform (write-time reconciliation). One arrow from each into a shrinking "derived store."
**Speaker notes:**
- Every leader spends its engineering budget on the same two moves: transform the conversation into a smaller derived representation, and resolve contradictions before the answerer runs.
- The shared, rarely-stated belief: better memory means cleverer memory.
- Set up the turn: what if the cleverness is the problem?
## Slide 4: The mechanism, clever becomes the weakness (1:30)
**Title:** The cleverness and the weakness are the same mechanism
**The one message:** The write-time reconciler that scores well on knowledge update is exactly what makes contradiction resolution the incumbent's worst ability.
**Visual:** A two-column contrast. Left: "Write-time reconcile: knowledge update 0.650 (strong)." Right: "Same reconciler: contradiction resolution 0.357 (worst)." An arrow labeled "same mechanism" connecting them.
**Speaker notes:**
- A write-time reconciler must pick one side to store. That helps knowledge update and destroys contradiction resolution.
- The weakness is not independent of the cleverness. It is a direct cost of it.
- This is the crack we drove a whole paper through.
## Slide 5: The one dumb idea (1:30)
**Title:** Conflict-aware raw-turn memory
**The one message:** Keep the raw dated turns, retrieve them verbatim, and when two disagree keep both.
**Visual:** Figure 1 (`figures/fig1-substrate.png`), the substrate pipeline: conversation to per-conversation dated raw-turn mind to top-30 retrieval to conflict-aware answer policy to answer.
**Speaker notes:**
- Three moves: per-conversation minds, dated verbatim raw turns as the primary lane, contradictions retained not reconciled.
- One transform at write time, resolving relative dates ("yesterday" becomes a real date), because that adds information rather than removing it.
- No graph, no fact distillation as the store, no write-time reconciliation, no learned router. The point is that it refuses to be clever.
## Slide 6: Measuring fairly (1:00)
**Title:** Cross-lab memory numbers are noise
**The one message:** We reproduce each incumbent's own pipeline under the same judge before we compare, then swap in our store and change nothing else.
**Visual:** Table 2 condensed: three rows (LoCoMo, LongMemEval, BEAM) showing answerer and judge held identical to the incumbent. Callout: "README says gpt-4o, shipped result files say gpt-5."
**Speaker notes:**
- The leaderboards mix macro and micro metrics and gpt-5 and gpt-4o judges. Reading them as one number is a category error.
- We reproduced Memori to 81.98 against their 81.95 to validate the harness end to end.
- One memory system, three benchmarks, zero per-benchmark tuning, most conservative protocol on the board.
## Slide 7: The measurement, 37 levers (1:00)
**Title:** We tried 37 ways to make it cleverer
**The one message:** The negative results are the product, not a footnote.
**Visual:** A grid of 37 tiles grouped by family: compression/distillation, structuring (KG/ledger), routing, prompt shaping, ensembling/voting. All tiles neutral for now (the reveal is next slide).
**Speaker notes:**
- Two independent programs: 23 levers on LongMemEval, 14 on BEAM.
- Each lever was a real attempt to beat the simple substrate: distill it, graph it, reconcile it, route it, ensemble it.
- Hold the suspense one beat before the reveal.
## Slide 8: The reveal, 33 lost (1:30)
**Title:** 33 of 37 made it worse
**The one message:** Every lever that transformed the store lost; the only four that helped never touched the store.
**Visual:** Figure 2 (`figures/fig2-falsification.png`), the falsification strip plot: 33 points on or below the raw-turn baseline, only 4 above it, all answer-side.
**Speaker notes:**
- The four survivors: self-consistency voting, a guarded split-vote recount, a preference grounding critic, a routing-matrix completion. All read the store more carefully; none rebuild it.
- Roughly 90 percent of everything we tried was self-harm.
- One concrete casualty: a distilled session outline cut summarization from 0.38 to 0.16. It was genuinely nice. It replaced high-resolution signal with a blurry copy.
## Slide 9: Receipt 1, LoCoMo (1:00)
**Title:** LoCoMo: 86.49 against 81.95
**The one message:** State of the art by 4.54 points, statistically solid, and it leads every question category.
**Visual:** Table 3 condensed to the top rows (Ours vs Memori vs full-context ceiling), single/multi/temporal/open/overall columns, our row in bold.
**Speaker notes:**
- 1332 of 1540, z = 4.64, p below ten to the minus five, against the reproduced same-judge Memori row.
- Open-domain accuracy is statistically indistinguishable from the full-context ceiling.
- We also corrected a column-scrambling error that had propagated through the LoCoMo literature.
## Slide 10: Receipt 2, LongMemEval (1:00)
**Title:** LongMemEval: 95.01 macro against 94.87
**The one message:** State of the art on the leader's own headline metric, under the most conservative judge on the board.
**Visual:** Table 4 intervention arc as a rising line from 75.9 percent to 95.01 macro, four adopted rungs labeled (vh5, klc, pfc, h3u).
**Speaker notes:**
- 468 of 500 under the official gpt-4o judge.
- The single biggest lever was recovering a dropped `question_date` field, which lifted temporal accuracy from 61 to 84 percent. No reasoning tuning can supply a reference point absent from the data.
- Every adopted lever is answer-side. None restructures the store.
## Slide 11: Receipt 3, BEAM and the 23 points (1:30)
**Title:** BEAM 1M: 74.0 percent pass, contradiction +23
**The one message:** Pass rate is a real win, and the one differentiator is conflict-awareness worth 23 points on the hardest ability.
**Visual:** Table 5 per-ability bars, ours vs mem0, with contradiction_resolution highlighted (0.588 against 0.357). Foot: pass 74.0 percent against 70.1 percent.
**Speaker notes:**
- 518 of 700, McNemar paired test z = 2.14 at p about 0.03. We rest the claim on pass rate; the average margin sits inside judge noise and we say so.
- Contradiction resolution: 0.588 against 0.357, a 23-point gap, on the benchmark's hardest ability.
- 7 times fewer retrieved items than the incumbent (30 raw turns against 200 facts). The whole BEAM program cost about 160 dollars.
## Slide 12: The mechanism, why simple wins (1:00)
**Title:** Every transform removes what the rubric scores
**The one message:** The benchmarks reward verbatim dates, numbers, and both sides of a conflict, which is exactly what distillation, graphs, and reconciliation delete.
**Visual (build a new one):** Left, a raw turn with date, number, and exact phrase intact. Right, three lossy arrows (distill, graph, reconcile) each dropping one of those tokens. Caption: "Detail removed by a transform cannot be recovered downstream."
**Speaker notes:**
- Retrieval-headroom audit: the supporting content is at rank 30 or better for about 90 percent of failed nuggets. The material is already in front of the answerer.
- So deeper retrieval and richer structure cannot help, and distillation only removes what the answerer already holds.
- The productive place to spend cleverness is read time, not write time. The survivors are the memory analog of test-time-compute scaling.
## Slide 13: Where it loses, honest disclosure (0:50)
**Title:** We report the losses as loudly as the wins
**The one message:** Simple concedes coverage-shaped breadth and inference cost, and we disclosed a bug we could have hidden.
**Visual:** Two small panels. Left: "Summarization 0.570 against 0.635 (we trail)." Right: "Empty-answer bug: 56 blanks, healed under identical config, broken snapshot published."
**Speaker notes:**
- Summarization trails because compound-nugget rubrics reward enumerated breadth, and a compact set of real turns lists fewer items than a padded fact dump. Forcing enumeration backfired every time.
- The bug: a token cap left 56 answers blank on our first BEAM run. We re-ran under the identical configuration and kept the broken snapshot as an artifact rather than quietly raising the cap.
- This disclosure is part of what makes the comparison trustworthy.
## Slide 14: The bitter lesson close (0:30)
**Title:** Keep the store dumb, spend cleverness on reads
**The one message:** The Bitter Lesson came for agent memory. Preserve the raw dated turns, keep both sides of a contradiction, and read more carefully instead of rebuilding.
**Visual:** The one-line takeaway centered, plus contact and links: paper, `github.com/marolinik/waggle-os`, email placeholder.
**Speaker notes:**
- The load-bearing moves: keep the raw data, resolve dates at write time because that adds information, preserve contradictions. Everything else is optional at best.
- We released the three protocols, every per-question judgment, the full ledger, and the harness. You do not need to re-derive the distillation tax; we paid it 33 times.
- Thank the audience, point to the repo, invite questions.
---
## Backup slides (for Q and A)
- **B1. Protocol-fidelity matrix (full Table 2):** answerer, judge, metric, and retrieval budget per system, including the README-versus-shipped judge swap on BEAM.
- **B2. Full falsification ledger (Table 6):** all 37 levers by family with the one-line reason each lost or survived.
- **B3. Retrieval-headroom numbers:** first-hit rank 30 or better for 88 percent (summarization), 90 percent (event ordering), 93 percent (multi-session); widening to top-100 moves only about 8 percent at 2 to 3 times the cost.
- **B4. Cost and tokens:** BEAM context about 27,000 tokens at top-30 against mem0's about 7,000; LoCoMo 3,747 tokens against Memori's 1,294; total BEAM spend about 160 dollars.
- **B5. LongMemEval appendix:** the 19 falsified levers enumerated individually.

BIN
docs/paper/figures/fig1-substrate.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 122 KiB

BIN
docs/paper/figures/fig2-falsification.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 127 KiB