oleg.m/waggle-os
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

moving
Some checks failed
Installer Smoke / installer-smoke (push) Has been cancelled
Details

Browse Source
This commit is contained in:
Oleg Maslov
2026-09-02 10:10:29 +02:00
commit 0c3e2ead3b
3841 changed files with 970576 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
docs/paper/dossier/README.md Normal file
View File

@@ -0,0 +1,59 @@
# Dossier: Clever Memory Loses
The complete recognition package for the paper **"Clever Memory Loses: A Single Simple Substrate Is State of the Art on LoCoMo, LongMemEval, and BEAM"** (Marko Marković, KORRO / hive-mind). This is the arsenal you deploy when someone says "tell me more about what you do": paper, blog, deck, posts, and press kit, ready to send in under five minutes.
The one insight, repackaged for every format: **keep the memory store dumb and spend your cleverness on reads.**
## The three numbers
| Benchmark | Best published incumbent | This work |
|---|---|---|
| LoCoMo (1,540 Q) | Memori 81.95% | 86.49% |
| LongMemEval-S (500 Q) | Mastra 94.87 macro | 95.01 macro / 93.60 micro |
| BEAM 1M (700 Q) | mem0 0.6409 avg / 70.1% pass | 0.6482 avg / 74.0% pass |
Single differentiator: conflict-awareness, worth 23 points on BEAM's contradiction ability. Falsification ledger: 37 interventions measured, 33 lost.
## Contents
| File | What it is |
|---|---|
| `README.md` | This index and the 5-minute send checklist. |
| `arxiv-metadata.md` | Submission-ready arXiv fields: title, author, verbatim abstract, categories (cs.CL primary; cs.AI and cs.IR cross-lists), comments field, license, and a full submission checklist. |
| `talk-deck-outline.md` | Slide-by-slide outline for a 15-minute conference talk (14 slides plus backups): one message, visual spec, and timed speaker notes per slide. |
| `announcement-posts.md` | Launch-day copy: LinkedIn post, an 8-tweet X thread, and a Show HN title with comment. No em dashes, no hype words. |
| `press-kit.md` | One-page press kit: what happened, the three numbers, an author quote, the falsification-ledger press hook, figure list, contact, and a five-question FAQ. |
### Companion assets (one level up, already shipped)
| Asset | Path |
|---|---|
| Research paper (markdown master) | `../2026-07-09-clever-memory-loses-draft.md` |
| Blog post (accessible version) | `../2026-07-09-clever-memory-loses-blog.md` |
| Figure 1, the substrate pipeline | `../figures/fig1-substrate.png` |
| Figure 2, the falsification strip plot | `../figures/fig2-falsification.png` |
| Consolidated results and reproduction commands | `../../../benchmarks/results/MEMORY-BENCHMARKS-CONSOLIDATED-2026-07.md` |
| Code, protocols, per-question artifacts | `github.com/marolinik/waggle-os` |
## What to send, by situation
| Situation | What to send |
|---|---|
| Speaking invitation request | Press kit one-pager + talk deck outline + link to the blog |
| Press or editor inquiry | Press kit + Figure 1 and Figure 2 + blog |
| Research job or collaboration | Paper + press kit + talk deck outline |
| Investor or partnership intro | Press kit one-pager + paper + the three numbers |
| Launch day (arXiv is live) | Announcement posts (LinkedIn, X, Show HN), all at once |
| "Tell me more about what you do" | Press kit one-pager + link to this dossier |
## The 5-minute send checklist
Before you send any subset of this dossier, confirm:
- [ ] **Pick the target.** Use the table above to select the minimum set of files for the situation; do not send everything by default.
- [ ] **Fill the placeholders.** Replace every `[LINK]` in `announcement-posts.md` with the live arXiv or blog URL, and the `[contact placeholder]` in `press-kit.md` with a real email.
- [ ] **Confirm the artifacts are live.** The repository at `github.com/marolinik/waggle-os` is public and contains the harness, per-question results, negative-results tables, and the pre-heal snapshot.
- [ ] **Attach the figures if the recipient is press.** `fig1-substrate.png` and `fig2-falsification.png` are legible at 100 percent and in grayscale.
- [ ] **Verify the numbers match the paper.** LoCoMo 86.49 vs 81.95; LongMemEval 95.01 macro vs 94.87; BEAM 74.0% pass vs 70.1%; contradiction +23. Nothing rounded differently across files.
- [ ] **Lead with the one-pager.** The press kit is read in 60 seconds and must stand alone; everything else is depth behind it.
- [ ] **Zip and send.** The dossier folder plus the two companion figures should be one archive, ready to attach in under five minutes.

95
docs/paper/dossier/announcement-posts.md Normal file
View File

@@ -0,0 +1,95 @@
# Announcement Posts
Three ready-to-send posts. Replace every `[LINK]` with the arXiv URL (or the blog URL) once the paper is announced. No em dashes anywhere. No hype words; the numbers carry it. Every figure quoted here matches the paper.
---
## (a) LinkedIn
Professional, roughly 200 words. Hook, then three numbers, then the link.
---
Everyone building AI memory is making it cleverer: knowledge graphs, extracted facts, systems that reconcile what you said last week against what you say today. We spent months building that cleverness, measured 37 versions of it against three public benchmarks, and watched 33 of them make the memory worse.
The design that won keeps the raw conversation, stamps every line with its date, and when two lines disagree, shows both. We call it conflict-aware raw-turn memory. The entire point is that it refuses to be clever.
Under each incumbent's own published protocol and judge, one substrate is state of the art on all three benchmarks:
- LoCoMo: 86.49% against the prior best 81.95%
- LongMemEval: 95.01 macro against 94.87
- BEAM 1M: 74.0% pass against 70.1%
The single decision that separates us from the field is conflict-awareness: keep both sides of a contradiction instead of resolving it at write time. A reconciler that runs at write time has to pick one value and delete the other before the question is ever asked. On contradiction resolution, the hardest ability on BEAM, that one choice is worth 23 points.
We released everything, including the 33 experiments that failed and a bug we could have hidden. The Bitter Lesson came for agent memory too.
Paper, code, and every per-question judgment: [LINK]
---
## (b) X / Twitter thread
Six to eight tweets. Tweet 1 stands alone and is numbers-forward; the honest-disclosure angle is the differentiator.
**1/**
We tried 37 ways to make AI memory smarter. 33 made it worse.
The design that won keeps the raw conversation, dates every line, and when two lines disagree keeps both. It is state of the art on LoCoMo, LongMemEval, and BEAM. We published all 33 failures.
**2/**
Everyone in agent memory builds the same thing: distill conversations into facts, build a knowledge graph, reconcile new facts against old ones at write time.
The shared belief is that a cleaner, smaller, reconciled store is a better store.
**3/**
We built the opposite. Keep the original turns, exactly as said, each stamped with its date. Retrieve the relevant ones per question. When two contradict, keep both and surface the conflict.
No graph. No fact extraction as the store. No reconciliation.
**4/**
One substrate, three benchmarks, each under the incumbent's own protocol and judge:
LoCoMo 86.49% vs 81.95%
LongMemEval 95.01 macro vs 94.87
BEAM 1M 74.0% pass vs 70.1%
Zero per-benchmark tuning.
**5/**
The one decision that separates us: keep both sides of a contradiction instead of resolving it at write time.
A reconciler that runs at write time must pick one value. A read-time policy can hold both. On BEAM's hardest ability that is worth 23 points.
**6/**
The 37 experiments are the real product. Every intervention that transformed the store lost: distill it, graph it, reconcile it, restructure it, all worse.
The only 4 that helped never touched the store. They just read it more carefully.
**7/**
Honest disclosure: our first BEAM run scored lower because a token cap left 56 answers blank.
We could have quietly raised the cap. Instead we re-ran under the identical config, healed the blanks, and published the broken snapshot as an artifact.
**8/**
Keep the store dumb, spend your cleverness on reads.
We released the three protocols, every per-question judgment, the full ledger of 37 interventions, and the harness. The Bitter Lesson came for agent memory too.
[LINK]
---
## (c) Hacker News (Show HN)
**Title:**
```
Show HN: Conflict-aware raw-turn memory, SOTA on LoCoMo, LongMemEval and BEAM
```
**Comment (post immediately after submitting):**
I spent months building clever agent memory (fact distillation, knowledge graphs, write-time reconciliation) and then measured 37 versions of it against three public long-term-memory benchmarks. 33 of the 37 made the memory worse. The design that won is almost embarrassingly simple: keep the raw conversation turns, stamp each with its date, retrieve the relevant ones per question, and when two turns contradict each other, keep both and surface the conflict instead of resolving it. Under each incumbent's own published protocol and judge, this one substrate is state of the art on all three: LoCoMo 86.49% (prior best 81.95%), LongMemEval 95.01 macro (prior best 94.87), and BEAM 1M 74.0% pass (prior best 70.1%). The single differentiator is conflict-awareness: a reconciler that resolves conflicts at write time has to delete one side before the question is ever asked, which is exactly what the contradiction rubric penalizes. That one decision is worth 23 points on BEAM's hardest ability.
The part I most want feedback on is the falsification ledger. I am publishing all 37 interventions, including the 33 that lost, because the negative results are the actual finding: every scheme that transformed the store (distillation, graphs, ledgers, routers, self-ensembles) lost, and the only four that helped left the store untouched and just read it more carefully. I am also disclosing a bug: my first BEAM run scored lower because a completion-token cap left 56 answers blank, and rather than quietly raise the cap I re-ran under the identical config and published the broken snapshot alongside the fixed one. All protocols, per-question judgments, and the harness are in the repo. I am happy to go deep on the protocol-fidelity methodology, since cross-lab memory numbers are mostly noise until you reproduce the incumbent under the same judge. [LINK]

76
docs/paper/dossier/arxiv-metadata.md Normal file
View File

@@ -0,0 +1,76 @@
# arXiv Submission Metadata
Submission-ready metadata for the paper. Copy each field into the arXiv submission form. Every number is drawn verbatim from the paper master (`../2026-07-09-clever-memory-loses-draft.md`); do not paraphrase the abstract.
---
## Title
```
Clever Memory Loses: A Single Simple Substrate Is State of the Art on LoCoMo, LongMemEval, and BEAM
```
## Authors
```
Marko Marković
```
Affiliation line (paper byline): KORRO / hive-mind.
## Abstract
Paste verbatim. This is the paper's Section 1 abstract, unaltered.
```
Long-term conversational memory, answering questions over weeks of prior dialogue, is the load-bearing capability for durable AI assistants, and three benchmarks (LoCoMo, LongMemEval, and BEAM) are the field's rulers. Every published leader adds structure to the memory path: atomic-fact distillation, temporal knowledge graphs, write-time entity reconciliation, and learned routing. Our insight is that these transforms are lossy in exactly the way the benchmarks penalize, because distillation strips the dates and specifics the rubrics score and write-time reconciliation silently resolves the contradictions the rubrics want surfaced. We show that one simple substrate, conflict-aware raw-turn memory (per-conversation minds, verbatim dated raw-turn retrieval, and a conflict-preserving answer policy), is state of the art on all three under each incumbent's own published protocol: LoCoMo 86.49 against 81.95, LongMemEval 95.01 macro against 94.87, and BEAM 0.6482 average and 74.0 percent pass against 0.6409 and 70.1 percent. We then measured 37 constructive interventions that tried to make the substrate cleverer; 33 lost, and we report them as a first-class falsification ledger. The single differentiator is conflict-awareness, worth 23 points on BEAM's contradiction ability. Because the leaderboards mix macro and micro metrics and gpt-5 and gpt-4o judges, we reproduce each incumbent's pipeline before comparing and report against the most conservative protocol on the board. All protocols, per-question artifacts, and the empty-answer-heal disclosure are released.
```
## Categories
| Field | Value | Rationale |
|---|---|---|
| Primary | `cs.CL` | Long-term conversational memory over natural-language dialogue; LLM answerers and LLM-as-judge scoring. |
| Cross-list | `cs.AI` | Agent memory architecture; the Bitter Lesson framing for autonomous agents. |
| Cross-list | `cs.IR` | Hybrid dense plus BM25 retrieval, reciprocal rank fusion, reranking, retrieval-headroom analysis. |
## Comments field
Paste into the "Comments" box (confirm the page count against the compiled PDF before submitting; the figure and table counts are fixed).
```
14 pages, 2 figures, 6 tables, 2 appendices. Code, protocols, per-question answers and judgments, the falsification ledger, and the empty-answer pre-heal snapshot: https://github.com/marolinik/waggle-os
```
## License
Suggested: **arXiv.org perpetual, non-exclusive license** (the default arXiv license). It permits arXiv to distribute the work while the author retains copyright, which is the correct choice for a preprint intended to be widely read and cited. Do not select a CC-BY or CC0 variant unless a target venue later requires it.
## MSC / ACM classification (optional fields)
- ACM class (optional): `I.2.7` (Natural Language Processing), `H.3.3` (Information Search and Retrieval).
---
## Submission checklist
### Endorsement
- [ ] Confirm endorsement status for `cs.CL`. A first-time submitter without an institutional arXiv history typically needs an endorsement from an established author in the category. If unendorsed, request endorsement from a coauthor or a cited author before uploading, and allow lead time.
- [ ] Verify the submitting email is on an endorsed or auto-endorsed domain if applicable.
### PDF and source requirements
- [ ] Preferred submission is LaTeX source (the markdown master converts to LaTeX; upload `.tex` plus the two `.png` figures, not a pre-built PDF, so arXiv can compile). A direct PDF upload is accepted but discouraged for text-heavy papers.
- [ ] Both figures embedded at print resolution: `figures/fig1-substrate.png` and `figures/fig2-falsification.png`. Confirm they are legible at 100 percent zoom and in grayscale.
- [ ] All six tables render inside the text column with no overflow (Tables 1 through 6).
- [ ] References compile cleanly; every arXiv identifier in the reference list resolves (2504.19413, 2501.13956, 2402.17753, 2410.10813, 2510.27246, 2512.12818, 2512.20237, 2310.08560).
- [ ] Title, author, and abstract in the form match the compiled PDF exactly.
- [ ] Page count in the Comments field matches the final PDF.
### Ancillary files
- [ ] Optional: attach the consolidated command list (`benchmarks/results/MEMORY-BENCHMARKS-CONSOLIDATED-2026-07.md`) as an ancillary file, or link it from the repository.
- [ ] Confirm the repository at `github.com/marolinik/waggle-os` is public and contains the harness, per-question result files, negative-results tables, and the pre-heal snapshot referenced in Appendix B, so the artifacts URL in the Comments field is live at submission time.
- [ ] No private data or API keys in any uploaded source; all three datasets are public and used as published.
### Post-submission
- [ ] After the paper is announced, add the arXiv ID to the blog post, the LinkedIn post, and the X thread (replace the `[LINK]` placeholders in `announcement-posts.md`).
- [ ] Submit the blog post to Hacker News and the paper to r/MachineLearning on the announcement day.

63
docs/paper/dossier/press-kit.md Normal file
View File

@@ -0,0 +1,63 @@
# Press Kit: Clever Memory Loses
One page for journalists, editors, and program chairs. Everything here is drawn from the paper and is quotable. No em dashes.
---
## What happened (3 sentences)
A single, deliberately simple memory substrate is state of the art on all three of the field's long-term conversational memory benchmarks, LoCoMo, LongMemEval, and BEAM, each measured under the incumbent leader's own published protocol and judge. Instead of distilling conversations into facts or knowledge graphs, it keeps the raw dated conversation turns and, when two statements contradict each other, retains both rather than resolving the conflict at write time. Alongside the three results, the author publishes a falsification ledger of 37 interventions that tried to make the substrate cleverer, of which 33 made it worse.
## The three numbers
| Benchmark | Best published incumbent | This work | Margin |
|---|---|---|---|
| LoCoMo (1,540 Q) | Memori 81.95% | 86.49% | +4.54pp (z=4.64, p<0.00001) |
| LongMemEval-S (500 Q) | Mastra 94.87 macro | 95.01 macro / 93.60 micro | +0.14 macro |
| BEAM 1M (700 Q) | mem0 0.6409 avg / 70.1% pass | 0.6482 avg / 74.0% pass | +3.9pp pass (McNemar z=2.14, p≈0.03) |
Single differentiator: conflict-awareness is worth 23 points on BEAM's contradiction ability (0.588 against 0.357). Total BEAM program cost: about 160 dollars.
## Quote from the author
Marko Marković (KORRO / hive-mind):
> "We paid the distillation tax 33 times so nobody else has to: keep the store dumb, keep both sides of a contradiction, and spend your cleverness on reading it, not rebuilding it."
## The press hook: they published everything that failed
Memory-systems papers typically report only the surviving configuration. This one ships the opposite: a first-class negative-results artifact of 37 interventions across two independent programs, 33 of which lost to the simple baseline, grouped by family so a reader can see that every scheme which transformed the store (distillation, graphs, ledgers, routers, self-ensembles) lost, and the only four that helped never touched the store. The author also volunteers a bug disclosure most papers would bury: a first BEAM run scored lower because a completion-token cap left 56 answers blank, and rather than quietly raise the cap, the author re-ran under the identical configuration and preserved the broken snapshot as a published artifact. The story is not only that simple beat clever; it is that the author documented every way clever lost.
## Figure files
- `figures/fig1-substrate.png` : the conflict-aware raw-turn substrate, left to right (write time: dated raw-turn minds with relative-date resolution; read time: hybrid top-k retrieval; answer policy: surface both sides on contradiction). Best for explainer and architecture context.
- `figures/fig2-falsification.png` : the falsification strip plot, 33 falsified levers on or below the raw-turn baseline and only 4 adopted answer-side levers above it. Best for the "complexity buys nothing" narrative.
(Both files live one level up at `../figures/` relative to this dossier.)
## Contact
- Author: Marko Marković, KORRO / hive-mind
- Email: [contact placeholder]
- Code, protocols, per-question artifacts, and the pre-heal snapshot: `github.com/marolinik/waggle-os`
- Paper: `../2026-07-09-clever-memory-loses-draft.md` (arXiv link to be added on announcement)
- Accessible write-up: `../2026-07-09-clever-memory-loses-blog.md`
---
## FAQ
**Is this cherry-picked?**
No, and the paper is built to make cherry-picking hard to hide. Each incumbent's own pipeline is reproduced under the same judge before any comparison (Memori's LoCoMo number was reproduced to 81.98 against their published 81.95). The losses are reported as loudly as the wins: summarization trails the incumbent at 0.570 against 0.635, and the BEAM average-score margin is stated as a statistical tie inside the judge-noise band. Every per-question judgment is released, along with a ledger of the 33 interventions that failed, and where a result was inflated by a bug the bug and both numbers are disclosed.
**Why should users care?**
An assistant that cannot recall what you said three weeks ago cannot be a durable collaborator. Today's memory systems quietly resolve your contradictions for you: tell one that your follower count changed and one number silently disappears from its store. This substrate keeps both dated statements and, at answer time, presents both and asks which is correct, which is what people actually want from a memory that is supposed to have been paying attention.
**What is conflict-aware memory?**
It keeps the original dated conversation turns instead of distilling them into facts or a graph, and when two statements disagree it retains both rather than reconciling them the moment they are written. At answer time it states the contradiction, presents both dated statements, and asks which is correct. A reconciler that runs at write time must pick one side to store; a policy that runs at read time can hold both and decide per question. That single decision is worth 23 points on BEAM's hardest ability.
**What did it cost?**
The entire BEAM program cost roughly 160 dollars in API spend. The memory engine runs local embeddings (nomic-embed-text via Ollama), so no memory content leaves the machine on the write path or the read path, and it retrieves 7 times fewer items than the incumbent (30 raw turns against 200 facts). The one axis it concedes is inference token cost: a raw-turn answer context is larger than a distilled-fact one (about 27,000 tokens against about 7,000 on BEAM), so the win is on answer quality and the concession is on efficiency.
**What is next?**
Three open questions. Can conflict preservation be made cheap enough to also win the coverage-shaped abilities such as summarization, where the substrate still trails. Does the raw-turn result hold under weaker and cheaper answerers, or is long-context capacity the true enabler. And what is the right rubric for a memory that surfaces a contradiction and asks for clarification, as against one that commits to a single value and happens to be right.

161
docs/paper/dossier/talk-deck-outline.md Normal file
View File

@@ -0,0 +1,161 @@
# Talk Deck Outline: 15-Minute Conference Talk
**Title:** Clever Memory Loses
**Subtitle:** A single simple substrate is state of the art on LoCoMo, LongMemEval, and BEAM
**Speaker:** Marko Marković (KORRO / hive-mind)
**Length:** 15 minutes, 14 slides.
Story arc (one narrative spine, no detours): cold open (the follower-count contradiction) to the wall (everyone builds cleverer memory) to the mechanism where clever becomes the weakness to the one dumb idea to fair measurement to the 37 levers to the reveal that 33 lost to the three receipts to the bitter-lesson close.
Design rules (from the presentation format guide): one message per slide, six lines of text maximum, 24pt minimum font, figures legible from the back, no em dashes, consistent color scheme with the paper. Total speaking budget below sums to roughly 15 minutes; the timings are a pacing guide, not a script.
---
## Slide 1: Title (0:10)
**Title:** Clever Memory Loses
**The one message:** The dumbest design on the board wins all three benchmarks.
**Visual:** Title, subtitle, author, venue. A faint background of dated raw turns (`[2023-03-14] ...`, `[2023-06-02] ...`) to preview the substrate.
**Speaker notes:**
- Say the title, then the thesis in one line: "We made memory dumber and it got better."
- Name the three benchmarks so the audience knows the scope up front.
## Slide 2: Cold open, the follower count (1:20)
**Title:** Your assistant deleted the evidence
**The one message:** Clever memory resolves a contradiction before you ever ask about it, and the number quietly disappears.
**Visual (build a new one):** A three-panel timeline. March: user says "10 followers." June: user says "15 followers." A write-time reconciler fires UPDATE and 10 vanishes from the store. Then a question mark: "Later: how did the count change?" with an empty store.
**Speaker notes:**
- Tell it as a story, no jargon: you say 10 in March, 15 in June, then someone asks about the change.
- A system that reconciles at write time already fired an UPDATE. 10 is gone. The contradiction the benchmark tests no longer exists.
- Land the line: "The clever system deleted the evidence." Pause. This is the whole talk in one anecdote.
## Slide 3: The wall (1:00)
**Title:** Everyone is building cleverer memory
**The one message:** The field agrees on one premise: a cleaner, smaller, reconciled store is a better store.
**Visual:** Four labeled boxes: Mem0 (atomic facts), Zep (temporal knowledge graph), Memori (dated triples), mem0 platform (write-time reconciliation). One arrow from each into a shrinking "derived store."
**Speaker notes:**
- Every leader spends its engineering budget on the same two moves: transform the conversation into a smaller derived representation, and resolve contradictions before the answerer runs.
- The shared, rarely-stated belief: better memory means cleverer memory.
- Set up the turn: what if the cleverness is the problem?
## Slide 4: The mechanism, clever becomes the weakness (1:30)
**Title:** The cleverness and the weakness are the same mechanism
**The one message:** The write-time reconciler that scores well on knowledge update is exactly what makes contradiction resolution the incumbent's worst ability.
**Visual:** A two-column contrast. Left: "Write-time reconcile: knowledge update 0.650 (strong)." Right: "Same reconciler: contradiction resolution 0.357 (worst)." An arrow labeled "same mechanism" connecting them.
**Speaker notes:**
- A write-time reconciler must pick one side to store. That helps knowledge update and destroys contradiction resolution.
- The weakness is not independent of the cleverness. It is a direct cost of it.
- This is the crack we drove a whole paper through.
## Slide 5: The one dumb idea (1:30)
**Title:** Conflict-aware raw-turn memory
**The one message:** Keep the raw dated turns, retrieve them verbatim, and when two disagree keep both.
**Visual:** Figure 1 (`figures/fig1-substrate.png`), the substrate pipeline: conversation to per-conversation dated raw-turn mind to top-30 retrieval to conflict-aware answer policy to answer.
**Speaker notes:**
- Three moves: per-conversation minds, dated verbatim raw turns as the primary lane, contradictions retained not reconciled.
- One transform at write time, resolving relative dates ("yesterday" becomes a real date), because that adds information rather than removing it.
- No graph, no fact distillation as the store, no write-time reconciliation, no learned router. The point is that it refuses to be clever.
## Slide 6: Measuring fairly (1:00)
**Title:** Cross-lab memory numbers are noise
**The one message:** We reproduce each incumbent's own pipeline under the same judge before we compare, then swap in our store and change nothing else.
**Visual:** Table 2 condensed: three rows (LoCoMo, LongMemEval, BEAM) showing answerer and judge held identical to the incumbent. Callout: "README says gpt-4o, shipped result files say gpt-5."
**Speaker notes:**
- The leaderboards mix macro and micro metrics and gpt-5 and gpt-4o judges. Reading them as one number is a category error.
- We reproduced Memori to 81.98 against their 81.95 to validate the harness end to end.
- One memory system, three benchmarks, zero per-benchmark tuning, most conservative protocol on the board.
## Slide 7: The measurement, 37 levers (1:00)
**Title:** We tried 37 ways to make it cleverer
**The one message:** The negative results are the product, not a footnote.
**Visual:** A grid of 37 tiles grouped by family: compression/distillation, structuring (KG/ledger), routing, prompt shaping, ensembling/voting. All tiles neutral for now (the reveal is next slide).
**Speaker notes:**
- Two independent programs: 23 levers on LongMemEval, 14 on BEAM.
- Each lever was a real attempt to beat the simple substrate: distill it, graph it, reconcile it, route it, ensemble it.
- Hold the suspense one beat before the reveal.
## Slide 8: The reveal, 33 lost (1:30)
**Title:** 33 of 37 made it worse
**The one message:** Every lever that transformed the store lost; the only four that helped never touched the store.
**Visual:** Figure 2 (`figures/fig2-falsification.png`), the falsification strip plot: 33 points on or below the raw-turn baseline, only 4 above it, all answer-side.
**Speaker notes:**
- The four survivors: self-consistency voting, a guarded split-vote recount, a preference grounding critic, a routing-matrix completion. All read the store more carefully; none rebuild it.
- Roughly 90 percent of everything we tried was self-harm.
- One concrete casualty: a distilled session outline cut summarization from 0.38 to 0.16. It was genuinely nice. It replaced high-resolution signal with a blurry copy.
## Slide 9: Receipt 1, LoCoMo (1:00)
**Title:** LoCoMo: 86.49 against 81.95
**The one message:** State of the art by 4.54 points, statistically solid, and it leads every question category.
**Visual:** Table 3 condensed to the top rows (Ours vs Memori vs full-context ceiling), single/multi/temporal/open/overall columns, our row in bold.
**Speaker notes:**
- 1332 of 1540, z = 4.64, p below ten to the minus five, against the reproduced same-judge Memori row.
- Open-domain accuracy is statistically indistinguishable from the full-context ceiling.
- We also corrected a column-scrambling error that had propagated through the LoCoMo literature.
## Slide 10: Receipt 2, LongMemEval (1:00)
**Title:** LongMemEval: 95.01 macro against 94.87
**The one message:** State of the art on the leader's own headline metric, under the most conservative judge on the board.
**Visual:** Table 4 intervention arc as a rising line from 75.9 percent to 95.01 macro, four adopted rungs labeled (vh5, klc, pfc, h3u).
**Speaker notes:**
- 468 of 500 under the official gpt-4o judge.
- The single biggest lever was recovering a dropped `question_date` field, which lifted temporal accuracy from 61 to 84 percent. No reasoning tuning can supply a reference point absent from the data.
- Every adopted lever is answer-side. None restructures the store.
## Slide 11: Receipt 3, BEAM and the 23 points (1:30)
**Title:** BEAM 1M: 74.0 percent pass, contradiction +23
**The one message:** Pass rate is a real win, and the one differentiator is conflict-awareness worth 23 points on the hardest ability.
**Visual:** Table 5 per-ability bars, ours vs mem0, with contradiction_resolution highlighted (0.588 against 0.357). Foot: pass 74.0 percent against 70.1 percent.
**Speaker notes:**
- 518 of 700, McNemar paired test z = 2.14 at p about 0.03. We rest the claim on pass rate; the average margin sits inside judge noise and we say so.
- Contradiction resolution: 0.588 against 0.357, a 23-point gap, on the benchmark's hardest ability.
- 7 times fewer retrieved items than the incumbent (30 raw turns against 200 facts). The whole BEAM program cost about 160 dollars.
## Slide 12: The mechanism, why simple wins (1:00)
**Title:** Every transform removes what the rubric scores
**The one message:** The benchmarks reward verbatim dates, numbers, and both sides of a conflict, which is exactly what distillation, graphs, and reconciliation delete.
**Visual (build a new one):** Left, a raw turn with date, number, and exact phrase intact. Right, three lossy arrows (distill, graph, reconcile) each dropping one of those tokens. Caption: "Detail removed by a transform cannot be recovered downstream."
**Speaker notes:**
- Retrieval-headroom audit: the supporting content is at rank 30 or better for about 90 percent of failed nuggets. The material is already in front of the answerer.
- So deeper retrieval and richer structure cannot help, and distillation only removes what the answerer already holds.
- The productive place to spend cleverness is read time, not write time. The survivors are the memory analog of test-time-compute scaling.
## Slide 13: Where it loses, honest disclosure (0:50)
**Title:** We report the losses as loudly as the wins
**The one message:** Simple concedes coverage-shaped breadth and inference cost, and we disclosed a bug we could have hidden.
**Visual:** Two small panels. Left: "Summarization 0.570 against 0.635 (we trail)." Right: "Empty-answer bug: 56 blanks, healed under identical config, broken snapshot published."
**Speaker notes:**
- Summarization trails because compound-nugget rubrics reward enumerated breadth, and a compact set of real turns lists fewer items than a padded fact dump. Forcing enumeration backfired every time.
- The bug: a token cap left 56 answers blank on our first BEAM run. We re-ran under the identical configuration and kept the broken snapshot as an artifact rather than quietly raising the cap.
- This disclosure is part of what makes the comparison trustworthy.
## Slide 14: The bitter lesson close (0:30)
**Title:** Keep the store dumb, spend cleverness on reads
**The one message:** The Bitter Lesson came for agent memory. Preserve the raw dated turns, keep both sides of a contradiction, and read more carefully instead of rebuilding.
**Visual:** The one-line takeaway centered, plus contact and links: paper, `github.com/marolinik/waggle-os`, email placeholder.
**Speaker notes:**
- The load-bearing moves: keep the raw data, resolve dates at write time because that adds information, preserve contradictions. Everything else is optional at best.
- We released the three protocols, every per-question judgment, the full ledger, and the harness. You do not need to re-derive the distillation tax; we paid it 33 times.
- Thank the audience, point to the repo, invite questions.
---
## Backup slides (for Q and A)
- **B1. Protocol-fidelity matrix (full Table 2):** answerer, judge, metric, and retrieval budget per system, including the README-versus-shipped judge swap on BEAM.
- **B2. Full falsification ledger (Table 6):** all 37 levers by family with the one-line reason each lost or survived.
- **B3. Retrieval-headroom numbers:** first-hit rank 30 or better for 88 percent (summarization), 90 percent (event ordering), 93 percent (multi-session); widening to top-100 moves only about 8 percent at 2 to 3 times the cost.
- **B4. Cost and tokens:** BEAM context about 27,000 tokens at top-30 against mem0's about 7,000; LoCoMo 3,747 tokens against Memori's 1,294; total BEAM spend about 160 dollars.
- **B5. LongMemEval appendix:** the 19 falsified levers enumerated individually.