waggle-os/docs/plans/MEMORY-SOTA-PROPOSAL-2026-06-10.md

Memory SOTA Proposal — 2026-06-10

Status: PROPOSAL (no code). Research basis: 6-agent workflow (substrate map, failure mining of 1,540 judged answers × 2 arms, Zep/Graphiti, LangMem, academic survey 2024–2026, open-domain deep-dive). Constraints: fully local (SQLite + sqlite-vec, Ollama embeddings, in-process ONNX cross-encoder, optional local LLM via Ollama), personal + workspace minds preserved, OSS subtree-split clean.

---

0. THE RE-BASELINE — we were chasing a phantom (read this first)

The Memori paper's baseline rows are column-scrambled. Its Table 1 says baselines were "retrieved from Du et al. [2025]" (= MemR3, arXiv:2512.20237). MemR3's column order is Multi | Temporal | Open | Single; Memori printed the same values under Single | Multi | Open | Temporal. Verified by extracting both PDFs + MemR3's §C.3 ("existing works have misaligned category labels") + Memori's own self-contradictory narrative ("72.70 trailing 61.06").

Audited our harness: our labels are CORRECT (temporal n=321 with "When did..." questions, open n=96, multi n=282, single n=841 — exact canon counts + semantic spot-checks pass). Our numbers stand. The competitor numbers move:

Corrected LoCoMo landscape (GPT-4.1-mini protocol)

Category	FC ceiling	MemR3*	Ours (P4)	Memori	Zep†	LangMem†	Mem0†
Temporal	86.82	82.14	80.06 ⭐	80.37	77.26	61.06	57.32
Open-domain	71.88	71.53	60.42 ❌	63.54	64.58	67.71	44.79
Single-hop	93.73	92.17	88.59 ⭐	87.87	83.49	86.92	66.47
Multi-hop	86.43	81.20	79.43 ⭐	72.70	72.34	74.47	62.41
Overall	—	—	83.38 ⭐	81.95	—	78.05	62.47

* MemR3 = agentic reflective-retrieval pipeline (different class, not a memory system). † Corrected per MemR3 Table 1. Zep's self-published 83.33/73.96 don't match any MemR3 version — provenance unclear; cross-lab LoCoMo numbers are noisy, which makes our in-harness same-judge comparison the defensible standard.

Corrected verdict: we are ALREADY the leading memory system on overall, single-hop, multi-hop, and (≈tied with Memori) temporal. The phantom "LangMem temporal 86.92" was LangMem's single-hop score; LangMem's real temporal is 61.06 — its extractor never receives conversation timestamps (verified in its source), which validates our write-time dating as the right design (+19pp over LangMem on temporal).

The ONE real gap: open-domain 60.42 vs LangMem 67.71 / FC 71.88. The second axis: tokens — 2,742/q vs Memori's 1,294.

Caveat: open-domain n=96 → SE ≈ ±5pp; deltas <8pp are noise-adjacent. All wave gates below use two-proportion z-tests on full N (no mid-run proxies — they burned us twice).

---

1. What the failure data says (1,540 judged answers × 2 arms, mined)

Open-domain (34 fails / 96)

• 16/34 are ABSTENTIONS ("Not stated in the retrieved context") on speculative questions ("Would Caroline be considered religious?"). The protocol dropped the adversarial category, so abstention is a guaranteed zero. In 4 cases the theirs arm answered the same question correctly from the SAME substrate → prompt-induced.
• ~12 wrong inferences (persona signal too dispersed; vocabulary mismatch: "console" never co-occurs with "Xenoblade").
• ~3 counting errors from episodic duplicates (same hike narrated twice → "five" vs gold "four").
• Question type: ~50% persona/preference inference, ~25% world-knowledge bridging, ~10% entity-ID, ~15% aggregation. It is persona synthesis + licensed speculation, not retrieval. Fact-list systems (us, Memori) bottom out here; profile/summary systems (LangMem, Zep, MIRIX) lead.

Temporal (71 fails / 321) — ~60% prompt-side, ~40% substrate

• 15 precision-miscalibration fails: we emit a confident exact ISO date 1–7 days off where the judge accepts coarse answers — 22 of 31 ours-only fails PASSED in the theirs arm with "Early June 2023"-style granularity.
• 11 session-date echoes: gold is "the week before "; our write-time resolution stamped the mention date (forward resolution exists, backward ranges don't).
• 18 wrong event bindings (Tokyo vs Boston; reversed adoption order) — episodic duplicates + no date-window filtering at retrieval.
• 11 refusals (dated event not retrieved), 6 duration fencepost errors.

Multi-hop (71 fails) — dominant cluster: partial enumeration on cross-session

aggregation ("Which US cities...?" → returns 1 of 3) + duplicate-inflated counting.

Single-hop (95 fails) — fine-grained detail lost by distillation (gold: "painting

inspired by sunsets with pink sky"; we retrieve the distilled "an abstract painting").

Substrate map findings (production-relevant)

• KnowledgeGraph contributes ZERO to recall — bfsDistances→contextual-score wiring exists in scoring.ts but no caller passes graphDistances → 20% of the 'balanced' relevance weight is permanently 0.
• since/until SQL filters exist in HybridSearch + FrameStore — never called by anyone.
• Production recall (orchestrator.ts) has NO reranker/chunking/distilled/episodic layers — those exist only in the OSS repo + benchmark harness. Production lags the benchmark substrate substantially.
• Scoring "temporal" dimension decays on last_accessed (access recency) — constant noise on a 2023 corpus, not event time.

---

2. The proposal — four waves, each gated by a full-N z-tested re-run

WAVE 1 — Answer-policy fixes (prompt-only, zero substrate risk, ~1 day)

Targets the measured prompt tax. No regression risk to the substrate.

1. Conditional abstention: speculative/inferential questions ("would/might/could/ likely") → forbid refusal, force committed best-effort inference from retrieved evidence + world knowledge. Factual questions keep abstention (production safety). Evidence: 16 guaranteed-zero abstentions; judge demonstrably accepts directional guesses.
2. Granularity-calibrated dates: emit exact day ONLY when explicitly stated; otherwise answer at week/month granularity ("early June 2023"). Evidence: 22 ours-only temporal fails passed in theirs arm with coarser answers.
3. Duration brevity + endpoint few-shot: final value only (verbose multi-date reasoning triggers harsh judging); fencepost examples.
4. Commit-to-one-option: forbid hedged dual answers ("both") on either/or questions.
5. Parametric-knowledge gating (arXiv:2510.23730): for world-knowledge-bridging questions, instruct "combine retrieved facts with general world knowledge" — retrieval-only instructions measurably suppress the model's own knowledge (FC 56.4 vs RAG 49.5 F1 on this category).

Expected: open +6–10pp, temporal +4–6pp, overall → ~85. Cost: ~$6 re-run.

WAVE 2 — Profile cards + episodic hygiene (write-time substrate, local LLM)

The dominant open-domain lever, converging from three independent sources (Zep entity summaries, LangMem profiles, MIRIX core-memory; all profile-carrying systems lead this category).

1. Per-speaker rolling profile cards: ~500-char abstractive profile per speaker/entity, updated incrementally at ingest (Graphiti fast-path: append facts without LLM call while under cap; consolidate-compress via Ollama when over). Rendered as a "PERSONA" block in context. Per-scope (personal + per-workspace) — isomorphic with our existing split.
2. Episodic event canonicalization: dedup same-event-renarrated rows at ingest (fixes counting failures in open + multi).
3. Event-date RANGES: backward resolution for retrospective narration — store [event_date_min, event_date_max] + mention date ("last week" → 7-day window), render ranges; answer at range granularity (pairs with Wave-1 #2).

Expected: open → ≥70 (combined with Wave 1), temporal +2–3pp, tokens −10–20% (one profile card replaces many weak-signal facts).

WAVE 3 — Retrieval lanes (query-time; mostly wiring existing code)

1. Temporal retrieval lane (MRAG arXiv:2412.15540 / Hindsight TEMPR): parse the query's temporal constraint (deterministic, extends resolve-relative-date.ts to query side) → pass the already-existing-but-never-called since/until filters → add a date-window lane into RRF fusion before the cross-encoder. MRAG: +9.3% top-1 recall on temporal QA.
2. Entity-keyed exhaustive retrieval for enumeration/counting questions: pull ALL episodic rows for the focal entity (not top-K), dedup-by-event before answering. Targets the dominant multi-hop cluster (~16/24 sampled fails).
3. Raw-detail escalation lane: when the question asks for concrete perceptual detail and distilled facts match only generically, fetch the raw session turn around the matching fact (256-token chunks; sqlite-vec + FTS — our existing stack). Targets the dominant single-hop cluster.
4. Wire graphDistances into scoring (the dead 20% weight) + BFS-expansion lane self-seeded from search-hit entities (depth ≤2, recursive CTE — sub-ms at our scale).

Expected: temporal → ~84–86 (FC ceiling is 86.82), multi → ~82–84, single → ~90.

WAVE 4 — Bi-temporal substrate + production parity (architecture; product-first)

1. Bi-temporal validity on facts (Zep model): valid_at/invalid_at (event time) + created_at/expired_at (transaction time); facts never deleted, only closed.
2. Ingest-time invalidation: one small-Ollama call per new fact against same-entity
   • RRF-similar existing facts → duplicate[]/contradicted[]; a deterministic temporal-overlap rule does the actual invalidation (LLM proposes, arithmetic disposes). Gives latest-wins for "what is X now" while preserving "what was true then" — fixes the knowledge-update losses we measured on LongMemEval too.
3. Production parity: port the benchmark-proven stack (reranker, distilled facts, episodic dated events, date rendering, TEMPORAL_GUIDANCE, profile cards) into the production recall path (orchestrator.ts) — production currently has none of it.
4. Token-budget context packing (Zep's 1.6k-token block beats 115k full-context): target ≤1,500 tokens/q — closes the efficiency gap with Memori while raising scores.

Expected: durable product wins beyond LoCoMo; tokens → ~1.5k/q.

---

3. Projected end-state (honest ranges, ±noise)

Category	Now	After W1	After W2	After W3/W4	FC ceiling
Temporal	80.06	~84	~85	85–87	86.82
Open-domain	60.42	~67	70–74	72–75	71.88
Single-hop	88.59	88.5	~89	90–92	93.73
Multi-hop	79.43	~80	~81	82–84	86.43
Overall	83.38	~85.3	~86.3	87–88.5	—
tokens/q	2,742	2,742	~2,300	≤1,500	—

At ~86–88 overall we'd clear every published memory system by a decisive (significant) margin and approach Hindsight's local-model result (85.67 w/ GPT-OSS-20B) — whose architecture (same CE reranker + temporal lane + observation summaries + graph lane) independently validates this exact roadmap, with local open-weight models.

4. Anti-goals (lessons paid for)

• Never strip write-time dating (LangMem's undated extraction = its 61.06 temporal).
• No relevance-ranked-only episodic (P5 proved top-K filtering reverts the timeline- scaffold benefit; keep the chronological block, dedup it instead).
• No agentic multi-turn retrieval loops in production hot path (LangMem p95 ~60s). MemR3-style reflection is benchmark-viable but a latency hazard; defer, lane-gate.
• Keep abstention for factual unknowns in production (conditional policy only loosens speculative questions; adversarial robustness must not regress).
• No Neo4j / no cloud — everything above is SQLite tables + Ollama + in-process ONNX.

5. Verification protocol

Each wave: full N=1540 both-prompt-arms re-run on the Memori-protocol harness, ours-vs- previous two-proportion z-tests per category, gate = target category up significantly OR (up + nothing down >1.5pp). Plus LongMemEval N=100 spot-check after W2/W4 (knowledge- update + temporal-reasoning types) to confirm cross-benchmark transfer. Publish per-wave in benchmarks/results/.

6. Decision requested

Approve wave order? W1 is ~1 day and pure prompt; W2 is the substrate centerpiece (~2–4 days); W3 mostly wires existing dead code (~2–3 days); W4 is the long-pole architecture + production-parity arc (~1–2 weeks, product value beyond benchmarks).