# Memory SOTA Proposal — 2026-06-10

**Status:** PROPOSAL (no code). Research basis: 6-agent workflow (substrate map, failure
mining of 1,540 judged answers × 2 arms, Zep/Graphiti, LangMem, academic survey
2024–2026, open-domain deep-dive). Constraints: **fully local** (SQLite + sqlite-vec,
Ollama embeddings, in-process ONNX cross-encoder, optional local LLM via Ollama),
**personal + workspace minds preserved**, OSS subtree-split clean.

---

## 0. THE RE-BASELINE — we were chasing a phantom (read this first)

The Memori paper's baseline rows are **column-scrambled**. Its Table 1 says baselines
were "retrieved from Du et al. [2025]" (= MemR3, arXiv:2512.20237). MemR3's column
order is `Multi | Temporal | Open | Single`; Memori printed the same values under
`Single | Multi | Open | Temporal`. Verified by extracting both PDFs + MemR3's §C.3
("existing works have misaligned category labels") + Memori's own self-contradictory
narrative ("72.70 trailing 61.06").

**Audited our harness: our labels are CORRECT** (temporal n=321 with "When did..."
questions, open n=96, multi n=282, single n=841 — exact canon counts + semantic
spot-checks pass). Our numbers stand. The *competitor* numbers move:

### Corrected LoCoMo landscape (GPT-4.1-mini protocol)

| Category | FC ceiling | MemR3* | **Ours (P4)** | Memori | Zep† | LangMem† | Mem0† |
|---|---|---|---|---|---|---|---|
| Temporal | 86.82 | 82.14 | **80.06** ⭐ | 80.37 | 77.26 | 61.06 | 57.32 |
| Open-domain | 71.88 | 71.53 | 60.42 ❌ | 63.54 | 64.58 | **67.71** | 44.79 |
| Single-hop | 93.73 | 92.17 | **88.59** ⭐ | 87.87 | 83.49 | 86.92 | 66.47 |
| Multi-hop | 86.43 | 81.20 | **79.43** ⭐ | 72.70 | 72.34 | 74.47 | 62.41 |
| **Overall** | — | — | **83.38** ⭐ | 81.95 | — | 78.05 | 62.47 |

\* MemR3 = agentic reflective-retrieval pipeline (different class, not a memory system).
† Corrected per MemR3 Table 1. Zep's self-published 83.33/73.96 don't match any MemR3
version — provenance unclear; cross-lab LoCoMo numbers are noisy, which makes our
**in-harness same-judge comparison the defensible standard**.

**Corrected verdict: we are ALREADY the leading memory system on overall, single-hop,
multi-hop, and (≈tied with Memori) temporal.** The phantom "LangMem temporal 86.92"
was LangMem's *single-hop* score; LangMem's real temporal is 61.06 — its extractor
never receives conversation timestamps (verified in its source), which validates our
write-time dating as the right design (+19pp over LangMem on temporal).

**The ONE real gap: open-domain 60.42** vs LangMem 67.71 / FC 71.88.
**The second axis: tokens** — 2,742/q vs Memori's 1,294.

Caveat: open-domain n=96 → SE ≈ ±5pp; deltas <8pp are noise-adjacent. All wave gates
below use two-proportion z-tests on full N (no mid-run proxies — they burned us twice).

---

## 1. What the failure data says (1,540 judged answers × 2 arms, mined)

### Open-domain (34 fails / 96)
- **16/34 are ABSTENTIONS** ("Not stated in the retrieved context") on speculative
  questions ("Would Caroline be considered religious?"). The protocol dropped the
  adversarial category, so abstention is a guaranteed zero. In 4 cases the *theirs*
  arm answered the same question correctly from the SAME substrate → prompt-induced.
- ~12 wrong inferences (persona signal too dispersed; vocabulary mismatch:
  "console" never co-occurs with "Xenoblade").
- ~3 counting errors from **episodic duplicates** (same hike narrated twice → "five" vs gold "four").
- Question type: ~50% persona/preference inference, ~25% world-knowledge bridging,
  ~10% entity-ID, ~15% aggregation. **It is persona synthesis + licensed speculation,
  not retrieval.** Fact-list systems (us, Memori) bottom out here; profile/summary
  systems (LangMem, Zep, MIRIX) lead.

### Temporal (71 fails / 321) — ~60% prompt-side, ~40% substrate
- **15 precision-miscalibration fails**: we emit a confident exact ISO date 1–7 days
  off where the judge accepts coarse answers — 22 of 31 ours-only fails PASSED in the
  theirs arm with "Early June 2023"-style granularity.
- **11 session-date echoes**: gold is "the week before <session date>"; our write-time
  resolution stamped the mention date (forward resolution exists, backward ranges don't).
- **18 wrong event bindings** (Tokyo vs Boston; reversed adoption order) — episodic
  duplicates + no date-window filtering at retrieval.
- **11 refusals** (dated event not retrieved), 6 duration fencepost errors.

### Multi-hop (71 fails) — dominant cluster: partial enumeration on cross-session
aggregation ("Which US cities...?" → returns 1 of 3) + duplicate-inflated counting.
### Single-hop (95 fails) — fine-grained detail lost by distillation (gold: "painting
inspired by sunsets with pink sky"; we retrieve the distilled "an abstract painting").

### Substrate map findings (production-relevant)
- **KnowledgeGraph contributes ZERO to recall** — `bfsDistances→contextual-score`
  wiring exists in scoring.ts but no caller passes graphDistances → 20% of the
  'balanced' relevance weight is permanently 0.
- **since/until SQL filters exist in HybridSearch + FrameStore — never called by anyone.**
- Production recall (orchestrator.ts) has NO reranker/chunking/distilled/episodic
  layers — those exist only in the OSS repo + benchmark harness. Production lags the
  benchmark substrate substantially.
- Scoring "temporal" dimension decays on `last_accessed` (access recency) — constant
  noise on a 2023 corpus, not event time.

---

## 2. The proposal — four waves, each gated by a full-N z-tested re-run

### WAVE 1 — Answer-policy fixes (prompt-only, zero substrate risk, ~1 day)
Targets the measured prompt tax. No regression risk to the substrate.
1. **Conditional abstention**: speculative/inferential questions ("would/might/could/
   likely") → forbid refusal, force committed best-effort inference from retrieved
   evidence + world knowledge. Factual questions keep abstention (production safety).
   *Evidence: 16 guaranteed-zero abstentions; judge demonstrably accepts directional guesses.*
2. **Granularity-calibrated dates**: emit exact day ONLY when explicitly stated;
   otherwise answer at week/month granularity ("early June 2023").
   *Evidence: 22 ours-only temporal fails passed in theirs arm with coarser answers.*
3. **Duration brevity + endpoint few-shot**: final value only (verbose multi-date
   reasoning triggers harsh judging); fencepost examples.
4. **Commit-to-one-option**: forbid hedged dual answers ("both") on either/or questions.
5. **Parametric-knowledge gating** (arXiv:2510.23730): for world-knowledge-bridging
   questions, instruct "combine retrieved facts with general world knowledge" —
   retrieval-only instructions measurably suppress the model's own knowledge (FC 56.4
   vs RAG 49.5 F1 on this category).

**Expected: open +6–10pp, temporal +4–6pp, overall → ~85.** Cost: ~$6 re-run.

### WAVE 2 — Profile cards + episodic hygiene (write-time substrate, local LLM)
The dominant open-domain lever, converging from three independent sources (Zep entity
summaries, LangMem profiles, MIRIX core-memory; all profile-carrying systems lead this
category).
1. **Per-speaker rolling profile cards**: ~500-char abstractive profile per
   speaker/entity, updated incrementally at ingest (Graphiti fast-path: append facts
   without LLM call while under cap; consolidate-compress via Ollama when over).
   Rendered as a "PERSONA" block in context. Per-scope (personal + per-workspace) —
   isomorphic with our existing split.
2. **Episodic event canonicalization**: dedup same-event-renarrated rows at ingest
   (fixes counting failures in open + multi).
3. **Event-date RANGES**: backward resolution for retrospective narration — store
   `[event_date_min, event_date_max]` + mention date ("last week" → 7-day window),
   render ranges; answer at range granularity (pairs with Wave-1 #2).

**Expected: open → ≥70 (combined with Wave 1), temporal +2–3pp, tokens −10–20%**
(one profile card replaces many weak-signal facts).

### WAVE 3 — Retrieval lanes (query-time; mostly wiring existing code)
1. **Temporal retrieval lane** (MRAG arXiv:2412.15540 / Hindsight TEMPR): parse the
   query's temporal constraint (deterministic, extends resolve-relative-date.ts to
   query side) → **pass the already-existing-but-never-called since/until filters** →
   add a date-window lane into RRF fusion before the cross-encoder. MRAG: +9.3% top-1
   recall on temporal QA.
2. **Entity-keyed exhaustive retrieval** for enumeration/counting questions: pull ALL
   episodic rows for the focal entity (not top-K), dedup-by-event before answering.
   *Targets the dominant multi-hop cluster (~16/24 sampled fails).*
3. **Raw-detail escalation lane**: when the question asks for concrete perceptual
   detail and distilled facts match only generically, fetch the raw session turn
   around the matching fact (256-token chunks; sqlite-vec + FTS — our existing stack).
   *Targets the dominant single-hop cluster.*
4. **Wire graphDistances into scoring** (the dead 20% weight) + BFS-expansion lane
   self-seeded from search-hit entities (depth ≤2, recursive CTE — sub-ms at our scale).

**Expected: temporal → ~84–86 (FC ceiling is 86.82), multi → ~82–84, single → ~90.**

### WAVE 4 — Bi-temporal substrate + production parity (architecture; product-first)
1. **Bi-temporal validity on facts** (Zep model): `valid_at/invalid_at` (event time) +
   `created_at/expired_at` (transaction time); facts never deleted, only closed.
2. **Ingest-time invalidation**: one small-Ollama call per new fact against same-entity
   + RRF-similar existing facts → `duplicate[]`/`contradicted[]`; a deterministic
   temporal-overlap rule does the actual invalidation (LLM proposes, arithmetic disposes).
   Gives latest-wins for "what is X now" while preserving "what was true then" —
   fixes the knowledge-update losses we measured on LongMemEval too.
3. **Production parity**: port the benchmark-proven stack (reranker, distilled facts,
   episodic dated events, date rendering, TEMPORAL_GUIDANCE, profile cards) into the
   production recall path (orchestrator.ts) — production currently has none of it.
4. **Token-budget context packing** (Zep's 1.6k-token block beats 115k full-context):
   target ≤1,500 tokens/q — closes the efficiency gap with Memori while raising scores.

**Expected: durable product wins beyond LoCoMo; tokens → ~1.5k/q.**

---

## 3. Projected end-state (honest ranges, ±noise)

| Category | Now | After W1 | After W2 | After W3/W4 | FC ceiling |
|---|---|---|---|---|---|
| Temporal | 80.06 | ~84 | ~85 | **85–87** | 86.82 |
| Open-domain | 60.42 | ~67 | **70–74** | 72–75 | 71.88 |
| Single-hop | 88.59 | 88.5 | ~89 | **90–92** | 93.73 |
| Multi-hop | 79.43 | ~80 | ~81 | **82–84** | 86.43 |
| **Overall** | **83.38** | **~85.3** | **~86.3** | **87–88.5** | — |
| tokens/q | 2,742 | 2,742 | ~2,300 | **≤1,500** | — |

At ~86–88 overall we'd clear every published memory system by a decisive (significant)
margin and approach Hindsight's local-model result (85.67 w/ GPT-OSS-20B) — whose
architecture (same CE reranker + temporal lane + observation summaries + graph lane)
independently validates this exact roadmap, with local open-weight models.

## 4. Anti-goals (lessons paid for)
- **Never strip write-time dating** (LangMem's undated extraction = its 61.06 temporal).
- **No relevance-ranked-only episodic** (P5 proved top-K filtering reverts the timeline-
  scaffold benefit; keep the chronological block, dedup it instead).
- **No agentic multi-turn retrieval loops in production hot path** (LangMem p95 ~60s).
  MemR3-style reflection is benchmark-viable but a latency hazard; defer, lane-gate.
- **Keep abstention for factual unknowns in production** (conditional policy only
  loosens speculative questions; adversarial robustness must not regress).
- **No Neo4j / no cloud** — everything above is SQLite tables + Ollama + in-process ONNX.

## 5. Verification protocol
Each wave: full N=1540 both-prompt-arms re-run on the Memori-protocol harness, ours-vs-
previous two-proportion z-tests per category, gate = target category up significantly
OR (up + nothing down >1.5pp). Plus LongMemEval N=100 spot-check after W2/W4 (knowledge-
update + temporal-reasoning types) to confirm cross-benchmark transfer. Publish per-wave
in benchmarks/results/.

## 6. Decision requested
Approve wave order? W1 is ~1 day and pure prompt; W2 is the substrate centerpiece
(~2–4 days); W3 mostly wires existing dead code (~2–3 days); W4 is the long-pole
architecture + production-parity arc (~1–2 weeks, product value beyond benchmarks).