moving

docs/plans/PILLAR2-MEMORY-LONGMEMEVAL-PLAN-2026-05-22.md

@@ -0,0 +1,63 @@
+# Pillar 2 — Memory SOTA (LongMemEval) — plan
+
+> Companion to HARNESS-BENCHMARK-GOAL (two co-equal pillars). Pillar 1 (harness) done at N=40
+> (Waggle on par with Hermes). Pillar 2 proves the **memory substrate** is at/near SOTA — the
+> one axis where Waggle can *beat* frontier long-context, not just match.
+
+## Why LongMemEval (upgrade from the C-1 LoCoMo run)
+- LoCoMo (C-1, done): 67.8% trio-strict, +4.6pp over Mem0 paper. "Modest by 2026 standards."
+- **LongMemEval**: 500 questions, **5 abilities** — info-extraction, multi-session reasoning, temporal
+  reasoning, **knowledge-updates**, **abstention** (the last two LoCoMo lacks). The credible, citable
+  successor. (BEAM = the unsaturated flagship; later.)
+- SOTA reference: multi-session reasoning ~70% (Mem0); top systems ~94% overall. Mem0's own
+  numbers are the comparison anchor — but we run our own under matched protocol (no apples-to-oranges).
+
+## Reusable infra (LoCoMo harness — `D:/Projects/hive-mind-test/scripts/locomo/`)
+The full pipeline already exists and is directly adaptable:
+| LoCoMo script | Role | LongMemEval adaptation |
+|---|---|---|
+| `00-fetch-dataset.mjs` | download + SHA-pin dataset | swap source → LongMemEval (HF `xiaowu0162/LongMemEval` / official release); LongMemEval_S (~115K tok, 40 sessions) |
+| `01-prepare-workspace.mjs` | fresh hive-mind workspace | reuse; **per-question** haystack isolation (LongMemEval gives each Q its own session set) |
+| `02b-ingest-all-convs.mjs` | ingest sessions → FrameStore | adapt to LongMemEval session schema |
+| `03b-cognify-all.mjs` | memory extraction | reuse |
+| `12-cell-retrieval.mjs` | recall top-K → subject model answers (via `cli/recall-context.js`) | reuse core; subject = Sonnet 4.6 AND Qwen 3.6 35B (local — `34-*-qwen` precedent) |
+| `38-judge-trio-v5.mjs` + `38b-redo` | trio-strict judge | reuse; **add abstention handling** (judge must accept "I don't know" when gold = no-answer) |
+| `14/24/34-report` | aggregate + report | reuse; report PER-ABILITY (5 categories) + overall |
+
+## Adaptation specifics (the real work vs LoCoMo)
+1. **Per-question haystacks.** LongMemEval associates each question with its own set of haystack
+   sessions (some relevant, most distractors). Either (a) one workspace per question (clean isolation,
+   500 ingests) or (b) one big workspace + per-question session-scope filter. (a) is cleaner/defensible.
+2. **Abstention category.** Some questions have NO answer in the haystack — the system must abstain.
+   Retrieval + subject prompt must allow "not in memory"; judge must score abstention correctly.
+3. **Knowledge-updates.** Facts change across sessions; retrieval must surface the LATEST. Tests the
+   substrate's bitemporal/recency handling (KnowledgeGraph validity).
+4. **Token-cost reporting.** Per Mem0's 2026 framing, report accuracy AT a token budget (retrieval is
+   cheap vs full-context) — a Waggle advantage to surface.
+
+## Two subject-model lanes (mirror Pillar 1)
+- **Sonnet 4.6** (cloud) — the capability number.
+- **Qwen 3.6 35B thinking** (LOCAL) — the sovereign number ([[project-pillar1-qwen-local-followup]]).
+  The `34-cell-retrieval-v4-qwen` + `35-judge-*-qwen` scripts prove the local-Qwen path already works.
+
+## Dataset VERIFIED (2026-05-22)
+`longmemeval_s_cleaned.json` — 264 MB, **SHA256 `d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442`**, at `D:/Projects/hive-mind-test/scripts/longmemeval/data/`. **500 questions.**
+
+Schema per question: `question_id, question_type, question, question_date, answer, answer_session_ids, haystack_dates, haystack_session_ids, haystack_sessions`.
+- `haystack_sessions` = list of sessions; each session = list of turns `{role, content}` (standard chat).
+- **~53 sessions / question, ~127K tokens** of haystack (relevant + distractors). `answer_session_ids` marks the relevant session(s) → the retrieval target.
+- **question_type dist (the 5 abilities):** multi-session 133 · temporal-reasoning 133 · knowledge-update 78 · single-session-user 70 · single-session-assistant 56 · single-session-preference 30. (Abstention is a separate `_abs` subset, not in these 500.)
+
+**Ingestion scale:** N=50 probe ≈ 50×53 ≈ 2,650 session ingests; full 500 ≈ 26,500. Per-question isolated workspaces. This is the main cost (time, not $) — chunk it.
+
+## Execution steps
+1. **Fetch + SHA-pin LongMemEval** (adapt `00-fetch`). Confirm exact HF/GitHub source + schema.
+2. Build sample (start small — N=50 across the 5 abilities — for a probe before the full 500).
+3. Ingest haystacks → cognify (per-question workspaces).
+4. Retrieval cell (Sonnet) → answers; then Qwen-local cell.
+5. Trio-strict judge (+ abstention handling) → per-ability + overall.
+6. Report vs Mem0 anchor under matched protocol. Scale 50 → 500 if the probe holds.
+
+## Budget
+Retrieval+judge is cheap (LoCoMo full was ~$26). N=50 probe ≈ a few $; full 500 ≈ ~$30-50 incl trio.
+Far cheaper than the agentic pillar — and the higher-leverage "Waggle wins" claim.