4.9 KiB
Pillar 2 — Memory SOTA (LongMemEval) — plan
Companion to HARNESS-BENCHMARK-GOAL (two co-equal pillars). Pillar 1 (harness) done at N=40 (Waggle on par with Hermes). Pillar 2 proves the memory substrate is at/near SOTA — the one axis where Waggle can beat frontier long-context, not just match.
Why LongMemEval (upgrade from the C-1 LoCoMo run)
- LoCoMo (C-1, done): 67.8% trio-strict, +4.6pp over Mem0 paper. "Modest by 2026 standards."
- LongMemEval: 500 questions, 5 abilities — info-extraction, multi-session reasoning, temporal reasoning, knowledge-updates, abstention (the last two LoCoMo lacks). The credible, citable successor. (BEAM = the unsaturated flagship; later.)
- SOTA reference: multi-session reasoning ~70% (Mem0); top systems ~94% overall. Mem0's own numbers are the comparison anchor — but we run our own under matched protocol (no apples-to-oranges).
Reusable infra (LoCoMo harness — D:/Projects/hive-mind-test/scripts/locomo/)
The full pipeline already exists and is directly adaptable:
| LoCoMo script | Role | LongMemEval adaptation |
|---|---|---|
00-fetch-dataset.mjs |
download + SHA-pin dataset | swap source → LongMemEval (HF xiaowu0162/LongMemEval / official release); LongMemEval_S (~115K tok, 40 sessions) |
01-prepare-workspace.mjs |
fresh hive-mind workspace | reuse; per-question haystack isolation (LongMemEval gives each Q its own session set) |
02b-ingest-all-convs.mjs |
ingest sessions → FrameStore | adapt to LongMemEval session schema |
03b-cognify-all.mjs |
memory extraction | reuse |
12-cell-retrieval.mjs |
recall top-K → subject model answers (via cli/recall-context.js) |
reuse core; subject = Sonnet 4.6 AND Qwen 3.6 35B (local — 34-*-qwen precedent) |
38-judge-trio-v5.mjs + 38b-redo |
trio-strict judge | reuse; add abstention handling (judge must accept "I don't know" when gold = no-answer) |
14/24/34-report |
aggregate + report | reuse; report PER-ABILITY (5 categories) + overall |
Adaptation specifics (the real work vs LoCoMo)
- Per-question haystacks. LongMemEval associates each question with its own set of haystack sessions (some relevant, most distractors). Either (a) one workspace per question (clean isolation, 500 ingests) or (b) one big workspace + per-question session-scope filter. (a) is cleaner/defensible.
- Abstention category. Some questions have NO answer in the haystack — the system must abstain. Retrieval + subject prompt must allow "not in memory"; judge must score abstention correctly.
- Knowledge-updates. Facts change across sessions; retrieval must surface the LATEST. Tests the substrate's bitemporal/recency handling (KnowledgeGraph validity).
- Token-cost reporting. Per Mem0's 2026 framing, report accuracy AT a token budget (retrieval is cheap vs full-context) — a Waggle advantage to surface.
Two subject-model lanes (mirror Pillar 1)
- Sonnet 4.6 (cloud) — the capability number.
- Qwen 3.6 35B thinking (LOCAL) — the sovereign number (project-pillar1-qwen-local-followup).
The
34-cell-retrieval-v4-qwen+35-judge-*-qwenscripts prove the local-Qwen path already works.
Dataset VERIFIED (2026-05-22)
longmemeval_s_cleaned.json — 264 MB, SHA256 d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442, at D:/Projects/hive-mind-test/scripts/longmemeval/data/. 500 questions.
Schema per question: question_id, question_type, question, question_date, answer, answer_session_ids, haystack_dates, haystack_session_ids, haystack_sessions.
haystack_sessions= list of sessions; each session = list of turns{role, content}(standard chat).- ~53 sessions / question, ~127K tokens of haystack (relevant + distractors).
answer_session_idsmarks the relevant session(s) → the retrieval target. - question_type dist (the 5 abilities): multi-session 133 · temporal-reasoning 133 · knowledge-update 78 · single-session-user 70 · single-session-assistant 56 · single-session-preference 30. (Abstention is a separate
_abssubset, not in these 500.)
Ingestion scale: N=50 probe ≈ 50×53 ≈ 2,650 session ingests; full 500 ≈ 26,500. Per-question isolated workspaces. This is the main cost (time, not $) — chunk it.
Execution steps
- Fetch + SHA-pin LongMemEval (adapt
00-fetch). Confirm exact HF/GitHub source + schema. - Build sample (start small — N=50 across the 5 abilities — for a probe before the full 500).
- Ingest haystacks → cognify (per-question workspaces).
- Retrieval cell (Sonnet) → answers; then Qwen-local cell.
- Trio-strict judge (+ abstention handling) → per-ability + overall.
- Report vs Mem0 anchor under matched protocol. Scale 50 → 500 if the probe holds.
Budget
Retrieval+judge is cheap (LoCoMo full was ~26). N=50 probe ≈ a few ; full 500 ≈ ~$30-50 incl trio.
Far cheaper than the agentic pillar — and the higher-leverage "Waggle wins" claim.