Pillar 2 — Memory SOTA (LongMemEval) — plan

Companion to HARNESS-BENCHMARK-GOAL (two co-equal pillars). Pillar 1 (harness) done at N=40 (Waggle on par with Hermes). Pillar 2 proves the memory substrate is at/near SOTA — the one axis where Waggle can beat frontier long-context, not just match.

Why LongMemEval (upgrade from the C-1 LoCoMo run)

LoCoMo (C-1, done): 67.8% trio-strict, +4.6pp over Mem0 paper. "Modest by 2026 standards."
LongMemEval: 500 questions, 5 abilities — info-extraction, multi-session reasoning, temporal reasoning, knowledge-updates, abstention (the last two LoCoMo lacks). The credible, citable successor. (BEAM = the unsaturated flagship; later.)
SOTA reference: multi-session reasoning ~70% (Mem0); top systems ~94% overall. Mem0's own numbers are the comparison anchor — but we run our own under matched protocol (no apples-to-oranges).

Reusable infra (LoCoMo harness — `D:/Projects/hive-mind-test/scripts/locomo/`)

The full pipeline already exists and is directly adaptable:

LoCoMo script	Role	LongMemEval adaptation
`00-fetch-dataset.mjs`	download + SHA-pin dataset	swap source → LongMemEval (HF `xiaowu0162/LongMemEval` / official release); LongMemEval_S (~115K tok, 40 sessions)
`01-prepare-workspace.mjs`	fresh hive-mind workspace	reuse; per-question haystack isolation (LongMemEval gives each Q its own session set)
`02b-ingest-all-convs.mjs`	ingest sessions → FrameStore	adapt to LongMemEval session schema
`03b-cognify-all.mjs`	memory extraction	reuse
`12-cell-retrieval.mjs`	recall top-K → subject model answers (via `cli/recall-context.js`)	reuse core; subject = Sonnet 4.6 AND Qwen 3.6 35B (local — `34-*-qwen` precedent)
`38-judge-trio-v5.mjs` + `38b-redo`	trio-strict judge	reuse; add abstention handling (judge must accept "I don't know" when gold = no-answer)
`14/24/34-report`	aggregate + report	reuse; report PER-ABILITY (5 categories) + overall

Adaptation specifics (the real work vs LoCoMo)

Per-question haystacks. LongMemEval associates each question with its own set of haystack sessions (some relevant, most distractors). Either (a) one workspace per question (clean isolation, 500 ingests) or (b) one big workspace + per-question session-scope filter. (a) is cleaner/defensible.
Abstention category. Some questions have NO answer in the haystack — the system must abstain. Retrieval + subject prompt must allow "not in memory"; judge must score abstention correctly.
Knowledge-updates. Facts change across sessions; retrieval must surface the LATEST. Tests the substrate's bitemporal/recency handling (KnowledgeGraph validity).
Token-cost reporting. Per Mem0's 2026 framing, report accuracy AT a token budget (retrieval is cheap vs full-context) — a Waggle advantage to surface.

Two subject-model lanes (mirror Pillar 1)

Sonnet 4.6 (cloud) — the capability number.
Qwen 3.6 35B thinking (LOCAL) — the sovereign number (project-pillar1-qwen-local-followup). The 34-cell-retrieval-v4-qwen + 35-judge-*-qwen scripts prove the local-Qwen path already works.

Dataset VERIFIED (2026-05-22)

longmemeval_s_cleaned.json — 264 MB, SHA256 d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442, at D:/Projects/hive-mind-test/scripts/longmemeval/data/. 500 questions.

Schema per question: question_id, question_type, question, question_date, answer, answer_session_ids, haystack_dates, haystack_session_ids, haystack_sessions.

haystack_sessions = list of sessions; each session = list of turns {role, content} (standard chat).
~53 sessions / question, ~127K tokens of haystack (relevant + distractors). answer_session_ids marks the relevant session(s) → the retrieval target.
question_type dist (the 5 abilities): multi-session 133 · temporal-reasoning 133 · knowledge-update 78 · single-session-user 70 · single-session-assistant 56 · single-session-preference 30. (Abstention is a separate _abs subset, not in these 500.)

Ingestion scale: N=50 probe ≈ 50×53 ≈ 2,650 session ingests; full 500 ≈ 26,500. Per-question isolated workspaces. This is the main cost (time, not $) — chunk it.

Execution steps

Fetch + SHA-pin LongMemEval (adapt 00-fetch). Confirm exact HF/GitHub source + schema.
Build sample (start small — N=50 across the 5 abilities — for a probe before the full 500).
Ingest haystacks → cognify (per-question workspaces).
Retrieval cell (Sonnet) → answers; then Qwen-local cell.
Trio-strict judge (+ abstention handling) → per-ability + overall.
Report vs Mem0 anchor under matched protocol. Scale 50 → 500 if the probe holds.

Budget

Retrieval+judge is cheap (LoCoMo full was ~26). N=50 probe ≈ a few ; full 500 ≈ ~$30-50 incl trio. Far cheaper than the agentic pillar — and the higher-leverage "Waggle wins" claim.

4.9 KiB Raw Blame History Unescape Escape