64 lines
4.9 KiB
Markdown
64 lines
4.9 KiB
Markdown
# Pillar 2 — Memory SOTA (LongMemEval) — plan
|
||
|
||
> Companion to HARNESS-BENCHMARK-GOAL (two co-equal pillars). Pillar 1 (harness) done at N=40
|
||
> (Waggle on par with Hermes). Pillar 2 proves the **memory substrate** is at/near SOTA — the
|
||
> one axis where Waggle can *beat* frontier long-context, not just match.
|
||
|
||
## Why LongMemEval (upgrade from the C-1 LoCoMo run)
|
||
- LoCoMo (C-1, done): 67.8% trio-strict, +4.6pp over Mem0 paper. "Modest by 2026 standards."
|
||
- **LongMemEval**: 500 questions, **5 abilities** — info-extraction, multi-session reasoning, temporal
|
||
reasoning, **knowledge-updates**, **abstention** (the last two LoCoMo lacks). The credible, citable
|
||
successor. (BEAM = the unsaturated flagship; later.)
|
||
- SOTA reference: multi-session reasoning ~70% (Mem0); top systems ~94% overall. Mem0's own
|
||
numbers are the comparison anchor — but we run our own under matched protocol (no apples-to-oranges).
|
||
|
||
## Reusable infra (LoCoMo harness — `D:/Projects/hive-mind-test/scripts/locomo/`)
|
||
The full pipeline already exists and is directly adaptable:
|
||
| LoCoMo script | Role | LongMemEval adaptation |
|
||
|---|---|---|
|
||
| `00-fetch-dataset.mjs` | download + SHA-pin dataset | swap source → LongMemEval (HF `xiaowu0162/LongMemEval` / official release); LongMemEval_S (~115K tok, 40 sessions) |
|
||
| `01-prepare-workspace.mjs` | fresh hive-mind workspace | reuse; **per-question** haystack isolation (LongMemEval gives each Q its own session set) |
|
||
| `02b-ingest-all-convs.mjs` | ingest sessions → FrameStore | adapt to LongMemEval session schema |
|
||
| `03b-cognify-all.mjs` | memory extraction | reuse |
|
||
| `12-cell-retrieval.mjs` | recall top-K → subject model answers (via `cli/recall-context.js`) | reuse core; subject = Sonnet 4.6 AND Qwen 3.6 35B (local — `34-*-qwen` precedent) |
|
||
| `38-judge-trio-v5.mjs` + `38b-redo` | trio-strict judge | reuse; **add abstention handling** (judge must accept "I don't know" when gold = no-answer) |
|
||
| `14/24/34-report` | aggregate + report | reuse; report PER-ABILITY (5 categories) + overall |
|
||
|
||
## Adaptation specifics (the real work vs LoCoMo)
|
||
1. **Per-question haystacks.** LongMemEval associates each question with its own set of haystack
|
||
sessions (some relevant, most distractors). Either (a) one workspace per question (clean isolation,
|
||
500 ingests) or (b) one big workspace + per-question session-scope filter. (a) is cleaner/defensible.
|
||
2. **Abstention category.** Some questions have NO answer in the haystack — the system must abstain.
|
||
Retrieval + subject prompt must allow "not in memory"; judge must score abstention correctly.
|
||
3. **Knowledge-updates.** Facts change across sessions; retrieval must surface the LATEST. Tests the
|
||
substrate's bitemporal/recency handling (KnowledgeGraph validity).
|
||
4. **Token-cost reporting.** Per Mem0's 2026 framing, report accuracy AT a token budget (retrieval is
|
||
cheap vs full-context) — a Waggle advantage to surface.
|
||
|
||
## Two subject-model lanes (mirror Pillar 1)
|
||
- **Sonnet 4.6** (cloud) — the capability number.
|
||
- **Qwen 3.6 35B thinking** (LOCAL) — the sovereign number ([[project-pillar1-qwen-local-followup]]).
|
||
The `34-cell-retrieval-v4-qwen` + `35-judge-*-qwen` scripts prove the local-Qwen path already works.
|
||
|
||
## Dataset VERIFIED (2026-05-22)
|
||
`longmemeval_s_cleaned.json` — 264 MB, **SHA256 `d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442`**, at `D:/Projects/hive-mind-test/scripts/longmemeval/data/`. **500 questions.**
|
||
|
||
Schema per question: `question_id, question_type, question, question_date, answer, answer_session_ids, haystack_dates, haystack_session_ids, haystack_sessions`.
|
||
- `haystack_sessions` = list of sessions; each session = list of turns `{role, content}` (standard chat).
|
||
- **~53 sessions / question, ~127K tokens** of haystack (relevant + distractors). `answer_session_ids` marks the relevant session(s) → the retrieval target.
|
||
- **question_type dist (the 5 abilities):** multi-session 133 · temporal-reasoning 133 · knowledge-update 78 · single-session-user 70 · single-session-assistant 56 · single-session-preference 30. (Abstention is a separate `_abs` subset, not in these 500.)
|
||
|
||
**Ingestion scale:** N=50 probe ≈ 50×53 ≈ 2,650 session ingests; full 500 ≈ 26,500. Per-question isolated workspaces. This is the main cost (time, not $) — chunk it.
|
||
|
||
## Execution steps
|
||
1. **Fetch + SHA-pin LongMemEval** (adapt `00-fetch`). Confirm exact HF/GitHub source + schema.
|
||
2. Build sample (start small — N=50 across the 5 abilities — for a probe before the full 500).
|
||
3. Ingest haystacks → cognify (per-question workspaces).
|
||
4. Retrieval cell (Sonnet) → answers; then Qwen-local cell.
|
||
5. Trio-strict judge (+ abstention handling) → per-ability + overall.
|
||
6. Report vs Mem0 anchor under matched protocol. Scale 50 → 500 if the probe holds.
|
||
|
||
## Budget
|
||
Retrieval+judge is cheap (LoCoMo full was ~$26). N=50 probe ≈ a few $; full 500 ≈ ~$30-50 incl trio.
|
||
Far cheaper than the agentic pillar — and the higher-leverage "Waggle wins" claim.
|