moving

benchmarks/results/locomo-sota-2026-06/INDEX.md

@@ -0,0 +1,54 @@
+# Hive Mind — LoCoMo SOTA results index (single source of truth)
+
+> Tracked, in-repo, reproducible. **Headline = 86.49%** (7-lane W4, N=1540, same-judge vs Memori).
+> Verify: `node recount.mjs`. Supersedes the non-reproducible 87.66% (see correction note below).
+
+**Headline:** On LoCoMo under Memori's own protocol (gpt-4.1-mini answerer + judge, N=1540),
+Hive Mind = **86.49% overall, +4.54pp over Memori 81.95 (z=4.64, p<10⁻⁵)**, leading/tying every
+category. Two same-judge head-to-heads anchor it (Memori + Mem0). Fully local substrate.
+
+## Correction note (2026-07-01)
+The prior headline **87.66%** (2026-06-11 report) does **not reproduce**. Fresh 7-lane W4 + fresh
+judge = **85.19%** on its own archived 2026-06-11 substrate, **86.49%** on the current substrate.
+Cause: stale-verdict-replay bug (harness note 2026-06-15) inflated the original judge pass; that
+1350-correct judgment set is lost. Adopted 86.49% as canonical (founder decision 2026-07-01).
+
+## The numbers (gpt-4.1-mini answerer+judge, N=1540)
+
+### 7-lane W4 vs Memori — same-judge
+| Category | Ours | Memori (pub) | Δ |
+|---|--:|--:|--:|
+| single-hop | 92.27 | 87.87 | +4.40 |
+| multi-hop | 80.50 | 72.70 | +7.80 |
+| temporal | 81.62 | 80.37 | +1.25 |
+| open-ended | 69.79 | 63.54 | +6.25 |
+| **overall** | **86.49** | **81.95** | **+4.54 (z=4.64, p<10⁻⁵)** |
+
+### Mem0 — same-judge, our ruler
+Ours 86.49 vs Mem0 73.96 = **+12.53pp** overall; temporal 81.62 vs 50.78 = **+30.84pp**
+(write-time dating vs ingestion-time). Mem0 judgments 1139/1540 committed.
+
+### Token efficiency
+Prior Config-D knee (−28% tokens for ~−1pp) was measured on an earlier answer set — **re-measure
+on the current substrate before re-citing**. Do not carry the old 87.66-anchored Pareto numbers.
+
+## Where everything lives
+
+### Committed + reproducible in THIS monorepo
+- `benchmarks/results/locomo-sota-2026-06/` — **this dir**: report + INDEX + `recount.mjs` +
+  `data/{answers,judgments}/…N1540.jsonl` (the pinned raw evidence, recount = 1332/1540).
+- Substrate code: `packages/hive-mind-core/src/mind/{inprocess-reranker,search,resolve-relative-date,parse-date-window,raw-detail-lane,recall-context}.ts`.
+
+### Reproduction harness (`hive-mind-test/scripts/locomo`)
+`40-cell-retrieval-gpt41mini.mjs` (7-lane: PROFILES/DATEWIN/EPISODIC/RAWDETAIL), `41-judge-memori-gpt41mini.mjs`,
+`42-report-memori.mjs`. **Always fresh `OUT_TAG`** to avoid stale-verdict replay.
+
+### OSS public (`marolinik/hive-mind`)
+Substrate code @ `bc4eba1` (PR #14). Benchmark results being updated 73.1% → 86.49% (this arc).
+
+## Gotchas (cost real time)
+- **Stale-verdict replay:** `41-judge` resumes by row count on the OUT_FILE → reusing a judgments
+  file replays old verdicts. This inflated 87.66. Use a fresh `OUT_TAG` every run.
+- `42-report` tokens/query line is a display bug; real tokens = `context_tokens` in answer rows.
+- Mem0 shares `~/.mem0/migrations_qdrant` lock → run per-conv.
+- Zep + LangMem stay literature-sourced (founder 2026-06-16).

benchmarks/results/locomo-sota-2026-06/LOCOMO-SOTA-86.49-vs-Memori.md

@@ -0,0 +1,59 @@
+# LoCoMo Memory-SOTA — 86.49% (7-lane W4, N=1540, same-judge vs Memori)
+
+**Reproduced & pinned 2026-07-01.** Subject = gpt-4.1-mini, judge = gpt-4.1-mini + Memori's
+verbatim "be generous" ACCURACY_PROMPT, natural full-N distribution, overall = count-weighted
+micro-average. Full 7-lane W4 stack (distilled + semantic + importance-K5 + episodic + profiles +
+date-window + raw-detail/CE-rerank), uncapped.
+
+> **Correction note.** The original 2026-06-11 report claimed **87.66%**. That number did **not
+> reproduce** on a fresh judge pass — on its own archived 2026-06-11 substrate it re-scores 85.19%,
+> and on the current substrate 86.49% — consistent with the stale-verdict-replay bug documented in
+> the harness on 2026-06-15 (`judge resumes by question_id and replayed stale verdicts`). The
+> original 1350-correct judgment set is lost and unreproducible. **86.49% is the honest, fresh,
+> reproducible figure** (verify with `node recount.mjs`). See
+> `docs/analysis/locomo-87.66-vs-85.26-integrity-2026-06-30.md`.
+
+## Head-to-head vs Memori Table 1 (N=1540)
+
+| Category | Memori (Table 1) | **Our substrate** (gpt-4.1-mini) | Δ vs Memori | n |
+|---|--:|--:|--:|--:|
+| single-hop | 87.87 | **92.27%** (776/841) | +4.40pp | 841 |
+| multi-hop | 72.70 | **80.50%** (227/282) | +7.80pp | 282 |
+| temporal | 80.37 | **81.62%** (262/321) | +1.25pp | 321 |
+| open-ended | 63.54 | **69.79%** (67/96) | +6.25pp | 96 |
+| **Overall** (count-weighted) | **81.95** | **86.49%** (1332/1540) | **+4.54pp** | 1540 |
+
+One-sample proportion test vs Memori's fixed 81.95%: **z = 4.64, p < 10⁻⁵** (one-sided).
+Memori baseline reproduced first on our own harness: nb02 = 81.98 vs published 81.95 (ruler validated).
+
+## Competitor re-run — Mem0 (same answerer + judge, our ruler)
+
+| Category | **Ours (7-lane W4)** | **Mem0 (our ruler)** | Δ |
+|---|--:|--:|--:|
+| single-hop | 92.27 | 83.59 | +8.68 |
+| multi-hop | 80.50 | 74.82 | +5.68 |
+| temporal | 81.62 | **50.78** | **+30.84** |
+| open-ended | 69.79 | 64.58 | +5.21 |
+| **overall** | **86.49** | **73.96** | **+12.53** |
+
+Lead every category on Mem0's own protocol; temporal by +30.84pp — write-time-dated episodic
+timeline vs Mem0's ingestion-time stamping. (Mem0 judgments: 1139/1540 = 73.96%, committed.)
+
+## Tokens/query
+avg context_tokens ≈ **3,100** on this run (per-row `context_tokens` in the answers file; the
+`42-report` tokens/query line is a known display bug — ignore it). The token-Pareto knee (Config D,
+−28% tokens for ~−1pp) was measured on a prior answer set and should be **re-measured on this
+substrate** before re-citing exact numbers.
+
+## Reproduce
+- **Offline (zero API):** `node recount.mjs` — recounts `data/judgments/…-N1540.jsonl` → 1332/1540.
+- **Full regen:** in `hive-mind-test/scripts/locomo`,
+  `PROMPT_MODE=ours PROFILES=1 DATEWIN=1 EPISODIC=1 RAWDETAIL=1 OUT_TAG=<fresh> node 40-cell-retrieval-gpt41mini.mjs`
+  then `41-judge --in=…-<fresh>.jsonl`. **Always use a fresh `OUT_TAG`** — reusing an existing
+  judgments file triggers the stale-verdict replay that produced the bogus 87.66.
+
+## Provenance (pinned together — this is the fix for the drift)
+- Answers: `data/answers/locomo-7lane-w4-answers-N1540.jsonl` (7-lane: raw_detail≈16, importance≈4.5).
+- Judgments: `data/judgments/locomo-7lane-w4-judgments-N1540.jsonl` (recount = 1332/1540).
+- Substrate: 10 LoCoMo workspace minds (`~/.hive-mind/workspaces/proj-locomo-*`), current (2026-06-29) build.
+- Zep + LangMem stay literature-sourced (founder decision 2026-06-16).

benchmarks/results/locomo-sota-2026-06/README.md

@@ -0,0 +1,52 @@
+# LoCoMo Memory-SOTA — canonical evidence (86.49%, reproducible + pinned)
+
+**Headline (defensible, same-judge, reproducible):** On LoCoMo under Memori's own published
+protocol (gpt-4.1-mini answerer **and** judge, verbatim "be generous" ACCURACY_PROMPT, natural
+full distribution, N=1,540), the Hive Mind 7-lane W4 substrate scores **86.49% overall
+(1,332/1,540), +4.54pp over Memori's 81.95%** (one-sample z=4.64, p<10⁻⁵), leading every category.
+Verify offline with `node recount.mjs`.
+
+> **Supersedes the 87.66% claim.** 87.66% was the 2026-06-11 report; it does not reproduce on a
+> fresh judge pass (85.19% on its own archived substrate, 86.49% on the current one) and was
+> inflated by the stale-verdict-replay bug the harness documented on 2026-06-15. 86.49% is the
+> honest, fresh, substrate-pinned number. Full story:
+> `docs/analysis/locomo-87.66-vs-85.26-integrity-2026-06-30.md`.
+
+## Contents
+| File | What |
+|---|---|
+| `LOCOMO-SOTA-86.49-vs-Memori.md` | Canonical report — Memori head-to-head + Mem0 + reproduction. |
+| `recount.mjs` | Offline verification (zero API): recounts the judgments → 1332/1540 = 86.49%. |
+| `INDEX.md` | Single-source-of-truth index: every number + where each artifact lives. |
+| `data/answers/…N1540.jsonl` | The 7-lane W4 per-question answers (raw_detail≈16, importance≈4.5). |
+| `data/judgments/…N1540.jsonl` | The per-question judge verdicts — recount source of the 86.49%. |
+
+## Per-category (7-lane W4, N=1,540, gpt-4.1-mini answerer+judge)
+| Category | Ours | Memori (pub) | Δ | n |
+|---|--:|--:|--:|--:|
+| single-hop | 92.27 | 87.87 | +4.40 | 841 |
+| multi-hop | 80.50 | 72.70 | +7.80 | 282 |
+| temporal | 81.62 | 80.37 | +1.25 | 321 |
+| open-ended | 69.79 | 63.54 | +6.25 | 96 |
+| **overall** | **86.49** | **81.95** | **+4.54** | 1540 |
+
+vs Mem0 (same-judge, our ruler): **86.49 vs 73.96, +12.53pp overall; temporal +30.84pp.**
+
+## Substrate code that produces this (in this monorepo)
+All under `packages/hive-mind-core/src/mind/`: `inprocess-reranker.ts`, `search.ts` (reranker
+wiring), `resolve-relative-date.ts` + `parse-date-window.ts` (write-time temporal dating),
+`raw-detail-lane.ts`, `recall-context.ts`. 7 lanes = distilled + semantic + importance-K5 +
+episodic + profiles + date-window + raw-detail (CE-reranked).
+
+## Reproduce
+- **Offline (zero API):** `node recount.mjs` → asserts 1332/1540 per-category.
+- **Full regen** (in `hive-mind-test/scripts/locomo`):
+  `PROMPT_MODE=ours PROFILES=1 DATEWIN=1 EPISODIC=1 RAWDETAIL=1 OUT_TAG=<fresh> node 40-cell-retrieval-gpt41mini.mjs`
+  → `41-judge --in=…-<fresh>.jsonl`. **Use a fresh `OUT_TAG`** (reusing a judgments file replays
+  stale verdicts — that is what produced the bogus 87.66).
+
+## Why this directory exists
+The 86.49% claim previously lived only on local disk (git-ignored by `**/benchmarks/results/*`) +
+the throwaway `hive-mind-test` repo, and its *number* had drifted from its *evidence*. This dir
+pins report + answers + judgments + a recount check together, git-tracked via a `.gitignore`
+negation exception. Do **not** remove that exception.

benchmarks/results/locomo-sota-2026-06/recount.mjs

@@ -0,0 +1,44 @@
+#!/usr/bin/env node
+// Offline verification of the LoCoMo SOTA headline — ZERO API calls.
+// Recounts the committed judgments and asserts the overall + per-category tallies
+// match the report. This is the regression baseline: if the substrate or scoring
+// changes, re-generate and update EXPECT. Run: `node recount.mjs`
+import { readFileSync } from 'node:fs';
+import { dirname, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const __dirname = dirname(fileURLToPath(import.meta.url));
+const JF = resolve(__dirname, 'data/judgments/locomo-7lane-w4-judgments-N1540.jsonl');
+
+// Expected tallies — 7-lane W4, current substrate, fresh gpt-4.1-mini judge (2026-07-01).
+const EXPECT = {
+  overall:      [1332, 1540],
+  'single-hop': [776, 841],
+  'multi-hop':  [227, 282],
+  temporal:     [262, 321],
+  'open-ended': [67, 96],
+};
+
+const rows = readFileSync(JF, 'utf8').trim().split('\n').filter(Boolean).map(l => JSON.parse(l));
+let n = 0, c = 0; const cat = {};
+for (const r of rows) {
+  n++; const cl = r.category_label || '?';
+  (cat[cl] ??= [0, 0])[1]++;
+  if (String(r.verdict).trim() === '1') { c++; cat[cl][0]++; }
+}
+
+const p0 = 0.8195, ph = c / n, z = (ph - p0) / Math.sqrt(p0 * (1 - p0) / n);
+console.log(`overall ${c}/${n} = ${(100 * ph).toFixed(2)}%   z=${z.toFixed(2)} vs Memori 81.95% (+${(100 * (ph - p0)).toFixed(2)}pp)`);
+
+let fail = false;
+for (const [k, [ec, en]] of Object.entries(EXPECT)) {
+  const got = k === 'overall' ? [c, n] : cat[k];
+  if (!got || got[0] !== ec || got[1] !== en) {
+    console.error(`  MISMATCH ${k}: got ${got ? got.join('/') : 'none'} expected ${ec}/${en}`);
+    fail = true;
+  } else {
+    console.log(`  OK ${k.padEnd(11)} ${ec}/${en} = ${(100 * ec / en).toFixed(2)}%`);
+  }
+}
+if (fail) { console.error('\nRECOUNT FAILED'); process.exit(1); }
+console.log('\nRECOUNT OK — committed judgments reproduce 86.49%.');