docs: drop references to the removed packages (P2.2)

AGENTS.md records that the excluded packages left in P2.2. SECURITY.md
scopes memory to memory-contract and memory-mem0. THREAT_MODEL.md points
the injection scanner at @waggle/core, marks the MCP tool scope gate as
removed with the memory MCP servers, and drops the hive-mind copy of the
URL egress guard. docs/ARCHITECTURE.md lists the memory packages instead
of the hive-mind substrate. docs/methodology.md, the LoCoMo write-up of
the hive-mind engine whose reproduction steps pointed at the
maintainer's hive-mind repository, is removed.
This commit is contained in:
shumeiko.it
2026-11-01 00:50:06 +01:00
parent 9276723f25
commit 72a2a895a5
5 changed files with 21 additions and 313 deletions

View File

@@ -12,8 +12,8 @@
This branch turns the commercial Waggle OS codebase into an open-source **community edition**. This branch turns the commercial Waggle OS codebase into an open-source **community edition**.
The conversion is in progress. Memory runs on local mem0 through the memory contract (P1.5); The conversion is in progress. Memory runs on local mem0 through the memory contract (P1.5);
team mode stays in postponed packages, and the excluded memory packages remain in the team mode stays in postponed packages, and the excluded packages left the repository in P2.2.
repository, unbuilt, until P2.2. Work packages remove or replace the rest one by one. Work packages remove or replace the rest one by one.
Sources of truth, in order: Sources of truth, in order:
1. The agreed scope ("Waggle OSS Agreed Scope Annex", held by the project owner). 1. The agreed scope ("Waggle OSS Agreed Scope Annex", held by the project owner).
@@ -64,8 +64,7 @@ npm workspaces with the root `package-lock.json`. Node `^20.19.0 || >=22.12.0`.
| `packages/agent`, `core`, `server`, `shared`, `sdk`, `marketplace`, `cli`, `launcher` | Keep. | | `packages/agent`, `core`, `server`, `shared`, `sdk`, `marketplace`, `cli`, `launcher` | Keep. |
| `packages/memory-contract`, `packages/memory-mem0` | Keep. The memory contract and its mem0 implementation (ADR-0007). | | `packages/memory-contract`, `packages/memory-mem0` | Keep. The memory contract and its mem0 implementation (ADR-0007). |
| `packages/team-server`, `packages/waggle-dance`, `packages/worker` | Postponed: team mode stays in the repo, is not built or shipped (ADR-0003). | | `packages/team-server`, `packages/waggle-dance`, `packages/worker` | Postponed: team mode stays in the repo, is not built or shipped (ADR-0003). |
| `packages/hive-mind-*`, `memory-mcp`, `wiki-compiler` | Excluded. Nothing in scope imports them; they are not built or tested and leave in P2.2. | | `packages/hive-mind-*`, `memory-mcp`, `wiki-compiler`, `admin-web`, `apps/www` | Excluded; removed in P2.2. The old memory engine is maintained separately by the owner. |
| `packages/admin-web`, `apps/www` | Excluded. |
| `apps/browser-ext` | Not decided. Do not extend it. | | `apps/browser-ext` | Not decided. Do not extend it. |
| `docs/` | `docs/oss/` holds the community-edition records. The remaining guides, reference, architecture, getting-started and contributing pages predate this edition and are rewritten in P2.4. Internal history (`sidecar/`, `benchmarks/`, `judging/`, internal docs) was removed in P2.2. | | `docs/` | `docs/oss/` holds the community-edition records. The remaining guides, reference, architecture, getting-started and contributing pages predate this edition and are rewritten in P2.4. Internal history (`sidecar/`, `benchmarks/`, `judging/`, internal docs) was removed in P2.2. |

View File

@@ -43,7 +43,7 @@ In scope:
- The desktop app (Tauri shell + bundled sidecar), the web app, and the - The desktop app (Tauri shell + bundled sidecar), the web app, and the
workspace packages under `packages/` and `apps/`. workspace packages under `packages/` and `apps/`.
- The memory substrate (`packages/hive-mind-core`) and its MCP servers. - Memory (`packages/memory-contract`, `packages/memory-mem0`).
- Prompt-injection paths, secret handling (the vault), the filesystem boundary, - Prompt-injection paths, secret handling (the vault), the filesystem boundary,
and the capability/approval gates — see `THREAT_MODEL.md` for the details. and the capability/approval gates — see `THREAT_MODEL.md` for the details.

View File

@@ -31,7 +31,7 @@ override, prompt-extraction) and make it act against the operator's intent.
## Controls (implemented) ## Controls (implemented)
### 1. Injection scanning at every ingress — `scanForInjection` ### 1. Injection scanning at every ingress — `scanForInjection`
`packages/hive-mind-core/src/injection-scanner.ts` (re-exported via `packages/core/src/injection-scanner.ts` (re-exported via
`packages/agent/src/injection-scanner.ts`). Three pattern sets — role-override, `packages/agent/src/injection-scanner.ts`). Three pattern sets — role-override,
prompt-extraction, instruction-injection — produce a score; `safe = score < 0.3` prompt-extraction, instruction-injection — produce a score; `safe = score < 0.3`
(instruction-injection is weighted higher, 0.6, for `tool_output` context). It gates the (instruction-injection is weighted higher, 0.6, for `tool_output` context). It gates the
@@ -82,14 +82,9 @@ belong in the vault or a local `.env` (never committed). `.env.example` contains
names plus non-secret development defaults, never live credentials. Callers must names plus non-secret development defaults, never live credentials. Callers must
not persist secret values into prompts, logs, or memory frames. not persist secret values into prompts, logs, or memory frames.
### 6. MCP tool scope gate — `scope.ts` ### 6. MCP tool scope gate (removed)
`packages/memory-mcp/src/scope.ts` + `packages/hive-mind-mcp-server/src/scope.ts`. An The memory MCP servers this gate protected are not part of the community edition
external agent (Claude Code/Codex) granted stdio access to the memory substrate can be (ADR-0002), so no external agent gets stdio access to memory.
scoped **read-only** via `WAGGLE_MCP_SCOPES` / `HIVE_MIND_SCOPES`: a `memory:read` scope
registers only the read tools, so a read-only client literally cannot call
save/cleanup/ingest. Default (unset) stays full read+write for backward-compat;
`memory:write` implies `memory:read`. This keeps a poisoned or buggy external agent from
writing junk into the substrate.
### 7. Workspace filesystem boundary + secret deny — `file-store.ts` ### 7. Workspace filesystem boundary + secret deny — `file-store.ts`
`packages/core/src/file-store.ts`. Every FileStore op resolves the caller path and asserts `packages/core/src/file-store.ts`. Every FileStore op resolves the caller path and asserts
@@ -106,10 +101,7 @@ sandbox:** the deny is a curated list (it cannot enumerate every secret a home d
is deny-by-default with no per-workspace override yet. is deny-by-default with no per-workspace override yet.
### 8. SSRF egress guard — `url-egress-guard.ts` ### 8. SSRF egress guard — `url-egress-guard.ts`
`packages/agent/src/url-egress-guard.ts` (guarding `web_fetch` in `system-tools.ts`) and `packages/agent/src/url-egress-guard.ts` (guarding `web_fetch` in `system-tools.ts`). A URL named by untrusted
the structurally-identical `packages/hive-mind-core/src/harvest/url-egress-guard.ts`
(guarding `UrlAdapter.fetchAndParse`, reached by the MCP `ingest_source` url path in
`packages/{memory-mcp,hive-mind-mcp-server}/src/tools/ingest.ts`). A URL named by untrusted
content or a user is resolved to concrete IP(s); the fetch is **refused if any resolved content or a user is resolved to concrete IP(s); the fetch is **refused if any resolved
address is loopback / private (RFC1918 + CGNAT) / link-local / unique-local / multicast / address is loopback / private (RFC1918 + CGNAT) / link-local / unique-local / multicast /
reserved / unspecified**. This closes the **cloud instance-metadata** exfiltration path — reserved / unspecified**. This closes the **cloud instance-metadata** exfiltration path —
@@ -170,6 +162,4 @@ Coverage:
full closure needs IP-pinning (connect to the validated address) which `fetch`+HTTPS can't full closure needs IP-pinning (connect to the validated address) which `fetch`+HTTPS can't
do portably without breaking TLS SNI/cert validation. Also out of scope: the guard bounds do portably without breaking TLS SNI/cert validation. Also out of scope: the guard bounds
the *target* address, not response size/content, and does not defend a genuinely the *target* address, not response size/content, and does not defend a genuinely
public-but-malicious endpoint. The duplicated agent/hive-mind-core guard copies share one public-but-malicious endpoint.
spec and must be kept in sync (they cannot share a module — hive-mind-core is OSS-mirrored
and must not import `@waggle/agent`).

View File

@@ -2,11 +2,10 @@
Waggle is a monorepo with **28 packages** under `packages/` organized around a layered architecture: the memory substrate, agent intelligence, server API, and UI presentation. This document covers the package structure, data flow, and extension points. Waggle is a monorepo with **28 packages** under `packages/` organized around a layered architecture: the memory substrate, agent intelligence, server API, and UI presentation. This document covers the package structure, data flow, and extension points.
> **Note (2026-04-30 monorepo migration):** the persistent-memory substrate > **Note:** memory goes through `@waggle/memory-contract`, implemented on local
> (`mind/` + `harvest/`) moved out of `@waggle/core` into > mem0 by `@waggle/memory-mem0` (ADR-0007). The React UI is **not** a package — it
> `@waggle/hive-mind-core` (`packages/hive-mind-core/src/{mind,harvest}`). The > lives in `apps/web/src`. There is no `@waggle/ui` package. This document predates
> React UI is **not** a package — it lives in `apps/web/src`. There is no > the community edition and is rewritten in P2.4.
> `@waggle/ui` package.
## Package Overview ## Package Overview
@@ -14,7 +13,6 @@ Waggle is a monorepo with **28 packages** under `packages/` organized around a l
waggle-os/ waggle-os/
apps/ apps/
web/ # Main web app UI (React 19 + Vite + Tailwind 4 + base-ui/react) web/ # Main web app UI (React 19 + Vite + Tailwind 4 + base-ui/react)
www/ # Marketing site (Next.js)
browser-ext/ # Browser extension (unpacked; not an npm workspace) browser-ext/ # Browser extension (unpacked; not an npm workspace)
packages/ packages/
# Product packages (MIT) # Product packages (MIT)
@@ -30,40 +28,19 @@ waggle-os/
sdk/ # Plugin/skill SDK, capability packs, starter skills sdk/ # Plugin/skill SDK, capability packs, starter skills
cli/ # Command-line REPL cli/ # Command-line REPL
launcher/ # AI-tool launcher / dock backend launcher/ # AI-tool launcher / dock backend
admin-web/ # Admin dashboard for team deployments memory-contract/ # MemoryStore interface, record model, capability map
wiki-compiler/ # Knowledge / wiki compiler memory-mem0/ # MemoryStore on local mem0
memory-mcp/ # MCP server exposing the memory substrate to external agents
# Memory substrate — hive-mind-* (Apache-2.0), mirrored to marolinik/hive-mind
hive-mind-core/ # The substrate: FrameStore, HybridSearch, KnowledgeGraph, Identity/Awareness, Harvest (src/mind + src/harvest)
hive-mind-cli/ # CLI for the substrate
hive-mind-mcp-server/ # MCP server for the substrate
hive-mind-shim-core/ # Signal-emitter shim library
hive-mind-wiki-compiler/# Wiki compiler (OSS)
hive-mind-hooks-core/ # Shared hook library
hive-mind-hooks-*/ # Per-tool capture hooks: claude-code, claude-desktop, codex,
# codex-desktop, cursor, hermes, openclaw
sidecar/ # Node.js sidecar for Tauri desktop app sidecar/ # Node.js sidecar for Tauri desktop app
app/ # Tauri 2.0 desktop shell (Rust + WebView2) — loads the apps/web build app/ # Tauri 2.0 desktop shell (Rust + WebView2) — loads the apps/web build
``` ```
## Package Details ## Package Details
### @waggle/hive-mind-core ### @waggle/memory-contract and @waggle/memory-mem0
The persistent-memory substrate (Apache-2.0; mirrored to the public `MemoryStore` (remember, search, list, get, update, forget, history) with its record
[`marolinik/hive-mind`](https://github.com/marolinik/hive-mind) repo). Zero model and capability map, and its implementation on the open-source mem0 library,
network dependencies; runs on SQLite + sqlite-vec. running locally in the sidecar (ADR-0007).
- **MindDB** (`src/mind/db.ts`, `schema.ts`): SQLite wrapper for `.mind` files. Tables for memory frames, knowledge-graph entities/relations, embeddings, sessions, and improvement signals.
- **FrameStore** (`src/mind/frames.ts`): CRUD on memory frames with FTS5 full-text search, importance ranking, and access counting.
- **HybridSearch** (`src/mind/search.ts`): vector + keyword retrieval, with an optional cross-encoder reranker.
- **KnowledgeGraph** (`src/mind/knowledge.ts`): entity-relation graph with temporal validity (`valid_from`/`valid_to`).
- **IdentityLayer / AwarenessLayer** (`src/mind/identity.ts`, `awareness.ts`): personal-identity persistence and active task/state tracking.
- **Embeddings** (`src/mind/*-embedder.ts`): pluggable providers — in-process, Ollama, Voyage, OpenAI, mock.
- **Harvest** (`src/harvest/`): conversation/file ingestion adapters (ChatGPT, Claude, Claude Code, Gemini, Perplexity, PDF, markdown, URL, plaintext) plus the dedup pipeline.
> Develop the substrate **here** and mirror it out — never the reverse. See the
> "Memory Substrate Sync" section of the root [`CLAUDE.md`](../CLAUDE.md).
### @waggle/core ### @waggle/core
@@ -71,7 +48,7 @@ The foundation layer for the desktop/server runtime. Zero network dependencies.
- **WaggleConfig**: Configuration management (`~/.waggle/config.json`). Provider keys, default model, team server config. - **WaggleConfig**: Configuration management (`~/.waggle/config.json`). Provider keys, default model, team server config.
- **Vault**: AES-256-GCM encrypted secret storage. Stores API keys, connector credentials, and sensitive metadata. - **Vault**: AES-256-GCM encrypted secret storage. Stores API keys, connector credentials, and sensitive metadata.
- **MultiMind**: Manages personal + workspace minds simultaneously — routes searches to both and merges results. Wraps the `@waggle/hive-mind-core` substrate. - **MultiMind**: Manages personal + workspace minds simultaneously — routes searches to both and merges results.
- **FileStore**: Workspace filesystem access with a segment-boundary + symlink-aware containment guard and a secret deny-list (see the [threat model](../THREAT_MODEL.md)). - **FileStore**: Workspace filesystem access with a segment-boundary + symlink-aware containment guard and a secret deny-list (see the [threat model](../THREAT_MODEL.md)).
- **CronStore**: Schedule management for the cron service. CRUD on cron expressions with last/next run tracking. - **CronStore**: Schedule management for the cron service. CRUD on cron expressions with last/next run tracking.
- **ImportParser** (`memory-import.ts`): parses ChatGPT and Claude export files into importable knowledge items. - **ImportParser** (`memory-import.ts`): parses ChatGPT and Claude export files into importable knowledge items.

View File

@@ -1,258 +0,0 @@
# Waggle Methodology Documentation
**Date:** 2026-06-29 (v3)
**Subject of this document:** how Waggle's published LoCoMo memory-substrate numbers are produced, what they mean, and how to reproduce them offline.
> **v4 update (2026-07-01) — headline corrected 87.66% → 86.49%.** The 87.66% figure did not reproduce on a fresh judge pass (stale-verdict-replay inflation; it re-scores 85.19% on its own archived substrate and 86.49% on the current one). The reproducible 7-lane W4 number is **86.49%** — pinned + offline-verifiable at `benchmarks/results/locomo-sota-2026-06/` (`node recount.mjs`); full record in `docs/analysis/locomo-87.66-vs-85.26-integrity-2026-06-30.md`.
> **v3 update — canonical headline.** The current state-of-the-art result is **86.49% on LoCoMo**, measured under the previous leader's (Memori's) own same-judge protocol — see **§0** below, which now leads this document. Sections 1–8 document the earlier **v5 / N=320** conservative-protocol arc (trio-strict 67.8%, self-judge 73.1%); those numbers remain valid and honestly reported *for that arc*, but **§0 is the headline.** The full W3.3 methodology, run record, and artifacts live in the hive-mind research repo and `docs/paper/` (pointers in §0).
> **v2 honesty note (retained).** v2 aligned the page to the canonical v5 run record (`benchmarks/locomo/RESULTS.md`, run v5, 2026-05-11, trio re-judge 2026-05-21) and removed a withdrawn oracle-ceiling headline, a "clean Mem0 win", and a "+27.35-point methodology gap" brand. Those corrections still stand.
---
## 0. Current canonical headline — 86.49% (7-lane W4, same-judge, N=1,540)
**On LoCoMo under the previous SOTA's own protocol — GPT-4.1-mini as both answerer and judge, N=1,540 — Waggle's memory substrate (7-lane W4) scores 86.49% overall, a new state of the art: +4.54 points over the prior best (Memori, 81.95%), z = 4.64, p < 10⁻⁵.** We first reproduced Memori's own pipeline in-harness (our reproduction 81.98% vs their published 81.95%, within 0.03 points) before running the comparison on the identical ruler. Waggle leads or ties every question category:
| Category | Waggle (7-lane W4) | Memori (pub) | Δ |
|---|---:|---:|---:|
| single-hop | 92.27 | 87.87 | +4.40 |
| multi-hop | 80.50 | 72.70 | +7.80 |
| temporal | 81.62 | 80.37 | +1.25 |
| open-ended | 69.79 | 63.54 | +6.25 |
| **overall** | **86.49** | **81.95** | **+4.54** |
**Competitor re-run on the identical ruler (Mem0, same answerer + judge).** Waggle 86.49% vs Mem0 73.96% overall (+12.53), leading every category — the largest gap is temporal, **81.62% vs 50.78% (+30.84pp)**, attributable to write-time event dating versus Mem0's ingestion-time stamping.
**Token cost (honest caveat).** This headline runs at ~3,100 tokens/question versus Memori's ~1,294 — Waggle trades tokens for accuracy. A token-efficiency Pareto was measured on an earlier answer set (a leaner "Config D" packing traded ~−28% tokens for ~−1–2pp accuracy); those exact figures were anchored to the withdrawn 87.66 run and should be **re-measured on the current substrate** before re-citing. The retrieval lanes are load-bearing.
**Protocol note.** 86.49% is a **same-judge** number (one judge, GPT-4.1-mini, which is also the answerer) — chosen because it exactly matches the protocol the prior SOTA published, making it an apples-to-apples ruler. It is a different, more permissive protocol than the cross-vendor **trio-strict** bar used in §1–§4 (which was run on the earlier v5 / N=320 answer set and scores 67.8%). When comparing across papers, always match the judge methodology.
**Evidence & reproduction.** Pinned, reproducible, in-repo: `benchmarks/results/locomo-sota-2026-06/` — the report, `INDEX.md`, committed answers + judgments, and `recount.mjs` (offline, zero-API: recounts 1,332/1,540 = 86.49%). Investigation record: `docs/analysis/locomo-87.66-vs-85.26-integrity-2026-06-30.md`. Paper + arXiv draft: `docs/paper/`. The winning stack is open source at github.com/marolinik/hive-mind (PR #14, @ `bc4eba1`), Apache 2.0.
---
## Summary
We measured Waggle's memory substrate on the LoCoMo long-term conversational memory benchmark (N=320 stratified) under multiple evaluation protocols. *(This section documents the earlier v5 / N=320 arc; the current overall headline is **§0** above — 86.49%, 7-lane W4, same-judge, N=1,540.)* Within that arc, the defensible, reproducible result is the **trio-strict** number: under a 3-vendor judge ensemble (Anthropic Opus 4.7 + OpenAI GPT-5.5 + MiniMax M2.7, scored as a logical AND of all three), the substrate scores **67.8% (217/320)**. This number is cross-vendor, conservative by design, and re-derivable offline from committed judgments with zero API calls.
The load-bearing scientific finding is **substrate ≈ subject**: two very different SOTA subject models — Anthropic Opus 4.7 and a ~35B open-weights local model (Qwen3.6-35B-A3B) — converge to within **0.3 percentage points** on the *same* retrieval substrate under self-judge (73.1% vs 73.4%). The binding constraint on accuracy is the memory layer, not the model. *The layer, not the model.*
The strongest category is fully reliable across vendors: **single-hop trio-strict 87.5%** exactly matches its self-judge score (0pp inflation). The hardest cell is disclosed honestly: **multi-hop trio-strict 61.3%**, where one judge most often dissents.
This document covers (1) the LoCoMo evaluation protocol, (2) the trio-strict judge-ensemble rationale and inter-judge agreement, (3) the self-judge bias quantification (+5.3pp on this system), (4) the Mem0 matched-protocol comparison, (5) offline reproducibility instructions, and (6) limitations including a documented negative result.
---
## 1. LoCoMo Evaluation Protocol
### 1.1 Dataset and provenance
We evaluate on the LoCoMo benchmark (`snap-research/locomo`, file `locomo10.json`), pinned by SHA-256 `79fa87e9…ea698ff4` (2,805,274 bytes), sourced from the upstream public release. LoCoMo provides multi-session conversations with question-answer pairs designed to test memory retrieval at conversational distance.
### 1.2 Sample size and stratification
Primary results are reported on **N=320**, stratified **80 per category** across four question categories: multi-hop, temporal, open-ended, single-hop. The LoCoMo "adversarial" category (category 5) is excluded from the 4-way split. Sampling is deterministic and reconstructible: `xorshift32(seed=42)` + Fisher-Yates per bucket, bucket order [1,2,3,4], sorted by `instance_id` ascending. The sample, dataset SHA, and seed are committed so the test set itself is reconstructible.
### 1.3 Subject model and substrate
The headline subject model is `claude-opus-4-7` (Anthropic Messages API). The retrieval substrate is the **v4 frozen architecture** (the eval generation is run v5): distilled-dense facts (~53 per conversation) + K=5 importance retrieval + K=10 semantic retrieval + cross-encoder reranker + a synthesis-encouraging system prompt.
### 1.4 Headline result
**Under the trio-strict 3-vendor judge ensemble, Waggle's substrate scores 67.8% (217/320) on LoCoMo (N=320 stratified).** This is the conservative, cross-vendor, offline-reproducible number and the headline *for this v5 / N=320 arc* (the current overall SOTA headline is **§0**, 86.49%). The same answers under a less strict majority rule (≥2 of 3 judges) score 70.0% (224/320). The self-judge reference (Opus judging Opus) is 73.1% (234/320) and is disclosed as inflated — never the headline.
| Metric | Value |
|---|---:|
| **Trio-strict (AND of 3) — headline** | **217 / 320 = 67.8%** |
| Trio-majority (≥2 of 3) | 224 / 320 = 70.0% |
| Self-judge (Opus alone, reference only) | 234 / 320 = 73.1% |
| Self-judge inflation | **+5.3 pp** |
| Parse failures | 4 / 320 = 1.25% (irrecoverable noise) |
---
## 2. The "Layer, Not the Model" Finding (Substrate ≈ Subject)
The central scientific result is that the **substrate is the binding constraint, not the subject LLM**. We ran the same v5 retrieval substrate under two very different SOTA subject models and scored both under the identical self-judge protocol:
| Subject model | multi | temporal | open-ended | single | **TOTAL** |
|---|---:|---:|---:|---:|---:|
| Opus 4.7 (frontier cloud) | 75.0% | 67.5% | 62.5% | 87.5% | **73.1%** |
| Qwen3.6-35B-A3B (~35B open-weights, local) | 78.8% | 67.1% | 58.8% | 88.8% | **73.4%** |
| **Δ (Opus − Qwen)** | −3.8 | +0.4 | +3.8 | −1.3 | **−0.3 pp** |
The two models land **within 0.3 percentage points** of each other on identical retrieval. Opus trades ~5pp on lookup-style questions for +3.8pp on the synthesis-heavy open-ended category — a different failure mode, the same envelope. A frontier cloud model and a ~35B model you can run on-premises reach the same accuracy on the same memory layer.
This is the architectural basis for sovereign / regulated deployment: organizations that cannot run frontier-cloud models for compliance, data-residency, or sovereignty reasons can pair a local ~35B model with the Waggle substrate and reach comparable accuracy on these conditions. The driver of quality is the memory layer.
> Note on scope: this convergence is measured under the **self-judge** protocol (73.1% / 73.4%) because both subject runs were scored that way; the cross-vendor trio-strict ensemble was run on the Opus answer set. The ~0.3pp convergence is the verified claim; it should be cited as a self-judge result, not as a trio-strict result.
---
## 3. Judge Ensemble Methodology
### 3.1 Why a trio-strict ensemble
A persistent issue in long-term-memory benchmark evaluation is **self-judging bias** — the same LLM family acts as both the answer generator and the answer evaluator, which inflates scores relative to held-out-judge protocols. To control for single-family bias, Waggle uses a three-judge ensemble drawn from independent model vendors and scores a response **correct only when all three judges agree** (logical AND, "trio-strict"). This is conservative by design — it is the harder bar, not the easier one.
The three judges (each polled with the Mem0 verbatim "be generous" accuracy prompt):
| Role | Model |
|---|---|
| Anthropic | `claude-opus-4-7` |
| OpenAI | `gpt-5.5-2026-04-23` |
| MiniMax | `MiniMax-M2.7` |
A row where any judge fails to parse is excluded — counted as not-correct, with the denominator held at 320. Four rows (1.25%) were irrecoverable parse noise.
### 3.2 Inter-judge agreement
For the canonical v5 run we report **pairwise agreement** between judges (the fraction of co-parsed rows on which two judges return the same verdict):
| Pair | Agreement |
|---|---:|
| Opus ↔ GPT | 98.3% (286/291) |
| Opus ↔ MiniMax | 93.5% (260/278) |
| GPT ↔ MiniMax | 95.1% (250/263) |
The three judges also land within ~4pp of one another on overall correctness (Opus 230/320, GPT 223/320, MiniMax 236/320), so the ensemble agrees on aggregate quality even where individual rows are disputed. Disagreement is concentrated in the multi-hop and open-ended categories, where a binary "correct" verdict is genuinely fuzzy.
> For the canonical v5 judge set we report **pairwise agreement only** (above); no Cohen's κ is published for this ensemble. Pairwise agreement is the inter-judge metric carried in the run record.
### 3.3 Parser-fix audit trail
We disclose the full audit trail of the trio re-judge. A first trio run (v1, 2026-05-21 morning) produced 184/320 = 57.5% because of a parser bug: MiniMax's `max_tokens` budget (800) was too small and its reasoning tokens consumed the budget before the verdict label was emitted, and the parser did not accept the natural-language "INCORRECT" as a WRONG synonym. This produced 63 parse failures (19.7%), which were (wrongly) treated as not-correct and inflated the apparent self-judge gap. The fix raised MiniMax to `max_tokens: 3000` (and Opus/GPT to 500), taught the parser to accept "INCORRECT", and re-judged only the 63 failed rows. After the fix, 59 of 63 resolved, leaving 4 irrecoverable. The canonical, post-fix result is **217/320 = 67.8%** (the v2 file). We keep the buggy v1 judgments committed as an audit trail.
---
## 4. Self-Judge Bias Quantification (+5.3pp on this system)
When the same answers are scored by Opus-alone (self-judge) versus the trio-strict ensemble, the self-judge score is **+5.3 percentage points higher** (73.1% → 67.8%). This is well within cross-LLM-benchmark norms and is the honest, measured inflation for this system.
> Earlier internal (PM-Waggle-OS) work carried a "+27.35pp methodology gap" estimate. The canonical v5 run **measured the inflation directly on this system at +5.3pp** — substantially less than that prior estimate. We do not carry the +27.35pp figure forward, and we have dropped the "+27.35-Point Methodology Gap" framing entirely. The honest number is +5.3pp.
### 4.1 Per-category inflation
The inflation is not uniform — it is concentrated exactly where binary correctness is fuzzy:
| Category | Trio-strict | Self-judge | Inflation |
|---|---:|---:|---:|
| single-hop | **87.5%** (70/80) | 87.5% | **0 pp** (exact match) |
| temporal | 65.0% (52/80) | 67.5% | +2.5 pp |
| open-ended | 57.5% (46/80) | 62.5% | +5.0 pp |
| multi-hop | **61.3%** (49/80) | 75.0% | +13.7 pp |
**Single-hop trio-strict (87.5%) exactly matches self-judge** — the substrate's strongest category is fully reliable across all three vendors with zero inflation. **Multi-hop is the hard cell** (61.3% trio-strict) and the most inflated (+13.7pp), because multi-hop reasoning is where one judge most frequently dissents. We disclose multi-hop as the honest weak point rather than averaging it away.
### 4.2 Implications
Single-judge LoCoMo scores reported elsewhere in the 80–95% range are not directly comparable to trio-strict scores in the 60–70% range. Cross-paper comparisons require judge-methodology disclosure. We offer trio-strict (or a comparable held-out-judge protocol) as a community-friendly bar, and we publish both our trio-strict headline and our self-judge reference so a reader can map onto whichever methodology a venue uses.
---
## 5. Mem0 Comparison (Matched Protocol)
We compare against Mem0's published LoCoMo number under a **matched protocol**, and we are explicit about what is and is not a clean win.
| Comparison | Waggle | Mem0 paper | Δ |
|---|---:|---:|---:|
| Same-protocol self-judge (same dataset, same protocol, same judge prompt) | 73.1% | 68.5% | **+4.6 pp (Waggle)** |
| Waggle **trio-strict** vs Mem0 **self-judge** | 67.8% | 68.5% | −0.7 pp (essentially tied) |
Two honest readings:
1. **Under matched self-judge methodology** (the apples-to-apples comparison — same dataset, same "be generous" prompt, same single-judge protocol Mem0 used), Waggle scores **+4.6pp over Mem0's published 68.5%**, cross-validated on two subject models (Opus 73.1%, Qwen 73.4%).
2. **When Waggle is held to the stricter trio-strict bar while Mem0 keeps its single permissive self-judge**, the two are **essentially tied** (−0.7pp). This comparison is deliberately in Waggle's disadvantage — a 3-vendor AND-of-3 ensemble vs a single permissive judge — so the "tied" reading is conservative.
**We do not claim a clean 7.1-point win over Mem0.** The previous "74% beats Mem0's 66.9% by 7.1 points" headline compared a Waggle oracle-ceiling number against a re-judged Mem0 figure and is withdrawn. The defensible statement is: under matched self-judge protocol Waggle is modestly ahead (+4.6pp); under the stricter trio-strict bar the substrates are roughly tied.
> On the "74%" number: that figure is a **full-context oracle ceiling** (the subject model is given the entire conversation history, not substrate-mediated retrieval), measured in an earlier Stage-3 run. It is a headroom indicator, **not** the substrate-retrieval accuracy, and it is **not** the headline on this page. The headline accuracy is the trio-strict 67.8%.
---
## 6. Reproducibility
### 6.1 Offline reproduction (zero API calls)
The headline number is re-derivable offline from committed judgments, with no network and no model calls:
```bash
git clone https://github.com/marolinik/hive-mind.git
cd hive-mind
node benchmarks/locomo/rescore.mjs
```
Expected output (exit 0):
```
Trio-strict (AND of 3) 217/320 = 67.8%
Trio-majority (>=2 of 3) 224/320 = 70.0%
Parse failures 4/320
single-hop 70/80 multi-hop 49/80 temporal 52/80 open-ended 46/80
opus 230/320 gpt 223/320 mm 236/320
```
The rescore is adversarial about its own inputs: it (1) verifies each artifact's SHA-256 against `MANIFEST.json` (tamper-evidence), (2) independently recomputes every per-row verdict from the raw per-judge verdicts and cross-checks against the committed fields (0 mismatches expected), and (3) asserts the strict/majority/per-category/per-judge tallies match the MANIFEST exactly, exiting non-zero on any drift. This is the intended verification route, and it costs **$0**.
### 6.2 Committed artifacts
The committed reproducibility set (`benchmarks/locomo/artifacts/`):
| File | Role |
|---|---|
| `trio-judgments-v5-retrieval.v2.jsonl` | canonical per-row trio judgments (the rescore input, post parser-fix) |
| `cell-retrieval-v5-claude.jsonl` | the v5 retrieval answers that were judged |
| `sample-cells-23-N320.jsonl` | the N=320 stratified question sample (the test set) |
| `dataset-MANIFEST.json` | upstream LoCoMo dataset provenance (source URL + SHA + shape) |
| `sample-MANIFEST.json` | sampling provenance (seed=42 + algorithm + per-bucket SHAs) |
| `MANIFEST.json` | pins all artifact SHA-256s + dataset SHA + seed + the expected-results contract |
### 6.3 Full re-run from scratch
Regenerating the answers and judgments from the numbered `benchmarks/locomo/*.mjs` harness requires API keys and roughly **$23–26** of spend (v5 retrieval ~$5; trio judging, including the redo of parse-failures, ~$14–17; original self-judge ~$9). The offline `rescore.mjs` path needs neither network nor keys.
### 6.4 Code license
The hive-mind substrate (bitemporal knowledge graph retrieval, frame compression, reranker) and the evaluation harness are open source under Apache 2.0 at github.com/marolinik/hive-mind.
---
## 7. Limitations and Negative Results
### 7.1 What is not in this measurement
- **Single configuration, no ablation.** The full stack is on; we have not yet ablated the reranker / chunker / 8k embedder / distilled facts one at a time to attribute the contribution of each lever.
- **Confidence intervals are eyeballed.** Rough binomial spread is ~2pp at N=80 per cell, ~1pp at N=320 total. We have not computed formal CIs.
- **CLI library-mode only.** The MCP path is not yet benchmarked (its reranker wiring is incomplete); no MCP-path number is published.
- **κ not published for the v5 judge set** (see §3.2). We report pairwise agreement only for the headline run.
### 7.2 Multiplier pilot — negative result (disclosed honestly)
A separate "multiplier hypothesis" pilot — testing whether the Waggle memory layer would more than double downstream agentic task performance on real PM / research / engineering scenarios — produced negative results in an early small-N pilot and was **not** confirmed. We disclose this explicitly: honest negatives build credibility, and this negative does not affect the LoCoMo substrate measurement above.
> By design we state only the **directional** finding — the multiplier hypothesis was not confirmed in an early pilot. Specific per-scenario tallies are deliberately not restated here, pending a properly powered re-test.
### 7.3 GEPA cross-family validation — not published here
Earlier internal material reported a GEPA (reflective prompt-evolution) cross-family uplift table. Those figures are not part of the canonical LoCoMo run record and are **not** published on this page. The verified, LoCoMo-grounded version of the "an open model reaches frontier quality on the same memory layer" story is the **substrate ≈ subject** convergence (Opus 73.1% vs Qwen 73.4%, Δ−0.3pp, §2) — cite that instead.
### 7.4 Judge-ensemble cost
Trio-strict evaluation is roughly 3× the API cost of single-judge evaluation (triple the judge calls). For early-stage exploration a single judge is fine; trio-strict is reserved for publication-grade claims, where the higher-confidence cross-vendor bar is worth the cost.
---
## 8. References
- LoCoMo benchmark: Maharana et al., 2024 — dataset `snap-research/locomo`, `locomo10.json`.
- Mem0 (peer-reviewed): Mem0.ai/research, arXiv:2504.19413 — published LoCoMo self-judge figure 68.5%.
- GEPA (Genetic-Pareto reflective prompt optimization): Agrawal et al., 2025, arXiv:2507.19457.
- Canonical run record: `benchmarks/locomo/RESULTS.md` and `benchmarks/locomo/artifacts/MANIFEST.json` in github.com/marolinik/hive-mind.
For the most current evaluation results and reproducibility artifacts, consult the hive-mind research repository (canonical OSS substrate distribution). When the run record and any prose disagree, **the run record (`RESULTS.md` / `MANIFEST.json`, re-derivable by `rescore.mjs`) is canonical.**
---
**Document maintenance:** This document is versioned. v2 (2026-06-01) realigns the public methodology page to the canonical v5 LoCoMo run record and removes the withdrawn "+27.35-Point Methodology Gap" / "7.1-point Mem0 win" / GEPA-uplift framings. Subsequent versions will incorporate ablation results, formal confidence intervals, an MCP-path benchmark, and any additional benchmarks once they are reproduced in-repo.
License: This documentation is released under Creative Commons Attribution 4.0 International (CC BY 4.0). Code and evaluation artifacts referenced are Apache 2.0 licensed.