docs: drop references to the removed packages (P2.2)
AGENTS.md records that the excluded packages left in P2.2. SECURITY.md scopes memory to memory-contract and memory-mem0. THREAT_MODEL.md points the injection scanner at @waggle/core, marks the MCP tool scope gate as removed with the memory MCP servers, and drops the hive-mind copy of the URL egress guard. docs/ARCHITECTURE.md lists the memory packages instead of the hive-mind substrate. docs/methodology.md, the LoCoMo write-up of the hive-mind engine whose reproduction steps pointed at the maintainer's hive-mind repository, is removed.
This commit is contained in:
@@ -12,8 +12,8 @@
|
|||||||
|
|
||||||
This branch turns the commercial Waggle OS codebase into an open-source **community edition**.
|
This branch turns the commercial Waggle OS codebase into an open-source **community edition**.
|
||||||
The conversion is in progress. Memory runs on local mem0 through the memory contract (P1.5);
|
The conversion is in progress. Memory runs on local mem0 through the memory contract (P1.5);
|
||||||
team mode stays in postponed packages, and the excluded memory packages remain in the
|
team mode stays in postponed packages, and the excluded packages left the repository in P2.2.
|
||||||
repository, unbuilt, until P2.2. Work packages remove or replace the rest one by one.
|
Work packages remove or replace the rest one by one.
|
||||||
|
|
||||||
Sources of truth, in order:
|
Sources of truth, in order:
|
||||||
1. The agreed scope ("Waggle OSS Agreed Scope Annex", held by the project owner).
|
1. The agreed scope ("Waggle OSS Agreed Scope Annex", held by the project owner).
|
||||||
@@ -64,8 +64,7 @@ npm workspaces with the root `package-lock.json`. Node `^20.19.0 || >=22.12.0`.
|
|||||||
| `packages/agent`, `core`, `server`, `shared`, `sdk`, `marketplace`, `cli`, `launcher` | Keep. |
|
| `packages/agent`, `core`, `server`, `shared`, `sdk`, `marketplace`, `cli`, `launcher` | Keep. |
|
||||||
| `packages/memory-contract`, `packages/memory-mem0` | Keep. The memory contract and its mem0 implementation (ADR-0007). |
|
| `packages/memory-contract`, `packages/memory-mem0` | Keep. The memory contract and its mem0 implementation (ADR-0007). |
|
||||||
| `packages/team-server`, `packages/waggle-dance`, `packages/worker` | Postponed: team mode stays in the repo, is not built or shipped (ADR-0003). |
|
| `packages/team-server`, `packages/waggle-dance`, `packages/worker` | Postponed: team mode stays in the repo, is not built or shipped (ADR-0003). |
|
||||||
| `packages/hive-mind-*`, `memory-mcp`, `wiki-compiler` | Excluded. Nothing in scope imports them; they are not built or tested and leave in P2.2. |
|
| `packages/hive-mind-*`, `memory-mcp`, `wiki-compiler`, `admin-web`, `apps/www` | Excluded; removed in P2.2. The old memory engine is maintained separately by the owner. |
|
||||||
| `packages/admin-web`, `apps/www` | Excluded. |
|
|
||||||
| `apps/browser-ext` | Not decided. Do not extend it. |
|
| `apps/browser-ext` | Not decided. Do not extend it. |
|
||||||
| `docs/` | `docs/oss/` holds the community-edition records. The remaining guides, reference, architecture, getting-started and contributing pages predate this edition and are rewritten in P2.4. Internal history (`sidecar/`, `benchmarks/`, `judging/`, internal docs) was removed in P2.2. |
|
| `docs/` | `docs/oss/` holds the community-edition records. The remaining guides, reference, architecture, getting-started and contributing pages predate this edition and are rewritten in P2.4. Internal history (`sidecar/`, `benchmarks/`, `judging/`, internal docs) was removed in P2.2. |
|
||||||
|
|
||||||
|
|||||||
@@ -43,7 +43,7 @@ In scope:
|
|||||||
|
|
||||||
- The desktop app (Tauri shell + bundled sidecar), the web app, and the
|
- The desktop app (Tauri shell + bundled sidecar), the web app, and the
|
||||||
workspace packages under `packages/` and `apps/`.
|
workspace packages under `packages/` and `apps/`.
|
||||||
- The memory substrate (`packages/hive-mind-core`) and its MCP servers.
|
- Memory (`packages/memory-contract`, `packages/memory-mem0`).
|
||||||
- Prompt-injection paths, secret handling (the vault), the filesystem boundary,
|
- Prompt-injection paths, secret handling (the vault), the filesystem boundary,
|
||||||
and the capability/approval gates — see `THREAT_MODEL.md` for the details.
|
and the capability/approval gates — see `THREAT_MODEL.md` for the details.
|
||||||
|
|
||||||
|
|||||||
@@ -31,7 +31,7 @@ override, prompt-extraction) and make it act against the operator's intent.
|
|||||||
## Controls (implemented)
|
## Controls (implemented)
|
||||||
|
|
||||||
### 1. Injection scanning at every ingress — `scanForInjection`
|
### 1. Injection scanning at every ingress — `scanForInjection`
|
||||||
`packages/hive-mind-core/src/injection-scanner.ts` (re-exported via
|
`packages/core/src/injection-scanner.ts` (re-exported via
|
||||||
`packages/agent/src/injection-scanner.ts`). Three pattern sets — role-override,
|
`packages/agent/src/injection-scanner.ts`). Three pattern sets — role-override,
|
||||||
prompt-extraction, instruction-injection — produce a score; `safe = score < 0.3`
|
prompt-extraction, instruction-injection — produce a score; `safe = score < 0.3`
|
||||||
(instruction-injection is weighted higher, 0.6, for `tool_output` context). It gates the
|
(instruction-injection is weighted higher, 0.6, for `tool_output` context). It gates the
|
||||||
@@ -82,14 +82,9 @@ belong in the vault or a local `.env` (never committed). `.env.example` contains
|
|||||||
names plus non-secret development defaults, never live credentials. Callers must
|
names plus non-secret development defaults, never live credentials. Callers must
|
||||||
not persist secret values into prompts, logs, or memory frames.
|
not persist secret values into prompts, logs, or memory frames.
|
||||||
|
|
||||||
### 6. MCP tool scope gate — `scope.ts`
|
### 6. MCP tool scope gate (removed)
|
||||||
`packages/memory-mcp/src/scope.ts` + `packages/hive-mind-mcp-server/src/scope.ts`. An
|
The memory MCP servers this gate protected are not part of the community edition
|
||||||
external agent (Claude Code/Codex) granted stdio access to the memory substrate can be
|
(ADR-0002), so no external agent gets stdio access to memory.
|
||||||
scoped **read-only** via `WAGGLE_MCP_SCOPES` / `HIVE_MIND_SCOPES`: a `memory:read` scope
|
|
||||||
registers only the read tools, so a read-only client literally cannot call
|
|
||||||
save/cleanup/ingest. Default (unset) stays full read+write for backward-compat;
|
|
||||||
`memory:write` implies `memory:read`. This keeps a poisoned or buggy external agent from
|
|
||||||
writing junk into the substrate.
|
|
||||||
|
|
||||||
### 7. Workspace filesystem boundary + secret deny — `file-store.ts`
|
### 7. Workspace filesystem boundary + secret deny — `file-store.ts`
|
||||||
`packages/core/src/file-store.ts`. Every FileStore op resolves the caller path and asserts
|
`packages/core/src/file-store.ts`. Every FileStore op resolves the caller path and asserts
|
||||||
@@ -106,10 +101,7 @@ sandbox:** the deny is a curated list (it cannot enumerate every secret a home d
|
|||||||
is deny-by-default with no per-workspace override yet.
|
is deny-by-default with no per-workspace override yet.
|
||||||
|
|
||||||
### 8. SSRF egress guard — `url-egress-guard.ts`
|
### 8. SSRF egress guard — `url-egress-guard.ts`
|
||||||
`packages/agent/src/url-egress-guard.ts` (guarding `web_fetch` in `system-tools.ts`) and
|
`packages/agent/src/url-egress-guard.ts` (guarding `web_fetch` in `system-tools.ts`). A URL named by untrusted
|
||||||
the structurally-identical `packages/hive-mind-core/src/harvest/url-egress-guard.ts`
|
|
||||||
(guarding `UrlAdapter.fetchAndParse`, reached by the MCP `ingest_source` url path in
|
|
||||||
`packages/{memory-mcp,hive-mind-mcp-server}/src/tools/ingest.ts`). A URL named by untrusted
|
|
||||||
content or a user is resolved to concrete IP(s); the fetch is **refused if any resolved
|
content or a user is resolved to concrete IP(s); the fetch is **refused if any resolved
|
||||||
address is loopback / private (RFC1918 + CGNAT) / link-local / unique-local / multicast /
|
address is loopback / private (RFC1918 + CGNAT) / link-local / unique-local / multicast /
|
||||||
reserved / unspecified**. This closes the **cloud instance-metadata** exfiltration path —
|
reserved / unspecified**. This closes the **cloud instance-metadata** exfiltration path —
|
||||||
@@ -170,6 +162,4 @@ Coverage:
|
|||||||
full closure needs IP-pinning (connect to the validated address) which `fetch`+HTTPS can't
|
full closure needs IP-pinning (connect to the validated address) which `fetch`+HTTPS can't
|
||||||
do portably without breaking TLS SNI/cert validation. Also out of scope: the guard bounds
|
do portably without breaking TLS SNI/cert validation. Also out of scope: the guard bounds
|
||||||
the *target* address, not response size/content, and does not defend a genuinely
|
the *target* address, not response size/content, and does not defend a genuinely
|
||||||
public-but-malicious endpoint. The duplicated agent/hive-mind-core guard copies share one
|
public-but-malicious endpoint.
|
||||||
spec and must be kept in sync (they cannot share a module — hive-mind-core is OSS-mirrored
|
|
||||||
and must not import `@waggle/agent`).
|
|
||||||
|
|||||||
@@ -2,11 +2,10 @@
|
|||||||
|
|
||||||
Waggle is a monorepo with **28 packages** under `packages/` organized around a layered architecture: the memory substrate, agent intelligence, server API, and UI presentation. This document covers the package structure, data flow, and extension points.
|
Waggle is a monorepo with **28 packages** under `packages/` organized around a layered architecture: the memory substrate, agent intelligence, server API, and UI presentation. This document covers the package structure, data flow, and extension points.
|
||||||
|
|
||||||
> **Note (2026-04-30 monorepo migration):** the persistent-memory substrate
|
> **Note:** memory goes through `@waggle/memory-contract`, implemented on local
|
||||||
> (`mind/` + `harvest/`) moved out of `@waggle/core` into
|
> mem0 by `@waggle/memory-mem0` (ADR-0007). The React UI is **not** a package — it
|
||||||
> `@waggle/hive-mind-core` (`packages/hive-mind-core/src/{mind,harvest}`). The
|
> lives in `apps/web/src`. There is no `@waggle/ui` package. This document predates
|
||||||
> React UI is **not** a package — it lives in `apps/web/src`. There is no
|
> the community edition and is rewritten in P2.4.
|
||||||
> `@waggle/ui` package.
|
|
||||||
|
|
||||||
## Package Overview
|
## Package Overview
|
||||||
|
|
||||||
@@ -14,7 +13,6 @@ Waggle is a monorepo with **28 packages** under `packages/` organized around a l
|
|||||||
waggle-os/
|
waggle-os/
|
||||||
apps/
|
apps/
|
||||||
web/ # Main web app UI (React 19 + Vite + Tailwind 4 + base-ui/react)
|
web/ # Main web app UI (React 19 + Vite + Tailwind 4 + base-ui/react)
|
||||||
www/ # Marketing site (Next.js)
|
|
||||||
browser-ext/ # Browser extension (unpacked; not an npm workspace)
|
browser-ext/ # Browser extension (unpacked; not an npm workspace)
|
||||||
packages/
|
packages/
|
||||||
# Product packages (MIT)
|
# Product packages (MIT)
|
||||||
@@ -30,40 +28,19 @@ waggle-os/
|
|||||||
sdk/ # Plugin/skill SDK, capability packs, starter skills
|
sdk/ # Plugin/skill SDK, capability packs, starter skills
|
||||||
cli/ # Command-line REPL
|
cli/ # Command-line REPL
|
||||||
launcher/ # AI-tool launcher / dock backend
|
launcher/ # AI-tool launcher / dock backend
|
||||||
admin-web/ # Admin dashboard for team deployments
|
memory-contract/ # MemoryStore interface, record model, capability map
|
||||||
wiki-compiler/ # Knowledge / wiki compiler
|
memory-mem0/ # MemoryStore on local mem0
|
||||||
memory-mcp/ # MCP server exposing the memory substrate to external agents
|
|
||||||
# Memory substrate — hive-mind-* (Apache-2.0), mirrored to marolinik/hive-mind
|
|
||||||
hive-mind-core/ # The substrate: FrameStore, HybridSearch, KnowledgeGraph, Identity/Awareness, Harvest (src/mind + src/harvest)
|
|
||||||
hive-mind-cli/ # CLI for the substrate
|
|
||||||
hive-mind-mcp-server/ # MCP server for the substrate
|
|
||||||
hive-mind-shim-core/ # Signal-emitter shim library
|
|
||||||
hive-mind-wiki-compiler/# Wiki compiler (OSS)
|
|
||||||
hive-mind-hooks-core/ # Shared hook library
|
|
||||||
hive-mind-hooks-*/ # Per-tool capture hooks: claude-code, claude-desktop, codex,
|
|
||||||
# codex-desktop, cursor, hermes, openclaw
|
|
||||||
sidecar/ # Node.js sidecar for Tauri desktop app
|
sidecar/ # Node.js sidecar for Tauri desktop app
|
||||||
app/ # Tauri 2.0 desktop shell (Rust + WebView2) — loads the apps/web build
|
app/ # Tauri 2.0 desktop shell (Rust + WebView2) — loads the apps/web build
|
||||||
```
|
```
|
||||||
|
|
||||||
## Package Details
|
## Package Details
|
||||||
|
|
||||||
### @waggle/hive-mind-core
|
### @waggle/memory-contract and @waggle/memory-mem0
|
||||||
|
|
||||||
The persistent-memory substrate (Apache-2.0; mirrored to the public
|
`MemoryStore` (remember, search, list, get, update, forget, history) with its record
|
||||||
[`marolinik/hive-mind`](https://github.com/marolinik/hive-mind) repo). Zero
|
model and capability map, and its implementation on the open-source mem0 library,
|
||||||
network dependencies; runs on SQLite + sqlite-vec.
|
running locally in the sidecar (ADR-0007).
|
||||||
|
|
||||||
- **MindDB** (`src/mind/db.ts`, `schema.ts`): SQLite wrapper for `.mind` files. Tables for memory frames, knowledge-graph entities/relations, embeddings, sessions, and improvement signals.
|
|
||||||
- **FrameStore** (`src/mind/frames.ts`): CRUD on memory frames with FTS5 full-text search, importance ranking, and access counting.
|
|
||||||
- **HybridSearch** (`src/mind/search.ts`): vector + keyword retrieval, with an optional cross-encoder reranker.
|
|
||||||
- **KnowledgeGraph** (`src/mind/knowledge.ts`): entity-relation graph with temporal validity (`valid_from`/`valid_to`).
|
|
||||||
- **IdentityLayer / AwarenessLayer** (`src/mind/identity.ts`, `awareness.ts`): personal-identity persistence and active task/state tracking.
|
|
||||||
- **Embeddings** (`src/mind/*-embedder.ts`): pluggable providers — in-process, Ollama, Voyage, OpenAI, mock.
|
|
||||||
- **Harvest** (`src/harvest/`): conversation/file ingestion adapters (ChatGPT, Claude, Claude Code, Gemini, Perplexity, PDF, markdown, URL, plaintext) plus the dedup pipeline.
|
|
||||||
|
|
||||||
> Develop the substrate **here** and mirror it out — never the reverse. See the
|
|
||||||
> "Memory Substrate Sync" section of the root [`CLAUDE.md`](../CLAUDE.md).
|
|
||||||
|
|
||||||
### @waggle/core
|
### @waggle/core
|
||||||
|
|
||||||
@@ -71,7 +48,7 @@ The foundation layer for the desktop/server runtime. Zero network dependencies.
|
|||||||
|
|
||||||
- **WaggleConfig**: Configuration management (`~/.waggle/config.json`). Provider keys, default model, team server config.
|
- **WaggleConfig**: Configuration management (`~/.waggle/config.json`). Provider keys, default model, team server config.
|
||||||
- **Vault**: AES-256-GCM encrypted secret storage. Stores API keys, connector credentials, and sensitive metadata.
|
- **Vault**: AES-256-GCM encrypted secret storage. Stores API keys, connector credentials, and sensitive metadata.
|
||||||
- **MultiMind**: Manages personal + workspace minds simultaneously — routes searches to both and merges results. Wraps the `@waggle/hive-mind-core` substrate.
|
- **MultiMind**: Manages personal + workspace minds simultaneously — routes searches to both and merges results.
|
||||||
- **FileStore**: Workspace filesystem access with a segment-boundary + symlink-aware containment guard and a secret deny-list (see the [threat model](../THREAT_MODEL.md)).
|
- **FileStore**: Workspace filesystem access with a segment-boundary + symlink-aware containment guard and a secret deny-list (see the [threat model](../THREAT_MODEL.md)).
|
||||||
- **CronStore**: Schedule management for the cron service. CRUD on cron expressions with last/next run tracking.
|
- **CronStore**: Schedule management for the cron service. CRUD on cron expressions with last/next run tracking.
|
||||||
- **ImportParser** (`memory-import.ts`): parses ChatGPT and Claude export files into importable knowledge items.
|
- **ImportParser** (`memory-import.ts`): parses ChatGPT and Claude export files into importable knowledge items.
|
||||||
|
|||||||
@@ -1,258 +0,0 @@
|
|||||||
# Waggle Methodology Documentation
|
|
||||||
|
|
||||||
**Date:** 2026-06-29 (v3)
|
|
||||||
**Subject of this document:** how Waggle's published LoCoMo memory-substrate numbers are produced, what they mean, and how to reproduce them offline.
|
|
||||||
|
|
||||||
> **v4 update (2026-07-01) — headline corrected 87.66% → 86.49%.** The 87.66% figure did not reproduce on a fresh judge pass (stale-verdict-replay inflation; it re-scores 85.19% on its own archived substrate and 86.49% on the current one). The reproducible 7-lane W4 number is **86.49%** — pinned + offline-verifiable at `benchmarks/results/locomo-sota-2026-06/` (`node recount.mjs`); full record in `docs/analysis/locomo-87.66-vs-85.26-integrity-2026-06-30.md`.
|
|
||||||
> **v3 update — canonical headline.** The current state-of-the-art result is **86.49% on LoCoMo**, measured under the previous leader's (Memori's) own same-judge protocol — see **§0** below, which now leads this document. Sections 1–8 document the earlier **v5 / N=320** conservative-protocol arc (trio-strict 67.8%, self-judge 73.1%); those numbers remain valid and honestly reported *for that arc*, but **§0 is the headline.** The full W3.3 methodology, run record, and artifacts live in the hive-mind research repo and `docs/paper/` (pointers in §0).
|
|
||||||
> **v2 honesty note (retained).** v2 aligned the page to the canonical v5 run record (`benchmarks/locomo/RESULTS.md`, run v5, 2026-05-11, trio re-judge 2026-05-21) and removed a withdrawn oracle-ceiling headline, a "clean Mem0 win", and a "+27.35-point methodology gap" brand. Those corrections still stand.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 0. Current canonical headline — 86.49% (7-lane W4, same-judge, N=1,540)
|
|
||||||
|
|
||||||
**On LoCoMo under the previous SOTA's own protocol — GPT-4.1-mini as both answerer and judge, N=1,540 — Waggle's memory substrate (7-lane W4) scores 86.49% overall, a new state of the art: +4.54 points over the prior best (Memori, 81.95%), z = 4.64, p < 10⁻⁵.** We first reproduced Memori's own pipeline in-harness (our reproduction 81.98% vs their published 81.95%, within 0.03 points) before running the comparison on the identical ruler. Waggle leads or ties every question category:
|
|
||||||
|
|
||||||
| Category | Waggle (7-lane W4) | Memori (pub) | Δ |
|
|
||||||
|---|---:|---:|---:|
|
|
||||||
| single-hop | 92.27 | 87.87 | +4.40 |
|
|
||||||
| multi-hop | 80.50 | 72.70 | +7.80 |
|
|
||||||
| temporal | 81.62 | 80.37 | +1.25 |
|
|
||||||
| open-ended | 69.79 | 63.54 | +6.25 |
|
|
||||||
| **overall** | **86.49** | **81.95** | **+4.54** |
|
|
||||||
|
|
||||||
**Competitor re-run on the identical ruler (Mem0, same answerer + judge).** Waggle 86.49% vs Mem0 73.96% overall (+12.53), leading every category — the largest gap is temporal, **81.62% vs 50.78% (+30.84pp)**, attributable to write-time event dating versus Mem0's ingestion-time stamping.
|
|
||||||
|
|
||||||
**Token cost (honest caveat).** This headline runs at ~3,100 tokens/question versus Memori's ~1,294 — Waggle trades tokens for accuracy. A token-efficiency Pareto was measured on an earlier answer set (a leaner "Config D" packing traded ~−28% tokens for ~−1–2pp accuracy); those exact figures were anchored to the withdrawn 87.66 run and should be **re-measured on the current substrate** before re-citing. The retrieval lanes are load-bearing.
|
|
||||||
|
|
||||||
**Protocol note.** 86.49% is a **same-judge** number (one judge, GPT-4.1-mini, which is also the answerer) — chosen because it exactly matches the protocol the prior SOTA published, making it an apples-to-apples ruler. It is a different, more permissive protocol than the cross-vendor **trio-strict** bar used in §1–§4 (which was run on the earlier v5 / N=320 answer set and scores 67.8%). When comparing across papers, always match the judge methodology.
|
|
||||||
|
|
||||||
**Evidence & reproduction.** Pinned, reproducible, in-repo: `benchmarks/results/locomo-sota-2026-06/` — the report, `INDEX.md`, committed answers + judgments, and `recount.mjs` (offline, zero-API: recounts 1,332/1,540 = 86.49%). Investigation record: `docs/analysis/locomo-87.66-vs-85.26-integrity-2026-06-30.md`. Paper + arXiv draft: `docs/paper/`. The winning stack is open source at github.com/marolinik/hive-mind (PR #14, @ `bc4eba1`), Apache 2.0.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
We measured Waggle's memory substrate on the LoCoMo long-term conversational memory benchmark (N=320 stratified) under multiple evaluation protocols. *(This section documents the earlier v5 / N=320 arc; the current overall headline is **§0** above — 86.49%, 7-lane W4, same-judge, N=1,540.)* Within that arc, the defensible, reproducible result is the **trio-strict** number: under a 3-vendor judge ensemble (Anthropic Opus 4.7 + OpenAI GPT-5.5 + MiniMax M2.7, scored as a logical AND of all three), the substrate scores **67.8% (217/320)**. This number is cross-vendor, conservative by design, and re-derivable offline from committed judgments with zero API calls.
|
|
||||||
|
|
||||||
The load-bearing scientific finding is **substrate ≈ subject**: two very different SOTA subject models — Anthropic Opus 4.7 and a ~35B open-weights local model (Qwen3.6-35B-A3B) — converge to within **0.3 percentage points** on the *same* retrieval substrate under self-judge (73.1% vs 73.4%). The binding constraint on accuracy is the memory layer, not the model. *The layer, not the model.*
|
|
||||||
|
|
||||||
The strongest category is fully reliable across vendors: **single-hop trio-strict 87.5%** exactly matches its self-judge score (0pp inflation). The hardest cell is disclosed honestly: **multi-hop trio-strict 61.3%**, where one judge most often dissents.
|
|
||||||
|
|
||||||
This document covers (1) the LoCoMo evaluation protocol, (2) the trio-strict judge-ensemble rationale and inter-judge agreement, (3) the self-judge bias quantification (+5.3pp on this system), (4) the Mem0 matched-protocol comparison, (5) offline reproducibility instructions, and (6) limitations including a documented negative result.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1. LoCoMo Evaluation Protocol
|
|
||||||
|
|
||||||
### 1.1 Dataset and provenance
|
|
||||||
|
|
||||||
We evaluate on the LoCoMo benchmark (`snap-research/locomo`, file `locomo10.json`), pinned by SHA-256 `79fa87e9…ea698ff4` (2,805,274 bytes), sourced from the upstream public release. LoCoMo provides multi-session conversations with question-answer pairs designed to test memory retrieval at conversational distance.
|
|
||||||
|
|
||||||
### 1.2 Sample size and stratification
|
|
||||||
|
|
||||||
Primary results are reported on **N=320**, stratified **80 per category** across four question categories: multi-hop, temporal, open-ended, single-hop. The LoCoMo "adversarial" category (category 5) is excluded from the 4-way split. Sampling is deterministic and reconstructible: `xorshift32(seed=42)` + Fisher-Yates per bucket, bucket order [1,2,3,4], sorted by `instance_id` ascending. The sample, dataset SHA, and seed are committed so the test set itself is reconstructible.
|
|
||||||
|
|
||||||
### 1.3 Subject model and substrate
|
|
||||||
|
|
||||||
The headline subject model is `claude-opus-4-7` (Anthropic Messages API). The retrieval substrate is the **v4 frozen architecture** (the eval generation is run v5): distilled-dense facts (~53 per conversation) + K=5 importance retrieval + K=10 semantic retrieval + cross-encoder reranker + a synthesis-encouraging system prompt.
|
|
||||||
|
|
||||||
### 1.4 Headline result
|
|
||||||
|
|
||||||
**Under the trio-strict 3-vendor judge ensemble, Waggle's substrate scores 67.8% (217/320) on LoCoMo (N=320 stratified).** This is the conservative, cross-vendor, offline-reproducible number and the headline *for this v5 / N=320 arc* (the current overall SOTA headline is **§0**, 86.49%). The same answers under a less strict majority rule (≥2 of 3 judges) score 70.0% (224/320). The self-judge reference (Opus judging Opus) is 73.1% (234/320) and is disclosed as inflated — never the headline.
|
|
||||||
|
|
||||||
| Metric | Value |
|
|
||||||
|---|---:|
|
|
||||||
| **Trio-strict (AND of 3) — headline** | **217 / 320 = 67.8%** |
|
|
||||||
| Trio-majority (≥2 of 3) | 224 / 320 = 70.0% |
|
|
||||||
| Self-judge (Opus alone, reference only) | 234 / 320 = 73.1% |
|
|
||||||
| Self-judge inflation | **+5.3 pp** |
|
|
||||||
| Parse failures | 4 / 320 = 1.25% (irrecoverable noise) |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2. The "Layer, Not the Model" Finding (Substrate ≈ Subject)
|
|
||||||
|
|
||||||
The central scientific result is that the **substrate is the binding constraint, not the subject LLM**. We ran the same v5 retrieval substrate under two very different SOTA subject models and scored both under the identical self-judge protocol:
|
|
||||||
|
|
||||||
| Subject model | multi | temporal | open-ended | single | **TOTAL** |
|
|
||||||
|---|---:|---:|---:|---:|---:|
|
|
||||||
| Opus 4.7 (frontier cloud) | 75.0% | 67.5% | 62.5% | 87.5% | **73.1%** |
|
|
||||||
| Qwen3.6-35B-A3B (~35B open-weights, local) | 78.8% | 67.1% | 58.8% | 88.8% | **73.4%** |
|
|
||||||
| **Δ (Opus − Qwen)** | −3.8 | +0.4 | +3.8 | −1.3 | **−0.3 pp** |
|
|
||||||
|
|
||||||
The two models land **within 0.3 percentage points** of each other on identical retrieval. Opus trades ~5pp on lookup-style questions for +3.8pp on the synthesis-heavy open-ended category — a different failure mode, the same envelope. A frontier cloud model and a ~35B model you can run on-premises reach the same accuracy on the same memory layer.
|
|
||||||
|
|
||||||
This is the architectural basis for sovereign / regulated deployment: organizations that cannot run frontier-cloud models for compliance, data-residency, or sovereignty reasons can pair a local ~35B model with the Waggle substrate and reach comparable accuracy on these conditions. The driver of quality is the memory layer.
|
|
||||||
|
|
||||||
> Note on scope: this convergence is measured under the **self-judge** protocol (73.1% / 73.4%) because both subject runs were scored that way; the cross-vendor trio-strict ensemble was run on the Opus answer set. The ~0.3pp convergence is the verified claim; it should be cited as a self-judge result, not as a trio-strict result.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. Judge Ensemble Methodology
|
|
||||||
|
|
||||||
### 3.1 Why a trio-strict ensemble
|
|
||||||
|
|
||||||
A persistent issue in long-term-memory benchmark evaluation is **self-judging bias** — the same LLM family acts as both the answer generator and the answer evaluator, which inflates scores relative to held-out-judge protocols. To control for single-family bias, Waggle uses a three-judge ensemble drawn from independent model vendors and scores a response **correct only when all three judges agree** (logical AND, "trio-strict"). This is conservative by design — it is the harder bar, not the easier one.
|
|
||||||
|
|
||||||
The three judges (each polled with the Mem0 verbatim "be generous" accuracy prompt):
|
|
||||||
|
|
||||||
| Role | Model |
|
|
||||||
|---|---|
|
|
||||||
| Anthropic | `claude-opus-4-7` |
|
|
||||||
| OpenAI | `gpt-5.5-2026-04-23` |
|
|
||||||
| MiniMax | `MiniMax-M2.7` |
|
|
||||||
|
|
||||||
A row where any judge fails to parse is excluded — counted as not-correct, with the denominator held at 320. Four rows (1.25%) were irrecoverable parse noise.
|
|
||||||
|
|
||||||
### 3.2 Inter-judge agreement
|
|
||||||
|
|
||||||
For the canonical v5 run we report **pairwise agreement** between judges (the fraction of co-parsed rows on which two judges return the same verdict):
|
|
||||||
|
|
||||||
| Pair | Agreement |
|
|
||||||
|---|---:|
|
|
||||||
| Opus ↔ GPT | 98.3% (286/291) |
|
|
||||||
| Opus ↔ MiniMax | 93.5% (260/278) |
|
|
||||||
| GPT ↔ MiniMax | 95.1% (250/263) |
|
|
||||||
|
|
||||||
The three judges also land within ~4pp of one another on overall correctness (Opus 230/320, GPT 223/320, MiniMax 236/320), so the ensemble agrees on aggregate quality even where individual rows are disputed. Disagreement is concentrated in the multi-hop and open-ended categories, where a binary "correct" verdict is genuinely fuzzy.
|
|
||||||
|
|
||||||
> For the canonical v5 judge set we report **pairwise agreement only** (above); no Cohen's κ is published for this ensemble. Pairwise agreement is the inter-judge metric carried in the run record.
|
|
||||||
|
|
||||||
### 3.3 Parser-fix audit trail
|
|
||||||
|
|
||||||
We disclose the full audit trail of the trio re-judge. A first trio run (v1, 2026-05-21 morning) produced 184/320 = 57.5% because of a parser bug: MiniMax's `max_tokens` budget (800) was too small and its reasoning tokens consumed the budget before the verdict label was emitted, and the parser did not accept the natural-language "INCORRECT" as a WRONG synonym. This produced 63 parse failures (19.7%), which were (wrongly) treated as not-correct and inflated the apparent self-judge gap. The fix raised MiniMax to `max_tokens: 3000` (and Opus/GPT to 500), taught the parser to accept "INCORRECT", and re-judged only the 63 failed rows. After the fix, 59 of 63 resolved, leaving 4 irrecoverable. The canonical, post-fix result is **217/320 = 67.8%** (the v2 file). We keep the buggy v1 judgments committed as an audit trail.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. Self-Judge Bias Quantification (+5.3pp on this system)
|
|
||||||
|
|
||||||
When the same answers are scored by Opus-alone (self-judge) versus the trio-strict ensemble, the self-judge score is **+5.3 percentage points higher** (73.1% → 67.8%). This is well within cross-LLM-benchmark norms and is the honest, measured inflation for this system.
|
|
||||||
|
|
||||||
> Earlier internal (PM-Waggle-OS) work carried a "+27.35pp methodology gap" estimate. The canonical v5 run **measured the inflation directly on this system at +5.3pp** — substantially less than that prior estimate. We do not carry the +27.35pp figure forward, and we have dropped the "+27.35-Point Methodology Gap" framing entirely. The honest number is +5.3pp.
|
|
||||||
|
|
||||||
### 4.1 Per-category inflation
|
|
||||||
|
|
||||||
The inflation is not uniform — it is concentrated exactly where binary correctness is fuzzy:
|
|
||||||
|
|
||||||
| Category | Trio-strict | Self-judge | Inflation |
|
|
||||||
|---|---:|---:|---:|
|
|
||||||
| single-hop | **87.5%** (70/80) | 87.5% | **0 pp** (exact match) |
|
|
||||||
| temporal | 65.0% (52/80) | 67.5% | +2.5 pp |
|
|
||||||
| open-ended | 57.5% (46/80) | 62.5% | +5.0 pp |
|
|
||||||
| multi-hop | **61.3%** (49/80) | 75.0% | +13.7 pp |
|
|
||||||
|
|
||||||
**Single-hop trio-strict (87.5%) exactly matches self-judge** — the substrate's strongest category is fully reliable across all three vendors with zero inflation. **Multi-hop is the hard cell** (61.3% trio-strict) and the most inflated (+13.7pp), because multi-hop reasoning is where one judge most frequently dissents. We disclose multi-hop as the honest weak point rather than averaging it away.
|
|
||||||
|
|
||||||
### 4.2 Implications
|
|
||||||
|
|
||||||
Single-judge LoCoMo scores reported elsewhere in the 80–95% range are not directly comparable to trio-strict scores in the 60–70% range. Cross-paper comparisons require judge-methodology disclosure. We offer trio-strict (or a comparable held-out-judge protocol) as a community-friendly bar, and we publish both our trio-strict headline and our self-judge reference so a reader can map onto whichever methodology a venue uses.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5. Mem0 Comparison (Matched Protocol)
|
|
||||||
|
|
||||||
We compare against Mem0's published LoCoMo number under a **matched protocol**, and we are explicit about what is and is not a clean win.
|
|
||||||
|
|
||||||
| Comparison | Waggle | Mem0 paper | Δ |
|
|
||||||
|---|---:|---:|---:|
|
|
||||||
| Same-protocol self-judge (same dataset, same protocol, same judge prompt) | 73.1% | 68.5% | **+4.6 pp (Waggle)** |
|
|
||||||
| Waggle **trio-strict** vs Mem0 **self-judge** | 67.8% | 68.5% | −0.7 pp (essentially tied) |
|
|
||||||
|
|
||||||
Two honest readings:
|
|
||||||
|
|
||||||
1. **Under matched self-judge methodology** (the apples-to-apples comparison — same dataset, same "be generous" prompt, same single-judge protocol Mem0 used), Waggle scores **+4.6pp over Mem0's published 68.5%**, cross-validated on two subject models (Opus 73.1%, Qwen 73.4%).
|
|
||||||
2. **When Waggle is held to the stricter trio-strict bar while Mem0 keeps its single permissive self-judge**, the two are **essentially tied** (−0.7pp). This comparison is deliberately in Waggle's disadvantage — a 3-vendor AND-of-3 ensemble vs a single permissive judge — so the "tied" reading is conservative.
|
|
||||||
|
|
||||||
**We do not claim a clean 7.1-point win over Mem0.** The previous "74% beats Mem0's 66.9% by 7.1 points" headline compared a Waggle oracle-ceiling number against a re-judged Mem0 figure and is withdrawn. The defensible statement is: under matched self-judge protocol Waggle is modestly ahead (+4.6pp); under the stricter trio-strict bar the substrates are roughly tied.
|
|
||||||
|
|
||||||
> On the "74%" number: that figure is a **full-context oracle ceiling** (the subject model is given the entire conversation history, not substrate-mediated retrieval), measured in an earlier Stage-3 run. It is a headroom indicator, **not** the substrate-retrieval accuracy, and it is **not** the headline on this page. The headline accuracy is the trio-strict 67.8%.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 6. Reproducibility
|
|
||||||
|
|
||||||
### 6.1 Offline reproduction (zero API calls)
|
|
||||||
|
|
||||||
The headline number is re-derivable offline from committed judgments, with no network and no model calls:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git clone https://github.com/marolinik/hive-mind.git
|
|
||||||
cd hive-mind
|
|
||||||
node benchmarks/locomo/rescore.mjs
|
|
||||||
```
|
|
||||||
|
|
||||||
Expected output (exit 0):
|
|
||||||
|
|
||||||
```
|
|
||||||
Trio-strict (AND of 3) 217/320 = 67.8%
|
|
||||||
Trio-majority (>=2 of 3) 224/320 = 70.0%
|
|
||||||
Parse failures 4/320
|
|
||||||
single-hop 70/80 multi-hop 49/80 temporal 52/80 open-ended 46/80
|
|
||||||
opus 230/320 gpt 223/320 mm 236/320
|
|
||||||
```
|
|
||||||
|
|
||||||
The rescore is adversarial about its own inputs: it (1) verifies each artifact's SHA-256 against `MANIFEST.json` (tamper-evidence), (2) independently recomputes every per-row verdict from the raw per-judge verdicts and cross-checks against the committed fields (0 mismatches expected), and (3) asserts the strict/majority/per-category/per-judge tallies match the MANIFEST exactly, exiting non-zero on any drift. This is the intended verification route, and it costs **$0**.
|
|
||||||
|
|
||||||
### 6.2 Committed artifacts
|
|
||||||
|
|
||||||
The committed reproducibility set (`benchmarks/locomo/artifacts/`):
|
|
||||||
|
|
||||||
| File | Role |
|
|
||||||
|---|---|
|
|
||||||
| `trio-judgments-v5-retrieval.v2.jsonl` | canonical per-row trio judgments (the rescore input, post parser-fix) |
|
|
||||||
| `cell-retrieval-v5-claude.jsonl` | the v5 retrieval answers that were judged |
|
|
||||||
| `sample-cells-23-N320.jsonl` | the N=320 stratified question sample (the test set) |
|
|
||||||
| `dataset-MANIFEST.json` | upstream LoCoMo dataset provenance (source URL + SHA + shape) |
|
|
||||||
| `sample-MANIFEST.json` | sampling provenance (seed=42 + algorithm + per-bucket SHAs) |
|
|
||||||
| `MANIFEST.json` | pins all artifact SHA-256s + dataset SHA + seed + the expected-results contract |
|
|
||||||
|
|
||||||
### 6.3 Full re-run from scratch
|
|
||||||
|
|
||||||
Regenerating the answers and judgments from the numbered `benchmarks/locomo/*.mjs` harness requires API keys and roughly **$23–26** of spend (v5 retrieval ~$5; trio judging, including the redo of parse-failures, ~$14–17; original self-judge ~$9). The offline `rescore.mjs` path needs neither network nor keys.
|
|
||||||
|
|
||||||
### 6.4 Code license
|
|
||||||
|
|
||||||
The hive-mind substrate (bitemporal knowledge graph retrieval, frame compression, reranker) and the evaluation harness are open source under Apache 2.0 at github.com/marolinik/hive-mind.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 7. Limitations and Negative Results
|
|
||||||
|
|
||||||
### 7.1 What is not in this measurement
|
|
||||||
|
|
||||||
- **Single configuration, no ablation.** The full stack is on; we have not yet ablated the reranker / chunker / 8k embedder / distilled facts one at a time to attribute the contribution of each lever.
|
|
||||||
- **Confidence intervals are eyeballed.** Rough binomial spread is ~2pp at N=80 per cell, ~1pp at N=320 total. We have not computed formal CIs.
|
|
||||||
- **CLI library-mode only.** The MCP path is not yet benchmarked (its reranker wiring is incomplete); no MCP-path number is published.
|
|
||||||
- **κ not published for the v5 judge set** (see §3.2). We report pairwise agreement only for the headline run.
|
|
||||||
|
|
||||||
### 7.2 Multiplier pilot — negative result (disclosed honestly)
|
|
||||||
|
|
||||||
A separate "multiplier hypothesis" pilot — testing whether the Waggle memory layer would more than double downstream agentic task performance on real PM / research / engineering scenarios — produced negative results in an early small-N pilot and was **not** confirmed. We disclose this explicitly: honest negatives build credibility, and this negative does not affect the LoCoMo substrate measurement above.
|
|
||||||
|
|
||||||
> By design we state only the **directional** finding — the multiplier hypothesis was not confirmed in an early pilot. Specific per-scenario tallies are deliberately not restated here, pending a properly powered re-test.
|
|
||||||
|
|
||||||
### 7.3 GEPA cross-family validation — not published here
|
|
||||||
|
|
||||||
Earlier internal material reported a GEPA (reflective prompt-evolution) cross-family uplift table. Those figures are not part of the canonical LoCoMo run record and are **not** published on this page. The verified, LoCoMo-grounded version of the "an open model reaches frontier quality on the same memory layer" story is the **substrate ≈ subject** convergence (Opus 73.1% vs Qwen 73.4%, Δ−0.3pp, §2) — cite that instead.
|
|
||||||
|
|
||||||
### 7.4 Judge-ensemble cost
|
|
||||||
|
|
||||||
Trio-strict evaluation is roughly 3× the API cost of single-judge evaluation (triple the judge calls). For early-stage exploration a single judge is fine; trio-strict is reserved for publication-grade claims, where the higher-confidence cross-vendor bar is worth the cost.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 8. References
|
|
||||||
|
|
||||||
- LoCoMo benchmark: Maharana et al., 2024 — dataset `snap-research/locomo`, `locomo10.json`.
|
|
||||||
- Mem0 (peer-reviewed): Mem0.ai/research, arXiv:2504.19413 — published LoCoMo self-judge figure 68.5%.
|
|
||||||
- GEPA (Genetic-Pareto reflective prompt optimization): Agrawal et al., 2025, arXiv:2507.19457.
|
|
||||||
- Canonical run record: `benchmarks/locomo/RESULTS.md` and `benchmarks/locomo/artifacts/MANIFEST.json` in github.com/marolinik/hive-mind.
|
|
||||||
|
|
||||||
For the most current evaluation results and reproducibility artifacts, consult the hive-mind research repository (canonical OSS substrate distribution). When the run record and any prose disagree, **the run record (`RESULTS.md` / `MANIFEST.json`, re-derivable by `rescore.mjs`) is canonical.**
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Document maintenance:** This document is versioned. v2 (2026-06-01) realigns the public methodology page to the canonical v5 LoCoMo run record and removes the withdrawn "+27.35-Point Methodology Gap" / "7.1-point Mem0 win" / GEPA-uplift framings. Subsequent versions will incorporate ablation results, formal confidence intervals, an MCP-path benchmark, and any additional benchmarks once they are reproduced in-repo.
|
|
||||||
|
|
||||||
License: This documentation is released under Creative Commons Attribution 4.0 International (CC BY 4.0). Code and evaluation artifacts referenced are Apache 2.0 licensed.
|
|
||||||
Reference in New Issue
Block a user