14 KiB
Decision Matrix Amendment — Self-Judge Re-Eval Reframe
Date: 2026-04-26
Author: PM
Status: Amendment to 2026-04-25-launch-gate-reframe-decision-matrix.md (does NOT supersede; supplements with new ground truth layer)
Trigger: 2026-04-25 apples-to-apples self-judge re-evaluation (Stage 3 v6 outputs re-judged using Mem0 paper methodology) revealed methodology gap that changes interpretation of "PASS / PARTIAL / FAIL" bands without altering the bands themselves.
§0 — Why this amendment exists
The 04-25 Decision Matrix locked three pre-registered narrative bands tied to the 91.6% Mem0 marketing reference:
| Aggregate LoCoMo | Banner | Coupling decision |
|---|---|---|
| ≥ 91.6 | NEW_SOTA | Coupled launch |
| 85.0-91.5 | SOTA_IN_LOCAL_FIRST | Decoupled launch |
| < 85.0 | GO_NOGO_REVIEW | Halt SOTA narrative |
The 04-25 self-judge re-eval discovered that 91.6% is not a peer-reviewed number. It is a Mem0 marketing claim from mem0.ai/blog/state-of-ai-agent-memory-2026. The peer-reviewed Mem0 paper (arxiv:2504.19413) reports 66.9% basic / 68.4% graph on LoCoMo.
Apples-to-apples re-judging of our N=2000 Stage 3 v6 outputs using Mem0's exact methodology (GPT-4o-mini as both subject and single-vendor judge) produced aggregate scores +27.35 percentage points higher than our trio-strict ensemble. This methodology gap accounts for most of the spread between Mem0's marketing claim and Mem0's peer-reviewed paper.
Implication: comparing our trio-strict numbers against Mem0's marketing number is methodologically invalid. The fair comparison anchors are:
- Our trio-strict 74% (oracle ceiling) vs Mem0 peer-reviewed 66.9% / 68.4%
- Our trio-strict 48% (V1 retrieval) vs Mem0 peer-reviewed 66.9% / 68.4%
This amendment does not shift pre-registered thresholds (anti-pattern #4 honored). It adds an honest framing layer anchored on peer-reviewed comparison instead of marketing comparison.
§1 — Numerical anchor reset
| Dimension | Pre-04-25 anchor | Post-04-25 anchor (binding) |
|---|---|---|
| Mem0 SOTA reference | 91.6% (marketing) | 66.9% / 68.4% (peer-reviewed, arxiv:2504.19413) |
| Comparison methodology | informal | apples-to-apples trio-strict ensemble vs peer-reviewed paper |
| Our substrate ceiling | not measured | 74% (trio-strict, oracle context, N=400) |
| Our V1 retrieval | not measured separately | 48% (trio-strict, N=400) |
| Methodology bias measurement | unknown | +27.35pp (self-judge inflates aggregate vs trio-strict on our outputs) |
| Statistical significance H1 | pre-registered | Fisher one-sided p < 8.07e-18 (PASS) |
Result frame: substrate ceiling +7.1pp over peer-reviewed Mem0; V1 retrieval −18.9pp under peer-reviewed Mem0; ceiling-to-V1 gap 26pp is the open V2 work direction.
§2 — Scenario re-classification (8 dimensions revisited)
The 04-25 matrix asked: which of PASS / PARTIAL / FAIL scenario applies, given final aggregate LoCoMo number?
The 04-26 reframe asks: given that we have substrate ceiling 74% (PASS-shaped) and V1 retrieval 48% (FAIL-shaped against marketing claim, FAIL-shaped against peer-reviewed Mem0), which scenario is binding?
Resolution: dual-narrative — substrate quality narrative + retrieval honesty narrative. This is not a hybrid scenario; it is a re-anchoring against peer-reviewed baseline instead of marketing baseline.
Re-classified scenario: PASS-WITH-HONEST-FRAMING (PHF)
PHF replaces the 04-25 PASS/PARTIAL/FAIL trichotomy with a single coherent narrative anchored on:
- Substrate ceiling beats peer-reviewed Mem0 (74% vs 66.9% / 68.4%) — defensible, peer-review-survivable, anti-marketing.
- V1 retrieval honest disclosure (48%, 18.9pp under peer-reviewed, 26pp under our ceiling) — community invitation framing.
- Methodology contribution (+27.35pp self-judge bias quantification) — paper-grade epistemic value beyond product claim.
- Apache-2.0 + local-first — sovereignty axis intact regardless of retrieval V1 gap.
Why PHF is not a "post-hoc threshold shift"
The 04-25 thresholds were tied to marketing-anchor comparison. The new ground truth (peer-reviewed comparison + measured methodology bias) was discovered AFTER outputs were generated but BEFORE launch. Discovery of new measurement (apples-to-apples re-judging) is not a threshold shift — it is a methodology audit. The pre-registered bands remain binding for any future comparison against the marketing anchor; we just no longer use the marketing anchor as primary because it is methodologically invalid.
Audit-trail position: 2026-04-25-self-judge-rebench-results.json (pending — to be emitted during pilot completion adjudication or written separately) is the artifact that justifies the re-anchor. Anti-pattern #4 honored: thresholds did not move; the comparison anchor moved because the prior anchor was unreliable.
§3 — 8-dimension decision update
For each of 04-25 matrix's 8 decision dimensions, this section locks the PHF disposition.
Dimension 1 — Claim narrative (PHF binding)
Lead claim: "Hive-Mind exceeds peer-reviewed Mem0 baseline at substrate ceiling: 74% vs 66.9% on LoCoMo, apples-to-apples trio-strict judge ensemble. V1 retrieval is honest at 48% — V2 in progress, community invited under Apache-2.0."
Supporting:
- pre-registered manifest v6 (anchor dedd698)
- κ_trio = 0.7878 substantial agreement on judge ensemble
- Fisher one-sided p < 8.07e-18 (H1 PASS)
- methodology contribution: +27.35pp self-judge bias quantified
Honest caveats (always present):
- single-benchmark (LoCoMo only); LongMemEval cross-replication in progress
- substrate ≠ end-to-end product; V1 retrieval gap acknowledged
- N=400 sub-sample of LoCoMo-1540 (not full benchmark; pre-registered N)
What we do NOT claim:
- "we beat Mem0" without qualification (substrate ceiling vs peer-reviewed paper, not commercial product)
- "91.6% comparable" (methodology contribution shows that figure is +27pp inflated; we do not engage with the marketing number as an equivalent target)
- "SOTA on memory systems end-to-end" (we claim SOTA on substrate quality; product-level depends on retrieval V2)
Dimension 2 — Launch coupling (PHF binding)
Coupled launch — Day 0 ships everything in one window:
- arxiv preprint live (preferably Day 0 -3 days for pickup window)
- hive-mind core public (GitHub, Apache-2.0, npm + PyPI)
- hive-mind-clients monorepo public (2 MVP shims Day 0: Claude Code + Cursor; Hermes + others within 2 weeks)
- Waggle landing live (waggle-os.ai) with v3 copy
- Waggle desktop downloadable (Solo free)
- Stripe checkout active for Pro $19 / Teams $49
- Technical blog post + LinkedIn long-form + Twitter thread + HN/Reddit submissions synchronized within 30-min window
Why coupled (vs 04-25 PARTIAL decoupling): substrate-ceiling-beats-peer-reviewed claim is strong enough to anchor a coupled launch. We don't need to decouple to defend a smaller win because the substrate win IS the win — V1 retrieval gap is a feature of the open-source separation framing, not a weakness to hide.
Dimension 3 — Pricing (UNCHANGED, LOCKED 04-18)
Solo Free / Pro $19/mo / Teams $49/seat/mo. Self-judge re-eval did not change pricing rationale. Decision LOCKED.
Dimension 4 — Audience (PHF amendment)
Day 0 primary audience (sequencing matters):
- Technical AI engineers building production agents — they read papers, evaluate substrate vs retrieval distinctly, recognize the Apache-2.0 + local-first value.
- Regulated industry technologists — banking, healthcare, legal — sovereignty axis lands directly.
- Boutique consultants and executive advisors — Waggle Pro / Teams target audience.
Day 0 NOT primary audience:
- Memory-product-shopping casual builders (they will compare 91.6% marketing vs our 48% V1 and walk away; we don't compete on that comparison)
- Enterprise SaaS buyers (KVARK enterprise sovereign deployment is separate sales motion, post-launch)
Dimension 5 — Press / PR (PHF binding)
Tier 1 outreach Day 0:
- HN front page (technical engineers)
- Linkedin Marko long-form (executive advisor / consultant audience)
- arxiv preprint announcement on Twitter via author network
Tier 2 outreach Day 0+1 to Day 0+7:
- The Information / Stratechery (industry analyst tier)
- VentureBeat / TechCrunch AI section
- Selected newsletter authors (Latent Space, Last Week in AI)
Tier 3 — methodology contribution outreach (Day 0+14 onward):
- AI evaluation / benchmark authors (could land us in academic discussion of memory evaluation methodology)
- This is the lever for sustained credibility beyond launch news cycle
Dimension 6 — Hires (PHF amendment)
Pre-launch (no change from 04-25): no net new hires before launch. Marko + existing Egzakta team + AI assistance executes Day 0.
Post-launch (PHF specific): prioritize hiring 1 retrieval engineer to drive V2 work. The +26pp ceiling-to-V1 gap is the most-actionable engineering surface; closing 50% of that gap closes the gap to peer-reviewed Mem0 product-level. Single engineer can drive months of V2 work.
Secondary post-launch hire: 1 community / DevRel for Apache-2.0 + shim adoption. The OSS community contribution thesis depends on community presence we currently don't have full coverage on.
Dimension 7 — Investor (PHF binding)
Investor narrative (if/when investor conversation is appropriate):
"Hive-Mind is the architectural memory substrate that exceeds peer-reviewed Mem0 baseline. Open-source under Apache-2.0 with local-first sovereignty as default. We separate substrate from retrieval, which makes us the first memory system that can be honestly compared layer-by-layer. Waggle is the funded consumer product on top; KVARK is the enterprise sovereign deployment. Three products, one substrate, one founding team."
Defensive prep (anticipated investor objection):
- "Your retrieval V1 is below Mem0 product-level." Response: "Yes, by design. We open-sourced the substrate; retrieval is community-pluggable. Mem0 sells a closed bundle. Our durability is in the substrate, not the bundling."
- "91.6% sounds like SOTA." Response: "It's marketing, not peer-reviewed. Peer-reviewed Mem0 is 66.9%. We measured the methodology gap and published it. Investor due diligence should not anchor on marketing numbers."
Dimension 8 — KVARK timing (UNCHANGED, post-launch sequencing)
KVARK enterprise sovereign deployment GTM motion remains post-launch (week 6+). Hive-Mind + Waggle launch Day 0 generates demand; KVARK sales conversations begin once we have audit-able evidence of customer adoption + regulated-industry inbound. Decision LOCKED.
§4 — Cross-walk: 04-25 bands → 04-26 PHF
For audit-trail purposes, this section maps the 04-25 pre-registered bands to the 04-26 PHF disposition without shifting the bands themselves.
| 04-25 band | 04-25 result interpretation | 04-26 reframe |
|---|---|---|
| ≥ 91.6 NEW_SOTA | "Beat marketing reference" | Not applicable — marketing reference is invalid for comparison |
| 85.0-91.5 SOTA_IN_LOCAL_FIRST | "Clean win in local-first quadrant" | Subsumed into PHF; sovereignty axis preserved |
| < 85.0 GO_NOGO_REVIEW | "Reframe required" | Not applicable — we have substrate ceiling beat against valid peer-reviewed baseline |
The 04-25 trichotomy was conditioned on a measurement comparison that turned out to be invalid. PHF is the consequence of running the comparison properly (against peer-reviewed paper) rather than improperly (against marketing claim).
If someone in 6 months argues "you should have hit 91.6", the audit-trail response is:
- 91.6 was never peer-reviewed (link to Mem0 paper showing 66.9 / 68.4)
- We measured the methodology gap (link to self-judge re-eval data)
- We published the methodology contribution alongside the architectural contribution (link to arxiv paper)
- We anchored launch comms on peer-reviewed comparison (link to landing copy v3)
This is the disciplined path — not threshold-shifting, but anchor-correcting.
§5 — Locked elements (preserved verbatim from 04-25)
These remain binding. Self-judge re-eval did not affect them.
- Pricing: Solo Free / Pro $19 / Teams $49 (LOCKED 04-18)
- KVARK GTM timing: post-launch (LOCKED 04-25)
- Pre-registered manifest v6 anchor:
dedd698(LOCKED 04-24) - Trio judge ensemble: Opus 4.7 + GPT-5.4 + MiniMax M2.7 (LOCKED 04-24 v6 Phase 1)
- κ_trio threshold for trio-strict: ≥ 0.61 substantial agreement band (LOCKED Sprint 10)
- 5-cell ablation pre-registered: no-context / oracle / full-context / retrieval / agentic (LOCKED manifest v6 §3)
- Sample size N=400, seed 42 (LOCKED manifest v6 §5)
§6 — Open items pending Marko ratification
- Ratify PHF as binding scenario for launch posture (Y/N).
- Ratify dual-narrative claim structure (substrate ceiling + V1 honest + methodology contribution + sovereignty) as launch comms anchor (Y/N).
- Ratify coupled-launch sequencing (Dimension 2) given PHF — this differs from 04-25 PARTIAL decoupling recommendation.
- Ratify Day 0 audience prioritization (Dimension 4) — technical engineers + regulated tech + boutique consultants.
- Ratify post-launch hire prioritization (Dimension 6) — retrieval engineer first, DevRel second.
After Marko ratifies, this document becomes binding. Launch comms templates (overnight 04-25 brief) populate with verbatim PHF numbers + framing. Landing copy v3 (04-26 brief) is consistent with PHF scenario.
§7 — Cross-references
- Original Decision Matrix:
decisions/2026-04-25-launch-gate-reframe-decision-matrix.md - Pre-fill recommendations:
decisions/2026-04-25-pm-pre-fill-decision-matrix-recommendations.md - Overnight execution log:
decisions/2026-04-25-overnight-pm-execution-log.md - Pilot decision template (separate from this matrix):
decisions/2026-04-26-pilot-decision-template.md - Landing copy v3:
briefs/2026-04-26-landing-copy-v3.md - arxiv paper outline:
research/2026-04-26-arxiv-paper/00-paper-outline.md - arxiv paper skeleton:
research/2026-04-26-arxiv-paper/01-paper-skeleton.md