waggle-os/docs/briefs/2026-04-29-benchmark-portfolio-refresh-2026-venues.md

---
brief_id: 2026-04-29-benchmark-portfolio-refresh-2026-venues
date: 2026-04-29
session: PM coordination (Cowork)
mission: Refresh hive-mind / Waggle benchmark portfolio against Q1-Q2 2026 venue landscape; preserve PHF launch posture; sequence post-launch tracks.
predecessor_decisions:
  - decisions/2026-04-26-decision-matrix-self-judge-reframe.md   # PHF binding
  - decisions/2026-04-27-phase-2-acceptance-gate-PASS.md
  - decisions/2026-04-29-gepa-faza1-results.md
predecessor_strategy: strategy/BENCHMARK-STRATEGY.txt   # 2026-04-18; primary plan, NOT superseded
predecessor_evidence:
  - benchmarks/results/v6-self-judge-rebench/apples-to-apples-memo.md
  - benchmarks/results/v6-self-judge-rebench/self-judge-vs-trio-comparison.md
  - benchmarks/results/stage3-n400-v6-final-memo.md
  - gepa-phase-5/manifest.yaml   # claude::gen1-v1 + qwen-thinking::gen1-v1
status: AMENDMENT-PROPOSAL (does NOT supersede 04-18 BENCHMARK-STRATEGY; supplements with 2026 venue refresh + competitive intel update)
authority_required: PM (Marko Marković) ratification on §7 ratification asks
horizon: 12 weeks (pre-launch finalization → 6 weeks post-launch sequencing)
---

# PM Brief — Benchmark Portfolio Refresh: 2026 Venue Landscape

## TL;DR

The 04-18 BENCHMARK-STRATEGY.txt remains the binding primary plan. PHF launch posture (substrate ceiling 74.0 % vs Mem0 peer-reviewed 66.9 %, methodology contribution +27.35 pp) holds and ships Day 0 unchanged.

This brief proposes **three additive amendments** anchored on Q1 2026 benchmark venue developments that postdate the 04-18 strategy:

1. **Add Gaia2 (Meta SuperIntelligence Labs, arxiv 2602.11964, 12 Feb 2026)** as Phase 3 primary agent-harness venue — replaces SWE-ContextBench as headline target.
2. **Add τ³-bench banking_knowledge (Sierra, 18 Mar 2026)** as Phase 4 KVARK-track venue — replaces proprietary BPMN-workflow benchmark.
3. **Inherit ERL methodology (ICLR 2026 MemAgents Workshop)** as the publication framing for Waggle self-evolution claim — eliminates need to invent new "self-improvement convergence" metric.

Plus one update to competitive intelligence: **Hermes Agent (Nous Research, 25 Feb 2026)** is now an architectural-philosophy competitor to Waggle, not in 03-March intel doc.

Five ratification asks in §7. No code or run actions before PM response.

---

## §1 — What does NOT change

The following are LOCKED and this brief does not propose modifications:

- **PHF claim and Day 0 narrative** — substrate ceiling 74.0 % vs Mem0 peer-reviewed 66.9 % / 68.4 %, +27.35 pp methodology bias quantification, V1 retrieval honest 48.25 %. Source: `decisions/2026-04-26-decision-matrix-self-judge-reframe.md`.
- **Coupled launch sequencing** — arxiv preprint + hive-mind public + Waggle landing + Stripe in a single Day 0 window.
- **Pricing** — Solo Free / Pro $19 / Teams $49 (LOCKED 04-18).
- **GEPA Phase 5 canary deployment** — claude::gen1-v1 + qwen-thinking::gen1-v1 in flight, scope LOCKED. Cost amendment ratified 04-30 ("stavi visi slobodno").
- **Stage 3 v6 N=400 LoCoMo** — closed PASS-WITH-HONEST-FRAMING. No re-run proposed.
- **arxiv paper structure** — `research/2026-04-26-arxiv-paper/` outline and skeleton remain primary author surface.

The portfolio refresh is **post-launch sequencing**, not pre-launch revision.

---

## §2 — Landscape changes since 04-18 (binding new evidence)

### 2.1 Gaia2 (Meta SuperIntelligence Labs)

**Anchor:** arxiv 2602.11964 (Froger et al., 12 Feb 2026); ARE platform repo `facebookresearch/meta-agents-research-environments`.

**What it measures:** asynchronous agent capability in a simulated mobile environment with 12 applications and 101 tools. Agents must operate under temporal constraints, adapt to noisy/dynamic events, resolve ambiguity, and collaborate. Pass@1 with write-action verifier per scenario.

**Current SOTA (Feb-Apr 2026):**
- GPT-5 (high): 42 % pass@1 (best overall; fails on time-sensitive tasks)
- Claude-4 Sonnet: trades accuracy/speed/cost
- Kimi-K2: **21 % pass@1 — open-source SOTA**
- No Anthropic dominance; no saturation.

**Why it replaces SWE-ContextBench in our portfolio:**
- Fresh venue (post-04-18); SWE-ContextBench is now Q4 2025 vintage and OpenClaw / Hermes have not engaged it either, so first-mover narrative is weaker.
- Agent capability domain matches Waggle product surface (general agentic tool use with persistent memory) better than SWE-ContextBench (code-context retrieval narrow scope).
- Universes architecture (isolated data partitions exposing identical tools but disjoint task content) provides clean substrate for self-evolution measurement — see §2.3.
- Open-source SOTA threshold of 21 % is realistic to beat with Qwen 3.6 35B + GEPA-evolved `qwen-thinking::gen1-v1` (+12.5 pp uplift validated in-sample n=8 + held-out n=5). Target band: 30-35 % pass@1, which enters Claude/GPT-5 reference zone.

**Cost estimate:** N=200 agent-task instances × ~$0.05/instance subject + judge harness = ~$25-40 per full run. Compute envelope manageable within existing GEPA Phase 5 cost amendment.

### 2.2 τ³-bench banking_knowledge (Sierra)

**Anchor:** Sierra Research blog 18 Mar 2026; `sierra-research/tau2-bench` repo; τ-Knowledge paper (Shi et al., arxiv 2603.04370).

**What it measures:** RAG-augmented customer service in banking domain. Configurable retrieval pipelines (keyword search, embedding-based, long-context, agentic shell-based). Task success measured by correctness of backend database state changes (dispute opened, card frozen, credit issued), not conversation polish. Pass^k metric for reliability.

**Current SOTA:**
- GPT-5.2 with high reasoning: ~25 % task success.
- Even with exact required documents provided: ~40 %. Bottleneck is reasoning/execution, not retrieval.

**Why it replaces proprietary BPMN-workflow benchmark:**
- Sierra is a credentialed third-party venue; community-driven leaderboard at taubench.com with verified submissions via S3 bucket trajectories.
- Banking domain is direct match for KVARK enterprise sales (regulated industry, RAG over policy documents, audit trail of agent actions).
- Bottleneck is exactly where hive-mind should add value (frame importance weighting, bitemporal validity, I/P/B distinction for hypothesis-vs-fact reasoning).
- Headroom is large (~25 % SOTA → ceiling ~40 %); a measurable lift here is the easiest-to-defend KVARK pitch artifact for regulated buyers.
- Proprietary BPMN-workflow benchmark in 04-18 strategy has zero adoption, zero comparison anchor, zero credibility — even if we publish it, no one cites it.

**Cost estimate:** N=200 instances × ~$0.10/instance (longer dialogues with retrieval round-trips) = ~$30-50. Same envelope class as Gaia2.

### 2.3 ERL methodology (ICLR 2026 MemAgents Workshop)

**Anchor:** "Experiential Reflective Learning for Self-Improving LLM Agents" (arxiv 2603.24639, March 2026). Published as conference paper at the ICLR 2026 MemAgents Workshop.

**What it does:** retrieval of heuristics from accumulated experience, injected into agent's system prompt before execution. No modification to core ReAct loop. Evaluated on Gaia2 Search + Execution splits and τ²-bench (all three customer service domains).

**Reported result:** +7.8 % success rate uplift over ReAct baseline on Gaia2; large gains in task completion reliability; outperforms prior experiential learning methods (ExpeL, AutoGuide, Reflexion).

**Why this matters for our portfolio:**
- The 04-18 strategy implies Waggle self-evolution claim needs a custom evaluation methodology. ERL provides the methodology already, with a published baseline (+7.8 %) to beat.
- Our hive-mind frame architecture (I/P/B, importance weighting) maps cleanly onto ERL's "selective retrieval of transferable heuristics" framing — this is publishable as an ERL extension, not as a separate framework.
- MemAgents Workshop venue exists and accepts work; we have a valid conference submission target instead of inventing a venue.
- Avoids the "we invented a metric to measure ourselves" credibility problem flagged in earlier prep work.

**Implication for Waggle launch comms:** the self-evolution claim moves from "trust us, internal benchmark shows X" to "validated against published ERL baseline on Gaia2". Order-of-magnitude credibility upgrade.

---

## §3 — Competitive intelligence update

The current `Waggle_Competitive_Intelligence_Full_Landscape_March_2026.docx` is dated. One material gap requires update before Day 0 comms freeze.

### 3.1 Hermes Agent (Nous Research)

**Launch date:** 25 February 2026. **Star count:** 110 K within 10 weeks of launch. **License:** open source.

**Architecture:** closed learning loop, prompt memory (MEMORY.md, USER.md), episodic archive (SQLite FTS5), procedural skills (auto-generated markdown). Internal benchmarks claim 40 % speedup on repeat tasks.

**Why this is material:** Hermes Agent occupies the same architectural-philosophy space as Waggle. The Hermes pitch is "agent that gets better over time at your specific workflows through closed learning loop". This is functionally identical to our self-evolution narrative.

**Defensible Waggle differentiators against Hermes (must appear in Day 0 comms):**

1. Bitemporal knowledge graph (Hermes uses flat SQLite FTS5).
2. I/P/B frame model with importance weighting and superseding-via-correction (Hermes does not distinguish hypothesis from fact).
3. MPEG-4 frame architecture and wiki compiler (Hermes has neither).
4. Apache 2.0 hive-mind as standalone npm package (`@hive-mind/core` etc.) — Hermes is monolithic.
5. EU AI Act audit triggers built-in (Hermes does not address).
6. **Published peer-reviewed-style benchmark results (apples-to-apples Mem0 + ERL methodology + Gaia2 + τ³)** — Hermes publishes only internal benchmarks.

Differentiator #6 is the moat. Hermes Agent has not engaged any standardized public benchmark venue. If we ship arxiv + Gaia2 + τ³ within Q2, the gap is unbridgeable for them in 2026.

### 3.2 OpenClaw security posture (no new evidence required)

OpenClaw March 2026 CVE cluster (9 CVEs in 4 days, including CVSS 9.9; Snyk flagged 1,467 malicious skills on ClawHub) is already in the existing intelligence doc per CC-1 audit. Confirming it remains in Day 0 narrative for regulated-industry pitches as "incumbent insecurity" framing.

---

## §4 — Recommended portfolio amendment

Replace BENCHMARK-STRATEGY.txt §3.4 (Phase 3) and §3.5 (Phase 4) primary venues. All other sections remain intact.

| Phase | 04-18 strategy | Proposed amendment | Rationale |
|---|---|---|---|
| Phase 0 (now → launch) | Stripe priority; nothing else | **Unchanged.** | PHF posture stable. |
| Phase 1 (post-Stripe, hive-mind alpha) | LoCoMo + bootstrap | **Unchanged.** Stage 3 v6 already complete. | Status quo. |
| Phase 2 (hive-mind launch) | LongMemEval + blog + GitHub public | **Unchanged.** | Coupled launch as PHF binds. |
| Phase 3 (Waggle benchmark integration) | SWE-bench sequential + SWE-ContextBench | **REPLACE with Gaia2 Search + Execution splits**, GEPA-evolved variants, ERL methodology framing. Target: 30-35 % pass@1 (open-source SOTA = 21 %). | Fresher venue, better domain fit, ERL publication target. |
| Phase 4 (KVARK milestones) | Proprietary BPMN-workflow + scale benchmarks | **REPLACE BPMN with τ³-bench banking_knowledge**, retain scale + multi-tenant + compliance latency benchmarks. Target: top-3 open-source on banking_knowledge. | Real venue, real comparison, regulated-industry sales artifact. |

Stretch targets in 04-18 strategy (BEAM 1M-token, SWE-ContextBench Memory track) deferred to Q3 2026 review.

---

## §5 — Sequencing (12-week horizon)

**Weeks 0-2 (now → hive-mind alpha):** PHF locked artifacts ship — arxiv preprint, hive-mind public, Waggle landing, Stripe. **No new benchmark work in this window.**

**Weeks 2-4 (post-launch consolidation):** Update `Waggle_Competitive_Intelligence_Full_Landscape_*` with Hermes Agent entry. Re-run any pitch deck slides that reference the outdated competitive landscape.

**Weeks 4-8 (Phase 3 Gaia2 sprint):**
- Week 4: Set up ARE platform locally; verify GEPA-evolved `qwen-thinking::gen1-v1` runs against Gaia2 Search split with no harness modification.
- Week 5: ERL-style heuristic retrieval wiring from hive-mind into agent system prompt (existing `retrieval-agent-loop.ts` is the integration point — 38.3 KB file already does adjacent work).
- Week 6: N=200 dry run on Search split; cost validation under $50.
- Week 7: Full Search + Execution split run, both ReAct baseline and ERL-augmented; trio-strict + self-judge dual reporting per PHF methodology lesson.
- Week 8: Results memo + arxiv submission to MemAgents Workshop or follow-on venue.

**Weeks 8-12 (Phase 4 τ³ sprint, KVARK track):**
- Week 8: Set up tau2-bench locally with banking_knowledge extras (`uv sync --extra knowledge`).
- Week 9: hive-mind retrieval pipeline integration as RAG provider; verify it satisfies tau2-bench `RetrievalProvider` interface.
- Week 10: N=100 dry run; calibrate per-task cost and latency.
- Week 11: Full N=200 run, frontier subject (Opus 4.7 + GPT-5.4) + Qwen subject for sovereignty story.
- Week 12: Submit results to taubench.com community leaderboard; produce KVARK enterprise sales one-pager with verified third-party broj.

---

## §6 — Risks and out-of-scope items

### 6.1 Risk: GEPA +12.5 pp uplift on N=13 may not generalize to Gaia2 task distribution

The held-out validation is statistically thin (N=5 held-out + N=8 in-sample). Gaia2 task distribution differs materially from in-sample evolution corpus (mobile environment, 12 apps, 101 tools vs analytical scenarios). Mitigation: Phase 3 sprint Week 6 dry run is the explicit checkpoint; if uplift collapses, halt and PM-escalate before Week 7 full run. Cost exposure if abort: ~$15.

### 6.2 Risk: ERL methodology reference point may shift before Workshop submission

The +7.8 % uplift is from the ERL paper as published. Other ERL extensions may publish between now and our submission window. Mitigation: framing should be "we extend ERL with bitemporal-KG-conditioned retrieval", not "we beat ERL by X". Defensible regardless of intermediate competitor work.

### 6.3 Risk: Hermes Agent or OpenClaw publish on Gaia2 / τ³ before us

Probability: low for OpenClaw (CVE remediation is consuming community bandwidth); medium for Hermes (Nous Research has paper-publishing track record). Mitigation: weeks 4-8 timeline above is aggressive; if Hermes publishes first, framing pivots to "Waggle vs Hermes head-to-head on Gaia2" rather than first-mover. Either way the published broj is the enterprise sales artifact.

### 6.4 Out of scope (explicitly)

- Any change to PHF claim, Day 0 narrative, coupling decision, or pricing.
- Frontier subject re-run of Stage 3 LoCoMo. (Earlier consideration deprecated by 04-25 self-judge re-eval evidence.)
- SWE-bench sequential learning curve experiment (Phase 3 in 04-18 strategy). Deferred to Q3 review pending Phase 3 Gaia2 results.
- New benchmark venue invention (StuLife, J-TTL, FieldWorkArena). Stick to community-recognized venues.

---

## §7 — Ratification asks

PM ratification required on the following five items before any Phase 3 or Phase 4 sprint kickoff. None blocks Day 0 launch.

1. **Ratify Gaia2 as Phase 3 primary agent-harness benchmark venue**, replacing SWE-ContextBench. (Y/N)
2. **Ratify τ³-bench banking_knowledge as Phase 4 KVARK-track primary venue**, replacing proprietary BPMN-workflow benchmark. (Y/N)
3. **Ratify ERL methodology inheritance** as the framing for Waggle self-evolution claim, with publication target = ICLR 2026 MemAgents Workshop or comparable venue. (Y/N)
4. **Ratify Hermes Agent competitive intelligence amendment** (§3.1) as binding update to `Waggle_Competitive_Intelligence_Full_Landscape_*` document. PM authorizes Marketing-side rewrite or assigns to CC. (Y/N + assignee)
5. **Ratify 12-week sequencing** in §5, with Weeks 4-12 Phase 3 + Phase 4 sprints contingent on successful Day 0 launch and post-launch consolidation Weeks 2-4. (Y/N)

After ratification, this brief becomes binding addendum to BENCHMARK-STRATEGY.txt; phase tables in §3.4 and §3.5 of that document are superseded by §4 of this brief. All other sections of 04-18 strategy remain primary.

---

## §8 — Cross-references

- 04-18 primary strategy: `strategy/BENCHMARK-STRATEGY.txt` (NOT superseded; supplemented).
- PHF binding decision: `decisions/2026-04-26-decision-matrix-self-judge-reframe.md`.
- Apples-to-apples Mem0 evidence: `benchmarks/results/v6-self-judge-rebench/apples-to-apples-memo.md` (in waggle-os repo).
- GEPA Phase 5 substrate: `gepa-phase-5/manifest.yaml` + `gepa-phase-5/preflight-evidence.md` (in waggle-os repo).
- arxiv paper anchor: `research/2026-04-26-arxiv-paper/00-paper-outline.md`.
- Existing competitive intel: `Waggle_Competitive_Intelligence_Full_Landscape_March_2026.docx` (in waggle-os repo root).
- Gaia2 paper: arxiv 2602.11964.
- τ³-bench / τ-Knowledge paper: arxiv 2603.04370.
- ERL paper: arxiv 2603.24639.
- Sierra leaderboard: taubench.com.
- ARE platform: github.com/facebookresearch/meta-agents-research-environments.

---

(2,847 words)