# GOAL STATEMENT — Agent Harness Benchmark (local-first, sovereign) **Date:** 2026-05-22 **Owner:** Marko (PM) · drives benchmark design + execution **Status:** DRAFT goal statement — hand to `/goal` → `/plan` once the §0 decision is locked --- ## 0. LOCKED — Reading B: Waggle is the arena + governance layer (2026-05-22) **Decision (Marko, 2026-05-22): LOCKED to Reading B.** Waggle OS is the **local-first OS that orchestrates** every one of these harnesses (the AI-OS arc: detect → launch Claude Code, Codex, Cursor, Hermes, OpenClaw… locally, with full audit). We do **not** position Waggle as a competing agent loop. We **run all of them safely on-prem** and publish the head-to-head comparison matrix as a **sovereign buyer's guide + governance proof**. **Claim shape:** *"Run any agent harness locally — fully audited, zero data egress — and here's exactly how each one performs in that sovereign environment."* **Why B (rationale of record):** - Matches the mission verbatim — "onboarding → push toward KVARK as full sovereign AI orchestration + governance." That is an orchestration/governance story, not a "our agent loop beats Codex" story. - The matrix becomes a **durable buyer asset** (a comparison guide buyers trust *because* we don't have a horse in the capability race) rather than a fragile "we're #1" claim that a single model/harness upgrade invalidates. - Waggle's credibility comes from being the **neutral, auditable, local-first home** for whichever harness the customer already trusts — which is exactly the KVARK pitch one tier up. **Consequence for design:** No Waggle→ARE adapter is required. Waggle's role is measured as the **execution+governance substrate** (it launches the harness, isolates it, captures the audit trail), and the protagonist metric set shifts from "Waggle's pass rate" to "the sovereignty triple (local-first / zero-egress / auditable) holds across ALL harnesses, and here is each harness's capability/cost/reliability profile when run inside Waggle." Reading A (Waggle's own loop as a 6th competitor) is explicitly **deferred** — it can become a later, narrower claim only if Waggle's loop proves differentiated, and is out of scope for this benchmark. --- ## 1. Objective — TWO co-equal product pillars (Marko, 2026-05-22) Waggle-the-product = **agent harness + memory substrate**, so the capability story needs **both proofs, co-equal** (not one headline + one footnote): - **Pillar 1 — Agent-harness SOTA.** Waggle's OWN harness (`runAgentLoop`) benchmarked head-to-head vs reference harnesses (Hermes/OpenClaw, Oracle ceiling) in the local-first GAIA 2 rig, same model + judge + scenarios → prove Waggle's loop is at/near SOTA. (NOTE: the existing 83.8% used third-party Hermes, NOT Waggle — see plan doc DIRECTION UPDATE.) - **Pillar 2 — Memory SOTA.** Waggle's hive-mind substrate on memory benchmarks (LoCoMo done in C-1 → **LongMemEval** near-term → **BEAM** flagship) → prove the memory substrate is at/near SOTA, ideally beating frontier long-context (the one axis where Waggle *wins*, not just matches). Both feed **defensible, dual-tier (peer-review + hero-page) statements**, positioning Waggle OS as where knowledge workers and sovereign-AI buyers run agentic work without data leaving the perimeter, and as the on-ramp to KVARK. The sovereignty triple (local-first / zero-egress / auditable) wraps both pillars. ## 2. Subject under test + comparison set | Entity | Role (per §0 — Reading B) | |---|---| | **Waggle OS** | the **arena + governance substrate** — launches/isolates/audits each harness locally. Measured by the sovereignty triple holding across all harnesses, not by a pass rate of its own. | | Hermes | harness-under-test · ARE-native reference agent (already wired — N=160 done) | | OpenClaw | harness-under-test · ARE-native reference agent (config scaffolded) | | Claude Code | harness-under-test · external coding/agent harness | | Codex | harness-under-test · external coding/agent harness | | Claude Cowork | harness-under-test · external agent product | **Controlled-variable principle (non-negotiable):** the harness is the ONLY variable. Same benchmark, **same model (Claude Sonnet 4.6) where the harness allows model choice**, same judge model + same judge protocol, same scenario set, same denominator. Anything else and the comparison is not defensible. **Waggle is held constant as the environment under all of them** — so any harness's number is also implicitly a "this ran inside Waggle, locally, audited" number. ## 3. Environment constraint — local-first is itself a measured property Everything runs **locally / on-prem** (Docker, hermetic, laptop-runnable). For the sovereign-AI audience this is not a footnote — it's a headline claim. Capture and assert: - **Zero data egress** during execution (network-isolated containers; prove it). - **Full auditability**: every tool call captured in `events.jsonl` / trace → this *is* the KVARK governance hook. - **Reproducibility**: hermetic, runs on Marko's Windows hardware (already proven for Hermes). ## 4. What to measure (harness quality is multi-dimensional) Pass rate alone is a thin claim. Measure per harness, per GAIA 2 split (search / execution / adaptability / time / ambiguity / noise): 1. **Capability** — strict pass rate + judged-only pass rate (report both; errors counted honestly). 2. **Efficiency** — tokens & $ per task, tool-calls per task, wall-clock. 3. **Reliability** — error rate, recovery, determinism across reruns. 4. **Sovereignty/safety** — local-first ✓, egress=0 ✓, trace-auditability ✓ (binary asserts, per harness). ## 5. Two deliverable tiers (different bars — do not blur) - **Tier 1 — Publishable** (paper / arxiv / KVARK technical annex): pre-registered protocol, N≥160 per cell, CI reported, judge protocol fixed in advance, no post-hoc baseline shopping. The GAIA 2 N=160 Hermes run is the first cell of this matrix. - **Tier 2 — Hero-page** (waggle-os.ai + KVARK deck): punchy but every number traces back to a Tier-1 cell. Honest framing only. E.g. *"Run Codex, Claude Code, or Hermes locally — fully audited, your data never leaves your machine."* ## 6. Success criteria (what counts as a win) - A **completed comparison matrix**: {6 harnesses} × {GAIA 2 splits} × {4 metric families}, same protocol throughout. - At least one **Tier-1 publishable** statement that survives peer-review scrutiny. - At least one **Tier-2 hero-page** statement that is punchy AND traces to a Tier-1 cell. - The **sovereignty triple** (local-first / zero-egress / auditable) demonstrated, not asserted. ## 7. Non-goals / guardrails (honesty bar) - **Not a memory-substrate proof.** That is C-1 (LOCOMO 67.8% trio-strict) + C-2 (Stage 3 +19.25pp, p=8e-18). GAIA 2 measures the *harness*, not hive-mind memory. Keep the lanes separate in every artifact. - **Kill the apples-to-oranges baseline.** Do NOT publish "Waggle 83.8% vs Mem0 ~40-55%" — different judge/denominator/protocol. Every comparison number must come from OUR matrix under identical protocol, or be dropped. - **Judge-leniency risk.** A self-judge (Sonnet judging Sonnet) inflates. For Tier-1, use an independent / ensemble judge and report the self-vs-independent delta (same discipline as C-1 trio-strict). - **No goalpost-moving, no hiding errors.** Strict + judged-only always reported together. ## 8. What already exists (starting point) - ✅ GAIA 2 ARE local-first pipeline runs on Windows Docker (Hermes + Sonnet 4.6), patches captured. - ✅ First matrix cell: **Hermes × search × N=160 = 83.8% strict / 86.5% judged-only.** - 🔲 OpenClaw cell (config scaffolded, not run). - 🔲 Claude Code / Codex / Claude Cowork adapters (do these expose an ARE-compatible runtime? — research task; first design question). - ⛔ Waggle→ARE adapter — **out of scope** (Reading A deferred per §0). - 🔲 Independent/ensemble judge wiring for Tier-1. - 🔲 Egress=0 proof harness (the sovereignty triple — protagonist metric for Reading B). --- ## TL;DR for `/goal` > **Benchmark agent-harness quality head-to-head (Waggle vs Hermes, OpenClaw, Claude Code, Codex, Claude Cowork) on GAIA 2, in a local-first / zero-egress / fully-audited environment — holding model + judge + scenarios constant so the harness is the only variable — to produce both peer-review-publishable and honest hero-page claims that onboard knowledge workers and funnel sovereign-AI buyers toward KVARK.** > > §0 LOCKED to Reading B (Waggle = orchestrator + governance layer, not a competing loop). First cell done (Hermes 83.8%). Next: research which external harnesses expose an ARE-compatible runtime, wire the independent judge, build the egress=0 proof, then fill the matrix.