Phase 4 P4.5 — GAIA 2 search split FULL RUN N=160 RESULTS

Date: 2026-05-22 Branch: feature/gaia2-are-setup Agent: Hermes runtime (localhost/gaia2-hermes:latest) + Anthropic claude-sonnet-4-6, thinking=high Judge: claude-sonnet-4-6 (in-container daemon) Dataset: meta-agents-research-environments/gaia2-cli, split search, N=160 (full split) Cost: ~$91 extrapolated (~165 scenario-executions × ~$0.55), under the $100 hard cap (Marko amendment 2026-05-21)

Headline

Metric	Value
Strict pass rate	134/160 = 83.8%
Judged-only pass rate	134/155 = 86.5% (excludes 5 harness-undecidable)
FAIL	21
ERROR (no turn boundary)	5
Mem0 paper baseline (Hermes + Sonnet, search split)	~40–55% pass@1

The ARE-native architecture clears the Mem0 baseline by ~30–45pp at full N=160. CI at N=160 is ~±6pp (Wilson, p=0.84), so the result is robust — not a small-N artifact.

How this run completed (provenance — it was a two-part run)

First pass (waggle_phase4_full_n160.toml, concurrency=2, 2026-05-21 20:02 → 2026-05-22 01:43): reached 141/160 before the CC session crashed and killed the runner process. Partial: 117 PASS / 19 FAIL / 5 ERROR. All 141 result.json survived on disk.
Finish pass (waggle_phase4_finish_subset.toml, 2026-05-22 03:35 → 04:45): a deterministic 24-scenario subset (the 5 errored + 19 never-run), written to the SAME output dir so the results merge into a clean N=160 aggregate.

Why a subset manifest instead of `--retry`

The runner's built-in --retry over-selected: it picked 48 scenarios = the correct 24 (missing + errored) PLUS 24 already-terminal PASS/FAIL that it re-queued needlessly. That would have cost ~$26 (breaching the $100 cap) and re-rolled good verdicts. Root cause not fully isolated (paths are uniform search/<id>/result.json, so it is not a directory-structure mismatch — the _select_retry_scenarios selector misclassifies a specific 24-scenario subset). The subset-manifest path ([target].subset_manifest listing exact IDs) sidesteps it entirely and is deterministic.

The 5-error floor (honest framing)

Errors are all Daemon error: no turn boundary detected — the agent ran real work (12–41 tool calls) but the in-container judge daemon never observed a clean turn close before idle-timeout. Re-running swapped which scenarios error rather than reducing the count (2 of the original 5 resolved; 2 new appeared), so this is a stable ~3% runner turn-detection floor on long scenarios, not agent failure and not flaky randomness.

Persistent errors at N=160: 27_azdwq4, 28_ofdzab, 29_f3f586, 30_48xm0t, 30_o7av0c.

Honest reporting: 155 judged, 5 undecidable on this harness → report both the strict (83.8%, counts errors against us) and judged-only (86.5%, excludes them) numbers. Strict is the defensible public figure; judged-only is the architecture-capability figure.

Open follow-ups (not blocking the result)

4th Windows unicode gap: trace_viewer.py HTML generation fails on ▸ (▸) with cp1252 — emitted as WARNINGs, affects only the per-scenario trace HTML, NOT result.json grading data. Patch 3 (runner.py artifact writes) does not cover the trace viewer. Logged in WINDOWS-DOCKER-RUNNER-PATCHES.md as the 4th patch needed for the upstream PR.
Turn-boundary floor: worth an upstream issue — daemon should treat a long idle after N tool calls with no further events as a soft turn-close + judge, rather than erroring.

Authority chain

Budget: Marko $80→$100 amendment 2026-05-21
Architecture validation: PHASE-4-P4.2-PROGRESS-2026-05-21.md §2 (ARE-native, targeted API calls)
N=10 probe: PHASE-4-P4.2-PROGRESS-2026-05-21.md §8 (8/10 strict, 8/8 judged-only)
Runner patches: WINDOWS-DOCKER-RUNNER-PATCHES.md
Artifacts: runs/p4-full-hermes-n160/ (160 scenario dirs + index.html + results.jsonl)

4.0 KiB Raw Blame History Unescape Escape