3.0 KiB
Judge-Delta — GAIA 2 search N=160 self-judge vs independent trio
Date: 2026-05-22 · Phase 1 (judge integrity) of the harness benchmark plan
Purpose: the search split is judged ~entirely by the LLM user_message_checker
(semantic equivalence of the agent's final answer vs the oracle answer; no app-action
oracle events to hard-match). The production run self-judged (Sonnet 4.6 judging a
Sonnet 4.6 agent), so we re-judged with an independent trio to detect inflation.
Method (controlled — only the judge model varies)
Imported GAIA 2's own user_message_checker — same system prompt, same few-shot
examples, same [[Success]]/[[Failure]] parsing (gaia2_core.judge.prompts +
LLMChecker). For each of the 148 answerable scenarios extracted (task, agent final
message, oracle reference) and re-ran the checker with three independent judges. Only
the judge MODEL changed. Mirrors the C-1 LOCOMO trio-strict discipline.
Judges: Opus 4.7 · Gemini 2.5 Pro · GPT-5 (M6 roster, all independent of the
Sonnet 4.6 agent+self-judge). Engine omits temperature uniformly (Opus 4.7 / GPT-5
reject it); num_votes=1. Script: rejudge_user_message.py. Raw: runs/rejudge-search-n160.jsonl.
Denominator = 148 answerable (160 scenarios − 12 with no final answer / errored).
Results
| Judge | PASS | Rate (n=148) |
|---|---|---|
| Self — Sonnet 4.6 (production) | 134 | 90.5% |
| Opus 4.7 | 134 | 90.5% |
| Gemini 2.5 Pro | 133 | 89.9% |
| GPT-5 | 132 | 89.2% |
| Trio-strict (all 3 independent agree PASS) | 129 | 87.2% |
| Trio-majority (≥2/3 independent PASS) | 135 | 91.2% |
Agreement with the self-judge: Opus 98.6% · Gemini 98.0% · GPT-5 97.3%.
Verdict — self-judge is NOT inflated
- Independent single judges land within 0.6–1.3pp of the Sonnet self-judge.
- Trio-strict is only −3.3pp below self (90.5% → 87.2% on n=148) — well within cross-LLM-judge norms, and far tighter than LOCOMO's +5.3pp self-judge inflation.
- Only 6/148 (4%) scenarios are marginal (self-PASS but not unanimous across the trio):
21_csyctc, 22_52pwi3, 23_onhtod, 26_oqrx9a, 30_69r1z7, 30_9uo633.
The 83.8% GAIA 2 search result was not a self-grading artifact. The harness-quality signal is real and independently confirmed.
Mapping to the full split (N=160 denominator)
| Metric | Self (Sonnet) | Trio-strict |
|---|---|---|
| n=148 answerable | 90.5% | 87.2% |
| N=160 full (errors/unanswerable count against) | 83.8% | ~80.6% (129/160) |
Defensible public framing: "GAIA 2 search split, N=160 — 83.8% self-judged, independently confirmed at 80.6% trio-strict (Opus 4.7 + Gemini 2.5 Pro + GPT-5, GAIA 2's own checker, judges agree 97–99%)."
Caveat (carried forward)
This validates the harness-cell judging, but the harness was Hermes (a third-party
reference agent), NOT Waggle's own runAgentLoop. Pillar-1 (Waggle harness SOTA) still
requires the waggle_worker build to put Waggle's loop in the same rig. See plan doc.