63 lines
3.0 KiB
Markdown
63 lines
3.0 KiB
Markdown
# Judge-Delta — GAIA 2 search N=160 self-judge vs independent trio
|
||
|
||
**Date:** 2026-05-22 · Phase 1 (judge integrity) of the harness benchmark plan
|
||
**Purpose:** the search split is judged ~entirely by the LLM `user_message_checker`
|
||
(semantic equivalence of the agent's final answer vs the oracle answer; no app-action
|
||
oracle events to hard-match). The production run self-judged (Sonnet 4.6 judging a
|
||
Sonnet 4.6 agent), so we re-judged with an independent trio to detect inflation.
|
||
|
||
## Method (controlled — only the judge model varies)
|
||
|
||
Imported GAIA 2's **own** `user_message_checker` — same system prompt, same few-shot
|
||
examples, same `[[Success]]/[[Failure]]` parsing (`gaia2_core.judge.prompts` +
|
||
`LLMChecker`). For each of the 148 answerable scenarios extracted (task, agent final
|
||
message, oracle reference) and re-ran the checker with three independent judges. Only
|
||
the judge MODEL changed. Mirrors the C-1 LOCOMO trio-strict discipline.
|
||
|
||
Judges: **Opus 4.7 · Gemini 2.5 Pro · GPT-5** (M6 roster, all independent of the
|
||
Sonnet 4.6 agent+self-judge). Engine omits `temperature` uniformly (Opus 4.7 / GPT-5
|
||
reject it); `num_votes=1`. Script: `rejudge_user_message.py`. Raw: `runs/rejudge-search-n160.jsonl`.
|
||
|
||
Denominator = **148 answerable** (160 scenarios − 12 with no final answer / errored).
|
||
|
||
## Results
|
||
|
||
| Judge | PASS | Rate (n=148) |
|
||
|---|---:|---:|
|
||
| **Self — Sonnet 4.6** (production) | 134 | **90.5%** |
|
||
| Opus 4.7 | 134 | 90.5% |
|
||
| Gemini 2.5 Pro | 133 | 89.9% |
|
||
| GPT-5 | 132 | 89.2% |
|
||
| **Trio-strict** (all 3 independent agree PASS) | 129 | **87.2%** |
|
||
| Trio-majority (≥2/3 independent PASS) | 135 | 91.2% |
|
||
|
||
**Agreement with the self-judge:** Opus 98.6% · Gemini 98.0% · GPT-5 97.3%.
|
||
|
||
## Verdict — self-judge is NOT inflated
|
||
|
||
- Independent single judges land within **0.6–1.3pp** of the Sonnet self-judge.
|
||
- **Trio-strict is only −3.3pp** below self (90.5% → 87.2% on n=148) — well within
|
||
cross-LLM-judge norms, and far tighter than LOCOMO's +5.3pp self-judge inflation.
|
||
- Only **6/148 (4%)** scenarios are marginal (self-PASS but not unanimous across the trio):
|
||
`21_csyctc, 22_52pwi3, 23_onhtod, 26_oqrx9a, 30_69r1z7, 30_9uo633`.
|
||
|
||
**The 83.8% GAIA 2 search result was not a self-grading artifact.** The harness-quality
|
||
signal is real and independently confirmed.
|
||
|
||
## Mapping to the full split (N=160 denominator)
|
||
|
||
| Metric | Self (Sonnet) | Trio-strict |
|
||
|---|---:|---:|
|
||
| n=148 answerable | 90.5% | 87.2% |
|
||
| **N=160 full (errors/unanswerable count against)** | **83.8%** | **~80.6%** (129/160) |
|
||
|
||
**Defensible public framing:** *"GAIA 2 search split, N=160 — 83.8% self-judged,
|
||
independently confirmed at 80.6% trio-strict (Opus 4.7 + Gemini 2.5 Pro + GPT-5,
|
||
GAIA 2's own checker, judges agree 97–99%)."*
|
||
|
||
## Caveat (carried forward)
|
||
|
||
This validates the **harness-cell judging**, but the harness was **Hermes** (a third-party
|
||
reference agent), NOT Waggle's own `runAgentLoop`. Pillar-1 (Waggle harness SOTA) still
|
||
requires the `waggle_worker` build to put Waggle's loop in the same rig. See plan doc.
|