4.3 KiB
Self-judge vs Trio-strict — Side-by-Side Comparison
Date: 2026-04-25 · Companion: apples-to-apples-memo.md
Anchor: v6 N=400 final commit afe6422 (2000 subject responses unchanged; only judge methodology re-applied)
Self-judge model: qwen3.6-35b-a3b-via-dashscope-direct (single judge; same model class as subject; thinking disabled; temperature=0)
Self-judge prompt: Mem0-style binary correctness (character-exact per PM brief)
Trio-strict baseline: Claude Opus 4.7 + GPT-5.4 + MiniMax M2.7 (via OpenRouter); majority verdict; v6 §5.2.1 quorum policy
Subject responses: unchanged from v6 N=400; reused verbatim from *-locomo-2026-04-2{4,5}T*.jsonl
Per-cell side-by-side
| Cell | n | Self-judge ✓ | Self-judge acc | Trio-strict ✓ | Trio-strict acc | Δ (self − trio) |
|---|---|---|---|---|---|---|
| no-context | 400 | 53 | 0.1325 (13.25 %) | 12 | 0.0300 | +10.25 pp |
| oracle-context | 400 | 296 | 0.7400 (74.00 %) | 134 | 0.3350 | +40.50 pp |
| full-context | 400 | 248 | 0.6200 (62.00 %) | 109 | 0.2725 | +34.75 pp |
| retrieval | 400 | 193 | 0.4825 (48.25 %) | 89 | 0.2225 | +26.00 pp |
| agentic | 400 | 187 | 0.4675 (46.75 %) | 86 | 0.2150 | +25.25 pp |
| micro (all 5) | 2000 | 977 | 0.4885 (48.85 %) | 430 | 0.2150 | +27.35 pp |
| macro (cell mean) | — | — | 0.4885 (48.85 %) | — | 0.2150 | +27.35 pp |
(micro = pooled-over-instances; macro = simple mean of per-cell accuracies. Both equal because every cell has n=400.)
Methodology-bias finding
Trio-strict ensemble (Opus + GPT + MiniMax majority) systematically returns lower accuracy than single self-judge on the same 2000 subject responses by an average of +27.35 pp.
- Largest gap on oracle-context cell (+40.50 pp): the cell where the subject has the highest signal density; single self-judge accepts paraphrastic / partially-correct answers that trio-strict rejects.
- Smallest gap on no-context cell (+10.25 pp): when the subject has nearly nothing to work with, both judges agree most answers are wrong.
- The bias is monotone increasing with subject signal: higher-information cells produce wider self-judge / trio-strict gaps. This is the classic self-judging-bias signature.
By-question-category breakdown
Aggregated across all 5 cells, ground-truth category field from locomo-1540.jsonl:
| Category | n (×5 cells) | Notes |
|---|---|---|
| single-hop | 1070 (214 × 5) | Plurality; biggest absolute count of judge disagreements |
| multi-hop | 370 (74 × 5) | Reasoning-chain questions |
| temporal | 425 (85 × 5) | Date/time recall — typically narrowest accuracy across both judges |
| open-ended | 135 (27 × 5) | Free-form answers — widest ambiguity gap |
Per-cell-per-category numbers in the source aggregate JSON (tmp/stage3-runs/self-judge-aggregate.json).
Headline: in oracle-context, self-judge gives single-hop answers 90.2 % accept rate vs trio-strict 33.5 % overall — exposing the categorical tightness of the trio's grading on paraphrastic answers.
Operational stats (re-judge run)
| Metric | Value |
|---|---|
| Total instances re-judged | 2000 / 2000 |
Errors (self_judge_error) |
0 |
| Ambiguous parses (neither Yes nor No) | 0 |
| Median p50 latency | 754 ms |
| p95 latency | 852 ms |
| Total cost | $0.0782 |
| Wall clock | 301 s (5 min 1 s) at concurrency=5 |
| Budget cap | $10.00 (halt $7.00) — 0.78 % used |
Reproducibility footer
re_judge_model: qwen3.6-35b-a3b-via-dashscope-direct
re_judge_thinking: false
re_judge_temperature: 0.0
re_judge_max_tokens: 8
re_judge_concurrency: 5
re_judge_prompt_template: |
Question: <question>
Ground truth: <ground_truth>
Model answer: <model_answer>
Is the model answer correct? Output exactly 'Yes' or 'No'.
re_judge_results_jsonl: benchmarks/results/v6-self-judge-rebench/qwen-self-judge-results.jsonl
ground_truth_source: benchmarks/data/locomo/locomo-1540.jsonl
ground_truth_upstream: benchmarks/data/locomo10.json (raw archive, SHA 79fa87e9...)
subject_responses: unchanged from v6 N=400 final commit afe6422
trio_strict_anchor: afe6422 (final-5cell-summary.md)