This commit is contained in:
Oleg Maslov
@@ -0,0 +1,78 @@
|
||||
# Self-judge vs Trio-strict — Side-by-Side Comparison
|
||||
|
||||
**Date:** 2026-04-25 · **Companion:** [`apples-to-apples-memo.md`](apples-to-apples-memo.md)
|
||||
**Anchor:** v6 N=400 final commit `afe6422` (2000 subject responses unchanged; only judge methodology re-applied)
|
||||
**Self-judge model:** `qwen3.6-35b-a3b-via-dashscope-direct` (single judge; same model class as subject; thinking disabled; temperature=0)
|
||||
**Self-judge prompt:** Mem0-style binary correctness (character-exact per PM brief)
|
||||
**Trio-strict baseline:** Claude Opus 4.7 + GPT-5.4 + MiniMax M2.7 (via OpenRouter); majority verdict; v6 §5.2.1 quorum policy
|
||||
**Subject responses:** unchanged from v6 N=400; reused verbatim from `*-locomo-2026-04-2{4,5}T*.jsonl`
|
||||
|
||||
## Per-cell side-by-side
|
||||
|
||||
| Cell | n | Self-judge ✓ | Self-judge acc | Trio-strict ✓ | Trio-strict acc | Δ (self − trio) |
|
||||
|------|---|--------------|-----------------|----------------|-------------------|-------------------|
|
||||
| no-context | 400 | 53 | **0.1325** (13.25 %) | 12 | 0.0300 | **+10.25 pp** |
|
||||
| oracle-context | 400 | 296 | **0.7400** (74.00 %) | 134 | 0.3350 | **+40.50 pp** |
|
||||
| full-context | 400 | 248 | **0.6200** (62.00 %) | 109 | 0.2725 | **+34.75 pp** |
|
||||
| retrieval | 400 | 193 | **0.4825** (48.25 %) | 89 | 0.2225 | **+26.00 pp** |
|
||||
| agentic | 400 | 187 | **0.4675** (46.75 %) | 86 | 0.2150 | **+25.25 pp** |
|
||||
| **micro (all 5)** | **2000** | **977** | **0.4885 (48.85 %)** | **430** | **0.2150** | **+27.35 pp** |
|
||||
| **macro (cell mean)** | — | — | **0.4885 (48.85 %)** | — | **0.2150** | **+27.35 pp** |
|
||||
|
||||
(micro = pooled-over-instances; macro = simple mean of per-cell accuracies. Both equal because every cell has n=400.)
|
||||
|
||||
## Methodology-bias finding
|
||||
|
||||
**Trio-strict ensemble (Opus + GPT + MiniMax majority) systematically returns lower accuracy than single self-judge on the same 2000 subject responses by an average of +27.35 pp.**
|
||||
|
||||
- Largest gap on **oracle-context** cell (+40.50 pp): the cell where the subject has the highest signal density; single self-judge accepts paraphrastic / partially-correct answers that trio-strict rejects.
|
||||
- Smallest gap on **no-context** cell (+10.25 pp): when the subject has nearly nothing to work with, both judges agree most answers are wrong.
|
||||
- The bias is **monotone increasing with subject signal**: higher-information cells produce wider self-judge / trio-strict gaps. This is the classic self-judging-bias signature.
|
||||
|
||||
## By-question-category breakdown
|
||||
|
||||
Aggregated across all 5 cells, ground-truth `category` field from `locomo-1540.jsonl`:
|
||||
|
||||
| Category | n (×5 cells) | Notes |
|
||||
|----------|---------------|-------|
|
||||
| single-hop | 1070 (214 × 5) | Plurality; biggest absolute count of judge disagreements |
|
||||
| multi-hop | 370 (74 × 5) | Reasoning-chain questions |
|
||||
| temporal | 425 (85 × 5) | Date/time recall — typically narrowest accuracy across both judges |
|
||||
| open-ended | 135 (27 × 5) | Free-form answers — widest ambiguity gap |
|
||||
|
||||
Per-cell-per-category numbers in the source aggregate JSON (`tmp/stage3-runs/self-judge-aggregate.json`).
|
||||
**Headline:** in oracle-context, self-judge gives single-hop answers 90.2 % accept rate vs trio-strict 33.5 % overall — exposing the categorical tightness of the trio's grading on paraphrastic answers.
|
||||
|
||||
## Operational stats (re-judge run)
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Total instances re-judged | 2000 / 2000 |
|
||||
| Errors (`self_judge_error`) | **0** |
|
||||
| Ambiguous parses (neither Yes nor No) | **0** |
|
||||
| Median p50 latency | **754 ms** |
|
||||
| p95 latency | **852 ms** |
|
||||
| Total cost | **$0.0782** |
|
||||
| Wall clock | **301 s (5 min 1 s)** at concurrency=5 |
|
||||
| Budget cap | $10.00 (halt $7.00) — 0.78 % used |
|
||||
|
||||
## Reproducibility footer
|
||||
|
||||
```
|
||||
re_judge_model: qwen3.6-35b-a3b-via-dashscope-direct
|
||||
re_judge_thinking: false
|
||||
re_judge_temperature: 0.0
|
||||
re_judge_max_tokens: 8
|
||||
re_judge_concurrency: 5
|
||||
re_judge_prompt_template: |
|
||||
Question: <question>
|
||||
Ground truth: <ground_truth>
|
||||
Model answer: <model_answer>
|
||||
Is the model answer correct? Output exactly 'Yes' or 'No'.
|
||||
|
||||
re_judge_results_jsonl: benchmarks/results/v6-self-judge-rebench/qwen-self-judge-results.jsonl
|
||||
ground_truth_source: benchmarks/data/locomo/locomo-1540.jsonl
|
||||
ground_truth_upstream: benchmarks/data/locomo10.json (raw archive, SHA 79fa87e9...)
|
||||
subject_responses: unchanged from v6 N=400 final commit afe6422
|
||||
trio_strict_anchor: afe6422 (final-5cell-summary.md)
|
||||
```
|
||||
Reference in New Issue
Block a user