waggle-os/benchmarks/probes/judge-swap-validation/kappa-split-analysis.md

§1.3h Judge Swap Stratified Re-Probe — Dual-κ Split Analysis

Date: 2026-04-24 (evening) Parent commit: 8a2f0e6 (§1.3g anchor) Source: benchmarks/results/locomo-mini-n20-retry-2026-04-24T00-02-12Z.jsonl (100 rows, authoritative).

Split pool (Opus ≠ GPT): 7 instances (use-all-available per PM amendment §1.3H-POOL-SHORTAGE OPTION 1). Split cell distribution: agentic=2, full-context=2, oracle-context=3, no-context=0, retrieval=0. Structural observation: All 7 splits have Opus=correct / GPT=incorrect. Zero inverse splits.

PM-amended verdict caveat: n=7 < 12 minimum for meaningful discrimination. All findings carry n=7 pool-limited, observational not confirmatory caveat. Primary value = operational signal (parse + latency + MiniMax routing) + bias detection via p_opus/p_gpt balance.

---

§1 Per-candidate split metrics

Cand	Parse	n_eval	p_opus	p_gpt	κ_vs_opus	κ_vs_gpt	κ_cons	p50 lat	p95 lat	Routing	Split verdict
kimi	5/7 (71.4%)	5	80.0% (4/5)	20.0% (1/5)	0.0 deg	0.0 deg	N/A deg	32.0 s	51.0 s	direct	N/A deg
minimax	7/7 (100.0%)	7	85.7% (6/7)	14.3% (1/7)	0.0 deg	0.0 deg	N/A deg	16.6 s	21.2 s	openrouter	N/A deg
deepseek	5/7 (71.4%)	5	40.0% (2/5)	60.0% (3/5)	0.0 deg	0.0 deg	N/A deg	15.4 s	21.2 s	direct	N/A deg
zhipu	6/7 (85.7%)	6	0.0% (0/6)	100.0% (6/6)	0.0 deg	1.0 deg	N/A deg	17.9 s	22.5 s	direct	N/A deg

§1.1 Why κ on split-only is mathematically degenerate

All 7 splits are Opus=correct / GPT=incorrect. For a Cohen's κ on the split subset:

• κ_vs_opus: reference (Opus) has zero variance → ref_counts has single class → p_e = p_o → κ = (0 − 0)/(1 − 1) undefined → reported as 0.0 deg
• κ_vs_gpt: same pathology → 0.0 deg (except Zhipu, which perfectly matches GPT → κ = 1.0 deg by convention when p_o = p_e = 1)

κ_cons = min() of two degenerate values is itself degenerate. The informative signal on splits is NOT κ, it is the p_opus / p_gpt balance which reveals each candidate's calibration bias. The brief's κ-threshold classification (PASS/BORDERLINE/FAIL @ 0.70/0.60) does not apply to this structural case.

§1.2 BIAS INTERPRETATION (actionable)

Since all splits are Opus=correct / GPT=incorrect:

• p_opus=100% = candidate lock-stepped with Opus (maximally lenient, adds no new signal vs Opus)
• p_gpt=100% = candidate lock-stepped with GPT (maximally strict, adds no new signal vs GPT)
• p_opus ≈ p_gpt ≈ 50% = candidate judges independently (ideal for 3-judge ensemble)

Cand	p_opus	p_gpt	Calibration profile
DeepSeek	40%	60%	Most balanced — slight GPT lean, clearest independent judgment
Kimi	80%	20%	Opus-lenient (echoes Opus 4/5 times)
MiniMax	86%	14%	Strongly Opus-lenient (echoes Opus 6/7 times)
Zhipu	0%	100%	Pure GPT-echo — 6/6 parsed splits = GPT verbatim

This is the critical finding the §1.3g probe could not produce. On the 20 unanimous cases, all 4 candidates scored κ=1.0 — appeared equivalent. On split cases where judges diverge, DeepSeek uniquely exhibits independent calibration; Zhipu is effectively a GPT echo; Kimi/MiniMax echo Opus. For a 3-judge ensemble replacing Gemini, independence from existing judges is a feature, not a bug.

---

§2 Aggregate κ on combined sample (27 = 20 unanimous + 7 splits)

Unanimous portion reuses §1.3g verdicts (consensus-matched, contributes κ=1.0 content). Split portion uses dual-reference with conservative min. Combined sample is non-degenerate (both classes present in Opus and GPT reference columns on the aggregate).

Cand	n_combined	κ_agg_vs_opus	κ_agg_vs_gpt	κ_agg_cons	Aggregate verdict
deepseek	23	0.7089	0.7473	0.7089	PASS (only one ≥ 0.70)
kimi	25	0.9110	0.5763	0.5763	FAIL (κ_vs_gpt < 0.70)
minimax	27	0.9222	0.4564	0.4564	FAIL (κ_vs_gpt < 0.70)
zhipu	25	0.4444	1.0000	0.4444	FAIL (κ_vs_opus < 0.70)

Aggregate κ ranking (descending): deepseek, kimi, minimax, zhipu

The aggregate picture inverts the split-only one: DeepSeek is the only candidate clearing 0.70 on the conservative dual-reference combined sample. Zhipu's perfect κ_vs_gpt is offset by its ~0.44 κ_vs_opus — clear asymmetry consistent with the bias finding.

---

§3 Operational snapshot (split-case specific)

Cand	Split parse	p50 (split)	p95 (split)	Routing	MiniMax status
kimi	5/7 (71%)	32.0 s	51.0 s	direct	—
minimax	7/7 (100%)	16.6 s	21.2 s	openrouter	direct FAILED — international 401/err + legacy 401/err
deepseek	5/7 (71%)	15.4 s	21.2 s	direct	—
zhipu	6/7 (86%)	17.9 s	22.5 s	direct	—

§3.1 MiniMax direct routing verdict

Primary operational value focus per PM amendment. Result:

• direct_international (https://api.minimaxi.com/v1/text/chatcompletion_v2?GroupId=...) — FAILED
• direct_legacy (https://api.minimax.chat/v1/text/chatcompletion_v2?GroupId=...) — FAILED
• openrouter fallback — SUCCEEDED (7/7 parse)

Routing resolution: direct_failed_fell_back_openrouter. The newly-added MINIMAX_GROUP_ID did not unblock direct routing. Either the key is not provisioned for v2 endpoints, the endpoint paths / model names need further research (e.g. raw v1 chatcompletion_pro format differs from OpenAI-compatible v2), or the account tier lacks direct API access. Manifest v6 must route MiniMax via OpenRouter if MiniMax is selected — direct routing is not a viable assumption.

§3.2 Parse rate on challenging vs unanimous (regression check)

Cand	Unanimous parse (§1.3g)	Split parse (§1.3h)	Δ
kimi	20/20 (100%)	5/7 (71%)	−29 pp (2 timeout-retries exhausted on long-context instances)
minimax	20/20 (100%)	7/7 (100%)	0 pp
deepseek	18/20 (90%)	5/7 (71%)	−19 pp (2 verdict=None — likely malformed JSON or over-length)
zhipu	19/20 (95%)	6/7 (86%)	−9 pp

Challenging cases expose degradation in Kimi and DeepSeek parse reliability. Zhipu and MiniMax (via OR) hold up best on difficult instances.

---

§4 Aggregate verdict: INCONCLUSIVE_BUT_OPERATIONAL_SIGNAL

Per PM-amended §4.4. Reasons (any single one triggers):

1. Split pool n=7 < 12 minimum for confirmatory κ discrimination
2. Parse rate <80% on 2 candidates (Kimi, DeepSeek) on split subset
3. Split-only κ is mathematically degenerate (structural Opus=correct / GPT=incorrect orientation)

However, three operational-signal findings are fully actionable:

• A. Bias profile (novel finding): DeepSeek is the only candidate with independent calibration. Zhipu is a GPT-echo (0% agreement with Opus on splits). MiniMax/Kimi are Opus-echoes.
• B. Aggregate κ on combined 27: DeepSeek is the only candidate κ_cons ≥ 0.70 on the non-degenerate combined sample.
• C. MiniMax direct routing FAILED even with MINIMAX_GROUP_ID. Manifest v6 cannot rely on direct; OpenRouter fallback is the operational truth.

---

§5 Revised recommendation (integrating §1.3g + §1.3h)

The §1.3g heuristic-based recommendation (Zhipu primary, DeepSeek backup) is not supported by §1.3h findings.

§5.1 Two viable paths, each with a clear trade-off

Path 1 — Independence priority (DeepSeek primary):

• Pros: Only candidate with balanced independent calibration (p_opus=40%, p_gpt=60%); only one clearing κ_agg_cons ≥ 0.70 on combined 27; direct routing works.
• Cons: Split parse rate 71% (2/7 NULL verdicts) — operational risk for N=400 unless root cause of parse drops on hard cases is addressed (likely max_tokens=1024 ceiling being hit on reasoning chains, same pathology as Kimi; could be fixed by bumping to 2048/4096).
• Mitigation: Boost max_tokens to 2048+ for DeepSeek on judge role; re-verify parse rate on a follow-up probe if PM accepts this path.

Path 2 — Reliability priority (Zhipu primary, MiniMax-via-OR backup):

• Pros: Zhipu fastest p50 (9s from §1.3g, 18s on splits) + direct routing + parse 86% on splits; MiniMax 100% parse via OR.
• Cons: Zhipu is effectively a GPT-echo (0% independence on splits) — adds minimal new signal to an Opus+GPT+GPT-echo ensemble. Ensemble's value is correlated, not complementary; κ(Gemini-replacement, GPT) ≈ 1.0 violates the independence assumption that motivated having a third judge in the first place.

§5.2 PM adjudication required

Recommended PM pick based on §1.3h empirical data:

• Primary: DEEPSEEK with max_tokens=2048 bump (independence > 80% parse in a 3-judge ensemble where the third judge's role is adding new information)
• Backup: ZHIPU (if parse-reliability becomes binding, accept the independence compromise; it's better than no third judge)

Alternative if PM prefers §1.3g-style operational-first ranking:

• Primary: ZHIPU (parse + speed + direct), Backup: DEEPSEEK-boosted (independence insurance if Zhipu ensemble correlation becomes a problem at N=400)

If PM wants to fix the unknowns before committing:

• Run a mini-confirmation probe: DeepSeek with max_tokens=2048 on the same 7 splits + 13 additional split-candidates from a larger calibration pool (if one is built). Cost ~$0.50.

---

§6 Halt ping data (machine-readable summary)

• aggregate_verdict: INCONCLUSIVE_BUT_OPERATIONAL_SIGNAL
• ranking_by_k_agg_cons_desc: deepseek (0.709), kimi (0.576), minimax (0.456), zhipu (0.444)
• ranking_by_p_opus_balance: deepseek (best: 40/60), kimi (80/20), minimax (86/14), zhipu (worst: 0/100)
• recommended_primary (empirical): DEEPSEEK (with max_tokens bump) — supersedes §1.3g Zhipu pick
• recommended_primary (operational-first, if PM prefers): ZHIPU with independence caveat
• recommended_backup: ZHIPU or DEEPSEEK-boosted (mirror of primary choice)
• minimax_routing_resolution: direct_failed_fell_back_openrouter
• split_cases_selected: 7/7 (all available from 100-row authoritative source)
• empirical_finding_novel: Zhipu 100% GPT-lock-step on splits → low independence from existing GPT judge

Raw summary also saved to _summary-split.json.