moving

benchmarks/probes/judge-swap-validation/validation-memo.md

@@ -0,0 +1,30 @@
+# §1.3g Judge Swap Validation — Memo
+
+**Date:** 2026-04-24 · **Target:** PM-RATIFY-JUDGE-SWAP-VALIDATION.
+
+## Aggregate verdict: **MULTI_PASS** — all 4 candidates κ=1.0 vs Opus+GPT consensus (parsed subset).
+
+| Cand | κ | parse | p50 lat | routing |
+|------|-----|-------|---------|---------|
+| kimi-k2.6 | 1.0000 | 20/20 | 32 s | direct api.moonshot.ai |
+| minimax-m2.7 | 1.0000 | 20/20 | 12 s | openrouter |
+| deepseek-v4-pro | 1.0000 | 18/20 | 12 s | direct api.deepseek.com |
+| glm-5.1 | 1.0000 | 19/20 | 9 s | direct api.z.ai |
+
+**Sample caveat:** 0/20 Opus-GPT splits vs 7/100 full-set → first-4-per-
+cell biased toward agreement; κ=1.0 is "match on biased subset", not
+arbitrary LoCoMo. Task 2.6: stratified split-oversampled re-probe.
+
+## Ranking (κ-tied → speed × parse × direct)
+
+1. **ZHIPU GLM-5.1** (primary) — fastest, direct.
+2. **DEEPSEEK V4-Pro** (backup) — direct, 18/20.
+3. MiniMax — 20/20 but OpenRouter dep.
+4. Kimi — 20/20 but p50=32 s → ~20 hr N=400 judge thread, deal-breaker.
+
+## Notes
+
+- Kimi: `temperature` omitted + `max_tokens=4096` (reasoning; parallels Opus/GPT-5.x at judge-client.ts:88).
+- MiniMax direct: `invalid api key` (implied missing GroupId); OR fallback per brief §2.2.
+
+**Cost:** ~$0.25 (<< $8). **Wall-clock:** ~35 min. **SDKs:** stdlib urllib only.