moving

docs/briefs/2026-04-24-cc1-judge-swap-validation-probe-brief.md

@@ -0,0 +1,273 @@
+# CC-1 Brief — §1.3g Judge Swap Validation Probe (4-Candidate Roster: Kimi + MiniMax + DeepSeek + Zhipu)
+
+**Date**: 2026-04-24 (updated same day)
+**Status**: §1.3g sub-gate, Marko ratifikovao path re-prioritization + roster expansion
+**Authorized by**: Marko Marković (2026-04-24 evening, "ajmo da ih probamo... ne bih ja cekao google, i bolje da imamo kineza" + "Moonshot Kimi i Minimax ima oba ima i deepseek i z.ai")
+**Predecessor**: §1.3f Vertex Batch INFEASIBLE → Branch A CLOSED; Marko ratifies judge swap as primary path over Branch B waiting
+**PM**: claude-opus-4-7 (Cowork)
+
+## Roster (4 candidates)
+
+Marko confirmed direct API keys present in `waggle-os/.env` for all four Chinese GA-status flagship reasoning models:
+
+1. **Moonshot Kimi** (latest k2 / kimi-2.7)
+2. **MiniMax** (latest M series — M2.7 target)
+3. **DeepSeek** (latest V3.1 / V4 / R2 — whichever is current flagship reasoning)
+4. **Zhipu / z.ai** (latest GLM-4.6 / 4.7 / 5)
+
+Roster expansion rationale: DeepSeek especially has reasoning-tuned training paradigm; Zhipu is Tsinghua-origin cross-lingual flagship. Cost impact trivial (~$4-6 vs $2-3 for 2-candidate roster, well within $5 → $7 halt envelope adjusted to $8 → $10 halt for 4-candidate). Methodology benefit (4-candidate κ ranking) >> small cost delta.
+
+---
+
+## §0 Path re-prioritization
+
+Stage 3 N=400 re-kick path ovim se re-prioritizuje:
+
+**Novi primary path**: judge swap za Gemini 3.1 Pro na Chinese GA model (Kimi 2.7 ili MiniMax M2.7), gated na validation probe (this brief). Swap success → manifest v6 emit → full N=400 pod novim trojcem.
+
+**Novi backup**: Google quota ticket approval (pozicija umanjena sa primary na contingency). Ako stigne pre swap validation completion-a, PM će razmotriti alternate path.
+
+**Fallback-of-fallback**: Branch B (manifest v6 reduced Gemini coverage) — aktivira se samo ako judge swap probe FAIL za oba kandidata.
+
+**Narrativni benefit swap-a**: ensemble diversity (US + US + CN jurisdictions), GA-status judge stack, eliminisan preview-model quota dependency dugoročno.
+
+---
+
+## §1 Cilj
+
+Empirijski utvrditi koji od četiri Chinese GA-status flagship reasoning modela (Kimi, MiniMax, DeepSeek, Zhipu) ima SOTA-grade judge capability za LoCoMo scoring task. Success criterion = κ(candidate vs Opus+GPT consensus) ≥ 0.70 na 20 instances iz existing κ calibration set-a.
+
+Identifikacija ishodi (per-candidate):
+- **PASS**: κ ≥ 0.70 (substantial agreement)
+- **BORDERLINE**: κ 0.60-0.70 (moderate — PM adjudicates on context)
+- **FAIL**: κ < 0.60 (insufficient)
+
+Aggregate verdicts:
+- **MULTI_PASS**: dva ili više kandidata prolaze → rangiraj po κ score-u descending, top kandidat = primary recommendation, drugi = backup for manifest v6
+- **SINGLE_PASS**: samo jedan kandidat prolazi → taj = recommendation (no backup within Chinese roster)
+- **ALL_FAIL**: nijedan ne prolazi → swap path CLOSED, vraćamo se na Branch B ili čekanje Google-a
+- **INCONCLUSIVE**: parse failure rate >= 2/20 na nekom kandidatu, ili API availability issues
+
+---
+
+## §2 Model identifier resolution
+
+Kimi "2.7" i MiniMax "M2.7" su Marko-ova terminologija za "najnoviju najjaču varijantu". CC-1 mora **real-time verifikovati exact current latest model identifier** kroz direct API catalog discovery, ne pretpostavljati od imena.
+
+**Routing priority order** (Marko ratifikovao 2026-04-24):
+
+1. **PRIMARY — direct provider APIs**: Moonshot direct za Kimi, MiniMax direct za MiniMax. Native catalog access, najnoviji identifier-i, bez OpenRouter surcharge-a, direct quota ownership.
+2. **FALLBACK — OpenRouter**: koristi samo ako direct API key missing/invalid ili direct catalog ne sadrži latest flagship.
+
+### §2.1 Kimi — Moonshot direct (primary)
+
+Required env var: `MOONSHOT_API_KEY` (ili `KIMI_API_KEY` zavisno od Marko-ove naming conventions u .env).
+
+Endpoint candidates (try in order, use first that works):
+- `https://api.moonshot.ai/v1/chat/completions` (newer international)
+- `https://api.moonshot.cn/v1/chat/completions` (China mainland)
+
+API surface: OpenAI-compatible (Bearer auth, standard ChatCompletion schema).
+
+Catalog discovery:
+- GET `https://api.moonshot.ai/v1/models` (or .cn equivalent) sa Bearer auth
+- Identify latest flagship (prefer k2, kimi-2, kimi-2.7 namespace; reject moonshot-v1-* if newer exists)
+
+Fallback to OpenRouter if direct key missing/invalid or catalog incomplete:
+- `curl -s https://openrouter.ai/api/v1/models -H "Authorization: Bearer $OPENROUTER_API_KEY" | jq '.data[] | select(.id | contains("moonshot") or contains("kimi"))'`
+
+Document selected identifier + routing used (direct / openrouter) + context window + pricing u memo.
+
+### §2.2 MiniMax — MiniMax direct (primary)
+
+Required env var: `MINIMAX_API_KEY` (plus potentially `MINIMAX_GROUP_ID` — MiniMax API sometimes requires group_id header).
+
+Endpoint candidates:
+- `https://api.minimaxi.com/v1/text/chatcompletion_v2` (newer OpenAI-compatible)
+- `https://api.minimax.chat/v1/text/chatcompletion_pro` (legacy)
+
+API surface: check docs — may require `GroupId` query param or header, and response schema slightly different from OpenAI standard.
+
+Catalog discovery:
+- MiniMax typically doesn't expose /models endpoint; use docs + direct model identifier
+- Known flagship families: M1 series, M2 series, abab6.5, abab7. Prefer latest M2 variant or abab7 namespace.
+- If uncertain, try multiple identifiers sequentially (e.g., "MiniMax-M2.7", "MiniMax-M2", "abab7-chat-preview") and log which returned valid response
+
+Fallback to OpenRouter:
+- Filter OpenRouter catalog for "minimax"
+
+Document selected identifier + routing used + any special auth params (group_id, etc.) u memo.
+
+### §2.3 DeepSeek — DeepSeek direct (primary)
+
+Required env var: `DEEPSEEK_API_KEY`.
+
+Endpoint:
+- `https://api.deepseek.com/v1/chat/completions` (OpenAI-compatible)
+
+Catalog discovery:
+- GET `https://api.deepseek.com/v1/models` with Bearer auth
+- Identify latest flagship reasoning model (prefer V3.1, V4, R2, deepseek-reasoner namespace over V2 / deepseek-chat legacy)
+- Document selected identifier + context window + pricing
+
+Fallback to OpenRouter:
+- Filter OpenRouter catalog for "deepseek"
+
+### §2.4 Zhipu / z.ai — Zhipu direct (primary)
+
+Required env var: `ZHIPU_API_KEY` ili `ZAI_API_KEY` (check Marko .env for exact naming).
+
+Endpoint candidates:
+- `https://api.z.ai/v1/chat/completions` (newer international)
+- `https://open.bigmodel.cn/api/paas/v4/chat/completions` (China mainland, legacy branding as "智谱清言")
+
+API surface: OpenAI-compatible Bearer auth typical; verify newer z.ai endpoint schema.
+
+Catalog discovery:
+- GET `https://api.z.ai/v1/models` or equivalent with Bearer auth
+- Identify latest GLM flagship (prefer GLM-4.6, GLM-4.7, GLM-5 namespace over GLM-4 legacy)
+- Document selected identifier + context window + pricing
+
+Fallback to OpenRouter:
+- Filter OpenRouter catalog for "zhipu" or "glm"
+
+---
+
+## §3 Sample selection
+
+Uzeti **20 instances iz postojećeg κ calibration set-a** (ne iz N=400 canonical fixture). Location: identifikuj iz manifest v5 §N / v4 predecessor references (gde god je κ=0.7458 kalibrisan). **Kriterijum**:
+- 20 instances koji imaju već-postojeće Opus 4.7 + GPT-5.4 + Gemini 3.1 Pro verdicts u dataset-u
+- Stratified ako moguće: izbalansirano preko cells (no-context, retrieval, agentic, itd.) da probe ne pokrije samo jednu kategoriju
+- Identifikuj po canonical instance_id iz postojećeg κ calibration results JSONL
+
+Ako balanced selection nije moguć zbog dataset structure, uzmi prvih 20 iz κ calibration results file-a (deterministic ordering) i document u memo-u.
+
+**STRICT CONSTRAINT**: ne generisati nove judge verdicts od Opus ili GPT za ovo probe. Koristimo **njihove already-generated verdicts iz κ calibration set-a** kao ground truth reference. Nove judge pozive samo ka Kimi i MiniMax.
+
+---
+
+## §4 Probe execution
+
+### §4.1 Prompt template
+
+Koristi **verbatim judge prompt template iz `failure-mode-judge.ts:245-258`** (isti koji Gemini 3.1 Pro dobija trenutno). Ne modifikuj, ne trim-uj, ne adaptiraj. Identičan input za fair comparison.
+
+### §4.2 Call pattern
+
+- Per instance: 4 API calls (1 Kimi + 1 MiniMax + 1 DeepSeek + 1 Zhipu)
+- Total: 80 API calls
+- Sequential OK (volume trivial, no rate limit pressure at this scale)
+- Deterministic settings: `temperature=0.0`, `max_tokens` matched na current judge runner setting
+- Retries: up to 3 na transient errors (rate limit, network); log retry count per instance
+
+### §4.3 Output parsing
+
+Parse Kimi + MiniMax responses kroz **identičan parser** koji judge-runner koristi za Gemini 3.1 Pro responses. Ako parsing fail → count as "judge failure" (ne kao disagreement). Threshold: ≥18/20 successful parse per candidate for valid probe; <18 → INCONCLUSIVE verdict.
+
+---
+
+## §5 κ computation
+
+Computation matrix per candidate:
+- **Consensus reference**: Opus + GPT on that instance
+  - If Opus == GPT → consensus = that value
+  - If Opus != GPT → "split" (exclude from κ or treat as disagreement — standard practice: exclude, document count)
+- **Candidate verdict**: Kimi or MiniMax verdict on same instance
+- **Agreement matrix**: 2×2 (consensus × candidate) across instances
+
+Compute **Cohen's κ** for each candidate against Opus+GPT consensus.
+
+Success criteria:
+- κ ≥ 0.80 = excellent agreement (preferred)
+- κ ≥ 0.70 = substantial agreement (pass threshold)
+- κ ≥ 0.60 = moderate agreement (borderline — PM adjudicates)
+- κ < 0.60 = fair/poor (FAIL)
+
+Also compute:
+- Raw agreement % (for context)
+- Count of "split" consensus instances (excluded from κ)
+- Per-cell breakdown if stratified sample permits
+
+---
+
+## §6 Scope guards (identični §1.3f)
+
+- **Manifest v5 anchor `fc16925` immutable**. No v6 emit u probe phase.
+- **HEAD `373516c` + 8 commits od v4 anchor-a intact**. Last commit short SHA = `8ad0567` (§1.3f).
+- **§11 frozen paths netaknuti**: runner, judge-runner, failure-mode-judge, health-check, litellm-config.yaml.
+- **No new LiteLLM alias** u `litellm-config.yaml` za Kimi ili MiniMax tokom probe-a. Probe script direktno priča sa OpenRouter API (ili Moonshot/MiniMax direct) bez LiteLLM proxy layer-a.
+- **Novi folder**: `benchmarks/probes/judge-swap-validation/` (izvan §11 frozen paths).
+- **Package install pre-authorized**: ako je potrebno, `pip install openai` (ili ekvivalent za OpenRouter routing) — minimal add. Log installed versions u memo.
+
+---
+
+## §7 Deliverables
+
+Commit na `feature/c3-v3-wrapper`:
+
+1. `benchmarks/probes/judge-swap-validation/probe-script.py` (ili `.ts`) — probe code sa inline comments
+2. `benchmarks/probes/judge-swap-validation/sample-instances.jsonl` — 20 instances selekovanih sa instance_id + cell + Opus verdict + GPT verdict (ground truth reference)
+3. `benchmarks/probes/judge-swap-validation/kimi-responses.jsonl` — 20 Kimi verdicts (raw + parsed)
+4. `benchmarks/probes/judge-swap-validation/minimax-responses.jsonl` — 20 MiniMax verdicts (raw + parsed)
+5. `benchmarks/probes/judge-swap-validation/deepseek-responses.jsonl` — 20 DeepSeek verdicts (raw + parsed)
+6. `benchmarks/probes/judge-swap-validation/zhipu-responses.jsonl` — 20 Zhipu verdicts (raw + parsed)
+7. `benchmarks/probes/judge-swap-validation/kappa-analysis.md` — κ matrix (per-candidate) + raw agreement + split count + verdict + ranking table
+8. `benchmarks/probes/judge-swap-validation/validation-memo.md` — ≤200 reči summary sa explicit per-candidate verdict + aggregate verdict (MULTI_PASS / SINGLE_PASS / ALL_FAIL / INCONCLUSIVE), recommended primary + backup candidates, actual cost, wall-clock
+
+Anchor commit: `[probe] judge swap validation 4-candidate: kimi + minimax + deepseek + zhipu - <aggregate_verdict>`. SHA-256 svih artefakata u halt ping.
+
+---
+
+## §8 Budget i halt criteria
+
+- **Budget cap**: $8 (adjusted from $5 za 2-candidate → 4-candidate roster; realno ~$4-6 expected)
+- **Halt @ $10**: ako spend pređe, full escalation, no completion
+- **Per-call timeout**: 60s (judge calls su short)
+- **Total wall-clock cap**: 2.5h (setup + 80 calls + per-candidate κ compute + aggregate ranking + memo)
+
+---
+
+## §9 Halt ping format
+
+Emit na completion (success/fail/inconclusive):
+
+- `aggregate_verdict: MULTI_PASS | SINGLE_PASS | ALL_FAIL | INCONCLUSIVE`
+- Per-candidate block (repeat for all 4):
+  - `<candidate>_identifier_used: <string>`
+  - `<candidate>_routing: direct | openrouter`
+  - `kappa_<candidate>_vs_consensus: <float>`
+  - `raw_agreement_<candidate>: <float>`
+  - `successful_parse_<candidate>: <int>/20`
+  - `per_candidate_verdict: PASS | BORDERLINE | FAIL`
+- `split_consensus_excluded: <int>` (Opus≠GPT count, shared across candidates)
+- `ranking_by_kappa: [<candidate1_by_rank>, <candidate2>, <candidate3>, <candidate4>]`
+- `recommended_primary: KIMI | MINIMAX | DEEPSEEK | ZHIPU | NONE`
+- `recommended_backup: KIMI | MINIMAX | DEEPSEEK | ZHIPU | NONE` (if MULTI_PASS)
+- `anchor_commit: <full sha>`
+- `artefact_shas: { script, sample, kimi-responses, minimax-responses, deepseek-responses, zhipu-responses, kappa, memo }`
+- `wall_clock: <duration>`
+- `cost_actual: $<actual>` (vs $8 cap)
+- `next_step_request: PM-RATIFY-JUDGE-SWAP-VALIDATION`
+- `cc1_state: HALTED`
+
+CC-1 ne self-advances. Ne emituje manifest v6 swap. Ne modifikuje runner. Čeka PM ratifikaciju.
+
+---
+
+## §10 Task #29 trace update (post-probe)
+
+Posle halt ping-a, PM update:
+- §1.3g Judge swap validation: <verdict>
+- If BOTH_PASS / KIMI_PASS / MINIMAX_PASS: emit manifest v6 swap proposal brief za recommended candidate
+- If BOTH_FAIL: swap path CLOSED, fall back na Google ticket waiting + Branch B prep
+- If INCONCLUSIVE: PM odlučuje retry parameter-ima ili escalates
+
+Post-PASS path timeline: manifest v6 emit + full κ re-calibration ($15-25, ~2h) + N=400 run sa novim trojcem (standard tier quota, no 250 RPD cap) → ~1-1.5 days total to SOTA claim completion, bez scope footnote.
+
+---
+
+## §11 Authorized by
+
+PM Marko Marković, 2026-04-24 evening, verbatim: "Ahmo da ih probamo. I kimi i Minimax... najjaci modeli, ne bih ja cekao google, i bolje da imamo kineza"
+
+CC-1 može da počne odmah — gcloud tooling nije potreban za ovaj probe (OpenRouter alternative route).