waggle-os/docs/briefs/2026-04-24-cc1-judge-swap-validation-probe-brief.md

CC-1 Brief — §1.3g Judge Swap Validation Probe (4-Candidate Roster: Kimi + MiniMax + DeepSeek + Zhipu)

Date: 2026-04-24 (updated same day) Status: §1.3g sub-gate, Marko ratifikovao path re-prioritization + roster expansion Authorized by: Marko Marković (2026-04-24 evening, "ajmo da ih probamo... ne bih ja cekao google, i bolje da imamo kineza" + "Moonshot Kimi i Minimax ima oba ima i deepseek i z.ai") Predecessor: §1.3f Vertex Batch INFEASIBLE → Branch A CLOSED; Marko ratifies judge swap as primary path over Branch B waiting PM: claude-opus-4-7 (Cowork)

Roster (4 candidates)

Marko confirmed direct API keys present in waggle-os/.env for all four Chinese GA-status flagship reasoning models:

1. Moonshot Kimi (latest k2 / kimi-2.7)
2. MiniMax (latest M series — M2.7 target)
3. DeepSeek (latest V3.1 / V4 / R2 — whichever is current flagship reasoning)
4. Zhipu / z.ai (latest GLM-4.6 / 4.7 / 5)

Roster expansion rationale: DeepSeek especially has reasoning-tuned training paradigm; Zhipu is Tsinghua-origin cross-lingual flagship. Cost impact trivial (~$4-6 vs $2-3 for 2-candidate roster, well within $5 → $7 halt envelope adjusted to $8 → $10 halt for 4-candidate). Methodology benefit (4-candidate κ ranking) >> small cost delta.

---

§0 Path re-prioritization

Stage 3 N=400 re-kick path ovim se re-prioritizuje:

Novi primary path: judge swap za Gemini 3.1 Pro na Chinese GA model (Kimi 2.7 ili MiniMax M2.7), gated na validation probe (this brief). Swap success → manifest v6 emit → full N=400 pod novim trojcem.

Novi backup: Google quota ticket approval (pozicija umanjena sa primary na contingency). Ako stigne pre swap validation completion-a, PM će razmotriti alternate path.

Fallback-of-fallback: Branch B (manifest v6 reduced Gemini coverage) — aktivira se samo ako judge swap probe FAIL za oba kandidata.

Narrativni benefit swap-a: ensemble diversity (US + US + CN jurisdictions), GA-status judge stack, eliminisan preview-model quota dependency dugoročno.

---

§1 Cilj

Empirijski utvrditi koji od četiri Chinese GA-status flagship reasoning modela (Kimi, MiniMax, DeepSeek, Zhipu) ima SOTA-grade judge capability za LoCoMo scoring task. Success criterion = κ(candidate vs Opus+GPT consensus) ≥ 0.70 na 20 instances iz existing κ calibration set-a.

Identifikacija ishodi (per-candidate):

• PASS: κ ≥ 0.70 (substantial agreement)
• BORDERLINE: κ 0.60-0.70 (moderate — PM adjudicates on context)
• FAIL: κ < 0.60 (insufficient)

Aggregate verdicts:

• MULTI_PASS: dva ili više kandidata prolaze → rangiraj po κ score-u descending, top kandidat = primary recommendation, drugi = backup for manifest v6
• SINGLE_PASS: samo jedan kandidat prolazi → taj = recommendation (no backup within Chinese roster)
• ALL_FAIL: nijedan ne prolazi → swap path CLOSED, vraćamo se na Branch B ili čekanje Google-a
• INCONCLUSIVE: parse failure rate >= 2/20 na nekom kandidatu, ili API availability issues

---

§2 Model identifier resolution

Kimi "2.7" i MiniMax "M2.7" su Marko-ova terminologija za "najnoviju najjaču varijantu". CC-1 mora real-time verifikovati exact current latest model identifier kroz direct API catalog discovery, ne pretpostavljati od imena.

Routing priority order (Marko ratifikovao 2026-04-24):

1. PRIMARY — direct provider APIs: Moonshot direct za Kimi, MiniMax direct za MiniMax. Native catalog access, najnoviji identifier-i, bez OpenRouter surcharge-a, direct quota ownership.
2. FALLBACK — OpenRouter: koristi samo ako direct API key missing/invalid ili direct catalog ne sadrži latest flagship.

§2.1 Kimi — Moonshot direct (primary)

Required env var: MOONSHOT_API_KEY (ili KIMI_API_KEY zavisno od Marko-ove naming conventions u .env).

Endpoint candidates (try in order, use first that works):

• https://api.moonshot.ai/v1/chat/completions (newer international)
• https://api.moonshot.cn/v1/chat/completions (China mainland)

API surface: OpenAI-compatible (Bearer auth, standard ChatCompletion schema).

Catalog discovery:

• GET https://api.moonshot.ai/v1/models (or .cn equivalent) sa Bearer auth
• Identify latest flagship (prefer k2, kimi-2, kimi-2.7 namespace; reject moonshot-v1-* if newer exists)

Fallback to OpenRouter if direct key missing/invalid or catalog incomplete:

• curl -s https://openrouter.ai/api/v1/models -H "Authorization: Bearer $OPENROUTER_API_KEY" | jq '.data[] | select(.id | contains("moonshot") or contains("kimi"))'

Document selected identifier + routing used (direct / openrouter) + context window + pricing u memo.

§2.2 MiniMax — MiniMax direct (primary)

Required env var: MINIMAX_API_KEY (plus potentially MINIMAX_GROUP_ID — MiniMax API sometimes requires group_id header).

Endpoint candidates:

• https://api.minimaxi.com/v1/text/chatcompletion_v2 (newer OpenAI-compatible)
• https://api.minimax.chat/v1/text/chatcompletion_pro (legacy)

API surface: check docs — may require GroupId query param or header, and response schema slightly different from OpenAI standard.

Catalog discovery:

• MiniMax typically doesn't expose /models endpoint; use docs + direct model identifier
• Known flagship families: M1 series, M2 series, abab6.5, abab7. Prefer latest M2 variant or abab7 namespace.
• If uncertain, try multiple identifiers sequentially (e.g., "MiniMax-M2.7", "MiniMax-M2", "abab7-chat-preview") and log which returned valid response

Fallback to OpenRouter:

• Filter OpenRouter catalog for "minimax"

Document selected identifier + routing used + any special auth params (group_id, etc.) u memo.

§2.3 DeepSeek — DeepSeek direct (primary)

Required env var: DEEPSEEK_API_KEY.

Endpoint:

• https://api.deepseek.com/v1/chat/completions (OpenAI-compatible)

Catalog discovery:

• GET https://api.deepseek.com/v1/models with Bearer auth
• Identify latest flagship reasoning model (prefer V3.1, V4, R2, deepseek-reasoner namespace over V2 / deepseek-chat legacy)
• Document selected identifier + context window + pricing

Fallback to OpenRouter:

• Filter OpenRouter catalog for "deepseek"

§2.4 Zhipu / z.ai — Zhipu direct (primary)

Required env var: ZHIPU_API_KEY ili ZAI_API_KEY (check Marko .env for exact naming).

Endpoint candidates:

• https://api.z.ai/v1/chat/completions (newer international)
• https://open.bigmodel.cn/api/paas/v4/chat/completions (China mainland, legacy branding as "智谱清言")

API surface: OpenAI-compatible Bearer auth typical; verify newer z.ai endpoint schema.

Catalog discovery:

• GET https://api.z.ai/v1/models or equivalent with Bearer auth
• Identify latest GLM flagship (prefer GLM-4.6, GLM-4.7, GLM-5 namespace over GLM-4 legacy)
• Document selected identifier + context window + pricing

Fallback to OpenRouter:

• Filter OpenRouter catalog for "zhipu" or "glm"

---

§3 Sample selection

Uzeti 20 instances iz postojećeg κ calibration set-a (ne iz N=400 canonical fixture). Location: identifikuj iz manifest v5 §N / v4 predecessor references (gde god je κ=0.7458 kalibrisan). Kriterijum:

• 20 instances koji imaju već-postojeće Opus 4.7 + GPT-5.4 + Gemini 3.1 Pro verdicts u dataset-u
• Stratified ako moguće: izbalansirano preko cells (no-context, retrieval, agentic, itd.) da probe ne pokrije samo jednu kategoriju
• Identifikuj po canonical instance_id iz postojećeg κ calibration results JSONL

Ako balanced selection nije moguć zbog dataset structure, uzmi prvih 20 iz κ calibration results file-a (deterministic ordering) i document u memo-u.

STRICT CONSTRAINT: ne generisati nove judge verdicts od Opus ili GPT za ovo probe. Koristimo njihove already-generated verdicts iz κ calibration set-a kao ground truth reference. Nove judge pozive samo ka Kimi i MiniMax.

---

§4 Probe execution

§4.1 Prompt template

Koristi verbatim judge prompt template iz failure-mode-judge.ts:245-258 (isti koji Gemini 3.1 Pro dobija trenutno). Ne modifikuj, ne trim-uj, ne adaptiraj. Identičan input za fair comparison.

§4.2 Call pattern

• Per instance: 4 API calls (1 Kimi + 1 MiniMax + 1 DeepSeek + 1 Zhipu)
• Total: 80 API calls
• Sequential OK (volume trivial, no rate limit pressure at this scale)
• Deterministic settings: temperature=0.0, max_tokens matched na current judge runner setting
• Retries: up to 3 na transient errors (rate limit, network); log retry count per instance

§4.3 Output parsing

Parse Kimi + MiniMax responses kroz identičan parser koji judge-runner koristi za Gemini 3.1 Pro responses. Ako parsing fail → count as "judge failure" (ne kao disagreement). Threshold: ≥18/20 successful parse per candidate for valid probe; <18 → INCONCLUSIVE verdict.

---

§5 κ computation

Computation matrix per candidate:

• Consensus reference: Opus + GPT on that instance
  • If Opus == GPT → consensus = that value
  • If Opus != GPT → "split" (exclude from κ or treat as disagreement — standard practice: exclude, document count)
• Candidate verdict: Kimi or MiniMax verdict on same instance
• Agreement matrix: 2×2 (consensus × candidate) across instances

Compute Cohen's κ for each candidate against Opus+GPT consensus.

Success criteria:

• κ ≥ 0.80 = excellent agreement (preferred)
• κ ≥ 0.70 = substantial agreement (pass threshold)
• κ ≥ 0.60 = moderate agreement (borderline — PM adjudicates)
• κ < 0.60 = fair/poor (FAIL)

Also compute:

• Raw agreement % (for context)
• Count of "split" consensus instances (excluded from κ)
• Per-cell breakdown if stratified sample permits

---

§6 Scope guards (identični §1.3f)

• Manifest v5 anchor fc16925 immutable. No v6 emit u probe phase.
• HEAD 373516c + 8 commits od v4 anchor-a intact. Last commit short SHA = 8ad0567 (§1.3f).
• §11 frozen paths netaknuti: runner, judge-runner, failure-mode-judge, health-check, litellm-config.yaml.
• No new LiteLLM alias u litellm-config.yaml za Kimi ili MiniMax tokom probe-a. Probe script direktno priča sa OpenRouter API (ili Moonshot/MiniMax direct) bez LiteLLM proxy layer-a.
• Novi folder: benchmarks/probes/judge-swap-validation/ (izvan §11 frozen paths).
• Package install pre-authorized: ako je potrebno, pip install openai (ili ekvivalent za OpenRouter routing) — minimal add. Log installed versions u memo.

---

§7 Deliverables

Commit na feature/c3-v3-wrapper:

1. benchmarks/probes/judge-swap-validation/probe-script.py (ili .ts) — probe code sa inline comments
2. benchmarks/probes/judge-swap-validation/sample-instances.jsonl — 20 instances selekovanih sa instance_id + cell + Opus verdict + GPT verdict (ground truth reference)
3. benchmarks/probes/judge-swap-validation/kimi-responses.jsonl — 20 Kimi verdicts (raw + parsed)
4. benchmarks/probes/judge-swap-validation/minimax-responses.jsonl — 20 MiniMax verdicts (raw + parsed)
5. benchmarks/probes/judge-swap-validation/deepseek-responses.jsonl — 20 DeepSeek verdicts (raw + parsed)
6. benchmarks/probes/judge-swap-validation/zhipu-responses.jsonl — 20 Zhipu verdicts (raw + parsed)
7. benchmarks/probes/judge-swap-validation/kappa-analysis.md — κ matrix (per-candidate) + raw agreement + split count + verdict + ranking table
8. benchmarks/probes/judge-swap-validation/validation-memo.md — ≤200 reči summary sa explicit per-candidate verdict + aggregate verdict (MULTI_PASS / SINGLE_PASS / ALL_FAIL / INCONCLUSIVE), recommended primary + backup candidates, actual cost, wall-clock

Anchor commit: [probe] judge swap validation 4-candidate: kimi + minimax + deepseek + zhipu - <aggregate_verdict>. SHA-256 svih artefakata u halt ping.

---

§8 Budget i halt criteria

• Budget cap: $8 (adjusted from $5 za 2-candidate → 4-candidate roster; realno ~$4-6 expected)
• Halt @ $10: ako spend pređe, full escalation, no completion
• Per-call timeout: 60s (judge calls su short)
• Total wall-clock cap: 2.5h (setup + 80 calls + per-candidate κ compute + aggregate ranking + memo)

---

§9 Halt ping format

Emit na completion (success/fail/inconclusive):

• aggregate_verdict: MULTI_PASS | SINGLE_PASS | ALL_FAIL | INCONCLUSIVE
• Per-candidate block (repeat for all 4):
  • <candidate>_identifier_used: <string>
  • <candidate>_routing: direct | openrouter
  • kappa_<candidate>_vs_consensus: <float>
  • raw_agreement_<candidate>: <float>
  • successful_parse_<candidate>: <int>/20
  • per_candidate_verdict: PASS | BORDERLINE | FAIL
• split_consensus_excluded: <int> (Opus≠GPT count, shared across candidates)
• ranking_by_kappa: [<candidate1_by_rank>, <candidate2>, <candidate3>, <candidate4>]
• recommended_primary: KIMI | MINIMAX | DEEPSEEK | ZHIPU | NONE
• recommended_backup: KIMI | MINIMAX | DEEPSEEK | ZHIPU | NONE (if MULTI_PASS)
• anchor_commit: <full sha>
• artefact_shas: { script, sample, kimi-responses, minimax-responses, deepseek-responses, zhipu-responses, kappa, memo }
• wall_clock: <duration>
• cost_actual: $<actual> (vs $8 cap)
• next_step_request: PM-RATIFY-JUDGE-SWAP-VALIDATION
• cc1_state: HALTED

CC-1 ne self-advances. Ne emituje manifest v6 swap. Ne modifikuje runner. Čeka PM ratifikaciju.

---

§10 Task #29 trace update (post-probe)

Posle halt ping-a, PM update:

• §1.3g Judge swap validation:
• If BOTH_PASS / KIMI_PASS / MINIMAX_PASS: emit manifest v6 swap proposal brief za recommended candidate
• If BOTH_FAIL: swap path CLOSED, fall back na Google ticket waiting + Branch B prep
• If INCONCLUSIVE: PM odlučuje retry parameter-ima ili escalates

Post-PASS path timeline: manifest v6 emit + full κ re-calibration ($15-25, ~2h) + N=400 run sa novim trojcem (standard tier quota, no 250 RPD cap) → ~1-1.5 days total to SOTA claim completion, bez scope footnote.

---

§11 Authorized by

PM Marko Marković, 2026-04-24 evening, verbatim: "Ahmo da ih probamo. I kimi i Minimax... najjaci modeli, ne bih ja cekao google, i bolje da imamo kineza"

CC-1 može da počne odmah — gcloud tooling nije potreban za ovaj probe (OpenRouter alternative route).