oleg.m/waggle-os
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
waggle-os/docs/briefs/2026-04-24-cc1-judge-swap-stratified-reprobe-brief.md
Oleg Maslov 0c3e2ead3b
Some checks failed
Installer Smoke / installer-smoke (push) Has been cancelled
Details
moving
2026-09-02 10:10:29 +02:00

11 KiB
Raw Permalink Blame History

CC-1 Brief — §1.3h Judge Swap Stratified Discriminating Re-Probe

Date: 2026-04-24 (evening, post-§1.3g) Status: §1.3h sub-gate, PM-adjudicated Path 2 (stratified re-probe) over Path 1 (accept-by-operational-criteria) Authorized by: Marko Marković (2026-04-24, "ostalo se slazem pravi prompt" + MINIMAX_GROUP_ID added to .env) Predecessor: §1.3g MULTI_PASS sa methodological caveat — κ=1.0000 tie across all 4 candidates na first-4-per-cell sample biased toward unanimous cases (split_consensus_excluded=0/20 vs full-set split rate 7%). Operational ranking (Zhipu > DeepSeek > MiniMax > Kimi) heuristic ne diskriminativan empirical signal. PM: claude-opus-4-7 (Cowork)

§0 Cilj re-probe-a

Diskriminisati 4 judge candidate-a na challenging cases gde Opus ≠ GPT (split consensus). Unanimous cases daju κ=1.0 by construction — nisu informativni. Split cases su gde stvarno vidimo judge character + calibration quality + independence from reference judges.

Output: empirical κ ranking based on discriminating signal, plus operational metrics (parse rate, latency, routing) na svežem challenging sample.

§1 Path re-prioritization summary

§1.3g confirmed: all 4 candidates pass κ ≥ 0.70 threshold. §1.3h goal:

  • Primary: rank candidates on SPLIT-CASE κ (discriminating)
  • Secondary: re-measure parse rate + latency + routing na split cases (complexity may degrade metrics vs unanimous)
  • Tertiary: sanity-check full 40-instance aggregate κ (should match or slightly exceed original if sample is representative)

Post-§1.3h verdict scenarios:

  • SPLIT_DISCRIMINATING: κ values spread across candidates on splits → empirical ranking determines primary + backup
  • STILL_ALL_PASS: all candidates κ ≥ 0.70 on splits too → operational secondary ranking legitimately tie-breaks
  • PARTIAL_FAIL: some candidates κ < 0.70 on splits → subset passes, clean cut
  • ALL_FAIL_ON_SPLITS: no candidate ≥ 0.70 on splits → swap path CLOSED, return to Branch B or Google ticket wait

§2 Sample construction (40 instances, stratified)

§2.1 Preserved unanimous subset (20 instances)

Reuse existing §1.3g sample-instances.jsonl (SHA f4770fec...). 20 instances gde Opus = GPT = Gemini. No need to re-select. For these, candidates already have verdicts from §1.3g — reuse those verdicts, do NOT re-execute calls on unanimous subset. Existing data is idempotent and counted in aggregate.

§2.2 NEW split-cases subset (20 instances)

Select 20 instances iz full κ calibration set-a (the one that produced κ=0.7458 three-way; should be N≈100-300 range) gde Opus verdict ≠ GPT verdict. Full-set split rate = 7% per §1.3g memo; if full κ set = 286 instances, expected splits ≈ 20 (exactly our target), meaning selection may effectively be "all available splits".

Selection priority:

  1. Stratified across cells (no-context, retrieval, full-context, oracle-context, agentic) — prefer balanced representation
  2. If all-available-splits count < 20: use all, document shortage, proceed with smaller sample (minimum 12 for meaningful discrimination)
  3. If all-available-splits count > 20: stratified random selection with seed=20260424 for reproducibility

Saving the split sample: benchmarks/probes/judge-swap-validation/split-cases-sample.jsonl with fields: instance_id, cell, opus_verdict, gpt_verdict, gemini_verdict (if present in original κ set — reference only, NOT used as consensus).

Critical note on consensus definition for split cases: On splits, Opus+GPT consensus doesn't exist. Use dual reference measurement:

  • agreement_vs_opus: candidate == Opus verdict (0 or 1 per instance)
  • agreement_vs_gpt: candidate == GPT verdict (0 or 1 per instance)
  • Per candidate compute: p_opus = agreement_vs_opus_count / split_n, p_gpt = agreement_vs_gpt_count / split_n
  • Well-calibrated judge ≈ 50/50 split (independent judgment; doesn't mimic either reference)
  • Biased judge shows > 70/30 systematic lean (correlated with one reference style; less ideal for ensemble)
  • Compute κ per candidate vs Opus alone AND vs GPT alone separately; take min(κ_vs_opus, κ_vs_gpt) as conservative split-case κ

§3 MiniMax direct routing unblock

Marko confirmed: MINIMAX_GROUP_ID added to D:\Projects\waggle-os\.env between §1.3g and §1.3h sessions.

CC-1 sanity check on pre-flight:

  • Verify MINIMAX_GROUP_ID env var resolves to non-empty string
  • First MiniMax call routes direct (api.minimaxi.com or api.minimax.chat) with GroupId header/query param per MiniMax docs
  • If direct still fails: fall back to openrouter as in §1.3g, document reason, continue (don't block probe on MiniMax routing issue — unanimous subset already has openrouter-routed MiniMax verdicts)

§4 Execution

§4.1 Call pattern

  • Per NEW split instance: 4 API calls (Kimi + MiniMax + DeepSeek + Zhipu)
  • Total NEW API calls: 4 × 20 = 80
  • UNANIMOUS verdicts reused, zero new API calls for that subset
  • Sequential OK
  • Deterministic: temperature=0.0, matched max_tokens
  • Retries: up to 3 on transient errors
  • Kimi max_tokens=4096 carry-over from §1.3g (addressed parse issues there)

§4.2 Metrics per candidate

Compute on NEW split subset (20 instances):

  • parse_success: /20
  • agreement_vs_opus_count: /valid
  • agreement_vs_gpt_count: /valid
  • p_opus, p_gpt: percentages
  • kappa_vs_opus: Cohen's κ candidate-vs-Opus
  • kappa_vs_gpt: Cohen's κ candidate-vs-GPT
  • kappa_conservative: min(kappa_vs_opus, kappa_vs_gpt)
  • latency_p50_split: median latency on split cases
  • latency_p95_split: p95 latency on split cases (for N=400 cost projection)

Compute on AGGREGATE 40 (unanimous + split):

  • kappa_aggregate_vs_consensus: κ on aggregated sample treating unanimous consensus as reference and split cases as per §2.2 dual reference with conservative min
  • This combines known-high unanimous κ with discriminating split κ for holistic view

§4.3 Per-candidate verdict (on split subset)

  • κ_conservative ≥ 0.70 → PASS
  • 0.60 ≤ κ_conservative < 0.70 → BORDERLINE
  • κ_conservative < 0.60 → FAIL

§4.4 Aggregate verdict

  • SPLIT_DISCRIMINATING: spread in κ_conservative across candidates ≥ 0.15 (meaningful differentiation)
  • STILL_ALL_PASS: all 4 κ_conservative ≥ 0.70, spread < 0.15 (tie — operational criteria determine rank)
  • PARTIAL_FAIL: 1-3 candidates κ_conservative < 0.70 (subset passes, natural cut)
  • ALL_FAIL_ON_SPLITS: 0 candidates κ_conservative ≥ 0.70 (swap path CLOSED)
  • INCONCLUSIVE: parse rate < 80% on any candidate on splits; OR valid split sample size < 12 after selection

§5 Scope guards

  • Manifest v5 anchor fc16925 immutable. NO v6 emission.
  • HEAD 373516c + 9 commits (including 8a2f0e6 §1.3g anchor) intact. No drift.
  • §11 frozen paths untouched (runner, judge-runner, failure-mode-judge, health-check, litellm-config.yaml).
  • No new LiteLLM aliases; probe bypasses proxy.
  • Reuse existing Opus/GPT verdicts from full κ calibration set; NO new Opus/GPT calls.
  • Modifications limited to benchmarks/probes/judge-swap-validation/ (existing probe folder from §1.3g).

§6 Deliverables

Commit na feature/c3-v3-wrapper:

  1. benchmarks/probes/judge-swap-validation/split-cases-sample.jsonl — 20 new split-case instances with instance_id, cell, Opus verdict, GPT verdict, Gemini reference
  2. benchmarks/probes/judge-swap-validation/kimi-split-responses.jsonl — 20 raw + parsed verdicts
  3. benchmarks/probes/judge-swap-validation/minimax-split-responses.jsonl — 20 raw + parsed verdicts
  4. benchmarks/probes/judge-swap-validation/deepseek-split-responses.jsonl — 20 raw + parsed verdicts
  5. benchmarks/probes/judge-swap-validation/zhipu-split-responses.jsonl — 20 raw + parsed verdicts
  6. benchmarks/probes/judge-swap-validation/kappa-split-analysis.md — per-candidate split κ matrix (κ_vs_opus, κ_vs_gpt, κ_conservative, p_opus, p_gpt balance analysis) + aggregate κ (40) + ranking by κ_conservative descending + per-candidate split verdict
  7. benchmarks/probes/judge-swap-validation/reprobe-memo.md — ≤250 words, aggregate verdict (SPLIT_DISCRIMINATING / STILL_ALL_PASS / PARTIAL_FAIL / ALL_FAIL_ON_SPLITS / INCONCLUSIVE), final recommended primary + backup with reasoning, actual cost, wall-clock, MiniMax routing resolution (direct / openrouter / failed-back)

Existing §1.3g artefacts (probe-script.py, sample-instances.jsonl, 4× responses, kappa-analysis.md, validation-memo.md) remain unchanged. New artefacts are additive.

Anchor commit: [probe] judge swap stratified reprobe on split cases - <aggregate_verdict>. Parent = 8a2f0e6 (§1.3g).

§7 Budget i halt criteria

  • Budget cap: $3 (realno ~$1.50-2.50 expected: 80 calls × average $0.02)
  • Halt @ $5: full escalation
  • Per-call timeout: 60s (Kimi historical 32s; allow headroom)
  • Total wall-clock cap: 60 min (setup + 80 calls + dual-κ compute + memo)

§8 Halt ping format

Emit at completion:

  • aggregate_verdict: SPLIT_DISCRIMINATING | STILL_ALL_PASS | PARTIAL_FAIL | ALL_FAIL_ON_SPLITS | INCONCLUSIVE
  • Per-candidate block (all 4): <candidate>_parse_success: <int>/20 <candidate>_kappa_vs_opus: <float> <candidate>_kappa_vs_gpt: <float> <candidate>_kappa_conservative: <float> <candidate>_p_opus: <float> <candidate>_p_gpt: <float> <candidate>_latency_p50_split: <int>s <candidate>_latency_p95_split: <int>s <candidate>_routing_actual: direct | openrouter <candidate>_split_verdict: PASS | BORDERLINE | FAIL
  • aggregate_kappa_40: { zhipu, deepseek, minimax, kimi } (per candidate on combined 40)
  • ranking_by_kappa_conservative_desc: ordered list
  • split_cases_selected: <int>/20 (actual selection count, may be < 20 if pool small)
  • recommended_primary: KIMI|MINIMAX|DEEPSEEK|ZHIPU|NONE
  • recommended_backup: KIMI|MINIMAX|DEEPSEEK|ZHIPU|NONE
  • minimax_routing_resolution: direct_successful | direct_failed_fell_back | openrouter_only
  • anchor_commit: <full sha>
  • artefact_shas: { split-sample, 4× split-responses, kappa-split-analysis, reprobe-memo }
  • wall_clock: <duration>
  • cost_actual: $<actual> vs $3 cap
  • next_step_request: PM-RATIFY-JUDGE-SWAP-REPROBE
  • cc1_state: HALTED

CC-1 ne self-advances. Ne emituje manifest v6. Čeka PM ratifikaciju.

§9 Task #29 trace update (post-reprobe)

  • §1.1 ✓ §1.2 ✓ §1.3b IN_SCOPE ✓ §1.3c PASS ✓ §1.3e strict hold ✓ §1.3f INFEASIBLE ✓ §1.3g MULTI_PASS ✓ §1.3h
  • Post-§1.3h:
    • If SPLIT_DISCRIMINATING / PARTIAL_FAIL / STILL_ALL_PASS → PM emit manifest v6 swap proposal brief sa data-driven primary + backup
    • If ALL_FAIL_ON_SPLITS → swap path CLOSED; return to Google ticket waiting or Branch B prep
    • If INCONCLUSIVE → PM adjudicates: retry with larger sample, or alternate methodology

§10 Authorized by

PM Marko Marković, 2026-04-24 evening. Verbatim: "ostalo se slazem pravi prompt". MINIMAX_GROUP_ID added to .env between sessions.

CC-1 may begin immediately — all prereqs in place (direct keys, GroupId, prior §1.3g artefacts on disk, gcloud tooling retained from §1.3f as permanent operational asset).

Reference in New Issue View Git Blame Copy Permalink