This commit is contained in:
Oleg Maslov
@@ -0,0 +1,203 @@
|
||||
# CC-1 Brief — §1.3h Judge Swap Stratified Discriminating Re-Probe
|
||||
|
||||
**Date**: 2026-04-24 (evening, post-§1.3g)
|
||||
**Status**: §1.3h sub-gate, PM-adjudicated Path 2 (stratified re-probe) over Path 1 (accept-by-operational-criteria)
|
||||
**Authorized by**: Marko Marković (2026-04-24, "ostalo se slazem pravi prompt" + MINIMAX_GROUP_ID added to .env)
|
||||
**Predecessor**: §1.3g MULTI_PASS sa methodological caveat — κ=1.0000 tie across all 4 candidates na first-4-per-cell sample biased toward unanimous cases (split_consensus_excluded=0/20 vs full-set split rate 7%). Operational ranking (Zhipu > DeepSeek > MiniMax > Kimi) heuristic ne diskriminativan empirical signal.
|
||||
**PM**: claude-opus-4-7 (Cowork)
|
||||
|
||||
---
|
||||
|
||||
## §0 Cilj re-probe-a
|
||||
|
||||
Diskriminisati 4 judge candidate-a na **challenging cases gde Opus ≠ GPT** (split consensus). Unanimous cases daju κ=1.0 by construction — nisu informativni. Split cases su gde stvarno vidimo judge character + calibration quality + independence from reference judges.
|
||||
|
||||
Output: empirical κ ranking based on discriminating signal, plus operational metrics (parse rate, latency, routing) na svežem challenging sample.
|
||||
|
||||
---
|
||||
|
||||
## §1 Path re-prioritization summary
|
||||
|
||||
§1.3g confirmed: all 4 candidates pass κ ≥ 0.70 threshold. §1.3h goal:
|
||||
- Primary: rank candidates on SPLIT-CASE κ (discriminating)
|
||||
- Secondary: re-measure parse rate + latency + routing na split cases (complexity may degrade metrics vs unanimous)
|
||||
- Tertiary: sanity-check full 40-instance aggregate κ (should match or slightly exceed original if sample is representative)
|
||||
|
||||
**Post-§1.3h verdict scenarios**:
|
||||
- **SPLIT_DISCRIMINATING**: κ values spread across candidates on splits → empirical ranking determines primary + backup
|
||||
- **STILL_ALL_PASS**: all candidates κ ≥ 0.70 on splits too → operational secondary ranking legitimately tie-breaks
|
||||
- **PARTIAL_FAIL**: some candidates κ < 0.70 on splits → subset passes, clean cut
|
||||
- **ALL_FAIL_ON_SPLITS**: no candidate ≥ 0.70 on splits → swap path CLOSED, return to Branch B or Google ticket wait
|
||||
|
||||
---
|
||||
|
||||
## §2 Sample construction (40 instances, stratified)
|
||||
|
||||
### §2.1 Preserved unanimous subset (20 instances)
|
||||
|
||||
**Reuse existing §1.3g `sample-instances.jsonl`** (SHA `f4770fec...`). 20 instances gde Opus = GPT = Gemini. No need to re-select. For these, candidates already have verdicts from §1.3g — **reuse those verdicts, do NOT re-execute calls on unanimous subset**. Existing data is idempotent and counted in aggregate.
|
||||
|
||||
### §2.2 NEW split-cases subset (20 instances)
|
||||
|
||||
Select 20 instances iz full κ calibration set-a (the one that produced κ=0.7458 three-way; should be N≈100-300 range) gde **Opus verdict ≠ GPT verdict**. Full-set split rate = 7% per §1.3g memo; if full κ set = 286 instances, expected splits ≈ 20 (exactly our target), meaning selection may effectively be "all available splits".
|
||||
|
||||
Selection priority:
|
||||
1. Stratified across cells (no-context, retrieval, full-context, oracle-context, agentic) — prefer balanced representation
|
||||
2. If all-available-splits count < 20: use all, document shortage, proceed with smaller sample (minimum 12 for meaningful discrimination)
|
||||
3. If all-available-splits count > 20: stratified random selection with `seed=20260424` for reproducibility
|
||||
|
||||
**Saving the split sample**: `benchmarks/probes/judge-swap-validation/split-cases-sample.jsonl` with fields: `instance_id`, `cell`, `opus_verdict`, `gpt_verdict`, `gemini_verdict` (if present in original κ set — reference only, NOT used as consensus).
|
||||
|
||||
**Critical note on consensus definition for split cases**:
|
||||
On splits, Opus+GPT consensus doesn't exist. Use **dual reference measurement**:
|
||||
- `agreement_vs_opus`: candidate == Opus verdict (0 or 1 per instance)
|
||||
- `agreement_vs_gpt`: candidate == GPT verdict (0 or 1 per instance)
|
||||
- Per candidate compute: `p_opus = agreement_vs_opus_count / split_n`, `p_gpt = agreement_vs_gpt_count / split_n`
|
||||
- Well-calibrated judge ≈ 50/50 split (independent judgment; doesn't mimic either reference)
|
||||
- Biased judge shows > 70/30 systematic lean (correlated with one reference style; less ideal for ensemble)
|
||||
- Compute κ per candidate vs Opus alone AND vs GPT alone separately; take min(κ_vs_opus, κ_vs_gpt) as conservative split-case κ
|
||||
|
||||
---
|
||||
|
||||
## §3 MiniMax direct routing unblock
|
||||
|
||||
Marko confirmed: `MINIMAX_GROUP_ID` added to `D:\Projects\waggle-os\.env` between §1.3g and §1.3h sessions.
|
||||
|
||||
CC-1 sanity check on pre-flight:
|
||||
- Verify `MINIMAX_GROUP_ID` env var resolves to non-empty string
|
||||
- First MiniMax call routes direct (api.minimaxi.com or api.minimax.chat) with GroupId header/query param per MiniMax docs
|
||||
- If direct still fails: fall back to openrouter as in §1.3g, document reason, continue (don't block probe on MiniMax routing issue — unanimous subset already has openrouter-routed MiniMax verdicts)
|
||||
|
||||
---
|
||||
|
||||
## §4 Execution
|
||||
|
||||
### §4.1 Call pattern
|
||||
|
||||
- Per NEW split instance: 4 API calls (Kimi + MiniMax + DeepSeek + Zhipu)
|
||||
- Total NEW API calls: 4 × 20 = 80
|
||||
- UNANIMOUS verdicts reused, zero new API calls for that subset
|
||||
- Sequential OK
|
||||
- Deterministic: `temperature=0.0`, matched `max_tokens`
|
||||
- Retries: up to 3 on transient errors
|
||||
- Kimi `max_tokens=4096` carry-over from §1.3g (addressed parse issues there)
|
||||
|
||||
### §4.2 Metrics per candidate
|
||||
|
||||
Compute on NEW split subset (20 instances):
|
||||
- `parse_success`: <int>/20
|
||||
- `agreement_vs_opus_count`: <int>/valid
|
||||
- `agreement_vs_gpt_count`: <int>/valid
|
||||
- `p_opus`, `p_gpt`: percentages
|
||||
- `kappa_vs_opus`: Cohen's κ candidate-vs-Opus
|
||||
- `kappa_vs_gpt`: Cohen's κ candidate-vs-GPT
|
||||
- `kappa_conservative`: min(kappa_vs_opus, kappa_vs_gpt)
|
||||
- `latency_p50_split`: median latency on split cases
|
||||
- `latency_p95_split`: p95 latency on split cases (for N=400 cost projection)
|
||||
|
||||
Compute on AGGREGATE 40 (unanimous + split):
|
||||
- `kappa_aggregate_vs_consensus`: κ on aggregated sample treating unanimous consensus as reference and split cases as per §2.2 dual reference with conservative min
|
||||
- This combines known-high unanimous κ with discriminating split κ for holistic view
|
||||
|
||||
### §4.3 Per-candidate verdict (on split subset)
|
||||
|
||||
- κ_conservative ≥ 0.70 → PASS
|
||||
- 0.60 ≤ κ_conservative < 0.70 → BORDERLINE
|
||||
- κ_conservative < 0.60 → FAIL
|
||||
|
||||
### §4.4 Aggregate verdict
|
||||
|
||||
- **SPLIT_DISCRIMINATING**: spread in κ_conservative across candidates ≥ 0.15 (meaningful differentiation)
|
||||
- **STILL_ALL_PASS**: all 4 κ_conservative ≥ 0.70, spread < 0.15 (tie — operational criteria determine rank)
|
||||
- **PARTIAL_FAIL**: 1-3 candidates κ_conservative < 0.70 (subset passes, natural cut)
|
||||
- **ALL_FAIL_ON_SPLITS**: 0 candidates κ_conservative ≥ 0.70 (swap path CLOSED)
|
||||
- **INCONCLUSIVE**: parse rate < 80% on any candidate on splits; OR valid split sample size < 12 after selection
|
||||
|
||||
---
|
||||
|
||||
## §5 Scope guards
|
||||
|
||||
- Manifest v5 anchor `fc16925` immutable. NO v6 emission.
|
||||
- HEAD `373516c` + 9 commits (including `8a2f0e6` §1.3g anchor) intact. No drift.
|
||||
- §11 frozen paths untouched (runner, judge-runner, failure-mode-judge, health-check, litellm-config.yaml).
|
||||
- No new LiteLLM aliases; probe bypasses proxy.
|
||||
- Reuse existing Opus/GPT verdicts from full κ calibration set; NO new Opus/GPT calls.
|
||||
- Modifications limited to `benchmarks/probes/judge-swap-validation/` (existing probe folder from §1.3g).
|
||||
|
||||
---
|
||||
|
||||
## §6 Deliverables
|
||||
|
||||
Commit na `feature/c3-v3-wrapper`:
|
||||
|
||||
1. `benchmarks/probes/judge-swap-validation/split-cases-sample.jsonl` — 20 new split-case instances with instance_id, cell, Opus verdict, GPT verdict, Gemini reference
|
||||
2. `benchmarks/probes/judge-swap-validation/kimi-split-responses.jsonl` — 20 raw + parsed verdicts
|
||||
3. `benchmarks/probes/judge-swap-validation/minimax-split-responses.jsonl` — 20 raw + parsed verdicts
|
||||
4. `benchmarks/probes/judge-swap-validation/deepseek-split-responses.jsonl` — 20 raw + parsed verdicts
|
||||
5. `benchmarks/probes/judge-swap-validation/zhipu-split-responses.jsonl` — 20 raw + parsed verdicts
|
||||
6. `benchmarks/probes/judge-swap-validation/kappa-split-analysis.md` — per-candidate split κ matrix (κ_vs_opus, κ_vs_gpt, κ_conservative, p_opus, p_gpt balance analysis) + aggregate κ (40) + ranking by κ_conservative descending + per-candidate split verdict
|
||||
7. `benchmarks/probes/judge-swap-validation/reprobe-memo.md` — ≤250 words, aggregate verdict (SPLIT_DISCRIMINATING / STILL_ALL_PASS / PARTIAL_FAIL / ALL_FAIL_ON_SPLITS / INCONCLUSIVE), final recommended primary + backup with reasoning, actual cost, wall-clock, MiniMax routing resolution (direct / openrouter / failed-back)
|
||||
|
||||
Existing §1.3g artefacts (`probe-script.py`, `sample-instances.jsonl`, 4× responses, `kappa-analysis.md`, `validation-memo.md`) remain unchanged. New artefacts are additive.
|
||||
|
||||
Anchor commit: `[probe] judge swap stratified reprobe on split cases - <aggregate_verdict>`. Parent = `8a2f0e6` (§1.3g).
|
||||
|
||||
---
|
||||
|
||||
## §7 Budget i halt criteria
|
||||
|
||||
- **Budget cap**: $3 (realno ~$1.50-2.50 expected: 80 calls × average $0.02)
|
||||
- **Halt @ $5**: full escalation
|
||||
- **Per-call timeout**: 60s (Kimi historical 32s; allow headroom)
|
||||
- **Total wall-clock cap**: 60 min (setup + 80 calls + dual-κ compute + memo)
|
||||
|
||||
---
|
||||
|
||||
## §8 Halt ping format
|
||||
|
||||
Emit at completion:
|
||||
|
||||
- `aggregate_verdict: SPLIT_DISCRIMINATING | STILL_ALL_PASS | PARTIAL_FAIL | ALL_FAIL_ON_SPLITS | INCONCLUSIVE`
|
||||
- Per-candidate block (all 4):
|
||||
`<candidate>_parse_success: <int>/20`
|
||||
`<candidate>_kappa_vs_opus: <float>`
|
||||
`<candidate>_kappa_vs_gpt: <float>`
|
||||
`<candidate>_kappa_conservative: <float>`
|
||||
`<candidate>_p_opus: <float>`
|
||||
`<candidate>_p_gpt: <float>`
|
||||
`<candidate>_latency_p50_split: <int>s`
|
||||
`<candidate>_latency_p95_split: <int>s`
|
||||
`<candidate>_routing_actual: direct | openrouter`
|
||||
`<candidate>_split_verdict: PASS | BORDERLINE | FAIL`
|
||||
- `aggregate_kappa_40: { zhipu, deepseek, minimax, kimi }` (per candidate on combined 40)
|
||||
- `ranking_by_kappa_conservative_desc: ordered list`
|
||||
- `split_cases_selected: <int>/20` (actual selection count, may be < 20 if pool small)
|
||||
- `recommended_primary: KIMI|MINIMAX|DEEPSEEK|ZHIPU|NONE`
|
||||
- `recommended_backup: KIMI|MINIMAX|DEEPSEEK|ZHIPU|NONE`
|
||||
- `minimax_routing_resolution: direct_successful | direct_failed_fell_back | openrouter_only`
|
||||
- `anchor_commit: <full sha>`
|
||||
- `artefact_shas: { split-sample, 4× split-responses, kappa-split-analysis, reprobe-memo }`
|
||||
- `wall_clock: <duration>`
|
||||
- `cost_actual: $<actual>` vs $3 cap
|
||||
- `next_step_request: PM-RATIFY-JUDGE-SWAP-REPROBE`
|
||||
- `cc1_state: HALTED`
|
||||
|
||||
CC-1 ne self-advances. Ne emituje manifest v6. Čeka PM ratifikaciju.
|
||||
|
||||
---
|
||||
|
||||
## §9 Task #29 trace update (post-reprobe)
|
||||
|
||||
- §1.1 ✓ §1.2 ✓ §1.3b IN_SCOPE ✓ §1.3c PASS ✓ §1.3e strict hold ✓ §1.3f INFEASIBLE ✓ §1.3g MULTI_PASS ✓ §1.3h <verdict>
|
||||
- Post-§1.3h:
|
||||
- If SPLIT_DISCRIMINATING / PARTIAL_FAIL / STILL_ALL_PASS → PM emit manifest v6 swap proposal brief sa data-driven primary + backup
|
||||
- If ALL_FAIL_ON_SPLITS → swap path CLOSED; return to Google ticket waiting or Branch B prep
|
||||
- If INCONCLUSIVE → PM adjudicates: retry with larger sample, or alternate methodology
|
||||
|
||||
---
|
||||
|
||||
## §10 Authorized by
|
||||
|
||||
PM Marko Marković, 2026-04-24 evening. Verbatim: "ostalo se slazem pravi prompt". MINIMAX_GROUP_ID added to .env between sessions.
|
||||
|
||||
CC-1 may begin immediately — all prereqs in place (direct keys, GroupId, prior §1.3g artefacts on disk, gcloud tooling retained from §1.3f as permanent operational asset).
|
||||
Reference in New Issue
Block a user