moving

docs/reports/sonnet-calibration-2026-04-21T08-55-51Z.md

@@ -0,0 +1,84 @@
+# Sonnet 4.6 Calibration — Sprint 10 Task 1.3
+
+**Generated:** 2026-04-21T08:55:51Z
+**Calibration artifact:** `preflight-results/judge-calibration-sonnet-2026-04-21T08-55-51Z.json`
+**Labels source:** `D:/Projects/PM-Waggle-OS/calibration/2026-04-20-failure-mode-calibration-labels.md`
+**Judge model:** `claude-sonnet-4-6` (post Task 1.2 route repair, merge commit `a09831e`)
+**Instances:** 10
+
+---
+
+## Result
+
+**Match rate: 8/10 — verdict: PASS** (judge-calibration rubric: ≥8 = PASS, 6-7 = PARTIAL, <6 = FAIL).
+
+**Spend:** $0.027 of $0.50 Task 1.3 budget (5.4%).
+
+### Per-instance detail
+
+| # | Instance | PM label | Sonnet label | Outcome |
+|---|---|---|---|---|
+| 1 | `locomo_conv-26_q109` | correct/null | correct/null | MATCH |
+| 2 | `locomo_conv-41_q123` | incorrect/F3 | incorrect/F3 | MATCH |
+| 3 | `locomo_conv-50_q141` | incorrect/F4 | incorrect/F3 | DIFF |
+| 4 | `locomo_conv-42_q030` | incorrect/F2 | incorrect/F2 | MATCH |
+| 5 | `locomo_conv-49_q015` | correct/null | correct/null | MATCH |
+| 6 | `locomo_conv-41_q036` | incorrect/F5 | incorrect/F5 | MATCH |
+| 7 | `locomo_conv-42_q038` | incorrect/F3 | incorrect/F3 | MATCH |
+| 8 | `locomo_conv-41_q053` | incorrect/F4 | incorrect/F4 | MATCH |
+| 9 | `locomo_conv-50_q037` | correct/null | incorrect/F4 | DIFF |
+| 10 | `locomo_conv-47_q017` | incorrect/F1 | incorrect/F1 | MATCH |
+
+---
+
+## Decision tree (brief §1.3)
+
+Brief §1.3 decision matrix on Sonnet match rate:
+
+| Band | Action |
+|---|---|
+| ≥ 9/10 | Sonnet becomes Stage 2 primary default |
+| **7-8/10** | **Trigger Task 2.2 Fleiss' kappa probe (multi-vendor LOCKED — not Claude-only)** |
+| < 7/10 | Stick with Opus, PM review gate |
+
+Task 1.3 landed at **8/10** → **triggers Task 2.2 multi-vendor kappa** path per brief. Sprint 10 Task 2.1 (multi-vendor ensemble baseline on same 10 triples) already ran as Day-2 Step 3 — see `docs/reports/multi-vendor-ensemble-baseline-2026-04-21T08-56-43Z.md`.
+
+---
+
+## Disagreement analysis
+
+### Instance 3 — `locomo_conv-50_q141` (single-hop)
+
+- **Question:** "Which city is featured in the photograph Dave showed Calvin?"
+- **PM:** `incorrect/F4` (hallucination / fabrication)
+- **Sonnet:** `incorrect/F3` (wrong content / entity)
+- Both verdicts agree on **incorrect**; only the failure_mode taxonomy differs. Sonnet read the model's "Chicago" as a wrong-entity substitution (F3). PM labeled it as fabrication (F4).
+- Subtle taxonomy judgment call — F3 vs F4 on a wrong-city answer is defensible either way depending on how strictly one reads F4 as "unsupported by any context" vs F3 as "identifiable wrong entity".
+- **Impact on Stage 2 banner:** zero — both map to `incorrect` in the binary correctness rollup. Failure-mode breakdown shifts one count from F4 to F3.
+
+### Instance 9 — `locomo_conv-50_q037` (open-ended)
+
+- **Question:** "Does Calvin love music tours?"
+- **PM:** `correct/null`
+- **Sonnet:** `incorrect/F4` — "The model introduces fabricated details not present in the ground-truth context, specifically 'touring with Frank Ocean' and 'felt alive on stage in Tokyo'."
+- **Historical pattern on this instance:**
+  - Sprint 9 Haiku (Task 4 diagnostic): `incorrect/F4` — flagged the Frank Ocean fabrication.
+  - Sprint 9 Opus 4.7 (Task 4 production): `correct/null` — accepted PM's label.
+  - Sprint 10 Sonnet (this run): `incorrect/F4` — flags the fabrication.
+  - Sprint 10 ensemble Opus 4.7: `correct/null` (unchanged).
+  - Sprint 10 ensemble GPT-5.4: `incorrect/F4`.
+  - Sprint 10 ensemble Gemini 3.1 Pro: `incorrect/F4`.
+- **Aggregate cross-vendor disagreement with PM on instance 9:** 4 of 5 non-Opus-4.7 judgments (Haiku + Sonnet + GPT-5.4 + Gemini) flag F4; only Opus 4.7 agrees with PM's `correct/null`.
+- **Signal:** consistent cross-family disagreement on a specific PM label, not a per-judge weakness. PM label may warrant a re-review — flagged to PM in the Task 2.1 baseline report §4.
+
+---
+
+## Conclusion
+
+- Task 1.2 Sonnet route repair verified in production: 10/10 non-404 successful completions.
+- Task 1.3 calibration at 8/10 PASS is within the brief-§1.3 borderline band, which correctly triggered the Task 2.1 ensemble path (ran Day-2 Step 3).
+- Sonnet is production-viable as a Stage-2 judge candidate but does NOT auto-elevate to Stage 2 primary per brief conditional. Final Stage 2 primary selection waits on Task 2.2 full 15-triple Fleiss' kappa.
+
+---
+
+*End of Task 1.3 report. See Task 2.1 baseline for ensemble analysis and Stage 2 recommendations.*