waggle-os/docs/plans/SPRINT-10-CLOSEOUT-2026-04-22.md

# Sprint 10 Close-Out — Task 2.2 + Judge-Methodology Validation

**Datum:** 2026-04-21T13:07:02.515Z
**Artifact:** `preflight-results/judge-calibration-ensemble-14inst-2026-04-21T13-00-04Z.json`
**Labels source:** 14-instance merged set — 9 retained from Sprint 9 (instance #9 Frank Ocean dropped per PM Option C) + 5 new PM-authored triples finalized 2026-04-22.
**Ensemble vendors:** claude-opus-4-7, gpt-5.4, gemini-3.1-pro
**Total calls:** 42 (3 vendors × 14 instances) · **Spend:** $0.151110 of $0.20 Task 2.2 ceiling (75.6%)

---

## 1. Headline result

| Metric | Value | Interpretation |
|---|---|---|
| Majority match vs PM | **13/14** (92.9%) | well above 8/10 PASS threshold |
| Fleiss' κ — vendors only | **0.8784** | strong |
| Fleiss' κ — vendors + PM (4 raters) | **0.8640** | strong |
| Sprint 11 GO/NO-GO (judge-methodology axis) | **GO** | authorized |

### Interpretation band (brief §pre-registered)

| κ range | Band | Stage 2 implication |
|---|---|---|
| ≥ 0.80 | strong | ensemble verdict primary |
| 0.60 — 0.80 | substantial | ensemble ready + tie-breaker policy (documented Day-2 §5 of multi-vendor baseline) |
| 0.40 — 0.60 | moderate | PM review gate |
| < 0.40 | fair or worse | scope pivot to single-judge Opus |

**Delta vs Day-2 10-instance baseline:** Day-2 κ = 0.7458 (n=10) → Day-3 κ = 0.8784 (n=14). Band shifted; diagnostic below.

## 2. Per-vendor match rate vs PM

| Vendor | Match | Spend | Avg latency | Disagreements |
|---|---|---|---|---|
| `claude-opus-4-7` | 12/14 (85.7%) | $0.050184 | 2917ms | 2 |
| `gpt-5.4` | 13/14 (92.9%) | $0.034119 | 2093ms | 1 |
| `gemini-3.1-pro` | 12/14 (85.7%) | $0.066807 | 6973ms | 2 |

## 3. Per-pair Cohen's κ (inter-vendor agreement)

| Pair | κ | Band | Agree% |
|---|---|---|---|
| `claude-opus-4-7` ↔ `gpt-5.4` | 0.9103 | strong | 92.9% |
| `claude-opus-4-7` ↔ `gemini-3.1-pro` | 0.8170 | strong | 85.7% |
| `gpt-5.4` ↔ `gemini-3.1-pro` | 0.9085 | strong | 92.9% |

## 4. Per-category Fleiss' κ breakdown

Categories combine both LoCoMo-native labels (single-hop / multi-hop / temporal / open-ended) and new PM categories (temporal-scope / null-result / chain-of-anchor).

| Category | n | κ | Band |
|---|---|---|---|
| `single-hop` | 3 | 1.0000 | strong |
| `multi-hop` | 3 | 0.6897 | substantial |
| `temporal` | 2 | 0.4545 | moderate |
| `open-ended` | 1 | undefined | n<2, kappa undefined |
| `temporal-scope` | 2 | 1.0000 | strong |
| `null-result` | 2 | undefined | undefined |
| `chain-of-anchor` | 1 | undefined | n<2, kappa undefined |

## 5. Per-F-mode Fleiss' κ breakdown

F-mode taxonomy per judge rubric: F1 (valid abstain), F2 (partial coverage / omission), F3 (misread of substrate), F4 (fabrication), F5 (other). `correct/null` is the PM ground-truth label indicating a correct answer with no failure mode.

| F-mode | n | κ | Band |
|---|---|---|---|
| `F1` | 1 | undefined | n<2, kappa undefined |
| `F2` | 2 | undefined | undefined |
| `F3` | 4 | 0.6250 | substantial |
| `F4` | 4 | undefined | undefined |
| `F5` | 1 | undefined | n<2, kappa undefined |
| `correct/null` | 2 | undefined | undefined |

## 6. Disagreement log

| Vendor | Instance | PM | Vendor |
|---|---|---|---|
| `correct/null` | 7 (locomo_conv-42_q038) | `incorrect/F3` | `correct/null` |
| `correct/null` | 10 (locomo_conv-44_pm_2026-04-22_001) | `incorrect/F3` | `correct/null` |
| `correct/null` | 10 (locomo_conv-44_pm_2026-04-22_001) | `incorrect/F3` | `correct/null` |
| `incorrect/F4` | 6 (locomo_conv-41_q036) | `incorrect/F5` | `incorrect/F4` |
| `correct/null` | 10 (locomo_conv-44_pm_2026-04-22_001) | `incorrect/F3` | `correct/null` |

### Disagreement rationale detail

- **`claude-opus-4-7` on instance 7 (locomo_conv-42_q038)** — PM `incorrect/F3` vs vendor `correct/null`: *7 September 2022 was the Friday before 14 September 2022, matching the ground truth.*
- **`claude-opus-4-7` on instance 10 (locomo_conv-44_pm_2026-04-22_001)** — PM `incorrect/F3` vs vendor `correct/null`: *Early April 2023 is an acceptable equivalent formulation of around April 2, 2023.*
- **`gpt-5.4` on instance 10 (locomo_conv-44_pm_2026-04-22_001)** — PM `incorrect/F3` vs vendor `correct/null`: *The model's answer, 'early April 2023,' is a reasonable equivalent of the ground truth 'around April 2, 2023' and adds no incorrect information.*
- **`gemini-3.1-pro` on instance 6 (locomo_conv-41_q036)** — PM `incorrect/F5` vs vendor `incorrect/F4`: *The model fails to mention the music events John attended and instead hallucinates activities like walks and picnics that are not present in the ground-truth context.*
- **`gemini-3.1-pro` on instance 10 (locomo_conv-44_pm_2026-04-22_001)** — PM `incorrect/F3` vs vendor `correct/null`: *The model's answer of 'early April 2023' accurately reflects the ground truth date of 'around April 2, 2023'.*

## 7. GO/NO-GO signal for Sprint 11 LoCoMo SOTA

**Verdict: GO**

Fleiss' κ = 0.8784 ≥ 0.60 floor. Judge-methodology axis authorized per brief §pre-registered-threshold. Sprint 11 LoCoMo SOTA run cleared on the ensemble layer; pre-registered LoCoMo bands (≥91.6% NEW_SOTA / 85.0-91.5% SOTA_IN_LOCAL_FIRST / <85% GO_NOGO_REVIEW) remain LOCKED for the downstream Sprint 11 outcome.

**Pre-registered LoCoMo thresholds (carried from parent brief §5, LOCKED):**

| Sprint 11 final score | Banner | Consequence |
|---|---|---|
| ≥ 91.6% | `NEW_SOTA` | Full launch narrative (Opus-class multiplier claim) |
| 85.0 — 91.5% | `SOTA_IN_LOCAL_FIRST` | Narrower framing (sovereignty vs cloud-revenue positioning) |
| < 85.0% | `GO_NOGO_REVIEW` | Auto-halt; scope reclassification with PM pre public comms |

Anti-pattern #4 reminder: **thresholds do NOT shift post-hoc.** This clause remains the same as before any Task 2.2 result.

## 8. Sprint 10 scorecard

| Sprint 10 task | Status | Key deliverable |
|---|---|---|
| 1.2 Sonnet route repair | ✅ CLOSED | PR #1 merged `a09831e`; smoke PASS |
| 1.3 Sonnet calibration re-run | ✅ CLOSED | 8/10 match on repaired route, triggered multi-vendor path |
| 1.4 DashScope dual-route | ✅ CLOSED | 3/3 routes PASS; real qwen3.6-35b-a3b on intl tenant |
| 2.1 Tri-vendor ensemble setup | ✅ CLOSED | Fleiss' κ=0.7458 on 10-instance baseline, substantial band |
| 2.2 Full 14-instance Fleiss' κ | ✅ CLOSED | **κ=0.8784** · strong band · **Sprint 11 GO** |
| 1.1 Qwen stability matrix | ✅ CLOSED (PASS) — 36/40 converged, 5 safe configs emerged. Stage 2 primary config LOCKED at `thinking=off, max_tokens=16000` (cheapest 5/5 safe config at ≥16K ceiling per STAGE-2-PREP-BACKLOG exit criterion). Spend $0.085 of $1.50 cap. | `preflight-results/qwen-thinking-stability-2026-04-21T14-05-12-175Z.md` · CSV sibling · exit ping at `PM-Waggle-OS/sessions/2026-04-22-sprint-10-task-1-1-exit.md` |
| 1.5 Harvest Claude artifacts adapter | ✅ CLOSED — Phase 1 (zip verified, artifacts folder ABSENT), Phase 2 (PM ratified Option 4: partial adapter), Phase 3 (hive-mind `c363257` pushed to origin/master per PM ratification 2026-04-22). +7 tests, 312/312 passing, tsc clean. Stage 0 mech #3 ticket stays OPEN (session-artifact gap is Sprint 11+ vendor-path work). | `preflight-results/claude-ai-export-verification-2026-04-22.md` · hive-mind origin/master at `c363257` |

## 9. Cost accounting

| Line | Spend | Running total |
|---|---|---|
| Day-1 vendor probe | $0.001 | $0.001 |
| Day-2 Sonnet calibration | $0.027 | $0.028 |
| Day-2 Tri-vendor 10-instance baseline | $0.101 | $0.129 |
| Day-3 Task 2.2 14-instance ensemble | $0.151 | $0.280 |
| Day-3 Task 1.1 Qwen stability matrix live-run | $0.085 | $0.365 |
| Day-3 Task 1.5 Phase 1+2 (file inspection, 0 API) | $0.000 | $0.365 |
| Day-3 Task 1.5 Phase 3 (local commit, not executed) | $0.000 | $0.365 |

**Sprint 10 total: $0.365 of $15 hard-stop ceiling (2.4%)**

## 10. Anti-pattern #4 compliance check

- Pre-registered κ band floor (0.60) set BEFORE Task 2.2 ran. Verdict delivered against that floor unchanged.
- 14-instance dataset composition defined BEFORE ensemble run (Option C drop of #9, 5 ratified triples finalized, slot-fill via Draft #3). No post-hoc dataset shuffling.
- Single PM-vs-ensemble disagreement (instance 10, temporal precision) is logged, not hidden. Ensemble called "correct/null" where PM called F3 — interpretive disagreement on "early April" vs "around April 2", not a judge fabrication.
- LoCoMo Sprint-11 banner thresholds (≥91.6% / 85-91.5% / <85%) untouched.

## 10b. Task 1.1 stability matrix — Stage 2 primary config LOCKED

The Qwen3.6 thinking-mode stability matrix (40 cells · 2 thinking toggles × 4 max_tokens ceilings × 5 prompt shapes) returned **36/40 converged (90%)**. Five `(thinking, max_tokens)` rows achieved full 5/5 prompt-shape convergence:

| Config | Avg latency (all 5 shapes) | Notes |
|---|---|---|
| `on / 64K` | 17.9s | fastest, reasoning-token overhead |
| `off / 32K` | 22.8s | — |
| `off / 64K` | 23.0s | — |
| **`off / 16K`** | **27.6s** | **recommended — cheapest 5/5 at ≥16K per exit criterion** |
| `on / 16K` | 28.8s | — |

**LOCKED recommendation for Stage 2 LoCoMo full-run: `thinking=off, max_tokens=16000`.** Per `STAGE-2-PREP-BACKLOG.md` §exit-criterion ("any thinking-off ≥16K config that converges 5/5"), this config meets the trigger and costs the least per call.

**Stage-2-unsafe cells to avoid (4 of 40):**
- All `(*, 8K, temporal-scope)` combinations — temporal-scope shape consistently loops at 8K regardless of thinking toggle.
- `(on, 32K, temporal-scope)` — 180s timeout (thinking loop on this shape at 32K ceiling).
- `(on, 8K, direct-fact)` — HTTP 500 one-shot cold-connection hiccup; not a systemic defect (subsequent cells on the identical route succeeded). Caller-side single-retry recommended for Stage 2 first-call-per-batch hardening.

## 11. Ready-state for Sprint 11

- Judge methodology: **AUTHORIZED** at κ=0.8784 (strong band).
- Tri-vendor ensemble verified on 14 instances covering 6 F-mode categories across 7 question categories.
- Tie-breaker policy documented Day-2 (first-in-list today; escalate-to-PM recommended for Sprint 11 Stage-2 full-run to preserve multi-vendor defensibility).
- Task 1.1 stability matrix **CLOSED with PASS verdict** — Stage 2 Qwen primary config **LOCKED at `thinking=off, max_tokens=16000`** (27.6s avg latency, cheapest 5/5-safe config at ≥16K ceiling).
- Task 1.5 **fully CLOSED** — Phase 1 (artifacts-absent verification) + Phase 2 (Option 4 partial-adapter ratified) + Phase 3 (hive-mind commit `c363257` pushed to origin/master 2026-04-22 per PM ratification). ClaudeAdapter now covers project-docs + memories + design_chats streams; +7 tests (hive-mind suite 305 → 312), tsc clean.
- Stage 0 mechanism #3 (session-generated `/mnt/user-data/outputs/*` artifacts) remains **OPEN** as hive-mind BACKLOG P1. Sprint 11+ vendor-path item; not a Sprint-10 gate.

**Sprint 10 scorecard: 7 of 7 tasks CLOSED. Sprint 10 fully closed.**

---

*End of Sprint 10 close-out. Sprint 10 scope delivered. Handoff to PM for Sprint 11 kickoff decision.*