moving

2026-09-02 10:10:29 +02:00
commit 0c3e2ead3b
3841 changed files with 970576 additions and 0 deletions
--- a/docs/reports/multi-vendor-ensemble-baseline-2026-04-21T08-56-43Z.md
+++ b/docs/reports/multi-vendor-ensemble-baseline-2026-04-21T08-56-43Z.md
@@ -0,0 +1,146 @@
+# Multi-Vendor Ensemble Baseline — Sprint 10 Task 2.1
+
+**Generated:** 2026-04-21T09:02:49.039Z
+**Calibration artifact:** `preflight-results/judge-calibration-ensemble-2026-04-21T08-56-43Z.json`
+**Labels source:** `D:/Projects/PM-Waggle-OS/calibration/2026-04-20-failure-mode-calibration-labels.md`
+**Ensemble vendors:** claude-opus-4-7, gpt-5.4, gemini-3.1-pro
+**Instances:** 10
+
+---
+
+## 1. Per-vendor match rate vs PM ground truth
+
+| Vendor | Match rate | Spend | Avg latency | Disagreements |
+|---|---|---|---|---|
+| `claude-opus-4-7` | 9/10 | $0.031446 | 2577ms | 1 |
+| `gpt-5.4` | 9/10 | $0.021648 | 2891ms | 1 |
+| `gemini-3.1-pro` | 8/10 | $0.048090 | 7376ms | 2 |
+
+**Ensemble majority match rate:** 9/10
+**Total ensemble spend:** $0.101184 (30 calls across 3 vendors × 10 instances)
+
+---
+
+## 2. Pair-wise Cohen's kappa (inter-rater, vendors only)
+
+| Pair | Kappa | Band | Agree% |
+|---|---|---|---|
+| `claude-opus-4-7` ↔ `gpt-5.4` | 0.7531 | substantial | 80.0% |
+| `claude-opus-4-7` ↔ `gemini-3.1-pro` | 0.6250 | substantial | 70.0% |
+| `gpt-5.4` ↔ `gemini-3.1-pro` | 0.8718 | strong | 90.0% |
+
+### Dataset-wide Fleiss' kappa
+
+- **Vendors only (3 raters):** κ = **0.7458** → substantial
+- **Vendors + PM (4 raters):** κ = **0.7897** → substantial
+
+### Interpretation band (brief §2.2)
+
+| κ range | Band | Stage 2 implication |
+|---|---|---|
+| ≥ 0.80 | strong | ensemble ready; ensemble verdict primary |
+| 0.60 — 0.80 | substantial | ensemble ready + tie-breaker policy documented |
+| 0.40 — 0.60 | moderate | **PM review required before Stage 2 kickoff** |
+| < 0.40 | fair or worse | **go/no-go review**; scope pivot to single-judge Opus + rubric refinement |
+
+---
+
+## 3. Disagreement log (vendor vs PM)
+
+### `claude-opus-4-7` — 1 disagreement(s)
+
+**Instance 7** (locomo_conv-42_q038)
+- PM: `incorrect/F3`
+- Vendor: `correct/null`
+- Rationale: 7 September 2022 is the Friday before 14 September 2022, matching the ground truth.
+
+### `gpt-5.4` — 1 disagreement(s)
+
+**Instance 9** (locomo_conv-50_q037)
+- PM: `correct/null`
+- Vendor: `incorrect/F4`
+- Rationale: The yes/no answer matches the ground truth, but it adds unsupported claims such as touring with Frank Ocean and feeling alive on stage in Tokyo, which are not present in the provided context.
+
+### `gemini-3.1-pro` — 2 disagreement(s)
+
+**Instance 6** (locomo_conv-41_q036)
+- PM: `incorrect/F5`
+- Vendor: `incorrect/F4`
+- Rationale: The model fails to mention the music events John attended and instead hallucinates activities like walks and picnics that are not present in the ground-truth context.
+
+**Instance 9** (locomo_conv-50_q037)
+- PM: `correct/null`
+- Vendor: `incorrect/F4`
+- Rationale: The model includes a fabricated detail about touring with Frank Ocean, which does not appear in the ground-truth context.
+
+---
+
+## 4. Notes for Stage 2 primary-judge selection
+
+Sprint 10 brief §1.3 + §2.1 decision tree:
+- Task 1.3 Sonnet calibration produced 8/10 match → borderline 7-8 band → **multi-vendor kappa required before Stage 2 primary lock**.
+- Task 2.1 ensemble majority produced 9/10 match → near-unanimous with PM on the current 10-instance dataset.
+- Fleiss' κ (vendors only) = 0.746 → **substantial** band.
+
+**Task 2.2 scope:** brief §2.2 calls for 15 triples (10 Sprint-9 + 5 new PM-authored) to extend this baseline. The 10-instance result above is INDICATIVE, not final — full band assessment requires the additional 5 triples to avoid small-sample bias.
+
+**Open signal (flagged to PM):**
+- Instance 9 (`locomo_conv-50_q037`) — PM labeled `correct/null`; Haiku (Sprint 9 Task 4), Sonnet (Sprint 10 Task 1.3), and 2-of-3 ensemble vendors (GPT-5.4 + Gemini 3.1 Pro) flag `incorrect/F4` (fabrication: "touring with Frank Ocean" + "Tokyo stage").
+- Sprint 9 Task 4 Opus 4.7 solo agreed with PM. Today's ensemble Opus 4.7 also agrees with PM.
+- Signal: one PM label may warrant re-review. Not a judge weakness; a consistent-across-3-vendor-families disagreement on a specific instance.
+
+---
+
+## 5. Tie-breaker policy (brief §2.2 substantial-band requirement)
+
+The substantial band (`0.60 ≤ κ < 0.80`) requires a documented tie-breaker policy per brief §2.2. This section specifies what the current code does, and the recommended policy for Stage 2 full-run use.
+
+### 5.1 Current behaviour (`failure-mode-judge.ts::computeMajority`)
+
+1. Verdict + failure_mode are joined into a single key per vendor judgment.
+2. Vendor judgments are tallied.
+3. The key with the highest count wins outright if it has a strict plurality.
+4. **On any tie, the first model in the `judgeModels` list wins** (the "first-in-list" tie-breaker).
+
+In this Task 2.1 run the invocation order was `claude-opus-4-7, gpt-5.4, gemini-3.1-pro` → Opus 4.7 acts as tie-breaker on any 1-1-1 disagreement. This did not fire on the 10-instance dataset (the single mismatch with PM on instance 9 was a 2-of-3 plurality for `incorrect/F4`, not a tie).
+
+### 5.2 Implications of first-in-list tie-breaker on a multi-vendor ensemble
+
+The whole point of the tri-vendor ensemble is to neutralize same-family bias (brief §2 rejection rationale for Claude-only trio). If 1-1-1 ties are resolved by an Anthropic-family judge, Claude bias leaks back in via the tie-break channel. Fleiss' κ = 0.75 on this dataset means 1-1-1 ties are expected at low frequency but will occur over Stage 2's ~200 instances per cell.
+
+### 5.3 Recommended Stage 2 policy (pending PM ratification)
+
+For Stage 2 full-run use, **escalate 1-1-1 three-way disagreements to PM human review** instead of resolving via first-in-list. Rationale:
+
+- Rare event by construction (κ ≥ 0.60 substantial agreement + 3-way distinct-label cases are ~5-10% of all judgments at this band).
+- Preserves the "no Anthropic-family tie-breaker" defensibility claim for the launch narrative.
+- Stage 2 scope is 200 instances × 4 cells = 800 judgments; 5-10% = ~40-80 PM-review escalations over the run — tractable manual pass.
+- Non-Stage-2 runs (Week-1 smaller batches) can keep first-in-list for throughput; the escalation path is reserved for the launch-defensibility deliverable.
+
+Implementation sketch: extend `JudgeConfig` with an optional `tieBreakerPolicy: 'first-in-list' | 'escalate-to-pm'` (default `first-in-list` to preserve current behavior). When `escalate-to-pm`, return a `majority` with `verdict: 'tie_unresolved'` + full per-vendor rationale; the aggregator adds a counter + emits a flag in the markdown report for operator attention.
+
+**Not implementing in Sprint 10** — this is brief-§2.2 documentation-only. Implementation slot (if PM ratifies policy) opens in Sprint 11 or a bolt-on PR before Stage 2 kickoff.
+
+### 5.4 Joint-label tie on unanimous verdict
+
+A sub-class of ties to note: all three vendors agree on `verdict` (e.g., all `incorrect`) but disagree on `failure_mode` (e.g., F3/F4/F5 one-each). Current code treats these as full ties under the joint key; the first-in-list rule applies.
+
+For Stage-2 LoCoMo, the binary-correct-or-not summary statistic is what feeds the NEW_SOTA / SOTA_IN_LOCAL_FIRST / GO_NOGO_REVIEW banner (brief §5). Failure-mode distribution is a diagnostic rollup, not a pass/fail gate. On a unanimous-verdict / mixed-failure-mode case, the verdict is unambiguous for banner computation regardless of which failure_mode wins the tie. Safe to ignore this sub-class for Stage-2-banner purposes.
+
+---
+
+## 6. Task 2.1 CLOSED — acceptance check
+
+Per brief §2.1:
+
+| Acceptance criterion | Status |
+|---|---|
+| Sva tri vendora vraćaju parsable verdict sa istog prompt shape-a. | ✅ PASS — 30/30 parses |
+| Per-vendor match rate zabeležen. | ✅ §1 |
+| Ensemble Fleiss' kappa izračunat na 10 triples minimum pre pravog Stage 2 run-a. | ✅ §2 — κ=0.7458 (substantial) |
+| Baseline markdown report written. | ✅ this document |
+| Spend ≤ $5. | ✅ $0.101 of $5 (2.0%) |
+
+---
+
+*End of Task 2.1 baseline report. Task 2.2 (full 15-triple Fleiss' kappa) opens next after PM authors the 5 additional ground-truth triples.*
--- a/docs/reports/opus-4-6-route-audit-2026-04-22.md
+++ b/docs/reports/opus-4-6-route-audit-2026-04-22.md
@@ -0,0 +1,216 @@
+# Opus 4.6 / Sonnet 4.6 Route Audit
+
+**Datum:** 2026-04-22
+**Sprint:** 11 · Track B · Task B3
+**Authority:** `briefs/2026-04-22-cc-sprint-11-kickoff.md` §3 Track B B3
+**Author:** CC-1
+**Scope:** `packages/server/**` + `packages/cli/**` — grep for `claude-opus|claude-sonnet-4` references; classify; recommend naming LOCK.
+**Budget:** $0 (read-only audit)
+
+---
+
+## 0. Executive summary
+
+Total hits: **54** in `packages/server`; **0** in `packages/cli`.
+
+Classification:
+
+| Class | Count | Notes |
+|---|---|---|
+| (a) dated snapshot | 11 | **9 of 11 use `-20250514` suffix which was never valid for the Claude 4.6 family** per litellm-config.yaml line 2-7 Sprint 10 Task 1.2 comment. |
+| (b) floating alias | 39 | `claude-sonnet-4-6` / `claude-opus-4-6` / `claude-opus-4-7` / `claude-haiku-4-5`. Mostly correct; match litellm-config.yaml canonical entries. |
+| (c) provider-prefixed | 4 | `anthropic/claude-sonnet-4.6` (dot-notation) + `anthropic/claude-opus-4.6`. Dot-notation normalizes to dash-form via existing `mapModel` logic. |
+
+**Top 3 cleanup priorities** (elaborated §4):
+
+1. **`anthropic-proxy.ts:43-44`** — hardcoded mapping `claude-sonnet-4-6 → claude-sonnet-4-20250514` + `claude-opus-4-6 → claude-opus-4-20250514` sends **provably invalid** snapshot IDs to the Anthropic Messages API. Breaks chat completion when the proxy is the request path. **HIGH / user-facing runtime defect.**
+2. **`workspace-templates.ts:406`** — new workspaces created via template wizard default to `claude-sonnet-4-20250514` (same invalid ID class). New users bounce on first message. **MEDIUM / onboarding path.**
+3. **Test-fixture drift** — 7 test files use `claude-sonnet-4-20250514`. Tests pass because they mock providers, but they codify the invalid ID as canonical. **LOW / hygiene, no runtime impact today.**
+
+---
+
+## 1. Canonical state of truth (per `litellm-config.yaml` Sprint 10 Task 1.2)
+
+| Family | Plain alias (current ID) | Correct dated snapshot |
+|---|---|---|
+| Sonnet 4.6 | `claude-sonnet-4-6` | none — *per docs, plain alias IS the current ID; `-20250514` was never valid for the 4.6 family* |
+| Opus 4.6 | `claude-opus-4-6` | `claude-opus-4-6-20250610` |
+| Opus 4.7 | `claude-opus-4-7` | `claude-opus-4-7-20260201` (per model IDs reference) |
+| Haiku 4.5 | `claude-haiku-4-5` | `claude-haiku-4-5-20251001` |
+
+Source: `litellm-config.yaml` lines 9-39 + Sprint 10 Task 1.2 verification note (2026-04-21 docs probe).
+
+---
+
+## 2. Full hit table (non-test code)
+
+### 2.1 Runtime request-path references
+
+| File | Line | Reference | Class | Current behavior | Recommendation |
+|---|---|---|---|---|---|
+| `packages/server/src/local/routes/anthropic-proxy.ts` | 43 | `'claude-sonnet-4-6': 'claude-sonnet-4-20250514'` | (a) dated (**INVALID**) | Maps floating alias to a non-existent dated snapshot. Proxy sends the invalid ID to Anthropic; response is `404 model_not_found`. | **REMOVE entry.** The floating alias `claude-sonnet-4-6` IS the current Anthropic ID; no mapping needed. |
+| `packages/server/src/local/routes/anthropic-proxy.ts` | 44 | `'claude-opus-4-6': 'claude-opus-4-20250514'` | (a) dated (**INVALID**) | Same class. `-20250514` was never a valid Opus 4.6 snapshot. | **REPLACE with `claude-opus-4-6-20250610`** (canonical dated per litellm-config) OR **REMOVE** to let the floating alias pass through unchanged. |
+| `packages/server/src/local/routes/anthropic-proxy.ts` | 45 | `'claude-haiku-4-5': 'claude-haiku-4-5-20251001'` | (a) dated | Correct — `-20251001` is the canonical Haiku 4.5 snapshot per litellm-config. | **KEEP.** |
+| `packages/server/src/local/routes/anthropic-proxy.ts` | 46 | `'claude-haiku-4-5-20251001': 'claude-haiku-4-5-20251001'` | (a) dated | Identity mapping — stable. | **KEEP.** |
+| `packages/server/src/local/routes/anthropic-proxy.ts` | 48-50 | `'claude-haiku-4-6'`, `'claude-haiku-4.6'`, `'claude-haiku-4.5'` → `claude-haiku-4-5-20251001` | (b)/(c) misnames | Defensive aliases for common typos. | **KEEP.** |
+| `packages/server/src/local/routes/workspace-templates.ts` | 406 | `let model = 'claude-sonnet-4-20250514'` | (a) dated (**INVALID**) | Default value when `settings.json` has no `model` key. Affects new workspaces created via template wizard. | **REPLACE with `'claude-sonnet-4-6'`** (floating alias matches litellm-config canonical). |
+| `packages/server/src/local/routes/chat.ts` | 449 | `model ?? wsModelConfig ?? pilotConfig.getDefaultModel() ?? 'claude-sonnet-4-6'` | (b) floating | Final fallback when all other model sources are null. | **KEEP** — canonical floating alias. |
+| `packages/server/src/local/routes/chat.ts` | 1153 | `// Extract provider name from model ID (e.g., "anthropic" from "claude-sonnet-4-6")` | (b) floating | Comment only, no runtime effect. | **KEEP.** |
+| `packages/server/src/local/routes/personas.ts` | 51 | `modelPreference: body.modelPreference ?? 'claude-sonnet-4-6'` | (b) floating | Default model preference for new personas. | **KEEP.** |
+| `packages/server/src/local/routes/personas.ts` | 130 | `model: 'claude-sonnet-4-6'` | (b) floating | Persona response default. | **KEEP.** |
+| `packages/server/src/local/routes/providers.ts` | 46 | `{ id: 'claude-opus-4-7', name: 'Claude Opus 4.7', cost: '$$$', speed: 'slow' }` | (b) floating | UI provider catalog entry. | **KEEP.** |
+| `packages/server/src/local/routes/providers.ts` | 47 | `{ id: 'claude-opus-4-6', name: 'Claude Opus 4.6', cost: '$$$', speed: 'slow' }` | (b) floating | UI provider catalog entry. | **KEEP.** |
+| `packages/server/src/local/routes/providers.ts` | 48 | `{ id: 'claude-sonnet-4-6', name: 'Claude Sonnet 4.6', cost: '$$', speed: 'medium' }` | (b) floating | UI provider catalog entry. | **KEEP.** |
+| `packages/server/src/local/routes/workspaces.ts` | 163 | `` `Examples: claude-sonnet-4-6, gpt-4o, gemini-2.0-flash` `` | (b) floating | Error message example text. | **KEEP.** |
+| `packages/server/src/local/routes/workspaces.ts` | 611 | same | (b) floating | Same. | **KEEP.** |
+| `packages/server/src/local/routes/litellm.ts` | 64 | `{ model: 'claude-sonnet-4-6', inputPer1k: 0.003, outputPer1k: 0.015, provider: 'anthropic' }` | (b) floating | Pricing catalog entry. | **KEEP.** |
+| `packages/server/src/local/routes/litellm.ts` | 65 | `{ model: 'claude-haiku-4-6', inputPer1k: 0.0008, outputPer1k: 0.004, provider: 'anthropic' }` | (b) floating — **misname** | Haiku 4.6 does not exist per Anthropic docs; current latest is Haiku 4.5. Pricing entry would never match a real request. | **REPLACE id with `'claude-haiku-4-5'`** or **REMOVE** the entry. |
+| `packages/server/src/local/routes/litellm.ts` | 66 | `{ model: 'claude-opus-4-6', inputPer1k: 0.015, outputPer1k: 0.075, provider: 'anthropic' }` | (b) floating | Pricing catalog entry. | **KEEP.** |
+| `packages/server/src/local/index.ts` | 726 | `defaultModel: 'claude-sonnet-4-6'` | (b) floating | Server bootstrap default. | **KEEP.** |
+| `packages/server/src/local/index.ts` | 781 | `defaultModel: 'claude-sonnet-4-6'` | (b) floating | Same. | **KEEP.** |
+| `packages/server/src/local/index.ts` | 897 | `const currentModel = 'claude-sonnet-4-6'` | (b) floating | Agent state default. | **KEEP.** |
+| `packages/server/src/local/index.ts` | 990 | `defaultModel: 'claude-sonnet-4-6'` | (b) floating | Local mode boot default. | **KEEP.** |
+| `packages/server/src/local/index.ts` | 1009 | `defaultModel: 'claude-sonnet-4-6'` | (b) floating | Same. | **KEEP.** |
+| `packages/server/src/local/index.ts` | 1627 | `model: 'claude-sonnet-4-6'` | (b) floating | Agent-tool default. | **KEEP.** |
+| `packages/server/src/local/index.ts` | 1747 | `model: server.agentState.currentModel ?? 'claude-sonnet-4-6'` | (b) floating | State fallback. | **KEEP.** |
+
+### 2.2 Test-fixture references
+
+| File | Ref sample | Class | Notes | Recommendation |
+|---|---|---|---|---|
+| `packages/server/tests/backup-restore.test.ts:47` | `claude-sonnet-4-6` | (b) | Config fixture. | **KEEP.** |
+| `packages/server/tests/backup-streaming.test.ts:41` | `claude-sonnet-4-6` | (b) | Same. | **KEEP.** |
+| `packages/server/tests/benchmarks/failure-mode-judge.test.ts:121, 126, 142, 208, 256` | `claude-sonnet-4-6` | (b) | Judge fixtures (aligns with Sprint 10 Task 2.2 ratified primary judge). | **KEEP.** |
+| `packages/server/tests/data-export.test.ts:108, 109, 114` | `claude-sonnet-4-6` | (b) | Secret + model export fixture. | **KEEP.** |
+| `packages/server/tests/benchmarks/aggregate.test.ts:58` | `claude-sonnet-4-6` | (b) | Judge aggregate fixture. | **KEEP.** |
+| `packages/server/tests/local-mode.test.ts:291, 295, 300` | `claude-opus-4-6` | (b) | Opus fixture. | **KEEP.** |
+| `packages/server/tests/local/cost.test.ts:89, 90` | `claude-sonnet-4-6` | (b) | Cost tracker fixture. | **KEEP.** |
+| `packages/server/tests/local/team-integration.test.ts:31` | `claude-sonnet-4-6` | (b) | Team defaultModel. | **KEEP.** |
+| `packages/server/tests/local/providers.test.ts:255` | `claude-sonnet-4-6` | (b) | Workspace model. | **KEEP.** |
+| `packages/server/tests/local/providers.test.ts:262` | `anthropic/claude-sonnet-4.6` | (c) | Provider-prefixed dot-notation — exercises `mapModel` normalization. | **KEEP.** (This is a legitimate test of the normalizer.) |
+| `packages/server/tests/litellm-api.test.ts:162, 174` | `claude-sonnet-4-20250514` | (a) **INVALID** | Hardcodes the non-existent dated snapshot. Test passes because provider is mocked, but it codifies the bad ID as canonical. | **REPLACE with `claude-sonnet-4-6`** unless a test-specific reason forces the invalid literal (there isn't one visible). |
+| `packages/server/tests/local/anthropic-proxy.test.ts:91, 114, 124, 149, 167, 177, 204, 233, 243` | mix of `claude-sonnet-4-6` and `claude-sonnet-4-20250514` | (a)/(b) | Proxy-translation tests. Some lines assert the input → output mapping, so the invalid ID may appear as the *expected output* of the current buggy mapping. | **DEPENDS on priority #1** — once the proxy mapping is fixed, these test expectations update accordingly. |
+| `packages/server/tests/routes/agents.test.ts:86, 97` | `claude-sonnet-4-20250514` | (a) **INVALID** | Agent creation fixture. Same class as #1 test-drift. | **REPLACE with `claude-sonnet-4-6`.** |
+
+---
+
+## 3. Class (c) provider-prefixed references
+
+| File | Line | Reference | Status |
+|---|---|---|---|
+| `packages/server/tests/local/providers.test.ts` | 262 | `anthropic/claude-sonnet-4.6` | Legitimate — exercises `mapModel()` dot-to-dash normalizer. |
+| `litellm-config.yaml` | 31 | `anthropic/claude-sonnet-4-6` | LiteLLM route alias — fine. |
+| `litellm-config.yaml` | 36 | `anthropic/claude-opus-4.6` | LiteLLM dot-notation alias — fine (same normalizer handles it). |
+
+Recommendation: LOCK naming convention that **both** dash-form (`anthropic/claude-sonnet-4-6`) and dot-form (`anthropic/claude-sonnet-4.6`) are accepted input, with dash-form as the canonical internal representation. The existing `mapModel()` normalizer in `anthropic-proxy.ts:38-40` already implements this. Document the contract in `docs/BENCHMARK-INFRASTRUCTURE.md` or similar canonical reference.
+
+---
+
+## 4. Top 3 cleanup priorities
+
+### Priority 1 — `anthropic-proxy.ts:43-44` invalid dated snapshot mapping (HIGH)
+
+```typescript
+// packages/server/src/local/routes/anthropic-proxy.ts:43-44  (CURRENT — BUGGED)
+const mapping: Record<string, string> = {
+  'claude-sonnet-4-6': 'claude-sonnet-4-20250514',   // ← invalid snapshot
+  'claude-opus-4-6': 'claude-opus-4-20250514',       // ← invalid snapshot
+  ...
+};
+```
+
+**Evidence it's invalid:** `litellm-config.yaml` lines 2-7 Sprint 10 Task 1.2 comment (Marko's own note, 2026-04-21): *"`-20250514` was never a valid Claude API ID for the 4.6 family."* The Anthropic docs list plain `claude-sonnet-4-6` as the current ID.
+
+**Impact:** Any chat/completion request that goes through the local Anthropic proxy (rather than LiteLLM) will fail with `404 model_not_found` from Anthropic. The failure path is user-visible: assistant turns return error, not content.
+
+**Fix:**
+
+```typescript
+const mapping: Record<string, string> = {
+  // Sonnet 4.6 canonical ID — plain alias per Anthropic docs 2026-04-21. No dated snapshot maps are published for the 4.6 family.
+  'claude-sonnet-4-6': 'claude-sonnet-4-6',
+  // Opus 4.6 canonical dated snapshot. Plain alias also works — LiteLLM/Anthropic resolves.
+  'claude-opus-4-6': 'claude-opus-4-6-20250610',
+  // Haiku 4.5 — existing correct entries stay.
+  'claude-haiku-4-5': 'claude-haiku-4-5-20251001',
+  'claude-haiku-4-5-20251001': 'claude-haiku-4-5-20251001',
+  // Defensive misnames — keep.
+  'claude-haiku-4-6': 'claude-haiku-4-5-20251001',
+  'claude-haiku-4.6': 'claude-haiku-4-5-20251001',
+  'claude-haiku-4.5': 'claude-haiku-4-5-20251001',
+  // NEW: add Opus 4.7 (shipped per Sprint 10 Task 1.2 litellm-config.yaml entries).
+  'claude-opus-4-7': 'claude-opus-4-7',
+};
+```
+
+Plus update `packages/server/tests/local/anthropic-proxy.test.ts` to assert the corrected expected outputs.
+
+### Priority 2 — `workspace-templates.ts:406` invalid default (MEDIUM)
+
+```typescript
+// packages/server/src/local/routes/workspace-templates.ts:406  (CURRENT — BUGGED)
+let model = 'claude-sonnet-4-20250514';
+```
+
+**Impact:** First-run users who land on template-wizard workspace creation get a workspace with the invalid model ID burned in. Next message bounces.
+
+**Fix:** `let model = 'claude-sonnet-4-6';`
+
+### Priority 3 — Test-fixture drift (LOW)
+
+Files that hardcode the invalid `claude-sonnet-4-20250514`:
+
+- `packages/server/tests/litellm-api.test.ts:162, 174`
+- `packages/server/tests/routes/agents.test.ts:86, 97`
+- `packages/server/tests/local/anthropic-proxy.test.ts` — multiple lines (update contingent on Priority 1).
+
+**Fix:** Migrate to `claude-sonnet-4-6` floating alias unless a specific test exercises dated-snapshot handling.
+
+---
+
+## 5. Recommended naming LOCK for `docs/BENCHMARK-INFRASTRUCTURE.md` (or equivalent)
+
+**Canonical form for all application code and tests:**
+
+| Use case | Correct form | Rationale |
+|---|---|---|
+| Default model selection, fallback strings, UI catalog | **floating alias** — `claude-sonnet-4-6`, `claude-opus-4-6`, `claude-opus-4-7`, `claude-haiku-4-5` | Anthropic publishes the floating alias as the canonical ID. LiteLLM + Anthropic SDK resolve it to the current dated snapshot on-demand. |
+| Benchmark / judge / reproducibility contexts | **dated snapshot** — `claude-opus-4-6-20250610`, `claude-haiku-4-5-20251001`, `claude-opus-4-7-20260201` | Reproducible results demand a pinned version so a model rotation by Anthropic doesn't silently shift benchmark scores. |
+| Test fixtures (unless specifically testing date-pinning) | **floating alias** | Keeps tests from codifying provider-internal dated IDs that change. |
+
+**For the 4.6 family specifically:** no valid `-20250514` dated snapshot exists. Any occurrence of `claude-sonnet-4-20250514` in code is a bug.
+
+**Trigger-on-first-caller-trip observability:** LiteLLM's existing `drop_params: true` + 4xx response logging already surfaces model-not-found errors. Explicit alert on `404 model_not_found` in proxy response → structured log with `{provider: 'anthropic', model: X, caller: route}` makes the trip actionable. Not a Sprint 11 gate; proposed as a separate hardening ticket.
+
+---
+
+## 6. Deliverable summary
+
+- **Report:** this file, `docs/reports/opus-4-6-route-audit-2026-04-22.md`.
+- **Hit table:** §2 covers 54 references across 20 files.
+- **Top 3 cleanups:** §4. Priority 1 is a real user-facing runtime defect and should be the immediate follow-up ticket.
+- **Naming LOCK proposal:** §5, awaits PM ratification in `decisions/2026-04-22-model-route-naming-locked.md`.
+
+---
+
+## 7. Budget
+
+| Line | Value |
+|---|---|
+| B3 cap (brief) | $0.10 |
+| Actual spend | $0.00 |
+| % of cap | 0% |
+
+Pure read-only audit — no LLM calls.
+
+---
+
+## 8. Related
+
+- `briefs/2026-04-22-cc-sprint-11-kickoff.md` §3 Track B B3
+- `litellm-config.yaml` — canonical Sonnet/Opus/Haiku routing + Sprint 10 Task 1.2 note
+- `docs/plans/SPRINT-10-CLOSEOUT-2026-04-22.md` — Task 1.2 Sonnet route repair
+
+---
+
+**B3 audit CLOSED. Awaiting PM decision doc `decisions/2026-04-22-model-route-naming-locked.md` for naming LOCK + cleanup-ticket authorization.**
--- a/docs/reports/sonnet-calibration-2026-04-21T08-55-51Z.md
+++ b/docs/reports/sonnet-calibration-2026-04-21T08-55-51Z.md
@@ -0,0 +1,84 @@
+# Sonnet 4.6 Calibration — Sprint 10 Task 1.3
+
+**Generated:** 2026-04-21T08:55:51Z
+**Calibration artifact:** `preflight-results/judge-calibration-sonnet-2026-04-21T08-55-51Z.json`
+**Labels source:** `D:/Projects/PM-Waggle-OS/calibration/2026-04-20-failure-mode-calibration-labels.md`
+**Judge model:** `claude-sonnet-4-6` (post Task 1.2 route repair, merge commit `a09831e`)
+**Instances:** 10
+
+---
+
+## Result
+
+**Match rate: 8/10 — verdict: PASS** (judge-calibration rubric: ≥8 = PASS, 6-7 = PARTIAL, <6 = FAIL).
+
+**Spend:** $0.027 of $0.50 Task 1.3 budget (5.4%).
+
+### Per-instance detail
+
+| # | Instance | PM label | Sonnet label | Outcome |
+|---|---|---|---|---|
+| 1 | `locomo_conv-26_q109` | correct/null | correct/null | MATCH |
+| 2 | `locomo_conv-41_q123` | incorrect/F3 | incorrect/F3 | MATCH |
+| 3 | `locomo_conv-50_q141` | incorrect/F4 | incorrect/F3 | DIFF |
+| 4 | `locomo_conv-42_q030` | incorrect/F2 | incorrect/F2 | MATCH |
+| 5 | `locomo_conv-49_q015` | correct/null | correct/null | MATCH |
+| 6 | `locomo_conv-41_q036` | incorrect/F5 | incorrect/F5 | MATCH |
+| 7 | `locomo_conv-42_q038` | incorrect/F3 | incorrect/F3 | MATCH |
+| 8 | `locomo_conv-41_q053` | incorrect/F4 | incorrect/F4 | MATCH |
+| 9 | `locomo_conv-50_q037` | correct/null | incorrect/F4 | DIFF |
+| 10 | `locomo_conv-47_q017` | incorrect/F1 | incorrect/F1 | MATCH |
+
+---
+
+## Decision tree (brief §1.3)
+
+Brief §1.3 decision matrix on Sonnet match rate:
+
+| Band | Action |
+|---|---|
+| ≥ 9/10 | Sonnet becomes Stage 2 primary default |
+| **7-8/10** | **Trigger Task 2.2 Fleiss' kappa probe (multi-vendor LOCKED — not Claude-only)** |
+| < 7/10 | Stick with Opus, PM review gate |
+
+Task 1.3 landed at **8/10** → **triggers Task 2.2 multi-vendor kappa** path per brief. Sprint 10 Task 2.1 (multi-vendor ensemble baseline on same 10 triples) already ran as Day-2 Step 3 — see `docs/reports/multi-vendor-ensemble-baseline-2026-04-21T08-56-43Z.md`.
+
+---
+
+## Disagreement analysis
+
+### Instance 3 — `locomo_conv-50_q141` (single-hop)
+
+- **Question:** "Which city is featured in the photograph Dave showed Calvin?"
+- **PM:** `incorrect/F4` (hallucination / fabrication)
+- **Sonnet:** `incorrect/F3` (wrong content / entity)
+- Both verdicts agree on **incorrect**; only the failure_mode taxonomy differs. Sonnet read the model's "Chicago" as a wrong-entity substitution (F3). PM labeled it as fabrication (F4).
+- Subtle taxonomy judgment call — F3 vs F4 on a wrong-city answer is defensible either way depending on how strictly one reads F4 as "unsupported by any context" vs F3 as "identifiable wrong entity".
+- **Impact on Stage 2 banner:** zero — both map to `incorrect` in the binary correctness rollup. Failure-mode breakdown shifts one count from F4 to F3.
+
+### Instance 9 — `locomo_conv-50_q037` (open-ended)
+
+- **Question:** "Does Calvin love music tours?"
+- **PM:** `correct/null`
+- **Sonnet:** `incorrect/F4` — "The model introduces fabricated details not present in the ground-truth context, specifically 'touring with Frank Ocean' and 'felt alive on stage in Tokyo'."
+- **Historical pattern on this instance:**
+  - Sprint 9 Haiku (Task 4 diagnostic): `incorrect/F4` — flagged the Frank Ocean fabrication.
+  - Sprint 9 Opus 4.7 (Task 4 production): `correct/null` — accepted PM's label.
+  - Sprint 10 Sonnet (this run): `incorrect/F4` — flags the fabrication.
+  - Sprint 10 ensemble Opus 4.7: `correct/null` (unchanged).
+  - Sprint 10 ensemble GPT-5.4: `incorrect/F4`.
+  - Sprint 10 ensemble Gemini 3.1 Pro: `incorrect/F4`.
+- **Aggregate cross-vendor disagreement with PM on instance 9:** 4 of 5 non-Opus-4.7 judgments (Haiku + Sonnet + GPT-5.4 + Gemini) flag F4; only Opus 4.7 agrees with PM's `correct/null`.
+- **Signal:** consistent cross-family disagreement on a specific PM label, not a per-judge weakness. PM label may warrant a re-review — flagged to PM in the Task 2.1 baseline report §4.
+
+---
+
+## Conclusion
+
+- Task 1.2 Sonnet route repair verified in production: 10/10 non-404 successful completions.
+- Task 1.3 calibration at 8/10 PASS is within the brief-§1.3 borderline band, which correctly triggered the Task 2.1 ensemble path (ran Day-2 Step 3).
+- Sonnet is production-viable as a Stage-2 judge candidate but does NOT auto-elevate to Stage 2 primary per brief conditional. Final Stage 2 primary selection waits on Task 2.2 full 15-triple Fleiss' kappa.
+
+---
+
+*End of Task 1.3 report. See Task 2.1 baseline for ensemble analysis and Stage 2 recommendations.*