moving

docs/briefs/2026-04-20-cc-preflight-prep-tasks.md

@@ -0,0 +1,184 @@
+# CC Brief — Preflight Prep Mini-Sprint
+
+**Datum:** 2026-04-20 PM (post-Sprint-7 push)
+**Autor:** PM (Claude, za Marka → CC)
+**Scope:** four tasks to scaffold preflight gate Stage 0 → Stage 1 → Stage 2 execution. No preflight run in this sprint — this is code + data + test scaffolding only. Zero API spend.
+**Exit target:** sprint-8-exit ping file committed to `PM-Waggle-OS/sessions/2026-04-XX-sprint-8-exit.md` with pass/fail + commit SHAs per task.
+
+---
+
+## Context
+
+Sprint 7 (7 tasks, 7 commits, 14-file turnId propagation) pushed to origin/main 2026-04-20 PM. Four-cell harness scaffold, M-11 real embedder with fail-loud 503 contract, and H-AUDIT-1 code-backed traceability are live.
+
+Between sprint 7 close and now, PM locked **three new OQ resolution sets** (9 resolutions total). You have not seen them. Read them first — they introduce hard constraints that Tasks 1-4 must honor.
+
+## Read-first (sequential, in this order)
+
+1. `D:\Projects\PM-Waggle-OS\decisions\2026-04-20-preflight-oq-resolutions-locked.md` — Stage 2 sample structure (13/13/12/12), re-run policy (same sample, max 3 attempts, 3 formal exception types), budget amendment ($150 Block 4.3)
+2. `D:\Projects\PM-Waggle-OS\decisions\2026-04-20-verbose-fixed-oq-resolutions-locked.md` — verbose-fixed template language (English), version lock timing (after Week 2), unit test requirement (explicit, not runtime-only)
+3. `D:\Projects\PM-Waggle-OS\decisions\2026-04-20-failure-mode-oq-resolutions-locked.md` — F1-F5 MECE taxonomy, F3 bucket for mixed errors, calibration set n=10
+4. `D:\Projects\PM-Waggle-OS\strategy\2026-04-20-failure-mode-taxonomy.md` — full v1 spec with judge prompt §4 and rubric §5
+5. `D:\Projects\PM-Waggle-OS\strategy\2026-04-20-verbose-fixed-template.md` — 6-segment template, forbidden elements list, validation checklist
+6. `D:\Projects\PM-Waggle-OS\strategy\2026-04-20-preflight-gate-spec.md` — full preflight gate operational spec
+
+If anything contradicts this brief, the LOCKED decision files win. Flag the contradiction in the exit ping.
+
+---
+
+## Task 1 — Stage 2 sample lock file
+
+**Output:** `benchmarks/data/preflight-locomo-50.json` (or `packages/server/benchmarks/data/preflight-locomo-50.json` — pick whichever matches harness conventions from Sprint 7).
+
+**Source:** LoCoMo public benchmark dataset (Zhang et al. 2024, HuggingFace `snap-stanford/locomo` or equivalent canonical source).
+
+**Composition:** 50 instances, distribution **13 single-hop / 13 multi-hop / 12 temporal / 12 open-ended**. No deviation from these counts.
+
+**Selection:** deterministic stratified sample with `seed=42`. Document the selection algorithm in a comment header of the JSON file (e.g., "sorted by instance ID ascending within category, seed=42 stable selection of first N per category after Fisher-Yates shuffle").
+
+**Schema per instance:**
+```json
+{
+  "id": "<locomo_instance_id>",
+  "category": "single-hop" | "multi-hop" | "temporal" | "open-ended",
+  "context": "<ground-truth supporting conversation/excerpt shown to model>",
+  "question": "<question text>",
+  "ground_truth_answer": "<canonical answer>",
+  "locomo_metadata": { ... original LoCoMo fields preserved ... }
+}
+```
+
+**Acceptance:**
+- File committed to repo
+- Harness runtime assertion added (where `runner.ts` loads sample): if category distribution ≠ 13/13/12/12, throw with explicit error message `Pre-flight sample distribution mismatch: expected 13/13/12/12, got {actual}`
+- Smoke test in `benchmarks/harness/tests/smoke.test.ts` (extend existing, don't add new file) asserts the distribution
+
+**Reference:** `decisions/2026-04-20-preflight-oq-resolutions-locked.md` §OQ-PF-1
+
+---
+
+## Task 2 — Failure mode calibration set
+
+**Output:** `benchmarks/data/failure-mode-calibration-10.jsonl` (same directory as Task 1 sample, JSONL format — one instance per line).
+
+**Composition:** 10 LoCoMo instances **non-overlapping** with preflight-locomo-50.json (different instance IDs, same source dataset). Category mix: **3 single-hop / 3 multi-hop / 2 temporal / 2 open-ended**.
+
+**Selection:** deterministic with `seed=43` (different from Task 1 to ensure non-overlap); assert no ID overlap with Task 1 output.
+
+**Schema per line:**
+```json
+{"id": "<id>", "category": "<cat>", "context": "<ctx>", "question": "<q>", "ground_truth_answer": "<a>", "human_label": {"verdict": null, "failure_mode": null, "rationale": null}}
+```
+
+**Leave `human_label` fields null.** PM will fill them in a labeling pass.
+
+**Acceptance:**
+- File committed
+- Assertion in smoke test: no ID overlap with Task 1 file; category distribution matches 3/3/2/2
+
+**Reference:** `decisions/2026-04-20-failure-mode-oq-resolutions-locked.md` §OQ-FM-3
+
+---
+
+## Task 3 — Verbose-fixed cell isolation unit test
+
+**Output:** `packages/server/tests/benchmarks/verbose-fixed-cell-isolation.test.ts` (or nearest equivalent path matching existing vitest conventions).
+
+**Minimum 3 test cases:**
+
+1. `'verbose-fixed cell invokes zero retrieval calls'` — mock the retrieval stack (combined-retrieval, memory adapter), activate the verbose-fixed cell through the harness cell function, assert `retriever.search` call count === 0.
+
+2. `'verbose-fixed cell invokes zero wiki compiler calls'` — mock `wiki.compile`, activate verbose-fixed cell, assert call count === 0.
+
+3. `'verbose-fixed cell invokes zero memory read calls'` — mock the memory reader, activate verbose-fixed cell, assert call count === 0.
+
+**Framework:** vitest (match Sprint 7 convention from `turn-context.test.ts`).
+
+**Acceptance:**
+- All 3 tests green in CI
+- Any future PR breaking cell isolation triggers test failure
+- Test included in `vitest.config.ts` default test run (no special flag needed)
+
+**Reference:** `decisions/2026-04-20-verbose-fixed-oq-resolutions-locked.md` §OQ-VF-3
+
+---
+
+## Task 4 — Failure mode judge module (scaffold, not wired)
+
+**Output:** `packages/server/benchmarks/judge/failure-mode-judge.ts` (or nearest equivalent in harness layout).
+
+**Content:** pure TypeScript module exporting:
+
+```typescript
+export interface JudgeResult {
+  verdict: "correct" | "incorrect";
+  failure_mode: null | "F1" | "F2" | "F3" | "F4" | "F5";
+  rationale: string;
+  judge_model: string;
+}
+
+export async function judgeAnswer(params: {
+  question: string;
+  groundTruth: string;
+  contextExcerpt: string;
+  modelAnswer: string;
+  judgeModel: string;          // e.g. "claude-sonnet-4-6"
+  llmClient: LlmClient;
+}): Promise<JudgeResult>
+
+export async function judgeEnsemble(params: {
+  question: string;
+  groundTruth: string;
+  contextExcerpt: string;
+  modelAnswer: string;
+  judgeModels: string[];       // typically 4: Sonnet, Haiku, GPT-5, Gemini-Pro
+  llmClients: Map<string, LlmClient>;
+}): Promise<{
+  ensemble: JudgeResult[];
+  majority: JudgeResult;
+  fleissKappa: number;
+}>
+
+export function computeFleissKappa(ratings: JudgeResult[][]): number
+```
+
+**Judge prompt:** EXACT text from `strategy/2026-04-20-failure-mode-taxonomy.md` §4, interpolated with `{{question}}`, `{{ground_truth}}`, `{{context_excerpt}}`, `{{model_answer}}`. Do not modify the prompt.
+
+**JSON parsing:** strict schema validation via Zod. If parse fails, retry once with a reminder `"Your previous response was not valid JSON. Return only the JSON object, no prose."`. If second retry fails, throw `JudgeParseError` — do not silently default to incorrect.
+
+**Unit tests in same folder (`failure-mode-judge.test.ts`):**
+- Valid JSON parse with all 5 failure modes (5 fixture cases)
+- Invalid JSON triggers retry, retry success returns correct result
+- Invalid JSON on retry throws `JudgeParseError`
+- 4-judge ensemble majority computation (2-2 tie broken by Sonnet, 3-1 majority wins, 4-0 unanimous)
+- Fleiss' kappa computed correctly on a hand-crafted 4×10 ratings matrix (use a known test case from statistical literature; target value within 0.01 tolerance)
+
+**NOT in scope this sprint:** wiring judge into `runner.ts`, adding per-instance judge call to the JSONL output flow, calling judge during harness execution. That is Sprint 9.
+
+**Reference:** `strategy/2026-04-20-failure-mode-taxonomy.md` §4 (prompt), §6 (ensemble protocol), §8 (kappa thresholds)
+
+---
+
+## Exit gate
+
+Ping PM via `PM-Waggle-OS/sessions/2026-04-XX-sprint-8-exit.md` (ISO date of completion) with:
+
+- [ ] Task 1 sample file committed, SHA, smoke test assertion green
+- [ ] Task 2 calibration file committed (empty human_label), SHA, distribution assertion green
+- [ ] Task 3 verbose-fixed cell isolation tests — all 3 green, file path, SHA
+- [ ] Task 4 judge module + unit tests — all green, file paths, SHA
+- [ ] `tsc --noEmit` clean across all packages
+- [ ] Total test count before/after (e.g., 4902 → 4910)
+- [ ] Zero regression (re-run full suite, confirm 4901 pre-existing still pass)
+- [ ] Zero API spend (confirm no LlmClient calls made outside unit-test mocks)
+
+If any task blocks on a LOCKED decision ambiguity, stop and write a clarification-request ping to PM instead of improvising. LOCKED decisions are source of truth; the brief is summary.
+
+## Not in scope (explicit exclusions)
+
+- Stage 0 Dogfood execution (Marko's personal AI exports harvest + 3-question test)
+- Stage 1 mikro-eval run (12 tasks × 3 arms)
+- Stage 2 preflight 4-cell run on preflight-locomo-50.json
+- Judge wiring into runner.ts JSONL output
+- Real embedder key provisioning (that's operational, not engineering)
+- Week 1 Qwen3 35B-A3B × LoCoMo main run