moving

docs/decisions/2026-04-26-pilot-decision-template.md

@@ -0,0 +1,231 @@
+# Pilot Decision Template — Go/No-Go for Full N=400 Multiplier Benchmark
+
+**Authored:** 2026-04-26 (pre-results, in advance of pilot completion)
+**Decision owner:** Marko (ratifies); PM (drafts)
+**Trigger:** CC-1 emits `pilot-summary.json` after agentic knowledge work pilot completes
+**Pilot ID:** `agentic-knowledge-work-pilot-2026-04-26`
+**Manifest anchor:** `pilot-2026-04-26-v1` (amendment SHA `3946d3e0`)
+
+**Why this template exists pre-results:** Pre-built decision branches force honest threshold adherence. When results arrive, PM doesn't draft a memo from scratch under emotional pressure to rationalize the outcome — PM populates the appropriate branch with verbatim numbers. This is the discipline `feedback_substrate_readiness_gate` and `Anti-pattern #4 reminder: thresholds do not shift post-hoc` from manifest v6 are enforcing.
+
+---
+
+## §1 — Pre-registered hypotheses (do NOT modify)
+
+From `cc1-brief.md` §2 + amendment §1:
+
+- **H2 — Opus multiplier**: Cell B trio mean > Cell A trio mean by ≥ 0.30 Likert points, on ≥ 2 of 3 tasks
+- **H3 — Qwen multiplier**: Cell D trio mean > Cell C trio mean by ≥ 0.30 Likert points, on ≥ 2 of 3 tasks
+- **H4 — Sovereignty bridge**: Cell D trio mean ≥ Cell A trio mean, on ≥ 2 of 3 tasks
+
+**Pilot binary verdict:**
+- **PASS** = H2 + H3 + H4 each show directional sign on ≥ 2 of 3 tasks AND no critical failures (no cell scoring < 2.0 on majority of judges)
+- **FAIL** = otherwise
+
+---
+
+## §2 — Decision template — Branch A: PILOT PASS
+
+If pilot summary shows PASS, PM populates this branch and submits to Marko for ratification.
+
+### Memo header
+
+**Subject:** PM-RATIFY — Full N=400 multiplier benchmark authorization (pilot PASSED)
+**Date:** [populate from pilot completion timestamp]
+**Author:** PM
+**Decision asks:** (1) Authorize full N=400 multiplier scope; (2) Confirm budget envelope; (3) Ratify model roster; (4) Lock manifest v7 anchor
+
+### Memo body
+
+**Pilot result summary:**
+- H2 directional pass: [X of 3 tasks]
+- H3 directional pass: [X of 3 tasks]
+- H4 directional pass: [X of 3 tasks]
+- Critical failures: [count]
+- Pilot verdict: **PASS**
+- Wall-clock: [actual hh:mm]
+- Total cost: $[actual]
+- HEAD SHA at execution: [commit hash]
+
+**Cell-by-cell deltas (per task, trio means):**
+
+| Task | A (Opus solo) | B (Opus + harness) | C (Qwen solo) | D (Qwen + harness) | H2 Δ (B-A) | H3 Δ (D-C) | H4 Δ (D-A) |
+|---|---|---|---|---|---|---|---|
+| Task 1 | [X.X] | [X.X] | [X.X] | [X.X] | [+/-X.X] | [+/-X.X] | [+/-X.X] |
+| Task 2 | [X.X] | [X.X] | [X.X] | [X.X] | [+/-X.X] | [+/-X.X] | [+/-X.X] |
+| Task 3 | [X.X] | [X.X] | [X.X] | [X.X] | [+/-X.X] | [+/-X.X] | [+/-X.X] |
+
+### Recommended full N=400 multiplier scope
+
+**Sample size:** N=400 instances per cell, drawn from real-world knowledge work corpus (TBD construction — synthetic-realistic per pilot pattern, scaled to 400 instances)
+
+**Cells (3 cells, not 4):**
+- Cell A: Opus 4.7 solo
+- Cell B: Opus 4.7 + memory + agent loop
+- Cell C: Qwen 3.6 35B-A3B + memory + agent loop
+
+(H4 sovereignty bridge claim is most-actionable comparison; Cell C-Qwen-solo redundant if pilot demonstrates Qwen + harness ≥ Opus solo. Confirm with Marko whether Cell D-Qwen-solo retained as control.)
+
+**Optional addition — Cell D: GPT-5.4 + memory + agent loop** for cross-vendor frontier comparison.
+
+**Models tested:**
+1. claude-opus-4-7 (frontier proprietary)
+2. qwen3.6-35b-a3b-via-openrouter (sovereign reference)
+3. (optional) gpt-5.4 (frontier proprietary, second vendor)
+
+**Judge ensemble:** Same trio-strict (Opus + GPT + MiniMax) per pilot. Re-calibrate κ on N=14 synthesis-task subset before full launch.
+
+**Budget envelope:**
+- Candidate model spend: $40-90 (Opus dominates, 400 × multi-step × Opus rate)
+- Trio judge spend: $30-50 (1200-1600 judge calls)
+- Buffer: $30
+- Total cap: **$150 hard cap, $130 halt**
+
+**Wall-clock target:** 36-48 hours runner time (parallel cell execution where possible)
+
+**Pre-registration manifest v7:**
+- Anchor commit at full launch (TBD)
+- Hypothesis statements identical to pilot (H2, H3, H4) — unchanged
+- Cost cap, halt thresholds locked
+- Sample size N=400, seed 42
+- Output schema same as pilot
+- F-mode taxonomy reused
+
+### Decision asks (Marko ratifies)
+
+1. **Authorize full N=400 multiplier benchmark** with scope above? (Y/N)
+2. **Cell D-Qwen-solo retained or descoped?** Pilot showed [X of 3 sovereignty bridges]; descoping reduces cost ~$30. Recommendation: [retain/descope based on pilot results]
+3. **Add GPT-5.4 cell or stay 3-cell?** Adds ~$30 cost, strengthens cross-vendor frontier claim for paper. Recommendation: [add/skip based on paper plans]
+4. **Manifest v7 anchor commit** — current HEAD or fresh commit before kick? Recommendation: [based on tree state]
+5. **Run timing** — kick off [today/tomorrow/post-arxiv] given concurrent landing copy + arxiv work?
+
+---
+
+## §3 — Decision template — Branch B: PILOT FAIL
+
+If pilot summary shows FAIL, PM populates this branch.
+
+### Memo header
+
+**Subject:** PM-HALT — Multiplier expansion deferred (pilot FAILED)
+**Date:** [populate from pilot completion timestamp]
+**Author:** PM
+**Decision asks:** (1) Confirm halt; (2) Choose next-action path; (3) Update launch narrative if material
+
+### Memo body
+
+**Pilot result summary:**
+- H2 directional pass: [X of 3 tasks] — required ≥ 2
+- H3 directional pass: [X of 3 tasks] — required ≥ 2
+- H4 directional pass: [X of 3 tasks] — required ≥ 2
+- Critical failures: [count]
+- Pilot verdict: **FAIL**
+
+**Failure mode classification (which hypothesis failed and why):**
+
+#### Sub-branch B.1 — H2 (Opus multiplier) failed
+**Implication:** Adding hive-mind + agent loop to a frontier model does NOT reliably lift performance on knowledge work. This contradicts PA V5 finding (April 2026, Opus 4.6 +5.2pp publishable on H1). Possible causes:
+- V1 retrieval quality is insufficient; agent loop pulls noisy chunks and degrades vs. full-context Opus baseline
+- Multi-step agent loop overhead exceeds value-add at 5-step ceiling
+- Knowledge work tasks do not benefit from memory in the same way memory-recall tasks do
+
+**Recommended response:** Pause multiplier expansion. Prioritize retrieval V2 work (5 directions identified in arxiv §5.3). Re-run pilot post-V2.
+
+#### Sub-branch B.2 — H3 (Qwen multiplier) failed
+**Implication:** Qwen 35B-A3B + harness does NOT lift Qwen performance reliably. Possible causes:
+- Qwen 35B-A3B context utilization is already strong; full-context cell is competitive baseline
+- Agent loop self-prompting confuses Qwen more than it helps
+- Knowledge work tasks require cognitive capability that base Qwen struggles with regardless of harness
+
+**Recommended response:** Run Qwen-only ablation isolating each loop component (retrieval-only, self-prompt-only, full harness). Identify which component degrades vs. helps.
+
+#### Sub-branch B.3 — H4 (Sovereignty bridge) failed
+**Implication:** Qwen + harness does NOT match Opus solo on knowledge work. Sovereignty narrative weakens.
+
+**Recommended response:** Honest framing in launch — "sovereignty class leader, not frontier-equivalent" rather than "SOTA-on-local". Update landing copy v3 §3 Claim 3 accordingly. Continue retrieval V2 work as primary path to closing the gap.
+
+#### Sub-branch B.4 — Critical failures (any cell < 2.0)
+**Implication:** Pilot harness or retrieval is broken at base level. Cannot interpret hypothesis results because system was not functional.
+
+**Recommended response:** Halt pilot expansion. Diagnose specific failure mode. Likely candidates: agent loop crash, judge JSON parse failure, hive-mind retrieval contamination. Fix before re-running pilot.
+
+### Launch narrative impact (if material)
+
+If H2 fails: arxiv paper §5.4 (multiplier section) is dropped or deferred. Paper claim #2 changes from "substrate + harness lift frontier model performance" to TBD.
+If H3 fails: sovereignty multiplier framing in landing copy v3 weakens; emphasize substrate ceiling instead.
+If H4 fails: landing copy v3 §3 Claim 3 reframed.
+
+### Decision asks (Marko ratifies)
+
+1. **Confirm halt** of full N=400 multiplier benchmark? (Y/N)
+2. **Next-action path:**
+   - (a) Retrieval V2 work first, pilot retry after
+   - (b) Diagnose specific failure mode, fix, re-run pilot
+   - (c) Drop multiplier from paper, focus on substrate-only narrative
+3. **Launch comms update** — does pilot fail trigger landing copy v3 revision before launch? (Recommend: only if pilot fails AND original copy makes multiplier claim, which v3 currently does not.)
+
+---
+
+## §4 — Decision template — Branch C: PILOT PARTIAL (mixed signals)
+
+If pilot summary shows mixed results — e.g., H2 PASS, H3 FAIL, H4 PASS — PM uses this branch.
+
+### Default disposition
+
+Mixed signals are **inherently ambiguous on N=3**. Sample size is too small to distinguish "true mixed reality" from "noise on a small sample".
+
+**Default recommendation:** Run pilot retry at N=20-30 (not N=400) to reduce uncertainty before committing to full N=400 budget.
+
+### Sub-branch C.1 — Strong signal on majority, weak on minority
+If 2 of 3 hypotheses pass strongly + 1 fails marginally → recommend full N=400 with the failing hypothesis flagged as "exploratory" not "confirmatory" in paper. This requires Marko ratification because it's a methodological judgment call.
+
+### Sub-branch C.2 — Strong on minority, weak on majority
+If 1 of 3 hypotheses passes strongly + 2 fail marginally → recommend retrieval V2 work first. Multiplier story is too uncertain to publish.
+
+### Sub-branch C.3 — All marginal (none clearly pass, none clearly fail)
+Strict reading: pilot FAIL. But may indicate threshold (≥0.30 Likert delta) was too strict for synthesis tasks where judge variance is naturally higher. Marko + PM ratify whether to:
+- (a) Treat as FAIL per pre-registration discipline (recommended; preserves anti-pattern #4)
+- (b) Run pilot retry with calibrated threshold based on observed Likert variance
+
+Anti-pattern #4 reminder: thresholds do not shift post-hoc. Sub-branch C.3 (b) is a methodological deviation that requires explicit acknowledgment and Marko ratification — not a quiet adjustment.
+
+---
+
+## §5 — Cost reality check + audit trail
+
+**Pilot cost cap (per amendment §6):** $7.00 hard, $6.00 halt
+**Full N=400 cost cap (Branch A recommendation):** $150 hard, $130 halt
+**Ratio:** 21x scale-up in budget for ~33x scale-up in sample size (12 → 400 instances)
+**Implication:** per-instance cost decreases due to amortized fixed costs; consistent with pilot-validated economics
+
+**Audit trail requirement:**
+Every populated branch must include verbatim from pilot-summary.json:
+- `pilot_id`
+- `manifest_anchor`
+- `total_cost_usd`
+- `total_judge_calls`, `total_candidate_calls`
+- `pilot_verdict`
+- All 3 task results structures (cell trio means, deltas, directional passes)
+- Full HEAD SHA at execution
+- Amendment SHA `3946d3e0`
+
+If pilot-summary.json is missing any field required for branch population, PM halts and pings CC-1 to re-emit summary. Memo is not authored on incomplete data.
+
+---
+
+## §6 — Memory + decisions folder updates
+
+After Marko ratifies:
+
+- `decisions/2026-04-26-pilot-verdict-{PASS|FAIL|PARTIAL}.md` — populated branch saved as decision record
+- `.auto-memory/project_pilot_2026_04_26_result.md` — memory entry summarizing verdict + ratified next action
+- `MEMORY.md` index updated with pilot result entry
+
+If Branch A (PASS): also create:
+- `decisions/2026-04-26-full-n400-multiplier-authorized.md` — authorization record for full benchmark
+- Update `.auto-memory/project_benchmark_strategy.md` with full N=400 scope locked
+- New CC-1 brief at `briefs/2026-04-27-cc-multiplier-n400-brief.md` based on pilot wrapper learnings
+
+If Branch B (FAIL): also create:
+- `decisions/2026-04-26-multiplier-expansion-deferred.md` — halt record with diagnostics
+- `briefs/2026-04-27-retrieval-v2-priority.md` — V2 work priorities (assuming sub-branch B.1 or B.4)