waggle-os/docs/decisions/2026-04-26-pilot-decision-template.md

# Pilot Decision Template — Go/No-Go for Full N=400 Multiplier Benchmark

**Authored:** 2026-04-26 (pre-results, in advance of pilot completion)
**Decision owner:** Marko (ratifies); PM (drafts)
**Trigger:** CC-1 emits `pilot-summary.json` after agentic knowledge work pilot completes
**Pilot ID:** `agentic-knowledge-work-pilot-2026-04-26`
**Manifest anchor:** `pilot-2026-04-26-v1` (amendment SHA `3946d3e0`)

**Why this template exists pre-results:** Pre-built decision branches force honest threshold adherence. When results arrive, PM doesn't draft a memo from scratch under emotional pressure to rationalize the outcome — PM populates the appropriate branch with verbatim numbers. This is the discipline `feedback_substrate_readiness_gate` and `Anti-pattern #4 reminder: thresholds do not shift post-hoc` from manifest v6 are enforcing.

---

## §1 — Pre-registered hypotheses (do NOT modify)

From `cc1-brief.md` §2 + amendment §1:

- **H2 — Opus multiplier**: Cell B trio mean > Cell A trio mean by ≥ 0.30 Likert points, on ≥ 2 of 3 tasks
- **H3 — Qwen multiplier**: Cell D trio mean > Cell C trio mean by ≥ 0.30 Likert points, on ≥ 2 of 3 tasks
- **H4 — Sovereignty bridge**: Cell D trio mean ≥ Cell A trio mean, on ≥ 2 of 3 tasks

**Pilot binary verdict:**
- **PASS** = H2 + H3 + H4 each show directional sign on ≥ 2 of 3 tasks AND no critical failures (no cell scoring < 2.0 on majority of judges)
- **FAIL** = otherwise

---

## §2 — Decision template — Branch A: PILOT PASS

If pilot summary shows PASS, PM populates this branch and submits to Marko for ratification.

### Memo header

**Subject:** PM-RATIFY — Full N=400 multiplier benchmark authorization (pilot PASSED)
**Date:** [populate from pilot completion timestamp]
**Author:** PM
**Decision asks:** (1) Authorize full N=400 multiplier scope; (2) Confirm budget envelope; (3) Ratify model roster; (4) Lock manifest v7 anchor

### Memo body

**Pilot result summary:**
- H2 directional pass: [X of 3 tasks]
- H3 directional pass: [X of 3 tasks]
- H4 directional pass: [X of 3 tasks]
- Critical failures: [count]
- Pilot verdict: **PASS**
- Wall-clock: [actual hh:mm]
- Total cost: $[actual]
- HEAD SHA at execution: [commit hash]

**Cell-by-cell deltas (per task, trio means):**

| Task | A (Opus solo) | B (Opus + harness) | C (Qwen solo) | D (Qwen + harness) | H2 Δ (B-A) | H3 Δ (D-C) | H4 Δ (D-A) |
|---|---|---|---|---|---|---|---|
| Task 1 | [X.X] | [X.X] | [X.X] | [X.X] | [+/-X.X] | [+/-X.X] | [+/-X.X] |
| Task 2 | [X.X] | [X.X] | [X.X] | [X.X] | [+/-X.X] | [+/-X.X] | [+/-X.X] |
| Task 3 | [X.X] | [X.X] | [X.X] | [X.X] | [+/-X.X] | [+/-X.X] | [+/-X.X] |

### Recommended full N=400 multiplier scope

**Sample size:** N=400 instances per cell, drawn from real-world knowledge work corpus (TBD construction — synthetic-realistic per pilot pattern, scaled to 400 instances)

**Cells (3 cells, not 4):**
- Cell A: Opus 4.7 solo
- Cell B: Opus 4.7 + memory + agent loop
- Cell C: Qwen 3.6 35B-A3B + memory + agent loop

(H4 sovereignty bridge claim is most-actionable comparison; Cell C-Qwen-solo redundant if pilot demonstrates Qwen + harness ≥ Opus solo. Confirm with Marko whether Cell D-Qwen-solo retained as control.)

**Optional addition — Cell D: GPT-5.4 + memory + agent loop** for cross-vendor frontier comparison.

**Models tested:**
1. claude-opus-4-7 (frontier proprietary)
2. qwen3.6-35b-a3b-via-openrouter (sovereign reference)
3. (optional) gpt-5.4 (frontier proprietary, second vendor)

**Judge ensemble:** Same trio-strict (Opus + GPT + MiniMax) per pilot. Re-calibrate κ on N=14 synthesis-task subset before full launch.

**Budget envelope:**
- Candidate model spend: $40-90 (Opus dominates, 400 × multi-step × Opus rate)
- Trio judge spend: $30-50 (1200-1600 judge calls)
- Buffer: $30
- Total cap: **$150 hard cap, $130 halt**

**Wall-clock target:** 36-48 hours runner time (parallel cell execution where possible)

**Pre-registration manifest v7:**
- Anchor commit at full launch (TBD)
- Hypothesis statements identical to pilot (H2, H3, H4) — unchanged
- Cost cap, halt thresholds locked
- Sample size N=400, seed 42
- Output schema same as pilot
- F-mode taxonomy reused

### Decision asks (Marko ratifies)

1. **Authorize full N=400 multiplier benchmark** with scope above? (Y/N)
2. **Cell D-Qwen-solo retained or descoped?** Pilot showed [X of 3 sovereignty bridges]; descoping reduces cost ~$30. Recommendation: [retain/descope based on pilot results]
3. **Add GPT-5.4 cell or stay 3-cell?** Adds ~$30 cost, strengthens cross-vendor frontier claim for paper. Recommendation: [add/skip based on paper plans]
4. **Manifest v7 anchor commit** — current HEAD or fresh commit before kick? Recommendation: [based on tree state]
5. **Run timing** — kick off [today/tomorrow/post-arxiv] given concurrent landing copy + arxiv work?

---

## §3 — Decision template — Branch B: PILOT FAIL

If pilot summary shows FAIL, PM populates this branch.

### Memo header

**Subject:** PM-HALT — Multiplier expansion deferred (pilot FAILED)
**Date:** [populate from pilot completion timestamp]
**Author:** PM
**Decision asks:** (1) Confirm halt; (2) Choose next-action path; (3) Update launch narrative if material

### Memo body

**Pilot result summary:**
- H2 directional pass: [X of 3 tasks] — required ≥ 2
- H3 directional pass: [X of 3 tasks] — required ≥ 2
- H4 directional pass: [X of 3 tasks] — required ≥ 2
- Critical failures: [count]
- Pilot verdict: **FAIL**

**Failure mode classification (which hypothesis failed and why):**

#### Sub-branch B.1 — H2 (Opus multiplier) failed
**Implication:** Adding hive-mind + agent loop to a frontier model does NOT reliably lift performance on knowledge work. This contradicts PA V5 finding (April 2026, Opus 4.6 +5.2pp publishable on H1). Possible causes:
- V1 retrieval quality is insufficient; agent loop pulls noisy chunks and degrades vs. full-context Opus baseline
- Multi-step agent loop overhead exceeds value-add at 5-step ceiling
- Knowledge work tasks do not benefit from memory in the same way memory-recall tasks do

**Recommended response:** Pause multiplier expansion. Prioritize retrieval V2 work (5 directions identified in arxiv §5.3). Re-run pilot post-V2.

#### Sub-branch B.2 — H3 (Qwen multiplier) failed
**Implication:** Qwen 35B-A3B + harness does NOT lift Qwen performance reliably. Possible causes:
- Qwen 35B-A3B context utilization is already strong; full-context cell is competitive baseline
- Agent loop self-prompting confuses Qwen more than it helps
- Knowledge work tasks require cognitive capability that base Qwen struggles with regardless of harness

**Recommended response:** Run Qwen-only ablation isolating each loop component (retrieval-only, self-prompt-only, full harness). Identify which component degrades vs. helps.

#### Sub-branch B.3 — H4 (Sovereignty bridge) failed
**Implication:** Qwen + harness does NOT match Opus solo on knowledge work. Sovereignty narrative weakens.

**Recommended response:** Honest framing in launch — "sovereignty class leader, not frontier-equivalent" rather than "SOTA-on-local". Update landing copy v3 §3 Claim 3 accordingly. Continue retrieval V2 work as primary path to closing the gap.

#### Sub-branch B.4 — Critical failures (any cell < 2.0)
**Implication:** Pilot harness or retrieval is broken at base level. Cannot interpret hypothesis results because system was not functional.

**Recommended response:** Halt pilot expansion. Diagnose specific failure mode. Likely candidates: agent loop crash, judge JSON parse failure, hive-mind retrieval contamination. Fix before re-running pilot.

### Launch narrative impact (if material)

If H2 fails: arxiv paper §5.4 (multiplier section) is dropped or deferred. Paper claim #2 changes from "substrate + harness lift frontier model performance" to TBD.
If H3 fails: sovereignty multiplier framing in landing copy v3 weakens; emphasize substrate ceiling instead.
If H4 fails: landing copy v3 §3 Claim 3 reframed.

### Decision asks (Marko ratifies)

1. **Confirm halt** of full N=400 multiplier benchmark? (Y/N)
2. **Next-action path:**
   - (a) Retrieval V2 work first, pilot retry after
   - (b) Diagnose specific failure mode, fix, re-run pilot
   - (c) Drop multiplier from paper, focus on substrate-only narrative
3. **Launch comms update** — does pilot fail trigger landing copy v3 revision before launch? (Recommend: only if pilot fails AND original copy makes multiplier claim, which v3 currently does not.)

---

## §4 — Decision template — Branch C: PILOT PARTIAL (mixed signals)

If pilot summary shows mixed results — e.g., H2 PASS, H3 FAIL, H4 PASS — PM uses this branch.

### Default disposition

Mixed signals are **inherently ambiguous on N=3**. Sample size is too small to distinguish "true mixed reality" from "noise on a small sample".

**Default recommendation:** Run pilot retry at N=20-30 (not N=400) to reduce uncertainty before committing to full N=400 budget.

### Sub-branch C.1 — Strong signal on majority, weak on minority
If 2 of 3 hypotheses pass strongly + 1 fails marginally → recommend full N=400 with the failing hypothesis flagged as "exploratory" not "confirmatory" in paper. This requires Marko ratification because it's a methodological judgment call.

### Sub-branch C.2 — Strong on minority, weak on majority
If 1 of 3 hypotheses passes strongly + 2 fail marginally → recommend retrieval V2 work first. Multiplier story is too uncertain to publish.

### Sub-branch C.3 — All marginal (none clearly pass, none clearly fail)
Strict reading: pilot FAIL. But may indicate threshold (≥0.30 Likert delta) was too strict for synthesis tasks where judge variance is naturally higher. Marko + PM ratify whether to:
- (a) Treat as FAIL per pre-registration discipline (recommended; preserves anti-pattern #4)
- (b) Run pilot retry with calibrated threshold based on observed Likert variance

Anti-pattern #4 reminder: thresholds do not shift post-hoc. Sub-branch C.3 (b) is a methodological deviation that requires explicit acknowledgment and Marko ratification — not a quiet adjustment.

---

## §5 — Cost reality check + audit trail

**Pilot cost cap (per amendment §6):** $7.00 hard, $6.00 halt
**Full N=400 cost cap (Branch A recommendation):** $150 hard, $130 halt
**Ratio:** 21x scale-up in budget for ~33x scale-up in sample size (12 → 400 instances)
**Implication:** per-instance cost decreases due to amortized fixed costs; consistent with pilot-validated economics

**Audit trail requirement:**
Every populated branch must include verbatim from pilot-summary.json:
- `pilot_id`
- `manifest_anchor`
- `total_cost_usd`
- `total_judge_calls`, `total_candidate_calls`
- `pilot_verdict`
- All 3 task results structures (cell trio means, deltas, directional passes)
- Full HEAD SHA at execution
- Amendment SHA `3946d3e0`

If pilot-summary.json is missing any field required for branch population, PM halts and pings CC-1 to re-emit summary. Memo is not authored on incomplete data.

---

## §6 — Memory + decisions folder updates

After Marko ratifies:

- `decisions/2026-04-26-pilot-verdict-{PASS|FAIL|PARTIAL}.md` — populated branch saved as decision record
- `.auto-memory/project_pilot_2026_04_26_result.md` — memory entry summarizing verdict + ratified next action
- `MEMORY.md` index updated with pilot result entry

If Branch A (PASS): also create:
- `decisions/2026-04-26-full-n400-multiplier-authorized.md` — authorization record for full benchmark
- Update `.auto-memory/project_benchmark_strategy.md` with full N=400 scope locked
- New CC-1 brief at `briefs/2026-04-27-cc-multiplier-n400-brief.md` based on pilot wrapper learnings

If Branch B (FAIL): also create:
- `decisions/2026-04-26-multiplier-expansion-deferred.md` — halt record with diagnostics
- `briefs/2026-04-27-retrieval-v2-priority.md` — V2 work priorities (assuming sub-branch B.1 or B.4)