waggle-os/docs/decisions/2026-04-22-stage-2-full-kickoff-memo.md

Stage 2 Full (H-42a/b) Methodology Intent Memo — B4 FINAL

Status: FINAL — Sprint 11 close artifact. This memo declares H-42a/b run methodology, parameters, budget envelope, exit criteria, abort criteria, and risk register. It does NOT authorize execution. H-42a/b execution is gated on Sprint 12 Task 2 (C3 Stage 2 mini PASS) per Path C ratification 2026-04-22 PM.

Datum: 2026-04-22 Sprint: 11 · Track B · Task B4 Authority: PM (Marko Marković) — finalizovan kao Sprint 11 close deliverable Pre-req gate (B4 → execution-ready transition): Sprint 12 Task 2 C3 PASS sa κ ≥ 0.65 i tier signal compatible sa A3 LOCK §2 (mini je exploratory, ali signal mora biti coherent)

---

1. Purpose and binding scope

Ovaj memo definiše šta će se desiti kada se H-42a/b autorizuje za izvršenje, ne kada. Sprint 11 ga finalizuje kao methodology declaration; Sprint 12 ga aktivira kao execution authority posle C3 PASS gate-a.

Memo binds future H-42a/b kickoff to:

• A3 LOCK v1 per PM-Waggle-OS/decisions/2026-04-22-bench-spec-locked.md (intact, ne issuujemo v2 u Sprint 11; ako Sprint 12 C3 surface-uje material change, A3 v2 se issuuje kao separate decision doc i ovaj memo se amend-uje pre execution-a).
• B1 Stage 2 primary config + B2 tie-break + B3 Surface A/B naming + B3 §5 DashScope addendum (full LOCK chain inherited).
• Sprint 12 Task 1 infra-build PASS (svih 6 substrate gapova zatvoreno, runtime spreman za pre-registration-conformant execution).
• Sprint 12 Task 2 C3 readiness signal (PASS ili PASS-WITH-FLAG sa PM review).

2. Why C3-execution-gated, not C3-PASS-pending

Original B4 draft 2026-04-22 PM bio je strukturisan sa <TBD-from-C3> slotovima koji bi se popunili posle C3 exit ping-a u Sprint 11 Day 3. CC-1 pre-kick verification 2026-04-22 PM razotkrio je 6 substrate gapova (LoCoMo dataset absent, cell enum mismatch, pre-registration flagovi unbuilt, judge-ensemble literal invalid, judge models unregistered, DashScope Surface B ambiguity per B3 §5). Path C ratifikovan: C3 izvršenje defer-ovano u Sprint 12 Task 2 posle Task 1 infra-build-a.

Kao posledica: B4 ne može biti "final, pending C3 PASS" jer C3 exit ping ne postoji u Sprint 11 close window-u. Reframing rezultat: B4 finalizuje methodology intent (binding parameters, budget, exit/abort criteria, risk register) bez popunjavanja C3-derived polja. Ta polja se materializuju u Sprint 12 Task 3 (B4 execution authorization addendum) posle Task 2 C3 PASS exit ping-a.

3. H-42a/b run parameters (binding)

Parametri su locked u A3 LOCK v1; per-run manifest emit-uje se na kickoff-u per A3 §7. Ovaj memo ih reproduktovno citira radi single-source-of-truth navigacije.

Target arm (H-42a — Qwen primary):

• Model: qwen3.6-35b-a3b-stage2 resolved to Surface B identifier at kickoff. Per B3 LOCK §5 addendum (datum 2026-04-22), za DashScope-routed Qwen target Surface B može biti Surface A floating alias sa documented carve-out u manifest-u (DashScope ne expose-uje Anthropic-style dated snapshots) ili OpenRouter revision hash ako je dostupan kroz bridge route.
• Mode: thinking=on, 64K reasoning budget per B1.
• Dataset: LoCoMo full eval set, N=1540 (per A3 §3 instance count + dataset_version hash u manifest-u).
• Runs: 3 independent seeded runs (seeds 42, 142, 242).
• Total Qwen evaluations: 4620.

Control arm (H-42b — Opus 4.6 probe):

• Model: anthropic/claude-opus-4-6 resolved to Surface B dated snapshot at kickoff (subject to B3 LOCK §3 verification — must be valid against Anthropic's published snapshot list at PR review time).
• Dataset: LoCoMo stratified subsample N=500.
• Runs: 3 independent seeded runs (seeds 42, 142, 242).
• Total Opus evaluations: 1500.

Judge ensemble (both arms):

• Primary 3: claude-opus-4-7 + gpt-5.4 + gemini-3.1 (Surface B dated snapshots resolved at kickoff and pinned in per-run manifest per A3 §4 consistency constraint).
• Tie-break: xai/grok-4.20 (Surface B dated snapshot resolved at kickoff). Auto-fires na 1-1-1 split per B2 LOCK; 2-2 split escalates u PM_ESCALATION surface per B2 §3.
• Same physical judges across H-42a and H-42b (A3 §4 consistency constraint).

Aggregate evaluations: 6120 (4620 Qwen + 1500 Opus).

4. Budget envelope (binding)

Per A3 §3 manifest budget breakdown:

Component	Expected	Ceiling
Qwen primary (4620 evals)	$600–1100	$1400
Opus probe (1500 evals)	$200–350	$450
Judge triple (6120 × 3)	$450–750	$900
Tie-break grok (~200 fires)	$5–15	$40
Buffer / retries	$45–85	$110
Total	$1300–2300	$2600 hard

Hard abort if cumulative spend crosses $2600 at any mid-run checkpoint.

5. Exit criteria — tier verdict

H-42a/b je CLOSED sa jednim od četiri tier verdict-a per A3 LOCK §2, primenjeno na H-42a (Qwen) aggregate across 3 runs (4620 evaluations):

• STRONG-PUBLISHABLE — point ≥ 91.6% and Wilson lower ≥ 91.6% and bootstrap lower ≥ 91.6%. Launch claim ready.
• PUBLISHABLE — point ≥ 91.6% and Wilson lower ≥ 89.0%. Claim with appropriate CI disclosure.
• WEAK — point in [89.0%, 91.5%]. Do NOT claim SOTA. Triggers PM-review-gate before any external use.
• FAIL — point < 89.0%. Do NOT publish. Triggers post-mortem.

H-42b (Opus probe) tier: secondary descriptive read; not subject to confirmatory hypothesis. Result feeds the multiplier thesis dual-axis narrative as "Opus 4.6 directional comparison vs Mem0 reference" but does NOT gate the Qwen launch claim.

Conservative rule: ako Wilson i cluster-bootstrap CI nisu saglasni o tier-u za H-42a, prevladava konzervativniji tier per A3 §2.

Required exit ping fields at PM-Waggle-OS/sessions/2026-XX-XX-h-42a-b-exit.md:

• Tier verdict (jedan od četiri).
• Per-arm aggregate: point estimate, Wilson 95% CI, cluster-bootstrap 95% CI.
• Per-run breakdown for both arms (3 runs × 2 arms = 6 sub-tables).
• κ per run + drop-from-baseline per run.
• Failure code distribution per arm.
• F-other rationale sample (3 random if rate >10%).
• Tie-break fire count + breakdown (1-1-1 vs 2-2 escalation).
• Shape distribution per arm.
• Manifest hash match verification.
• Budget actual breakdown by call class.
• Tier 2 archive bundle pointer.

6. Abort criteria

Mid-run abort if ANY:

• Budget burn > $2600 cumulative at any partial-run checkpoint.
• κ < 0.60 OR drop > 10pp from Sprint 10 0.7458 baseline (i.e., κ < 0.6458) on any run's mid-checkpoint (every 500 evaluations).
• LiteLLM /health flips unhealthy and does not recover within 5 minutes of restart attempt.
• Persistent NETWORK_ERROR for >5 consecutive calls on same target model.
• reasoning_content_shape_unknown drift event fires more than 3x per run.
• Manifest hash mismatch between emit and committed YAML.
• Provider rotation event mid-run on any judge or target model floating alias (would invalidate per-run manifest pinning).

On abort: write sessions/<DATE>-h42-aborted-<reason>.md with captured state; preserve partial JSONL; notify PM immediately.

7. Risk register (full-run specific)

Tri rizika elevated above C3 thresholds because full-run scale magnifies impact:

R1 — Provider rotation mid-campaign. Full run takes ~24-72h wall-clock depending on rate limits. Probability that any of 5 model floating aliases (4 judges + 1 target) gets a snapshot rotation by provider during that window is non-trivial. Mitigation: per-run manifest pins all 5 Surface B identifiers at kickoff. Detection: pino events log resolved snapshot per call; CC-1 monitors for unexpected snapshot diversity in event stream and HALTs if seen. Recovery: ako se rotacija detektuje mid-run, HALT i odluka između (a) restart sa novim snapshot-om pinnovanim (full re-run), (b) restart sa starim snapshot-om ako je još accessible (preferred), (c) defer to next snapshot stability window.

R2 — Judge κ drift over 24-72h window. Sprint 10 κ=0.7458 izračunat na single sitting-u; 24-72h continuous evaluation može razotkriti degradation patterns invisible at small N. Mitigation: κ se računa per run (ne samo per campaign) tako da 3 runs daju 3 κ values. Ako bilo koji run individually drop-uje ispod threshold-a, taj run aborts; preostali runs continue ako issue ne generalizuje. PM reviews κ trend across the 3 runs before tier verdict.

R3 — Network volatility cumulative effect. C2 had 1 timeout in 10 instances. At 6120 evaluations, even a 5% transient error rate yields ~300 retries, which strains rate limits and inflates wall-clock. Mitigation: harness retry budget per-instance is honored; persistent errors trigger §6 abort threshold. Ako wall-clock pređe 96h, PM reviews da li continue ili restart sa adjusted concurrency.

R4 (NEW, identified post-C3-block) — Substrate regression after Sprint 12 Task 1 completion. H-42a/b se kick-uje tek posle Sprint 12 Task 1 PASS-a. Postoji rizik da između Task 1 commit-a i H-42a/b kickoff-a (Sprint 12 days 1-7) downstream PR-ovi inadvertently regress neku od substrate komponenti (dataset loader path, CLI flagovi, event emitter, model registry). Mitigation: pre H-42a/b kickoff-a, CC-1 ponavlja substrate readiness §0 grep evidence pass identičan onome što je proizveo C3 BLOCKED report; ako bilo šta drift-uje, HALT pre kickoff-a, fix u Task 1 retroaktivno, restart §0 check.

8. Post-PASS communication and artifacts

Upon H-42a tier verdict ∈ {STRONG-PUBLISHABLE, PUBLISHABLE}:

• Tier 2 archive bundle created at waggle-os/benchmarks/archive/2026-XX-XX-stage2-full.tar.gz per A3 §9 layout. Full JSONL with reasoning_content preserved.
• Audit log entry initialized at PM-Waggle-OS/audit-log/2026-XX-XX-stage2-full-archive-created.md with SHA-256 of bundle + retention horizon timer start.
• Tier verdict + Wilson + bootstrap CIs become input to launch claim copy on Waggle/KVARK landing per multiplier thesis dual-axis framing. PMM owns derivation; engineering does not.
• B4 execution authorization addendum committed (Sprint 12 deliverable, populates kontekst za Sprint 13 launch sequencing).
• Sprint 12 retrospective triggered.
• Sprint 13 planning anchored on H-42a/b results + launch sequencing.

Upon H-42a tier ∈ {WEAK, FAIL}:

• WEAK: PM-review-gate pre any external claim. Possible paths: (a) accept WEAK and frame launch as "approaching SOTA with explicit caveats", (b) iterate on harness or model config and rerun, (c) defer launch claim. Decision via dedicated PM doc within 7 days of exit.
• FAIL: post-mortem triggered. Investigation scope: harness defect vs model capability vs LoCoMo subset bias vs judge calibration. Multi-day cycle expected. Launch sequencing reconsidered.

H-42b Opus probe result attached as secondary input to multiplier thesis narrative regardless of H-42a tier.

9. Sprint 11 close

Upon B4 memo finalization (this doc, kao Sprint 11 deliverable), Sprint 11 reaches 9/10 exit criteria CLOSED:

• A1 ✅ Design doc ratified
• A2 ✅ reasoning_content wire LIVE
• A3 ✅ Bench-spec LOCKED v1 (intact)
• B1 ✅ Stage 2 primary config LOCKED
• B2 ✅ Tie-break policy LOCKED + LIVE u judge-runner
• B3 ✅ Opus 4.6 audit + cleanup + B3 §5 DashScope addendum
• B4 ✅ Stage 2 methodology intent memo (this doc, finalizovan)
• C1 ✅ Stage 0 baseline
• C2 ✅ Stage 1 mikro-eval PASS
• C3 ⏸️ DEFERRED → Sprint 12 Task 2 (per Path C ratifikacija 2026-04-22 PM, blocked na 6 substrate gapova koji se rešavaju u Sprint 12 Task 1)

Cumulative Sprint 11 spend at 9/10: ~$0.019 of ~$150 soft ceiling (0.013%). H-42a/b remains gated on Sprint 12 Task 1 + Task 2 PASS.

10. What this memo does NOT do

• Does NOT autorizovati H-42a/b execution. Autorizacija zahteva (a) Sprint 12 Task 1 PASS (substrate ready), (b) Sprint 12 Task 2 PASS (C3 mini exit verdict compatible sa A3 LOCK §2), (c) PM review + ratification addendum koji popunjava prethodne <TBD> polja.
• Does NOT commit budget for Sprint 12. Sprint 12 budget je separate planning artifact.
• Does NOT specify launch copy ili PMM derivation rules. Downstream PMM work consuming H-42a/b exit ping.
• Does NOT amend A3 LOCK. Ako Sprint 12 C3 surface-uje material change, A3 v2 je separate PM-ratified doc that supersedes A3 v1 — ovaj memo se tada amend-uje pre execution-a.
• Does NOT presume C3 has executed. C3 execution je Sprint 12 Task 2 prerequisite za H-42a/b authorization.

11. Related

• PM-Waggle-OS/decisions/2026-04-22-bench-spec-locked.md — A3 LOCK v1 (parent manifest, intact)
• PM-Waggle-OS/decisions/2026-04-22-bench-spec-locked.manifest.yaml — A3 YAML twin
• PM-Waggle-OS/decisions/2026-04-22-stage-2-primary-config-locked.md — B1
• PM-Waggle-OS/decisions/2026-04-22-tie-break-policy-locked.md — B2
• PM-Waggle-OS/decisions/2026-04-22-model-route-naming-locked.md — B3
• PM-Waggle-OS/decisions/2026-04-22-b3-lock-dashscope-addendum.md — B3 §5 addendum (this PM tranche)
• PM-Waggle-OS/briefs/2026-04-22-cc-c3-stage2-mini-kickoff.md — C3 brief (Sprint 12 reference artifact)
• PM-Waggle-OS/sessions/2026-04-22-c3-blocked-substrate-gap.md — CC-1 pre-kick verification
• PM-Waggle-OS/sessions/2026-04-22-c3-standdown-path-c-ratified.md — Path C ratification + standdown
• PM-Waggle-OS/briefs/2026-04-22-sprint-12-scope-draft.md — Sprint 12 scope skica (this PM tranche)
• PM-Waggle-OS/sessions/2026-04-22-sprint-11-c2-stage1-mikroeval-exit.md — C2 exit ping (forensic input + R3 baseline)

---

B4 FINAL — Sprint 11 close artifact. Methodology intent declared, execution C3-gated to Sprint 12. Sprint 11 9/10 CLOSED.