waggle-os/docs/briefs/2026-04-22-sprint-12-scope-draft.md

Sprint 12 Scope — Draft Skica

Datum: 2026-04-22 (Sprint 11 Day 2 PM, drafted u anticipaciji Sprint 12 kickoff-a) Author: PM (Marko Marković) Status: 📝 DRAFT — ratifikacija se očekuje na Sprint 12 kickoff session-u Trajanje: 5-7 radnih dana projected (Task 1 ~2-3 CC-1 sesije, Task 2 1 sesija, Task 4 anchor task 1-2 sesije) Total budget projected: $1420-2500 LLM spend (Task 2 + Task 4); $0 za Task 1 (infra-build, zero LLM)

---

1. Purpose

Ovaj dokument je working draft Sprint 12 scope skice, spreman za ratifikaciju na Sprint 12 kickoff session-u. Sprint 11 zatvara se 9/10 CLOSED sa C3 DEFERRED → Sprint 12 Task 1 per Path C verdict (sessions/2026-04-22-c3-standdown-path-c-ratified.md). Sprint 12 svrha je: (a) infra-backfill A3 LOCK v1 substrate, (b) izvršenje C3 Stage 2 mini kao pre-requisite za H-42a/b authorization, (c) izvršenje H-42a/b pune eval-e kao SOTA launch-gating signal.

Nije execution authorization. Nije LOCKED. Služi kao substrate iz kojeg se Sprint 12 kickoff brief derivuje.

2. Task taxonomy

Task	Naslov	Trajanje (CC-1 sesije)	LLM budget	Zavisnost
1	Infra-build (6 blockera + 2 non-blocking)	2-3	$0	Sprint 11 close ratified
2	C3 Stage 2 mini execution	1 (+1 buffer)	$120-200 cap $250	Task 1 CLOSED
3	B4 finalization + H-42a/b pre-flight gate	0.5 (PM-side)	$0	Task 2 PASS
4	H-42a/b execution (6120 evals)	1-2 (anchor)	$1300-2300 cap $2600	Task 3 authorization

3. Task 1 — Infra-build (6 blockera dependency-ordered)

Svrha: Backfill A3 LOCK v1 runtime substrate tako da C3 i H-42a/b mogu biti verbatim-executed bez brief relaxation-a. CC-1 engineering only, $0 LLM spend (sve lokalno development + test).

Acceptance gate za Task 1 CLOSURE: 6 blockera CLOSED + 2 non-blocking kompletisani + B3 addendum §5 implemented + smoke test za full invocation shape-a runner-a pass-uje na 10-instance dry run (bez actual LLM calls — mock client za substrate verification).

Dependency DAG:

#1 Dataset loader ──┐
#2 Cell enum rename ┤
                    ├──> #5 Metric computation (reads from loader + cells)
#3 Pre-reg CLI     ──┤
#4 Judge registry   ┤
                    ├──> #6 Failure taxonomy (reads from judge + metrics)
B3 addendum §4/§5 ──┘

Blocker #1 — LoCoMo dataset loader

Trajanje estimate: 3-4h Acceptance criteria:

• benchmarks/data/locomo/ sadrži N=1540 canonical LoCoMo instance fajlove ili single archive (JSONL/JSON) sa conversation_id + question + gold_answer + category poljima.
• datasets.ts:121 više ne silent-fallback-uje na 60-instance synthetic set. Loader throws DatasetMissingError ako canonical path missing.
• dataset_version hash (SHA-256 of canonical archive) emitted kao pre-registration manifest field i JSONL row header.
• 10-instance smoke test (bez LLM calls, samo loader verification) pass.

Why first: Sve ostale blocker-e zavise od real dataset-a. C3 mini (400 evals) bez real LoCoMo ne proizvodi SOTA signal.

Blocker #2 — Cell enum rename

Trajanje estimate: 2-3h Acceptance criteria:

• cells.ts:54-86 key-ovi promenjeni sa raw|memory-only|evolve-only|full-stack → raw|filtered|compressed|full-context per A3 LOCK §cell-taxonomy.
• isCellName tip guard updated.
• Svi cell-specific testovi pass.
• Sprint 10 JSONL artifakti re-keyed ili annotirani sa legacy_cell_name polje (PM odluka za Sprint 12 kickoff brief: hipoteza prepisivanje jer su Sprint 10 artefakti pre-publication; backward-compat fallback opcija samo ako PM eksplicitno ratifikuje).

Why parallel with #1: Nezavisan od dataset loader-a (pure type refactor), može ići paralelno sa #1 u istoj CC-1 sesiji.

Blocker #3 — Pre-registration CLI surface

Trajanje estimate: 4-5h Acceptance criteria:

• runner.ts parseArgs dodaje: --manifest-hash <sha256>, --emit-preregistration-event, --per-cell, --judge-tiebreak <policy> flagove.
• Novi modul benchmarks/harness/src/preregistration.ts sa pino-compatible event emitter za bench.preregistration.manifest_hash event.
• Emitter payload sadrži: manifest_sha256, manifest_path, run_id, timestamp_utc, dataset_version_hash, target_model_pinning block (per B3 addendum §4 YAML schema).
• H-AUDIT-2 spot-verification test harness može match-ovati run-start event sa committed YAML SHA-256 u A3 LOCK twin-u.

Why after #1/#2: Event emitter reads from loader + cells state, tako da mora biti after dataset + enum sanity.

Blocker #4 — Judge model registry

Trajanje estimate: 2h Acceptance criteria:

• config/models.json extended sa entry-ijima za claude-opus-4-7 (Surface B default + dated alias variant), gpt-5.4, gemini-3.1, grok-4.20.
• Svaki entry ima: LiteLLM route, pricing input/output per 1M token, pinning_surface (A ili B per B3 addendum), carve_out_reason gde primenljivo.
• createJudgeLlmClient resolves all four bez errors.
• Cost accounting attribution works end-to-end (test: 100-token mock call kroz svaku od 4 rute, total cost u USD computed correctly sa tolerance ±$0.0001).

Why parallel with #3: Nezavisan od CLI surface-a (config-only), može ići paralelno sa #3 u istoj CC-1 sesiji.

Blocker #5 — Metric computation

Trajanje estimate: 4-6h Acceptance criteria:

• Fleiss' κ mid-run sa baseline 0.7458 anchor (Sprint 10 baseline) + HALT threshold < 0.60 trigger.
• Wilson 95% score CI (per-cell aggregate score, asymmetric bounds za small-sample correctness).
• Cluster-bootstrap 95% CI (10K iterations, seed 42 fixed, cluster variable = conversation_id per A3 LOCK §7).
• Aggregate output JSONL emit-uje sve tri metrike alongside failure taxonomy distribution.
• Unit test: poznat input (50 mock evals sa known verdict distribution) reproduces analytically-expected κ, Wilson bounds, cluster-bootstrap bounds within numerical tolerance.

Why after #1+#2: Metric reads from dataset + cell-keyed results.

Blocker #6 — Failure taxonomy F1-F6 judge rubric integration

Trajanje estimate: 3-4h Acceptance criteria:

• judge-runner.ts judge prompt proširuje rubric sa F1-F6 + null + F-other kategorijama per A3 LOCK §6.
• JSONL rows emit-uju verdict + failure_code + failure_rationale polja per judge per instance.
• Unit test: mock judge response sa svakom od 8 kategorija (F1-F6, null, F-other) parse-uje correctno.

Why last: Judge runner reads from judge registry (#4) + cell-keyed inputs (#2) + metric aggregation schema (#5).

Non-blocking kompletacija A — B3 addendum §4/§5 implementation

Trajanje estimate: 1-2h (uglavnom piggy-backs na #3 i #4) Acceptance criteria:

• JSONL row schema dodaje model_pinning_surface, model_pinning_carve_out_reason, model_revision_hash polja.
• Pre-registration manifest emitter (#3) extend-uje target_model_pinning block per B3 addendum §4 YAML schemu.
• Integration test: single dry run sa DashScope Qwen + Anthropic Opus judge ensemble produces manifest + JSONL sa correct pinning surface annotations per row.

Non-blocking kompletacija B — Smoke test suite

Trajanje estimate: 1-2h Acceptance criteria:

• 10-instance dry-run (mock LLM clients) kroz full runner invocation shape per C3 brief §4.
• Verifies: dataset loaded (1540 count), cell enum correct keys, pre-reg event emitted sa valid YAML SHA-256, judge registry resolves all 4, metrics computed (κ/Wilson/bootstrap placeholders od mock data), failure taxonomy attached.
• $0 LLM spend (mock clients only). Execution time < 60 sekundi.

Why: Ovo je Task 1 CLOSURE gate — ako smoke test pass-uje, Task 1 je CLOSED i Task 2 može kick-ovati.

Task 1 agregirani estimate

• Total CC-1 hours: 15-22h (dependency-ordered, neke parallelizable — #1+#2 together, #3+#4 together, #5/#6 linear).
• Total CC-1 sessions: 2-3 (po 6-8h realnog engineering work-a sa break-ovima).
• LLM spend: $0 (mock clients samo, no real API calls u Task 1).
• Risk: Ako #1 dataset sourcing duži od očekivanog (LoCoMo canonical license/access issue), Task 1 se extend-uje. Fallback: Sprint 12 Task 1 brief eksplicitno imenuje data-sourcing kao #0 (pre-blocker-1).

4. Task 2 — C3 Stage 2 mini execution

Svrha: Izvršiti C3 brief (briefs/2026-04-22-cc-c3-stage2-mini-kickoff.md) verbatim posle Task 1 CLOSURE, uz minimalne updates za Blocker 2 rename (cell enum novi key-ovi) i Blocker 4 judge ensemble literal (eksplicitna concrete model ID lista).

Parametri:

• 4 cells × 100 instances = 400 total evals
• Judge ensemble: claude-opus-4-7,gpt-5.4,gemini-3.1 (Surface B za Opus, Surface A za GPT/Gemini per B3 addendum)
• Budget: $120-200 cap $250 per C3 brief §3
• Exit criteria: 11 per C3 brief §5 (aggregate score trend, κ ≥ 0.60, Wilson CI width ≤ 6pp per-cell, failure taxonomy distribution, et al.)
• Abort triggers: 6 per C3 brief §6 (κ collapse, cost overrun, judge availability)

Ratification gate: PM (Marko) ratifikuje Task 1 CLOSURE → CC-1 kickuje Task 2 per C3 brief §4 invocation template (minor flag adjustments za rename + judge literal).

Projected outcome: Ako PASS (per §5 thresholds), B4 final memo unlock-uje za H-42a/b execution authorization (Task 3).

5. Task 3 — B4 finalization + H-42a/b pre-flight gate

Svrha: Po C3 mini PASS-u, PM potpisuje B4 final memo kao "execution-approved" umesto "methodology intent". Dvojni dokument evolution:

1. Update B4 memo (decisions/2026-04-22-stage-2-full-kickoff-memo.md): Status "FINAL" → "EXECUTION-APPROVED". Section 2 reframe iz "zavisi od Sprint 12 Task 2" u "Task 2 PASS-ovan sa [scores/κ/CI], H-42a/b unlock authorized".
2. Pre-flight checklist dry run: Substrate readiness §0 re-check (per feedback_substrate_readiness_gate.md): grep evidence da sva Task 1 work ostaje intact (dataset hash match, enum unchanged, CLI flags functional, judge registry resolves, metrics testovi pass, event emitter live). Ako bilo koji substrate regression, R4 risk trigger-uje (per B4 memo §7 R4).

Trajanje: 30-60 min PM-side, 30-60 min CC-1 substrate re-check. Budget: $0.

6. Task 4 — H-42a/b execution

Svrha: Pre-registration-conformant puna eval-a za SOTA-gated launch signal.

Parametri:

• H-42a (Qwen primary): 4 cells × 1540 instances × 3 judges (sve judges agreement-based per B2 tie-break policy) = 4620 evals operational count (instance-judge pairs aggregate). Estimated LLM cost $1000-1700.
• H-42b (Opus probe): Reduced eval (1500 instances subset per B4 memo §3) za probe signal za cross-model validation. Estimated LLM cost $300-600.
• Total: 6120 evals cumulative operational count, $1300-2300 LLM spend cap $2600.

Exit criteria (per A3 LOCK §10 + B4 memo §5):

• STRONG-PUBLISHABLE: Aggregate score ≥ 91.6% (Mem0 reper matched/surpassed) + κ ≥ 0.75 + Wilson CI lower bound ≥ 88% per-cell + failure taxonomy distribution consistent with Sprint 10 baseline.
• PUBLISHABLE: Aggregate score 88-91.5% + κ ≥ 0.70 + Wilson CI lower bound ≥ 85%.
• WEAK: Aggregate score 85-87.9% + κ ≥ 0.65. Launch narrative adjusts tone — not "SOTA matched" nego "competitive with published".
• FAIL: Aggregate score < 85% or κ < 0.60. Halt. Triggeruje post-mortem + potencijalno Sprint 13 deep-dive.

Abort triggers:

• Cumulative cost exceeds $2600 hard cap → HALT.
• Live κ drops below 0.60 at 25% checkpoint → HALT, post-mortem.
• Any single judge model unavailable for > 30min → HALT, retry with reduced ensemble option (PM call).

Ratification gate: Task 3 B4 execution-approved + substrate §0 re-check passed → CC-1 kickuje H-42a, then H-42b.

7. Sprint 12 total budget

• Task 1: $0 LLM (pure infra)
• Task 2: $120-200 cap $250
• Task 3: $0
• Task 4: $1300-2300 cap $2600
• Total Sprint 12 projected spend: $1420-2500, hard cap $2850

Budget rationale: Sprint 12 je execution-heavy anchor sprint. Unlike Sprint 11 koji je budget-conservative ($0.018893 cumulative), Sprint 12 investira u signal-gating evidence za SOTA launch claim. Cost-per-signal je ~$0.40-0.50 per eval, acceptable za pre-registration-conformant benchmark.

8. Sprint 12 success criteria (exit condition)

1. Task 1 CLOSED (6 blockera + 2 non-blocking + B3 addendum implementation verified).
2. Task 2 PASS per C3 brief §5 thresholds.
3. Task 3 B4 execution-approved + substrate §0 re-check clean.
4. Task 4 H-42a/b completed sa verdict u STRONG-PUBLISHABLE ili PUBLISHABLE kategoriji.
5. Sprint 12 close memo documented sa scores + κ + CI + cost ledger + next-sprint implications (Sprint 13 likely = launch narrative + brand assets finalization ako Task 4 STRONG-PUBLISHABLE).

Ako Task 4 verdict je WEAK ili FAIL, Sprint 12 close memo documentuje to evenhandedly i Sprint 13 preuzima post-mortem + remediation scope umesto launch narrative.

9. Risks i mitigations

R1 — Task 1 duže nego projected (substrate issues surface kasno): Sprint 12 extend-uje za 1-2 dana. Mitigation: #0 pre-blocker dataset sourcing check na samom kickoff-u.

R2 — Task 2 FAIL na C3 mini: Ne ide u Task 4 dok se uzrok ne ustanovi. Mitigation: post-mortem obavezan, potencijalni re-run sa adjusted thresholds (PM call, NE brief relaxation bez explicit LOCK revisit).

R3 — Task 4 cost overrun: Hard cap $2600 halt-uje pre nego što budžet eksplodira. Mitigation: live cost tracking emitter, alert na 80% threshold.

R4 — Substrate regression između Task 1 i Task 4: B4 memo §7 R4 mitigation (substrate §0 re-check pre Task 4 kickoff-a).

R5 — Provider drift (κ collapse mid-run Task 4): HALT + post-mortem. A3 LOCK §10 protokol triggeruje.

R6 — Judge model unavailability (API outage): Retry-with-reduced-ensemble opcija + PM call za proceed/halt decision.

10. Open questions za Sprint 12 kickoff brief

1. Legacy JSONL handling (Blocker 2 rename): Prepisivanje Sprint 10 artefakata novim key-ovima, ili legacy_cell_name polje za backward-compat? Hipoteza prepisivanje (Sprint 10 je pre-publication), ali PM odluka obavezna pre Task 1 kickoff-a.
2. Canary drift-detection subset (B3 addendum §3): Da li Sprint 12 Task 1 uvodi 50-instance canary set za drift monitoring, ili se to defer-uje u Sprint 13? Trenutni default: defer.
3. Task 4 H-42b probe scope: 1500 instances subset je trenutni estimate; alternative 2000 instances za tighter Wilson CI (trošak +$80-120). PM call na kickoff-u.
4. B4 memo evolution path: Da li update DRAFT-u ili novi "EXECUTION-APPROVED" verzioni fajl? Preporuka: update in-place jer je B4 već jedna-datoteka autoritativ.
5. Sprint 13 preview: Ako Task 4 STRONG-PUBLISHABLE, Sprint 13 scope je launch narrative + brand + wiki publication. Ako WEAK, Sprint 13 je remediation. Oba skica ne drafted dok Task 4 verdict nije poznat.

11. Related

• decisions/2026-04-22-bench-spec-locked.md — A3 LOCK v1 (strategy intact, Sprint 12 Task 1 backfill-uje runtime)
• decisions/2026-04-22-bench-spec-locked.manifest.yaml — A3 LOCK YAML twin
• decisions/2026-04-22-stage-2-full-kickoff-memo.md — B4 final memo, reframed to "execution-gated"
• decisions/2026-04-22-b3-lock-dashscope-addendum.md — B3 addendum za non-Anthropic pinning surface
• briefs/2026-04-22-cc-c3-stage2-mini-kickoff.md — C3 brief source (execution-deferred, ne superseded)
• sessions/2026-04-22-c3-blocked-substrate-gap.md — CC-1 pre-kick verification + 6-blocker taxonomy
• sessions/2026-04-22-c3-standdown-path-c-ratified.md — Path C verdict
• .auto-memory/feedback_substrate_readiness_gate.md — §0 grep evidence protokol
• .auto-memory/project_sprint_11_c3_deferred_path_c.md — Sprint 11 close state

---

DRAFT — awaits PM ratification na Sprint 12 kickoff session-u. Open pitanja §10 blokiraju direct execution-LOCK; Task 1 kickoff čeka PM clarifikaciju na #1 (legacy JSONL) minimum. Remainder open pitanja mogu biti resolved inline tokom Sprint 12 tok-a.