239 lines
16 KiB
Markdown
239 lines
16 KiB
Markdown
# Sprint 12 Scope — Draft Skica
|
||
|
||
**Datum:** 2026-04-22 (Sprint 11 Day 2 PM, drafted u anticipaciji Sprint 12 kickoff-a)
|
||
**Author:** PM (Marko Marković)
|
||
**Status:** 📝 **DRAFT** — ratifikacija se očekuje na Sprint 12 kickoff session-u
|
||
**Trajanje:** 5-7 radnih dana projected (Task 1 ~2-3 CC-1 sesije, Task 2 1 sesija, Task 4 anchor task 1-2 sesije)
|
||
**Total budget projected:** $1420-2500 LLM spend (Task 2 + Task 4); $0 za Task 1 (infra-build, zero LLM)
|
||
|
||
---
|
||
|
||
## 1. Purpose
|
||
|
||
Ovaj dokument je **working draft Sprint 12 scope skice**, spreman za ratifikaciju na Sprint 12 kickoff session-u. Sprint 11 zatvara se 9/10 CLOSED sa C3 DEFERRED → Sprint 12 Task 1 per Path C verdict (`sessions/2026-04-22-c3-standdown-path-c-ratified.md`). Sprint 12 svrha je: (a) infra-backfill A3 LOCK v1 substrate, (b) izvršenje C3 Stage 2 mini kao pre-requisite za H-42a/b authorization, (c) izvršenje H-42a/b pune eval-e kao SOTA launch-gating signal.
|
||
|
||
Nije execution authorization. Nije LOCKED. Služi kao substrate iz kojeg se Sprint 12 kickoff brief derivuje.
|
||
|
||
## 2. Task taxonomy
|
||
|
||
| Task | Naslov | Trajanje (CC-1 sesije) | LLM budget | Zavisnost |
|
||
|------|--------|------------------------|-----------|-----------|
|
||
| 1 | Infra-build (6 blockera + 2 non-blocking) | 2-3 | $0 | Sprint 11 close ratified |
|
||
| 2 | C3 Stage 2 mini execution | 1 (+1 buffer) | $120-200 cap $250 | Task 1 CLOSED |
|
||
| 3 | B4 finalization + H-42a/b pre-flight gate | 0.5 (PM-side) | $0 | Task 2 PASS |
|
||
| 4 | H-42a/b execution (6120 evals) | 1-2 (anchor) | $1300-2300 cap $2600 | Task 3 authorization |
|
||
|
||
## 3. Task 1 — Infra-build (6 blockera dependency-ordered)
|
||
|
||
**Svrha:** Backfill A3 LOCK v1 runtime substrate tako da C3 i H-42a/b mogu biti verbatim-executed bez brief relaxation-a. CC-1 engineering only, $0 LLM spend (sve lokalno development + test).
|
||
|
||
**Acceptance gate za Task 1 CLOSURE:** 6 blockera CLOSED + 2 non-blocking kompletisani + B3 addendum §5 implemented + smoke test za full invocation shape-a runner-a pass-uje na 10-instance dry run (bez actual LLM calls — mock client za substrate verification).
|
||
|
||
**Dependency DAG:**
|
||
|
||
```
|
||
#1 Dataset loader ──┐
|
||
#2 Cell enum rename ┤
|
||
├──> #5 Metric computation (reads from loader + cells)
|
||
#3 Pre-reg CLI ──┤
|
||
#4 Judge registry ┤
|
||
├──> #6 Failure taxonomy (reads from judge + metrics)
|
||
B3 addendum §4/§5 ──┘
|
||
```
|
||
|
||
### Blocker #1 — LoCoMo dataset loader
|
||
|
||
**Trajanje estimate:** 3-4h
|
||
**Acceptance criteria:**
|
||
- `benchmarks/data/locomo/` sadrži N=1540 canonical LoCoMo instance fajlove ili single archive (JSONL/JSON) sa `conversation_id` + `question` + `gold_answer` + `category` poljima.
|
||
- `datasets.ts:121` više ne silent-fallback-uje na 60-instance synthetic set. Loader throws `DatasetMissingError` ako canonical path missing.
|
||
- `dataset_version` hash (SHA-256 of canonical archive) emitted kao pre-registration manifest field i JSONL row header.
|
||
- 10-instance smoke test (bez LLM calls, samo loader verification) pass.
|
||
|
||
**Why first:** Sve ostale blocker-e zavise od real dataset-a. C3 mini (400 evals) bez real LoCoMo ne proizvodi SOTA signal.
|
||
|
||
### Blocker #2 — Cell enum rename
|
||
|
||
**Trajanje estimate:** 2-3h
|
||
**Acceptance criteria:**
|
||
- `cells.ts:54-86` key-ovi promenjeni sa `raw|memory-only|evolve-only|full-stack` → `raw|filtered|compressed|full-context` per A3 LOCK §cell-taxonomy.
|
||
- `isCellName` tip guard updated.
|
||
- Svi cell-specific testovi pass.
|
||
- Sprint 10 JSONL artifakti re-keyed ili annotirani sa `legacy_cell_name` polje (PM odluka za Sprint 12 kickoff brief: hipoteza prepisivanje jer su Sprint 10 artefakti pre-publication; backward-compat fallback opcija samo ako PM eksplicitno ratifikuje).
|
||
|
||
**Why parallel with #1:** Nezavisan od dataset loader-a (pure type refactor), može ići paralelno sa #1 u istoj CC-1 sesiji.
|
||
|
||
### Blocker #3 — Pre-registration CLI surface
|
||
|
||
**Trajanje estimate:** 4-5h
|
||
**Acceptance criteria:**
|
||
- `runner.ts` parseArgs dodaje: `--manifest-hash <sha256>`, `--emit-preregistration-event`, `--per-cell`, `--judge-tiebreak <policy>` flagove.
|
||
- Novi modul `benchmarks/harness/src/preregistration.ts` sa pino-compatible event emitter za `bench.preregistration.manifest_hash` event.
|
||
- Emitter payload sadrži: `manifest_sha256`, `manifest_path`, `run_id`, `timestamp_utc`, `dataset_version_hash`, `target_model_pinning` block (per B3 addendum §4 YAML schema).
|
||
- H-AUDIT-2 spot-verification test harness može match-ovati run-start event sa committed YAML SHA-256 u A3 LOCK twin-u.
|
||
|
||
**Why after #1/#2:** Event emitter reads from loader + cells state, tako da mora biti after dataset + enum sanity.
|
||
|
||
### Blocker #4 — Judge model registry
|
||
|
||
**Trajanje estimate:** 2h
|
||
**Acceptance criteria:**
|
||
- `config/models.json` extended sa entry-ijima za `claude-opus-4-7` (Surface B default + dated alias variant), `gpt-5.4`, `gemini-3.1`, `grok-4.20`.
|
||
- Svaki entry ima: LiteLLM route, pricing input/output per 1M token, `pinning_surface` (A ili B per B3 addendum), `carve_out_reason` gde primenljivo.
|
||
- `createJudgeLlmClient` resolves all four bez errors.
|
||
- Cost accounting attribution works end-to-end (test: 100-token mock call kroz svaku od 4 rute, total cost u USD computed correctly sa tolerance ±$0.0001).
|
||
|
||
**Why parallel with #3:** Nezavisan od CLI surface-a (config-only), može ići paralelno sa #3 u istoj CC-1 sesiji.
|
||
|
||
### Blocker #5 — Metric computation
|
||
|
||
**Trajanje estimate:** 4-6h
|
||
**Acceptance criteria:**
|
||
- Fleiss' κ mid-run sa baseline 0.7458 anchor (Sprint 10 baseline) + HALT threshold < 0.60 trigger.
|
||
- Wilson 95% score CI (per-cell aggregate score, asymmetric bounds za small-sample correctness).
|
||
- Cluster-bootstrap 95% CI (10K iterations, seed 42 fixed, cluster variable = `conversation_id` per A3 LOCK §7).
|
||
- Aggregate output JSONL emit-uje sve tri metrike alongside failure taxonomy distribution.
|
||
- Unit test: poznat input (50 mock evals sa known verdict distribution) reproduces analytically-expected κ, Wilson bounds, cluster-bootstrap bounds within numerical tolerance.
|
||
|
||
**Why after #1+#2:** Metric reads from dataset + cell-keyed results.
|
||
|
||
### Blocker #6 — Failure taxonomy F1-F6 judge rubric integration
|
||
|
||
**Trajanje estimate:** 3-4h
|
||
**Acceptance criteria:**
|
||
- `judge-runner.ts` judge prompt proširuje rubric sa F1-F6 + null + F-other kategorijama per A3 LOCK §6.
|
||
- JSONL rows emit-uju `verdict` + `failure_code` + `failure_rationale` polja per judge per instance.
|
||
- Unit test: mock judge response sa svakom od 8 kategorija (F1-F6, null, F-other) parse-uje correctno.
|
||
|
||
**Why last:** Judge runner reads from judge registry (#4) + cell-keyed inputs (#2) + metric aggregation schema (#5).
|
||
|
||
### Non-blocking kompletacija A — B3 addendum §4/§5 implementation
|
||
|
||
**Trajanje estimate:** 1-2h (uglavnom piggy-backs na #3 i #4)
|
||
**Acceptance criteria:**
|
||
- JSONL row schema dodaje `model_pinning_surface`, `model_pinning_carve_out_reason`, `model_revision_hash` polja.
|
||
- Pre-registration manifest emitter (#3) extend-uje `target_model_pinning` block per B3 addendum §4 YAML schemu.
|
||
- Integration test: single dry run sa DashScope Qwen + Anthropic Opus judge ensemble produces manifest + JSONL sa correct pinning surface annotations per row.
|
||
|
||
### Non-blocking kompletacija B — Smoke test suite
|
||
|
||
**Trajanje estimate:** 1-2h
|
||
**Acceptance criteria:**
|
||
- 10-instance dry-run (mock LLM clients) kroz full runner invocation shape per C3 brief §4.
|
||
- Verifies: dataset loaded (1540 count), cell enum correct keys, pre-reg event emitted sa valid YAML SHA-256, judge registry resolves all 4, metrics computed (κ/Wilson/bootstrap placeholders od mock data), failure taxonomy attached.
|
||
- $0 LLM spend (mock clients only). Execution time < 60 sekundi.
|
||
|
||
**Why:** Ovo je Task 1 CLOSURE gate — ako smoke test pass-uje, Task 1 je CLOSED i Task 2 može kick-ovati.
|
||
|
||
### Task 1 agregirani estimate
|
||
|
||
- **Total CC-1 hours:** 15-22h (dependency-ordered, neke parallelizable — #1+#2 together, #3+#4 together, #5/#6 linear).
|
||
- **Total CC-1 sessions:** 2-3 (po 6-8h realnog engineering work-a sa break-ovima).
|
||
- **LLM spend:** $0 (mock clients samo, no real API calls u Task 1).
|
||
- **Risk:** Ako #1 dataset sourcing duži od očekivanog (LoCoMo canonical license/access issue), Task 1 se extend-uje. Fallback: Sprint 12 Task 1 brief eksplicitno imenuje data-sourcing kao #0 (pre-blocker-1).
|
||
|
||
## 4. Task 2 — C3 Stage 2 mini execution
|
||
|
||
**Svrha:** Izvršiti C3 brief (`briefs/2026-04-22-cc-c3-stage2-mini-kickoff.md`) verbatim posle Task 1 CLOSURE, uz minimalne updates za Blocker 2 rename (cell enum novi key-ovi) i Blocker 4 judge ensemble literal (eksplicitna concrete model ID lista).
|
||
|
||
**Parametri:**
|
||
- 4 cells × 100 instances = 400 total evals
|
||
- Judge ensemble: `claude-opus-4-7,gpt-5.4,gemini-3.1` (Surface B za Opus, Surface A za GPT/Gemini per B3 addendum)
|
||
- Budget: $120-200 cap $250 per C3 brief §3
|
||
- Exit criteria: 11 per C3 brief §5 (aggregate score trend, κ ≥ 0.60, Wilson CI width ≤ 6pp per-cell, failure taxonomy distribution, et al.)
|
||
- Abort triggers: 6 per C3 brief §6 (κ collapse, cost overrun, judge availability)
|
||
|
||
**Ratification gate:** PM (Marko) ratifikuje Task 1 CLOSURE → CC-1 kickuje Task 2 per C3 brief §4 invocation template (minor flag adjustments za rename + judge literal).
|
||
|
||
**Projected outcome:** Ako PASS (per §5 thresholds), B4 final memo unlock-uje za H-42a/b execution authorization (Task 3).
|
||
|
||
## 5. Task 3 — B4 finalization + H-42a/b pre-flight gate
|
||
|
||
**Svrha:** Po C3 mini PASS-u, PM potpisuje B4 final memo kao "execution-approved" umesto "methodology intent". Dvojni dokument evolution:
|
||
|
||
1. **Update B4 memo (`decisions/2026-04-22-stage-2-full-kickoff-memo.md`):** Status "FINAL" → "EXECUTION-APPROVED". Section 2 reframe iz "zavisi od Sprint 12 Task 2" u "Task 2 PASS-ovan sa [scores/κ/CI], H-42a/b unlock authorized".
|
||
2. **Pre-flight checklist dry run:** Substrate readiness §0 re-check (per `feedback_substrate_readiness_gate.md`): grep evidence da sva Task 1 work ostaje intact (dataset hash match, enum unchanged, CLI flags functional, judge registry resolves, metrics testovi pass, event emitter live). Ako bilo koji substrate regression, R4 risk trigger-uje (per B4 memo §7 R4).
|
||
|
||
**Trajanje:** 30-60 min PM-side, 30-60 min CC-1 substrate re-check.
|
||
**Budget:** $0.
|
||
|
||
## 6. Task 4 — H-42a/b execution
|
||
|
||
**Svrha:** Pre-registration-conformant puna eval-a za SOTA-gated launch signal.
|
||
|
||
**Parametri:**
|
||
- **H-42a (Qwen primary):** 4 cells × 1540 instances × 3 judges (sve judges agreement-based per B2 tie-break policy) = 4620 evals operational count (instance-judge pairs aggregate). Estimated LLM cost $1000-1700.
|
||
- **H-42b (Opus probe):** Reduced eval (1500 instances subset per B4 memo §3) za probe signal za cross-model validation. Estimated LLM cost $300-600.
|
||
- **Total:** 6120 evals cumulative operational count, $1300-2300 LLM spend cap $2600.
|
||
|
||
**Exit criteria (per A3 LOCK §10 + B4 memo §5):**
|
||
- **STRONG-PUBLISHABLE:** Aggregate score ≥ 91.6% (Mem0 reper matched/surpassed) + κ ≥ 0.75 + Wilson CI lower bound ≥ 88% per-cell + failure taxonomy distribution consistent with Sprint 10 baseline.
|
||
- **PUBLISHABLE:** Aggregate score 88-91.5% + κ ≥ 0.70 + Wilson CI lower bound ≥ 85%.
|
||
- **WEAK:** Aggregate score 85-87.9% + κ ≥ 0.65. Launch narrative adjusts tone — not "SOTA matched" nego "competitive with published".
|
||
- **FAIL:** Aggregate score < 85% or κ < 0.60. Halt. Triggeruje post-mortem + potencijalno Sprint 13 deep-dive.
|
||
|
||
**Abort triggers:**
|
||
- Cumulative cost exceeds $2600 hard cap → HALT.
|
||
- Live κ drops below 0.60 at 25% checkpoint → HALT, post-mortem.
|
||
- Any single judge model unavailable for > 30min → HALT, retry with reduced ensemble option (PM call).
|
||
|
||
**Ratification gate:** Task 3 B4 execution-approved + substrate §0 re-check passed → CC-1 kickuje H-42a, then H-42b.
|
||
|
||
## 7. Sprint 12 total budget
|
||
|
||
- **Task 1:** $0 LLM (pure infra)
|
||
- **Task 2:** $120-200 cap $250
|
||
- **Task 3:** $0
|
||
- **Task 4:** $1300-2300 cap $2600
|
||
- **Total Sprint 12 projected spend:** $1420-2500, hard cap $2850
|
||
|
||
Budget rationale: Sprint 12 je execution-heavy anchor sprint. Unlike Sprint 11 koji je budget-conservative ($0.018893 cumulative), Sprint 12 investira u signal-gating evidence za SOTA launch claim. Cost-per-signal je ~$0.40-0.50 per eval, acceptable za pre-registration-conformant benchmark.
|
||
|
||
## 8. Sprint 12 success criteria (exit condition)
|
||
|
||
1. Task 1 CLOSED (6 blockera + 2 non-blocking + B3 addendum implementation verified).
|
||
2. Task 2 PASS per C3 brief §5 thresholds.
|
||
3. Task 3 B4 execution-approved + substrate §0 re-check clean.
|
||
4. Task 4 H-42a/b completed sa verdict u STRONG-PUBLISHABLE ili PUBLISHABLE kategoriji.
|
||
5. Sprint 12 close memo documented sa scores + κ + CI + cost ledger + next-sprint implications (Sprint 13 likely = launch narrative + brand assets finalization ako Task 4 STRONG-PUBLISHABLE).
|
||
|
||
Ako Task 4 verdict je WEAK ili FAIL, Sprint 12 close memo documentuje to evenhandedly i Sprint 13 preuzima post-mortem + remediation scope umesto launch narrative.
|
||
|
||
## 9. Risks i mitigations
|
||
|
||
**R1 — Task 1 duže nego projected (substrate issues surface kasno):** Sprint 12 extend-uje za 1-2 dana. Mitigation: #0 pre-blocker dataset sourcing check na samom kickoff-u.
|
||
|
||
**R2 — Task 2 FAIL na C3 mini:** Ne ide u Task 4 dok se uzrok ne ustanovi. Mitigation: post-mortem obavezan, potencijalni re-run sa adjusted thresholds (PM call, NE brief relaxation bez explicit LOCK revisit).
|
||
|
||
**R3 — Task 4 cost overrun:** Hard cap $2600 halt-uje pre nego što budžet eksplodira. Mitigation: live cost tracking emitter, alert na 80% threshold.
|
||
|
||
**R4 — Substrate regression između Task 1 i Task 4:** B4 memo §7 R4 mitigation (substrate §0 re-check pre Task 4 kickoff-a).
|
||
|
||
**R5 — Provider drift (κ collapse mid-run Task 4):** HALT + post-mortem. A3 LOCK §10 protokol triggeruje.
|
||
|
||
**R6 — Judge model unavailability (API outage):** Retry-with-reduced-ensemble opcija + PM call za proceed/halt decision.
|
||
|
||
## 10. Open questions za Sprint 12 kickoff brief
|
||
|
||
1. **Legacy JSONL handling (Blocker 2 rename):** Prepisivanje Sprint 10 artefakata novim key-ovima, ili `legacy_cell_name` polje za backward-compat? Hipoteza prepisivanje (Sprint 10 je pre-publication), ali PM odluka obavezna pre Task 1 kickoff-a.
|
||
2. **Canary drift-detection subset (B3 addendum §3):** Da li Sprint 12 Task 1 uvodi 50-instance canary set za drift monitoring, ili se to defer-uje u Sprint 13? Trenutni default: defer.
|
||
3. **Task 4 H-42b probe scope:** 1500 instances subset je trenutni estimate; alternative 2000 instances za tighter Wilson CI (trošak +$80-120). PM call na kickoff-u.
|
||
4. **B4 memo evolution path:** Da li update DRAFT-u ili novi "EXECUTION-APPROVED" verzioni fajl? Preporuka: update in-place jer je B4 već jedna-datoteka autoritativ.
|
||
5. **Sprint 13 preview:** Ako Task 4 STRONG-PUBLISHABLE, Sprint 13 scope je launch narrative + brand + wiki publication. Ako WEAK, Sprint 13 je remediation. Oba skica ne drafted dok Task 4 verdict nije poznat.
|
||
|
||
## 11. Related
|
||
|
||
- `decisions/2026-04-22-bench-spec-locked.md` — A3 LOCK v1 (strategy intact, Sprint 12 Task 1 backfill-uje runtime)
|
||
- `decisions/2026-04-22-bench-spec-locked.manifest.yaml` — A3 LOCK YAML twin
|
||
- `decisions/2026-04-22-stage-2-full-kickoff-memo.md` — B4 final memo, reframed to "execution-gated"
|
||
- `decisions/2026-04-22-b3-lock-dashscope-addendum.md` — B3 addendum za non-Anthropic pinning surface
|
||
- `briefs/2026-04-22-cc-c3-stage2-mini-kickoff.md` — C3 brief source (execution-deferred, ne superseded)
|
||
- `sessions/2026-04-22-c3-blocked-substrate-gap.md` — CC-1 pre-kick verification + 6-blocker taxonomy
|
||
- `sessions/2026-04-22-c3-standdown-path-c-ratified.md` — Path C verdict
|
||
- `.auto-memory/feedback_substrate_readiness_gate.md` — §0 grep evidence protokol
|
||
- `.auto-memory/project_sprint_11_c3_deferred_path_c.md` — Sprint 11 close state
|
||
|
||
---
|
||
|
||
**DRAFT — awaits PM ratification na Sprint 12 kickoff session-u. Open pitanja §10 blokiraju direct execution-LOCK; Task 1 kickoff čeka PM clarifikaciju na #1 (legacy JSONL) minimum. Remainder open pitanja mogu biti resolved inline tokom Sprint 12 tok-a.**
|