moving

docs/briefs/2026-04-28-cc4-faza1-preflight-report.md

@@ -0,0 +1,238 @@
+---
+report_id: 2026-04-28-cc4-faza1-preflight-report
+date: 2026-04-28
+session: CC-4 (fresh)
+mission: GEPA Tier 2 Prompt-Shapes Evolution Faza 1
+predecessor_brief: briefs/2026-04-28-cc4-gepa-tier2-evolution-faza1-brief.md
+status: HALT-AND-PM (3 critical, 2 minor ratifications required before NULL-baseline kick)
+authority_required: PM (Marko Markovic) ratification on §6 ratification asks
+---
+
+# CC-4 Faza 1 — Pre-Flight Report
+
+## TL;DR
+
+Pre-flight checks executed per brief §6 (8 sub-rules). **6.8 PASS, 6.5/6.6/6.7 PASS-DESIGN-READY, 6.1/6.2/6.4 PARTIAL pending manifest v7 authoring, 6.3 FAIL-AMBIGUOUS — cannot proceed without PM disambiguation**. Three additional discoveries during scan also require PM ratification before NULL-baseline kick.
+
+**Recommendation:** halt-and-PM at this checkpoint per brief §6.3 protocol. Five ratification asks below in §6. No additional code or runs prior to PM response.
+
+---
+
+## §1 — Repository topology resolved
+
+| Repo | Role | Confirmed paths used |
+|---|---|---|
+| `D:\Projects\waggle-os` | Code, benchmarks, decisions, .mind/, prompt-shapes | manifest v6, Stage 3 results, pilot 2026-04-26 data, prompt-shapes |
+| `D:\Projects\PM-Waggle-OS` | Briefs, PM coordination, sessions | brief, this report, decisions/2026-04-28-phase-4-3-rescore-delta-report.md |
+
+**Cross-repo audit chain references in brief resolve to waggle-os**, despite harness CWD = PM-Waggle-OS. CC-4 session will operate in waggle-os for code/runs and PM-Waggle-OS for briefs/reports — same dual-repo workflow as recent CC-1 sessions per Phase 4.3 verdict doc.
+
+## §2 — Brief vs reality discrepancies (factual)
+
+### 2.1 — Path error in brief §2 + §8
+
+Brief writes **`packages/core/src/prompt-shapes/`** as the GEPA evolution target. Actual location verified:
+
+```
+D:\Projects\waggle-os\packages\agent\src\prompt-shapes\
+  ├── README.md (Phase 1.2 spec — empirical evidence_link rule)
+  ├── claude.ts (4096 max_tokens, thinking on)
+  ├── qwen-thinking.ts (16000 max_tokens, thinking on) ← H3 substrate target
+  ├── qwen-non-thinking.ts (3000 max_tokens)
+  ├── gpt.ts (4096 max_tokens)
+  ├── generic-simple.ts (4096 max_tokens, fallback)
+  ├── selector.ts (model-alias → shape resolution)
+  ├── types.ts (PromptShape interface + MULTI_STEP_ACTION_CONTRACT)
+  └── index.ts (re-exports)
+```
+
+`packages/core/src/` does NOT contain prompt-shapes (verified `ls`). Brief §2 + §8 should read `packages/agent/src/prompt-shapes/`. This is a typo, not a scope change.
+
+### 2.2 — Brief assumes `feedback_config_inheritance_audit.md` exists; it doesn't
+
+Brief §6 cites this file as the source of the 8 sub-rules and §11 cross-references it as `.auto-memory/feedback_config_inheritance_audit.md`. **No such file exists in either repo** (verified `find`). The 8 sub-rules ARE listed verbatim in brief §6 itself, so functionally the rules are accessible. CC-4 should author the missing memory file (rebuild from brief contents) so future sessions inherit the rules.
+
+### 2.3 — `.auto-memory/` directory does not exist
+
+Brief §8.5 specifies post-Checkpoint-C memory entry at `.auto-memory/project_gepa_faza1_results.md`. Directory absent in both repos. CC-4 will create when authoring memory entry post Checkpoint C (no PM action needed beyond knowing the path will be created).
+
+## §3 — Pre-flight check results
+
+### 3.1 — §6.1 Config inheritance audit: PARTIAL
+
+Manifest v6 §5.2 + §5.4 explicitly specifies model strings + temperature 0.0 + max_tokens per judge:
+- claude-opus-4-7: 1024
+- gpt-5.4: 1024
+- minimax-m27 / kimi-k26: 4096
+
+Brief §3.1 mandates **max_tokens=3000 per Stage 3 v6 fix** for all judges in trio-strict scoring — **this conflicts with manifest v6 values** (1024 for Opus/GPT, 4096 for MiniMax). PM ratification needed on which max_tokens governs Faza 1 (manifest v6 inherited values vs brief override 3000).
+
+Manifest v7 must explicitly redeclare these values + Qwen reasoning_effort + per-shape model parameters. Cannot inherit implicitly.
+
+**Status:** READY-PENDING-PM-DECISION on max_tokens reconciliation.
+
+### 3.2 — §6.2 Mixed-methodology baseline: READY
+
+NULL-baseline run will report trio-strict + self-judge **separately** (not aggregate). Acceptance rule will cite trio-strict only. Compliant with rule.
+
+### 3.3 — §6.3 Scope verification (H3 ≥40 instances): **FAIL — AMBIGUOUS**
+
+This is the **critical halt trigger**. Two semantically distinct "H3 cell" interpretations:
+
+| Interpretation | Source | Available instances | Phase 4.3 anchor compatibility |
+|---|---|---|---|
+| **A. Pilot synthesis "H3 hypothesis"** = Qwen solo on task-{1,2,3}/C | `benchmarks/results/pilot-2026-04-26/pilot-task-{1,2,3}-C.jsonl` | **3 instances total** | YES — directly maps to Phase 4.3 H3 verdict (66.7% T2) |
+| **B. Stage 3 v6 LoCoMo "agentic cell"** | `benchmarks/results/agentic-locomo-2026-04-25T16-13-29-924Z.jsonl` | 400 instances | NO — Stage 3 v6 cells are no-context/oracle/full/retrieval/agentic; no "H3" label exists in v6 |
+| **C. Hybrid: generate ≥40 new synthesis instances** | NEW corpus, same NorthLane/CFO task structure as pilot | 0 today; would need authoring | YES via stratified sampling |
+
+Brief is internally inconsistent on this:
+- §2 anchors to **Phase 4.3 verdict** → implies A
+- §3.4 says "15 instances per cell sampled deterministic-stratified iz **LoCoMo full corpus**" → implies B
+- §6.3 demands ≥40 instances → only B satisfies; A fails outright (3 << 40); C requires net-new corpus authoring
+
+**Verdict:** brief §6.3 cannot pass with current corpus + Interpretation A. Brief §6.3 requires PM disambiguation before NULL-baseline kick.
+
+### 3.4 — §6.4 Cell semantic preservation: DESIGN READY
+
+Mutation validator will diff GEPA candidate vs baseline shape and reject if any of these change:
+1. `MULTI_STEP_ACTION_CONTRACT` constant in `types.ts` (touched at all → INVALID)
+2. `types.ts` interfaces (`PromptShape`, `PromptShapeMetadata`, `*Input`)
+3. `selector.ts` (registry, resolution logic)
+4. `index.ts` exports
+5. Cell-level config in manifest v7 (cells_semantics block — locked from v6)
+6. Shape file outside the 4 method bodies (`systemPrompt`, `soloUserPrompt`, `multiStepKickoffUserPrompt`, `retrievalInjectionUserPrompt`) — i.e. metadata block is also off-limits except `evidence_link` which MUST be updated to point to GEPA Gen 1 results
+
+Allowed mutation surface = the 4 method bodies' string-building only.
+
+### 3.5 — §6.5 σ-aware acceptance documented: READY
+
+N=8 binomial CI = ±17pp at 95%. +5pp threshold = fitness signal indicator only, not statistically rigorous. Will be stated explicitly in launch decision §LOCK and Checkpoint C results memo.
+
+### 3.6 — §6.6 Trio-strict primary: CONFIRMED
+
+Acceptance §4 will cite trio-strict only. Self-judge supplementary diagnostic.
+
+### 3.7 — §6.7 Cost super-linear projection: READY
+
+Will use 1.5× baseline token count for cost projection. Mid-run threshold: halt if actual cost exceeds projection by >30%. Telemetry hook will fire at every 20 evaluations.
+
+### 3.8 — §6.8 Source data structure (agentic spot-check): PASS
+
+Verified `pilot-task-1-C.jsonl` and prompt archive. Confirmed:
+- Task structure = persona (CFO of NorthLane B2B SaaS) + scenario (Q2-Q4 risk memo) + 7 source documents (P&L, pipeline, churn, eng velocity, marketing, board notes, competitor intel) + open-ended Likert-scored question
+- Format = agentic knowledge work synthesis (NOT factoid LoCoMo Q&A)
+- Output = ~5300-token CFO memo with structured action plans
+- Judge dimensions: completeness, accuracy, synthesis, judgment, actionability, structure (6-dim Likert 1-5)
+- Trio uses **trio_mean** (Likert) + **trio_strict_pass** (binary, threshold UNDOCUMENTED in pilot artifact — see §4 below)
+
+PASS on agentic format. **Open question on metric definition** (§4).
+
+## §4 — Additional discoveries requiring PM ratification
+
+### 4.1 — Metric ambiguity: "trio-strict accuracy" on Likert tasks
+
+Brief §3.1 says fitness = "trio-strict accuracy (Opus 4.7 + GPT-5.4 + MiniMax M2.7 ensemble, 2/3 must agree, max_tokens=3000)".
+
+For LoCoMo binary correctness this is unambiguous (2 of 3 judges return correct=true → trio_strict).
+
+For pilot synthesis Likert, "agreement" is undefined. Two operationalization candidates:
+- **(i)** trio_strict_pass = ≥2 of 3 judge_means ≥ 4.0 (binary on per-judge mean)
+- **(ii)** trio_strict_pass = trio_mean ≥ threshold T (single binary on aggregate; T = 4.0 candidate)
+
+Pilot data already contains `trio_strict_pass` field (sample shows `true` for trio_mean=4.583 with judge_minimax failed). This implies operationalization (ii) with T probably = 4.0 (sample value 4.583 ≥ 4.0 = pass). **PM ratification needed on T value + which operationalization.**
+
+### 4.2 — Canonical κ baseline (brief §4) source
+
+Brief §4 condition 3: "Trio judge κ remains within ±0.05 of canonical 0.7878". 
+
+Manifest v6 §5.4 specifies:
+- pass_trio_kappa_gte: 0.70
+- borderline: [0.60, 0.70]
+- fail: <0.60
+
+Stage 3 v6 N=400 final-memo or kappa-recal artifact may carry the actual measured value 0.7878 — need to verify source. Brief value 0.7878 is plausibly the Phase 1 κ re-cal result. **PM cite needed** so manifest v7 can pin the canonical reference + audit chain.
+
+### 4.3 — Path correction authorization
+
+Brief §2 + §8 reference `packages/core/src/prompt-shapes/`. Actual = `packages/agent/src/prompt-shapes/`. **Authorize CC-4 to use actual path in manifest v7 + decisions + GEPA outputs?** (Recommended: yes, treat as typo correction, no scope change.)
+
+### 4.4 — feedback_config_inheritance_audit.md authorization
+
+File missing. Should CC-4 reconstruct from brief §6 verbatim and persist at `.auto-memory/feedback_config_inheritance_audit.md` in waggle-os? (Recommended: yes, as audit infrastructure.)
+
+### 4.5 — Substrate freeze verification
+
+Brief §3.5: "GEPA radi nad post-Phase 4.6 HEAD". Manifest v6 §11 freezes HEAD at `373516c`. Phase 4.3 verdict cites HEAD `c9bda3d` (Phase 4.7). **Branch is feature/c3-v3-wrapper.**
+
+CC-4 needs to verify current HEAD on this branch matches expectation (post-Phase-4.6, NOT post any Phase 5+ work). Quick check planned post-PM-ratify (single `git rev-parse HEAD` + `git log --oneline | head -5`).
+
+## §5 — Pre-flight check matrix summary
+
+| Check | ID | Status | Blocker? |
+|---|---|---|---|
+| Config inheritance | 6.1 | PARTIAL (max_tokens reconciliation needed) | NO (resolved in manifest v7) |
+| Mixed-methodology baseline | 6.2 | READY | NO |
+| **Scope verification (≥40 H3)** | **6.3** | **FAIL — AMBIGUOUS** | **YES** |
+| Cell semantic preservation | 6.4 | DESIGN READY | NO |
+| σ-aware acceptance | 6.5 | READY | NO |
+| Trio-strict primary | 6.6 | CONFIRMED | NO |
+| Cost super-linear | 6.7 | READY | NO |
+| Source data agentic format | 6.8 | PASS | NO |
+
+## §6 — Ratification asks (in order — A is critical path blocker)
+
+| # | Ask | Recommended option | Blocks |
+|---|---|---|---|
+| **A** | Disambiguate "H3 cell" semantics | C: generate ≥40 new synthesis instances using NorthLane-style task family (preserves Phase 4.3 anchor + satisfies §6.3) — adds 2-3 hours pre-work + small subject-LLM cost (~$5) | NULL-baseline kick |
+| **B** | Define `trio_strict_pass` operationalization for Likert synthesis | (ii) trio_mean ≥ T with T ratified explicitly (rec T=4.0 based on pilot sample) | NULL-baseline kick |
+| **C** | Confirm canonical κ value 0.7878 source | Cite Stage 3 v6 Phase 1 κ re-cal artifact path or override with actual measured value | manifest v7 LOCK |
+| **D** | Authorize path correction (`packages/core/` → `packages/agent/`) | YES (typo) | manifest v7 LOCK |
+| **E** | Authorize `.auto-memory/feedback_config_inheritance_audit.md` reconstruction | YES (audit infra) | optional, not blocker |
+
+If PM ratifies A as Option C (corpus expansion):
+- Sub-ask: target N for new H3 corpus = 50? (8 NULL + 24 Gen 1 + 5 held-out + 13 buffer = 50, comfortable margin over §6.3 ≥40)
+- Sub-ask: subject model for instance generation = Opus 4.7? (consistent with mutation oracle)
+- Sub-ask: stratification axes (task type / persona / domain)? Recommended: 5 task families × 10 instances each, mirroring pilot's task-1/task-2/task-3 structure.
+
+If PM ratifies A as Option B (LoCoMo agentic): Faza 1 disconnects from Phase 4.3 verdict; would need brief addendum reframing the rationale.
+
+If PM ratifies A as Option A (proceed with 3 instances): would violate brief §6.3 — would need brief amendment relaxing the threshold for Faza 1 specifically. Not recommended.
+
+## §7 — Cost & wall-clock impact of ratifications
+
+| Option | Pre-work cost | Pre-work wall-clock | Faza 1 wall-clock impact |
+|---|---|---|---|
+| A: Option C corpus expansion | ~$5 (50 synthesis-task generations × Opus 4.7) | ~2-3h CC time | +1 day total |
+| A: Option B LoCoMo pivot | $0 | 0 | -0.5 day (faster, 400 instances ready) |
+| A: Option A relax threshold | $0 | 0 | 0 (immediate kick possible) |
+| B+C+D+E | $0 | ~30 min CC time | 0 |
+
+## §8 — Status post-ratification → next moves
+
+Upon receiving PM ratification on asks A-E:
+1. CC-4 executes corpus expansion (if Option C) — gated by ratification
+2. CC-4 authors `manifest-v7-gepa-faza1.yaml` with explicit max_tokens reconciliation, κ baseline pin, path correction
+3. CC-4 authors `decisions/2026-04-28-gepa-faza1-launch.md` (LOCK on session start) per brief §8.3
+4. CC-4 reconstructs `feedback_config_inheritance_audit.md` (if E ratified)
+5. CC-4 verifies substrate HEAD on feature/c3-v3-wrapper
+6. CC-4 builds GEPA harness scaffold + tests (≥80% coverage)
+7. CC-4 kicks NULL-baseline → Checkpoint A halt
+
+No code authoring or LLM API calls before PM ratification.
+
+---
+
+## Audit chain
+
+| Item | Value |
+|---|---|
+| Pre-flight session date | 2026-04-28 |
+| Brief read | briefs/2026-04-28-cc4-gepa-tier2-evolution-faza1-brief.md (266 lines) |
+| Phase 4.3 verdict read | decisions/2026-04-28-phase-4-3-rescore-delta-report.md (172 lines) |
+| Stage 3 v6 5-cell summary read | benchmarks/results/stage3-n400-v6-final-5cell-summary.md (76 lines) |
+| Manifest v6 read | benchmarks/preregistration/manifest-v6-preregistration.yaml (688 lines) |
+| Prompt-shapes README + 5 shape files + selector + types read | packages/agent/src/prompt-shapes/ (verified inventory) |
+| Pilot-2026-04-26 sample data + prompt archive read | pilot-task-1-C.jsonl + prompts-archive/task-1-cell-C-prompt.md |
+| Pre-flight session cumulative cost | $0 (no LLM calls; only file reads) |
+
+**End of pre-flight report. Standing AWAITING PM ratification on §6 asks A-E before proceeding to manifest v7 authoring + NULL-baseline kick.**