266 lines
14 KiB
Markdown
266 lines
14 KiB
Markdown
# CC-4 Brief — GEPA Tier 2 Prompt-Shapes Evolution (Faza 1 pilot)
|
||
|
||
**Date:** 2026-04-28
|
||
**Author:** PM
|
||
**Status:** Authored, awaiting Marko ratification before paste-into-CC-4
|
||
**Sesija type:** CC-4 fresh (paralelno sa CC-1 Phase 4.4/4.5 + CC-3 memory shims)
|
||
**Critical path:** Korak 1 → GEPA Faza 1 → Faza 2 (gated) → Phase 5 GEPA-evolved
|
||
**Cost cap:** $100 hard, halt at $80
|
||
**Wall-clock projection (not trigger):** 3–5 dana CC time, ~2–3 dana wall-clock if no rate-limit blockers
|
||
|
||
---
|
||
|
||
## §1 — Context (zašto smo ovde)
|
||
|
||
Phase 4.3 verdikt (CC-1, commit be8f702→c9bda3d) empirijski potvrdio da H3/H4 fail je **72.2% Tier 2** (reasoning/planning failure koji zahteva prompt evoluciju), samo 5.6% Tier 1 (presentation artifact koji bi se popravio Phase 1.1 normalize). Phase 1.1 normalize delta = 0% kroz svih 12 cells. Worst-case T1 ceiling = 27.8%, ispod 30% threshold-a koji bi nam dao paper claim #2 multiplier signal. H4 je 100% T2 unanimnost.
|
||
|
||
Implikacija: agent fix sprint Phase 1-4 sam, ma koliko ga doteramo, ne može da spase multiplier tezu. Treba reasoning/planning evolucija nad prompt-ima — što je GEPA (Agrawal et al., genetic evolutionary prompt adaptation).
|
||
|
||
Ovo je gated work: Faza 1 = pilot proof-of-concept (1 cell, $100 cap). Ako Faza 1 PASS → Faza 2 expansion (sve T2-saturated cells, $200-300 cap, posebna ratifikacija). Ako Faza 1 FAIL → fallback na PHF (PASS-with-honest-framing) per Decision Matrix amendment.
|
||
|
||
---
|
||
|
||
## §2 — Scope LOCK (Faza 1)
|
||
|
||
**Što GEPA evoluira:** prompt-shapes templates u `packages/core/src/prompt-shapes/`. **NE** evoluira system prompts (cell semantics) — ti ostaju lock-ovani na manifest v6 specifikaciju da očuvamo apples-to-apples kontrolu sa Stage 3 v6 N=400 results.
|
||
|
||
**Razlog:** prompt-shapes evolucija je niži rizik za leakage cross-cell, evolution boundary je čist + auditable. System prompt evolucija bi compoundova confounders i ugrozila reproducibility paper claim #1 substrate (74% > Mem0 66.9%).
|
||
|
||
**Cell scope Faza 1:** **H3 only** (najjača T2 saturacija per Phase 4.3 verdict). H2 + H4 ulaze u Fazu 2 ako Faza 1 PASS. Zašto H3 prvi: maksimalni signal-to-cost ratio za "does GEPA help at all" gate.
|
||
|
||
**Prompt shapes scope:** svih 5 (claude / qwen-thinking / qwen-non-thinking / gpt / generic-simple). GEPA evoluira **per-shape**, ne unified. Selection metric = best-per-shape-per-cell (ne aggregate).
|
||
|
||
**N per evaluation:** 15 instanci per candidate per generation (per shape per cell). Insufficient za publishable paper claim, **dovoljno za GEPA fitness signal** (per Agrawal paper — fitness signal stabilizes around N=10-20 in inner loop).
|
||
|
||
**Generations:** 2 (initial population + 1 mutation round). Faza 1 = proof-of-concept, ne convergence search. Faza 2 expansion može da poveća na 3-5 generations.
|
||
|
||
**Population:** 3 candidates per shape per cell. Initial population = current shape (baseline) + 2 LLM-generated mutations (Opus 4.7 as mutation oracle).
|
||
|
||
---
|
||
|
||
## §3 — Methodology
|
||
|
||
### 3.1 — Fitness function
|
||
|
||
Composite score per candidate:
|
||
- **Primary:** trio-strict accuracy (Opus 4.7 + GPT-5.4 + MiniMax M2.7 ensemble, 2/3 must agree, max_tokens=3000 per Stage 3 v6 fix)
|
||
- **Cost penalty:** −0.5pp per $0.10 cost above baseline median (encourages efficiency)
|
||
- **Tie-breaker:** trio-soft accuracy (any 1/3 agreement)
|
||
|
||
Aggregate fitness per candidate = trio-strict − cost_penalty.
|
||
|
||
### 3.2 — NULL-baseline gate (binding)
|
||
|
||
Pre Faza 1 GEPA run, CC-4 mora da reproducira **NULL-baseline** = current prompt-shape on H3 cell, N=15, trio-judged. Ovo lockuje fitness floor pre evolucije.
|
||
|
||
Acceptance: NULL-baseline trio-strict mora pasti u predikcionom range-u iz Phase 4.3 (h3 cell est. 18-24% trio-strict baseline). Ako NULL-baseline ispod 15% ili iznad 30%, halt-and-PM (signal da nešto fundamentally drift-ovalo između Phase 4.3 i sad).
|
||
|
||
### 3.3 — Mutation oracle
|
||
|
||
Opus 4.7 generates 2 mutations per shape per generation, prompted with:
|
||
- Current shape template
|
||
- Failure mode summary (top-3 T2 failures from Phase 4.3 categorization)
|
||
- Constraint: preserve cell semantics (system prompt boundaries, output format contract)
|
||
- Mutation guidance: "modify reasoning scaffold, planning step structure, or chain-of-thought triggers — do not modify task framing or scoring criteria"
|
||
|
||
### 3.4 — Reproducibility
|
||
|
||
- **Seed:** GEPA evolution algorithm seeded with `42` (configurable in manifest v7)
|
||
- **Selection set:** 15 instances per cell sampled deterministic-stratified iz LoCoMo full corpus (same stratification as Stage 3 v6 manifest)
|
||
- **Manifest v7:** new manifest extending v6 sa GEPA section (population seed, mutation oracle SHA, generation count, candidate hashes)
|
||
- **Audit trail:** every candidate prompt hashed (SHA256), every evaluation logged sa raw judge outputs
|
||
|
||
### 3.5 — Substrate dependency
|
||
|
||
GEPA radi nad **post-Phase 4.6 HEAD** (mind/ + harness sa svih Phase 1-4 fix-eva integrated). Substrate version stays v6 (zero changes). Manifest v7 audit chain references manifest v6 anchor SHA.
|
||
|
||
---
|
||
|
||
## §4 — Acceptance criteria (Faza 1 → Faza 2 gate)
|
||
|
||
Faza 1 PASS conditions (binding, all 4 must hold):
|
||
|
||
1. **Best GEPA candidate per shape beats NULL-baseline by ≥ +5pp on trio-strict** (per H3 cell)
|
||
2. **At least 3/5 shapes show positive delta** (avoids cherry-picking single shape that lucked out)
|
||
3. **Trio judge κ remains within ±0.05 of canonical 0.7878** (validates judge ensemble didn't drift mid-run)
|
||
4. **Zero cell semantic violations detected** (audit step §6.4)
|
||
|
||
Faza 1 FAIL conditions (any single condition triggers FAIL verdict):
|
||
|
||
- Best candidate delta < +5pp on majority shapes
|
||
- Best candidate beats NULL-baseline only by overfitting evaluation set (detected via held-out 5 instances per cell)
|
||
- κ drift > 0.05 (judge ensemble unreliable)
|
||
- Cell semantic violation found
|
||
|
||
If FAIL → fallback PHF, GEPA work parked, paper claim #2 multiplier tezu reframe-uje na "demonstrated only under V2 retrieval + Tier 1 normalize, GEPA insufficient at this scope" (acceptable academic framing, ne mora se hide).
|
||
|
||
---
|
||
|
||
## §5 — Cost & halt
|
||
|
||
**Faza 1 cost projection (rigorous, not trigger):**
|
||
- NULL-baseline: 5 shapes × 15 instances × $0.50/run × 1 cell = **$37.50**
|
||
- GEPA Gen 1: 5 shapes × 3 candidates × 15 instances × $0.50 × 1 cell = **$112.50**
|
||
- Held-out validation: 5 shapes × top-1 × 5 instances × $0.50 = **$12.50**
|
||
|
||
Wait — that exceeds $100 cap. Recalc with Gen 1 reduced:
|
||
|
||
**Revised Faza 1 with $100 cap:**
|
||
- NULL-baseline: 5 shapes × 10 instances × $0.50 × 1 cell = **$25**
|
||
- GEPA Gen 1: 5 shapes × 3 candidates × 10 instances × $0.50 × 1 cell = **$75**
|
||
- Held-out: 5 shapes × top-1 × 5 instances × $0.50 = **$12.50**
|
||
|
||
**Total: ~$112** — still over. Final cut:
|
||
|
||
**Faza 1 LOCKED scope ($100 cap, $80 halt):**
|
||
- NULL-baseline: 5 shapes × **8 instances** × $0.50 × 1 cell = **$20**
|
||
- GEPA Gen 1: 5 shapes × 3 candidates × **8 instances** × $0.50 × 1 cell = **$60**
|
||
- Held-out: top-1 per shape × 5 instances × $0.50 = **$12.50**
|
||
|
||
**Total: $92.50 expected, $100 hard cap.** N=8 per evaluation is at lower bound of GEPA fitness signal — acceptable for proof-of-concept Faza 1, NOT acceptable for Faza 2 expansion (Faza 2 scales N to 20+).
|
||
|
||
**Halt triggers (any single triggers immediate halt + PM ratify):**
|
||
- Cumulative spend > $80
|
||
- κ drift detected mid-run (judge sample audit every 20 calls)
|
||
- Cell semantic violation detected (audit § 6.4)
|
||
- Mutation oracle (Opus 4.7) returns 2 consecutive invalid mutations (e.g. mutates system prompt instead of shape)
|
||
- Any LLM API blocker (rate-limit cascade, auth failure) — halt, restart with diagnostic pre-flight
|
||
|
||
---
|
||
|
||
## §6 — Pre-flight checks (binding, all 8 sub-rules from feedback memory)
|
||
|
||
Per `feedback_config_inheritance_audit.md` Extensions 1-6 + sub-rules 6-8, CC-4 mora da verifikuje pre run:
|
||
|
||
### 6.1 — Config inheritance audit
|
||
Eksplicitno specify Qwen + Opus + GPT + MiniMax model strings + reasoning_effort + max_tokens u manifest v7. Ne nasleduj iz manifest v6 implicitly. Naročito: Qwen reasoning mode (thinking vs non-thinking) MUST match per-shape configuration (qwen-thinking shape → reasoning enabled, qwen-non-thinking → disabled).
|
||
|
||
### 6.2 — Mixed-methodology baseline
|
||
NULL-baseline mora prijavljivati **trio-strict + self-judge razdvojeno** (ne shared aggregate). Phase 4.3 koristi trio-strict; pisanje "GEPA delta" mora citirati trio-strict numbers, ne self-judge.
|
||
|
||
### 6.3 — Scope verification
|
||
Pre run, CC-4 verifikuje da H3 cell ima ≥40 instanci u source corpus (potrebno za 8 NULL + 24 GEPA + 5 held-out = 37 instances + buffer). Ako H3 ima <40 instanci, halt-and-PM (signal da scope estimate pogrešan).
|
||
|
||
### 6.4 — Cell semantics prompt strictness preservation
|
||
Audit step pre commit Faza 1 results: za svaki GEPA candidate prompt, diff vs baseline. Diff mora biti samo unutar prompt-shape template body (between defined boundaries u shape file). Diff koji touch-uje cell.system_prompt ili cell.scoring_rubric = automatic INVALID, candidate dropped, mutation oracle re-prompted.
|
||
|
||
### 6.5 — σ-aware acceptance range
|
||
N=8 per cell daje cca CI ± 17pp at 95% (binomial), što je široko. **+5pp acceptance threshold je ne-statistički-rigorozan na N=8** — uzima se kao **fitness signal indicator**, ne kao publishable claim. To je razlog zašto Faza 1 = proof-of-concept, ne paper-ready evidence. Faza 2 scale-up je tek tu za publishable σ-bounded delta.
|
||
|
||
### 6.6 — Mixed-methodology variant
|
||
Trio-strict je primary; self-judge je supplementary diagnostic only. Faza 1 acceptance rule (§4) bazira se na trio-strict, ne self-judge.
|
||
|
||
### 6.7 — Cost super-linear input growth
|
||
GEPA candidates have variable token length (mutations may grow prompts). Cost calculation must use **worst-case 1.5× baseline token count** per candidate (encodes mutation overhead). If actual mid-run cost exceeds projection by >30%, halt.
|
||
|
||
### 6.8 — Source data structure
|
||
Verify H3 source data is **agentic knowledge work format** (not factoid LoCoMo). Phase 4.3 categorization confirms H3 = agentic. CC-4 spot-check 3 random H3 instances pre run, confirm task structure matches pilot 2026-04-26 corpus.
|
||
|
||
---
|
||
|
||
## §7 — Halt-and-PM checkpoints
|
||
|
||
Faza 1 ima 3 mandatory halt-and-PM points:
|
||
|
||
**Checkpoint A (post NULL-baseline, $25 cumulative):**
|
||
- Report NULL-baseline results per shape
|
||
- Confirm trio-strict in 18-24% range per shape
|
||
- Confirm κ within ±0.05 of canonical
|
||
- PM authorize GEPA Gen 1 kick
|
||
|
||
**Checkpoint B (mid-Gen 1, $50 cumulative):**
|
||
- Report intermediate κ from first 30 evaluations
|
||
- Report any cell semantic violation
|
||
- Report mutation oracle behavior (valid mutation rate)
|
||
- PM authorize completion of Gen 1
|
||
|
||
**Checkpoint C (post Gen 1 + held-out, ~$92 cumulative):**
|
||
- Final results per shape (NULL-baseline vs best GEPA candidate)
|
||
- κ stability report
|
||
- Acceptance rule (§4) verdict
|
||
- PM authorize either Faza 2 expansion OR FAIL fallback PHF
|
||
|
||
---
|
||
|
||
## §8 — Deliverables
|
||
|
||
1. **Code:**
|
||
- `packages/core/src/prompt-shapes/gepa-evolved/` — directory sa best-per-shape candidates (5 files)
|
||
- `packages/core/src/prompt-shapes/gepa-evolved/manifest.json` — selection metadata + audit chain
|
||
- `benchmarks/gepa/faza-1/` — run logs + raw judge outputs + κ audit + diff snapshots
|
||
|
||
2. **Manifest v7:**
|
||
- `benchmarks/preregistration/manifest-v7-gepa-faza1.yaml`
|
||
- Extends v6 sa GEPA section (seed, oracle SHA, candidate hashes, generation count)
|
||
- SHA256 logged u Checkpoint C report
|
||
|
||
3. **Decisions:**
|
||
- `decisions/2026-04-28-gepa-faza1-launch.md` (LOCK upon paste-into-CC-4)
|
||
- `decisions/2026-04-XX-gepa-faza1-results.md` (post-Checkpoint C)
|
||
|
||
4. **Test coverage:**
|
||
- GEPA selection logic unit-tested (≥80% coverage)
|
||
- Mutation validator (cell semantic check) unit-tested
|
||
- κ audit utility unit-tested
|
||
|
||
5. **Memory entry:**
|
||
- `.auto-memory/project_gepa_faza1_results.md` post Checkpoint C
|
||
|
||
---
|
||
|
||
## §9 — Out-of-scope (Faza 1)
|
||
|
||
Explicitly NOT in Faza 1:
|
||
|
||
- H2 + H4 cells (Faza 2 expansion)
|
||
- More than 2 GEPA generations
|
||
- Population > 3 candidates per shape
|
||
- N > 8 per evaluation
|
||
- System prompt evolucija (locked by §2 scope)
|
||
- Substrate (mind/) modifications (locked by §3.5)
|
||
- Apples-to-apples re-eval against pilot 2026-04-26 (separate Korak 12 work)
|
||
- Paper §5.4 framing update (post Phase 5 GEPA-evolved variant complete)
|
||
|
||
---
|
||
|
||
## §10 — Sequencing
|
||
|
||
**Predicates (must be done before CC-4 starts):**
|
||
- CC-1 Phase 4.3 verdict ratified ✅
|
||
- PM brief landed in `briefs/` ✅ (this file)
|
||
- Marko ratifikuje + executes paste-into-CC-4
|
||
|
||
**Successors (depend on Faza 1 outcome):**
|
||
- Faza 1 PASS → Faza 2 expansion brief authoring → CC-4 sledeća sesija
|
||
- Faza 1 PASS → Phase 5 GEPA-evolved variant brief (CC-1 sesija, post NULL-baseline)
|
||
- Faza 1 FAIL → PHF fallback decision memo + paper §5.4 reframe
|
||
|
||
**Parallel (CC-1 + CC-4 + CC-3):**
|
||
- CC-1: Phase 4.4 (skills sweep) → 4.5 (tools sweep) → Phase 5 NULL-baseline
|
||
- CC-4: Faza 1 GEPA pilot
|
||
- CC-3: Memory shims monorepo (Wave 1.3 next)
|
||
|
||
No code path conflicts between CC-1 and CC-4 (CC-1 touches harness around prompt-shapes, CC-4 produces new files in `gepa-evolved/` subdir).
|
||
|
||
---
|
||
|
||
## §11 — Cross-references
|
||
|
||
- Phase 4.3 results: `decisions/2026-04-28-phase-4-3-rescore-delta-report.md`
|
||
- Manifest v6 anchor: `benchmarks/preregistration/manifest-v6-preregistration.yaml`
|
||
- Stage 3 v6 5-cell summary: `D:\Projects\waggle-os\benchmarks\results\stage3-n400-v6-final-5cell-summary.md`
|
||
- Pilot 2026-04-26 result: `decisions/2026-04-26-pilot-verdict-FAIL.md`
|
||
- Decision Matrix PHF amendment: `decisions/2026-04-26-decision-matrix-self-judge-reframe.md`
|
||
- arxiv §5.4 multiplier framing: `research/2026-04-26-arxiv-paper/01-paper-skeleton.md`
|
||
- Feedback memory rules: `.auto-memory/feedback_config_inheritance_audit.md`
|
||
- 14-step launch plan: `.auto-memory/project_launch_plan_14_step_2026_04_27.md`
|
||
|
||
---
|
||
|
||
## §12 — Open questions for Marko
|
||
|
||
1. **Cost cap $100** — OK ili treba hard $80? PM rec $100 sa $80 internal halt.
|
||
2. **GEPA Faza 2 escalation budget** — predaj sad ili odluči post-Faza-1? PM rec post-Faza-1 (gated decision).
|
||
3. **Mutation oracle = Opus 4.7** — OK ili koristimo Sonnet 4.6 za cost reduction? PM rec Opus 4.7 (better mutation quality justifies cost; only 5×3×2 = 30 mutation calls total).
|
||
4. **Wall-clock priority** — paralelno sa CC-1 sweep ili sequential? PM rec paralelno (no conflict).
|