175 lines
8.1 KiB
Markdown
175 lines
8.1 KiB
Markdown
# CC Brief — Sesija C: Gaia2 ARE Setup + GEPA Dry Verification
|
||
|
||
**Brief ID:** `cc-sesija-c-gaia2-setup-dry-verification-v1`
|
||
**Date:** 2026-04-30
|
||
**Author:** PM
|
||
**Status:** LOCKED (Marko ratifikovao 2026-04-30 "sve yes potvrdjeno", benchmark portfolio refresh ratification ask #1 = YES)
|
||
**Stream:** CC Sesija C (paralelno sa Sesija A + Sesija B)
|
||
**Branch:** Kreirati `feature/gaia2-are-setup` iz `main` (ne zavisi od Sesija A ili B grana)
|
||
**Wall-clock:** 1-2 dana CC implementation (projection NOT trigger)
|
||
**Cost cap:** $15 hard / $10 halt / $5-8 expected (dry run only, no full benchmark)
|
||
**Authority chain:**
|
||
- `decisions/2026-04-30-pre-launch-sprint-consolidation-LOCKED.md`
|
||
- `briefs/2026-04-29-benchmark-portfolio-refresh-2026-venues.md` (§2.1 Gaia2 + §2.3 ERL)
|
||
- Faza 1 closure (substrate evidence): `decisions/2026-04-29-gepa-faza1-results.md`
|
||
|
||
---
|
||
|
||
## §0 — Pre-flight gates (BLOCKING — must PASS before §1)
|
||
|
||
### §0.1 — Gaia2 ARE platform availability
|
||
|
||
CC mora dokumentovati u `gaia2-setup-evidence.md`:
|
||
|
||
1. ARE platform repo accessible: `https://github.com/facebookresearch/meta-agents-research-environments`
|
||
2. Gaia2 paper anchor: arxiv 2602.11964 (Froger et al., 12 Feb 2026)
|
||
3. Gaia2 dataset license verified (research use OK za dry run + future Phase 3 sprint)
|
||
4. Gaia2 Search split task count verified (target N=50-100 tasks za dry run, full N=200+ deferred za Phase 3)
|
||
|
||
### §0.2 — GEPA-evolved variants accessible
|
||
|
||
1. claude::gen1-v1 + qwen-thinking::gen1-v1 shape definitions reachable u `packages/agent/src/prompt-shapes/` ili monorepo migrated location
|
||
2. `registerShape` canonical API working (Amendment 8 native)
|
||
3. runRetrievalAgentLoop accessible iz Gaia2 harness adapter
|
||
|
||
### §0.3 — Cost projection probe
|
||
|
||
3-request dry run probe sa Gaia2 sample tasks, beleziš per-request cost p50/p95. Compute total dry run projection. Halt-and-PM ako probe-validated total > $10.
|
||
|
||
---
|
||
|
||
## §1 — Scope declaration
|
||
|
||
CC setup-uje **Gaia2 ARE platform lokalno** + verifikuje da GEPA-evolved variants rade na Gaia2 Search split bez harness modifikacije + autoring ERL methodology integration plan u `retrieval-agent-loop.ts`. Output je preparation za post-launch Phase 3 sprint Week 4-8.
|
||
|
||
**Scope LOCKED:**
|
||
- ARE platform install (Python venv ili Docker)
|
||
- Gaia2 dataset download + preprocessing
|
||
- Adapter sloj koji wraps Gaia2 task → runRetrievalAgentLoop call sa selected shape
|
||
- Dry run N=10-20 tasks na Gaia2 Search split (subset, ne full)
|
||
- ERL methodology integration plan dokumentacija (kod ne implementira u ovom briefu)
|
||
|
||
**Out of scope (post-launch Phase 3 sprint):**
|
||
- Full N=200 Gaia2 Search + Execution split run
|
||
- ReAct baseline vs ERL-augmented A/B comparison
|
||
- Trio-strict + self-judge dual reporting
|
||
- arxiv submission ka MemAgents Workshop
|
||
|
||
---
|
||
|
||
## §2 — Implementation plan
|
||
|
||
### §2.1 — ARE platform install (Day 1 morning)
|
||
|
||
**Task C1:** Clone `facebookresearch/meta-agents-research-environments` u `D:/Projects/waggle-os/external/meta-agents-research-environments/` (or external/ submodule). Follow ARE installation README (Python venv, dependencies, dataset download).
|
||
|
||
**Task C2:** Verify ARE platform smoke test prema platform's own quick-start guide. Run their default agent on 1-2 sample Gaia2 tasks, verify expected output format.
|
||
|
||
### §2.2 — Adapter za GEPA-evolved variants (Day 1 afternoon)
|
||
|
||
**Task C3:** Adapter sloj `D:/Projects/waggle-os/benchmarks/gaia2/adapter.ts` koji:
|
||
- Loads Gaia2 task definicije iz dataset
|
||
- Wraps task za runRetrievalAgentLoop call
|
||
- Selects shape (`claude::gen1-v1` ili `qwen-thinking::gen1-v1`) iz config
|
||
- Captures response + write-action verifier output
|
||
- Logs to JSONL `D:/Projects/waggle-os/benchmarks/gaia2/runs/<ISO_date>/`
|
||
|
||
**Task C4:** Configuration file `D:/Projects/waggle-os/benchmarks/gaia2/config.yaml` sa:
|
||
- task_count_dry_run: 10-20
|
||
- shapes: ["claude::gen1-v1", "qwen-thinking::gen1-v1", "claude::base", "qwen-thinking::base"]
|
||
- baseline_shape: "claude::base" (control)
|
||
- judge_methodology: "self-judge-dry-run" (full trio-strict deferred Phase 3)
|
||
- cost_cap: 10
|
||
- halt_trigger: 8
|
||
|
||
### §2.3 — Dry run execution (Day 1 evening — Day 2 morning)
|
||
|
||
**Task C5:** Run 4 dry run scenarios:
|
||
1. claude::base baseline (10 tasks)
|
||
2. claude::gen1-v1 GEPA-evolved (10 tasks)
|
||
3. qwen-thinking::base baseline (10 tasks)
|
||
4. qwen-thinking::gen1-v1 GEPA-evolved (10 tasks)
|
||
|
||
Total 40 task invocations. Cost projection ~$5-8.
|
||
|
||
**Task C6:** Beleziš:
|
||
- Per-shape Pass@1 rate na 10-task subset
|
||
- Per-task cost (input + output tokens)
|
||
- Per-task latency
|
||
- Failure modes (loop_exhausted, timeout, parse_fail, judge_failure)
|
||
|
||
### §2.4 — ERL methodology integration plan (Day 2)
|
||
|
||
**Task C7:** ERL paper review (`arxiv:2603.24639`). Document u `benchmarks/gaia2/erl-integration-plan.md`:
|
||
- ERL methodology summary (retrieval of heuristics from accumulated experience)
|
||
- Integration point u Waggle: `packages/agent/src/retrieval-agent-loop.ts` (38.3 KB file koji već radi adjacent work)
|
||
- Heuristic source: hive-mind frame store (use I/P/B distinction, importance weighting)
|
||
- Injection point: agent system prompt enrichment pre execution
|
||
- Acceptance criteria za Phase 3 sprint: ERL-augmented variant +5pp Pass@1 over ReAct baseline na Gaia2 Search split
|
||
|
||
**Task C8:** ERL methodology poređenje sa Waggle native retrieval. Document differences:
|
||
- ERL retrieves "transferable heuristics" iz experience
|
||
- Waggle retrieves "facts/decisions/insights/tasks" frames sa importance scoring
|
||
- Mapping: Waggle insight frames + decision frames ≈ ERL heuristics
|
||
- Hypothesis: Waggle's bitemporal-KG-conditioned retrieval = ERL extension, ne replication
|
||
|
||
### §2.5 — Cost validation + final acceptance (Day 2)
|
||
|
||
**Task C9:** Cost reconciliation. Total spent vs projection. Acceptance: actual cost < $10 hard cap.
|
||
|
||
**Task C10:** Dry run results memo `benchmarks/gaia2/dry-run-results-memo.md`. Format:
|
||
- Per-shape Pass@1 (4 shapes × 10 tasks)
|
||
- Cost per task per shape
|
||
- Failure modes distribution
|
||
- Comparison sa Faza 1 in-sample evidence (na N=13 GEPA validation)
|
||
- Disposition: ako dry run signal je consistent sa Faza 1 (+12.5pp lift), Phase 3 sprint Week 4-8 kick-off authorized post-launch
|
||
|
||
**Task C11:** Final commit + emit "PHASE 5 SESIJA C COMPLETE — Gaia2 ARE setup + GEPA dry verification done, ERL integration plan authored, ready za post-launch Phase 3 sprint". Push grana origin.
|
||
|
||
---
|
||
|
||
## §3 — Halt-and-PM triggers
|
||
|
||
- §0 sub-gate FAIL (ARE platform inaccessible, GEPA shapes not loadable)
|
||
- Cost overshoot >$8 (halt) ili >$10 (hard cap)
|
||
- Discovery van scope-a (Gaia2 task format incompatible sa runRetrievalAgentLoop input expectations)
|
||
- Dry run signal contradicts Faza 1 evidence (npr. GEPA-evolved variants underperform baseline na Gaia2 — would require investigation pre Phase 3 sprint)
|
||
|
||
---
|
||
|
||
## §4 — Acceptance criteria (sve PASS pre §2.5 close)
|
||
|
||
1. ARE platform installed lokalno + smoke test pass
|
||
2. Adapter sloj funkcionalan, integrates Gaia2 task → runRetrievalAgentLoop
|
||
3. Dry run N=40 tasks complete (4 shapes × 10 tasks)
|
||
4. JSONL logs presented sa per-task evidence
|
||
5. Cost validation: actual < $10
|
||
6. ERL integration plan dokumentovan (8 sekcija minimum)
|
||
7. Dry run results memo emit-uje go/no-go signal za Phase 3 sprint
|
||
8. Tests passing (no regression u postojećim test suite)
|
||
9. Commit + push grana
|
||
|
||
---
|
||
|
||
## §5 — Cross-stream dependencies
|
||
|
||
**Sesija A + Sesija B — paralelno, no blocker:** Sesija C ne dotice se apps/web ili monorepo migration. Independent stream.
|
||
|
||
**Post-launch Phase 3 sprint (Week 4-8 per benchmark portfolio brief §5):** Build na Sesija C output. Pre-existing setup eliminira ~1 dan overhead u Phase 3 sprint Week 4.
|
||
|
||
---
|
||
|
||
## §6 — Audit trail anchors
|
||
|
||
- Pre-launch sprint consolidation: `decisions/2026-04-30-pre-launch-sprint-consolidation-LOCKED.md`
|
||
- Benchmark portfolio brief (Gaia2 + ERL section): `briefs/2026-04-29-benchmark-portfolio-refresh-2026-venues.md`
|
||
- Faza 1 closure (substrate evidence): `decisions/2026-04-29-gepa-faza1-results.md`
|
||
- ARE platform: github.com/facebookresearch/meta-agents-research-environments
|
||
- Gaia2 paper: arxiv 2602.11964
|
||
- ERL paper: arxiv 2603.24639
|
||
- This brief: `briefs/2026-04-30-cc-sesija-C-gaia2-setup-dry-verification.md`
|
||
|
||
---
|
||
|
||
**End of brief. Awaiting CC kick-off.**
|